Stage 4 - In-depth discussion

The evaluator then facilitates a discussion based on the scoring table. The function of the discussion is to get beneath the numbers to people's issues and views. The discussion is an open one and is based on a process which has been transparent and logical from the beginning. If the voting process was designed to ensure the people scored without regard to the views of others, this stage makes those involved listen to others and to value their opinions. It is East Oxford Action's experience that the democratic structure of the workshop means that there is less chance of programme staff either feeling threatened by the process or feeling the need to play power games. This is aided by the breaking down of the scoring into component parts for different criteria – as staff and others can see that scoring badly in one criterion does not mean a condemnation in others. Where tensions do arise these need to mediated by the evaluator, but they should not be ignored. The evaluator will often find themselves assisted in this by the other group members and where this happens the evaluator should encourage it – it is a sign that the group is beginning to take responsibility for the evaluation process.

Stage 3 - Scoring performance

Having prioritised the criteria/actions the participants individually are asked to score the project's current performance against the criteria. Scoring is out of ten, with ten being perfect. They do this on their own on a piece of paper and then either they read out their votes and the evaluator marks them on to a scoring grid (see fig1) or they write them on the grid themselves. This approach helps stop people from being influenced by what the others are scoring. In this and in the writing down of criteria (stage 2) the evaluator encourages the participants not to think too hard, but to write down what comes easily to mind. Whilst the workshop members are having a tea break the evaluator totals the scores both by criteria and participant.

This element of the workshop can be seen as producing a quantifiable scoring of performance that over a longitudinal evaluation can be used to measure progress. However caution should be exercised in doing this, as it should with all similar surveys and as is made clear in Stage 4 of the evaluation. It is the experience of East Oxford Action that programmes which include capacity and social capital building often do not score as might be expected – i.e. approval ratings do not rise as problems are tackled. The reason for this is that in disadvantaged communities there are often low expectations. If through the programme one raises expectations then the scoring of performance may be affected adversely. For example in East Oxford there was at the beginning of the regeneration programme an important piece of open space which was used as a dumping ground for rubbish, and as an area for drug abuse. The regeneration programme transformed the open space into a pleasant park. However, with their expectations raised, local people would still complain about the occasional dropping of litter.

Stage 2 - Prioritising criteria for success

In empowerment evaluation this is done through a voting system; equal weight is given to all participants. Each participant is given a set number of sticky dots (number dependent on the number of criteria to vote on, but usually 5 – 10) and each participant is asked to stick their dots next to their priority criteria – they can distribute the dots across the criteria or if they so feel put all the dots against just one. The participants come up to the flip chart and vote together, this again allows participants to think about the views of others on the subject. The evaluator then counts the dots and orders the criteria by numbers of vote. There is often a discussion at this point.

It is interesting in a longitudinal evaluation to see the variation between the results on stage 2 . This can show that concerns that featured highly at the beginning of the project have now been addressed and are no longer a consideration. For example in a training project evaluation that East Oxford Action undertook, for the first evaluation “recruitment of people onto the training course” ranked very highly, this had totally disappeared by the end evaluation workshop. Through the evaluation process however, the importance of recruitment at the beginning had been captured and so could be learnt from for future project planning.

Stage 2 - Criteria for success

In this stage the workshop focuses on the criteria for success in the evaluation. This is couched in terms of a question: “What needs to happen in order to achieve the agreed aim?” It is important to use accessible language and questions, hence “what needs to happen” instead of “what are the criteria of success”. It is important also for the evaluator to ensure that the language in the discussions is inclusive. This usually means asking for explanations and even getting the group to agree the banning of acronyms. Each group member is given five post-it notes and the group is asked individually to choose five things that need to happen and write one on each of the post-it notes. The participants then place their post-its on a flipchart. They are encouraged to bunch together notes which have the same criteria. The process allows the participants to see what others have written and to think. By writing the notes individually everyone's views are given equal weight. The evaluator then talks through the notes with the participants, seeking clarification as appropriate and the evaluator facilitates the group to agree the top things (Fetterman suggests ten to twenty) which need to happen. These are written up on a flipchart.

First Stage - Define the Aim

Stage 1

The single most important element is the definition of the aim of the programme or project. If an evaluation’s function is to measure the programme’s success, it is necessary to know what the programme aims to do. It is East Oxford Action's experience that this is where a lot of problems arise, both in terms of power - with different groups vying to define the aim, and in terms of lack of clarity – with different groups having different interpretations of the aim and thus different views on whether the project has been a success.

The function of the evaluator in this process is to mediate between those with differing views on the aim of the project, and to enable them to resolve their differences and to come if possible to a consensus on the aim. At the end of the stage the group should have negotiated and agreed one aim. Where the evaluation has not started at the beginning of the programme the differences over the aim between those involved can often explain the problems the project is encountering and part of this stage of the workshop much time may need to be devoted to exploring this. Workshops undertaken at the middle and end of the programme repeat the question. This reveals whether the aim has changed or perhaps even needs to be. In cases where there has been disagreement over the aims in the first evaluation it is useful to see whether these differences have been resolved.

The stages of an evaluation workshop

East Oxford Action (EOA) , has been carrying out empowerment evaluations for internal projects and for clients for the last three years. During this time we have adapted the approach to the circumstances of British regeneration programmes and project delivery. A team of empowerment evaluators drawn from groups and individuals who have worked with East Oxford Action has been trained.

At East Oxford Action we break the process down into five stages (expanded from Fetterman's three) :

• The first stage of an empowerment evaluation is to establish the aim of the project

• The second stage is to agree criteria for success

• The third stage is to score current performance against the criteria

• The fourth stage is a facilitated discussion around the scoring on the criteria, to get beneath the skin of the scoring

• The fifth stage is to identify actions to address the issues coming out of the workshop and to agree what actions are needed to a) address the issues and b) to evaluate performance.

Coming out of the evaluation workshop therefore are

• Agreement on project aim, criteria for success, ways of measuring success

• Agreement on priorities for future action in the project and evaluation

• A written record of the evaluation workshop.