EOA - evaluation services offered

EOA has a team of trained evaluators, led by myself, including specialists in housing, ethnic minority involvement and monitoring, managements systems, mediation, regeneration, social enterprise, enterprise development, social impact assessment, charitable law and culture. For most evaluations two evaluators, chosen to best suit the client’s needs, work together.

Evaluations start at £5,000 plus VAT and expenses. This includes preparation, a stakeholder workshop, some follow-up interviews and a written report. Longitudinal evaluations with a series of workshops and interviews spread over the lifetime of a programme are also available.

If you are interested, please contact me on zoe at eastoxford.com (replacing the "at" with an @ and deleting the spaces) or ring 01865 203367

Stage 4 - In-depth discussion

The evaluator then facilitates a discussion based on the scoring table. The function of the discussion is to get beneath the numbers to people's issues and views. The discussion is an open one and is based on a process which has been transparent and logical from the beginning. If the voting process was designed to ensure the people scored without regard to the views of others, this stage makes those involved listen to others and to value their opinions. It is East Oxford Action's experience that the democratic structure of the workshop means that there is less chance of programme staff either feeling threatened by the process or feeling the need to play power games. This is aided by the breaking down of the scoring into component parts for different criteria – as staff and others can see that scoring badly in one criterion does not mean a condemnation in others. Where tensions do arise these need to mediated by the evaluator, but they should not be ignored. The evaluator will often find themselves assisted in this by the other group members and where this happens the evaluator should encourage it – it is a sign that the group is beginning to take responsibility for the evaluation process.

Stage 2 - Prioritising criteria for success

In empowerment evaluation this is done through a voting system; equal weight is given to all participants. Each participant is given a set number of sticky dots (number dependent on the number of criteria to vote on, but usually 5 – 10) and each participant is asked to stick their dots next to their priority criteria – they can distribute the dots across the criteria or if they so feel put all the dots against just one. The participants come up to the flip chart and vote together, this again allows participants to think about the views of others on the subject. The evaluator then counts the dots and orders the criteria by numbers of vote. There is often a discussion at this point.

It is interesting in a longitudinal evaluation to see the variation between the results on stage 2 . This can show that concerns that featured highly at the beginning of the project have now been addressed and are no longer a consideration. For example in a training project evaluation that East Oxford Action undertook, for the first evaluation “recruitment of people onto the training course” ranked very highly, this had totally disappeared by the end evaluation workshop. Through the evaluation process however, the importance of recruitment at the beginning had been captured and so could be learnt from for future project planning.

First Stage - Define the Aim

Stage 1

The single most important element is the definition of the aim of the programme or project. If an evaluation’s function is to measure the programme’s success, it is necessary to know what the programme aims to do. It is East Oxford Action's experience that this is where a lot of problems arise, both in terms of power - with different groups vying to define the aim, and in terms of lack of clarity – with different groups having different interpretations of the aim and thus different views on whether the project has been a success.

The function of the evaluator in this process is to mediate between those with differing views on the aim of the project, and to enable them to resolve their differences and to come if possible to a consensus on the aim. At the end of the stage the group should have negotiated and agreed one aim. Where the evaluation has not started at the beginning of the programme the differences over the aim between those involved can often explain the problems the project is encountering and part of this stage of the workshop much time may need to be devoted to exploring this. Workshops undertaken at the middle and end of the programme repeat the question. This reveals whether the aim has changed or perhaps even needs to be. In cases where there has been disagreement over the aims in the first evaluation it is useful to see whether these differences have been resolved.

Scene Setting for an Empowerment Workshop

Scene setting for the evaluation workshop is crucial. Fetterman suggests that this take place in “an open session with as many staff members and participants as possible.” Fetterman, 2001. East Oxford Action's experience of facilitated workshops such as these are they are best when the numbers are limited to about fifteen. This raises problems for large programmes. Our experience is also that the process of inviting people to the workshop offers an opportunity for control – ie only inviting people who will support your point of view. Whilst all measures are taken to prevent this – through explaining the evaluation process to those doing the invitations – it is also useful to keep an eye open for this during the workshop and perhaps do some follow-up work if appropriate.

An agreement by all those involved to subscribe to openness and mutual respect needs to be established at the beginning – the evaluator making clear that everyone is equal. East Oxford Action has experienced problems with this, for example where cultural barriers prevent one group from speaking in front of another. Another example is that of a project evaluation taken with people with mental health problems, where it was felt that a fully mixed workshop would be too stressful for those involved. In this circumstance we ran a series of workshops with the different groups (if the participants are likely to be intimidated by the process), whilst ensuring that the different groups heard what each other are saying and eventually run a mixed group workshop once trust had been built up. With regeneration programmes made up of several projects delivering different elements of the programme, it is necessary to undertake several workshops looking at the overall aim and the projects' aims.

The choice of time and venue for the evaluation workshop is also important – where possible it should be designed to suit the needs of everyone, but where this is not possible to suit the needs of the most vulnerable in the group.

The stages of an evaluation workshop

East Oxford Action (EOA) , has been carrying out empowerment evaluations for internal projects and for clients for the last three years. During this time we have adapted the approach to the circumstances of British regeneration programmes and project delivery. A team of empowerment evaluators drawn from groups and individuals who have worked with East Oxford Action has been trained.

At East Oxford Action we break the process down into five stages (expanded from Fetterman's three) :

• The first stage of an empowerment evaluation is to establish the aim of the project

• The second stage is to agree criteria for success

• The third stage is to score current performance against the criteria

• The fourth stage is a facilitated discussion around the scoring on the criteria, to get beneath the skin of the scoring

• The fifth stage is to identify actions to address the issues coming out of the workshop and to agree what actions are needed to a) address the issues and b) to evaluate performance.

Coming out of the evaluation workshop therefore are

• Agreement on project aim, criteria for success, ways of measuring success

• Agreement on priorities for future action in the project and evaluation

• A written record of the evaluation workshop.