EOA - evaluation services offered

EOA has a team of trained evaluators, led by myself, including specialists in housing, ethnic minority involvement and monitoring, managements systems, mediation, regeneration, social enterprise, enterprise development, social impact assessment, charitable law and culture. For most evaluations two evaluators, chosen to best suit the client’s needs, work together.

Evaluations start at £5,000 plus VAT and expenses. This includes preparation, a stakeholder workshop, some follow-up interviews and a written report. Longitudinal evaluations with a series of workshops and interviews spread over the lifetime of a programme are also available.

If you are interested, please contact me on zoe at eastoxford.com (replacing the "at" with an @ and deleting the spaces) or ring 01865 203367

Stage 5 - Developing an action plan

This stage of an empwerment evaluation often will happen as a separate workshop at a later date or alternatively as a second half to the rest of the workshop, depending on the needs and make-up of the project. Using the actions and scoring developed so far as the basis the fifth stage aims to identify actions to address the issues coming out of the workshop and what actions are needed to a) to evaluate performance in future and b) plan for future actions. By using the scoring from the previous stages a continuity is created into the future. For each of the criteria identified further goals are identified, actions and strategies to deliver the goals and evaluation evidence to verify success. These then form the basis of either work within the project or programme or for future programmes of action. In the case of the latter this approach is particularly important in securing sustainability for programmes, creating a basis and argument – in terms of project and partnership development – for future fundraising bids or mainstreaming.

Stage 4 - In-depth discussion

The evaluator then facilitates a discussion based on the scoring table. The function of the discussion is to get beneath the numbers to people's issues and views. The discussion is an open one and is based on a process which has been transparent and logical from the beginning. If the voting process was designed to ensure the people scored without regard to the views of others, this stage makes those involved listen to others and to value their opinions. It is East Oxford Action's experience that the democratic structure of the workshop means that there is less chance of programme staff either feeling threatened by the process or feeling the need to play power games. This is aided by the breaking down of the scoring into component parts for different criteria – as staff and others can see that scoring badly in one criterion does not mean a condemnation in others. Where tensions do arise these need to mediated by the evaluator, but they should not be ignored. The evaluator will often find themselves assisted in this by the other group members and where this happens the evaluator should encourage it – it is a sign that the group is beginning to take responsibility for the evaluation process.

Stage 3 - Scoring performance

Having prioritised the criteria/actions the participants individually are asked to score the project's current performance against the criteria. Scoring is out of ten, with ten being perfect. They do this on their own on a piece of paper and then either they read out their votes and the evaluator marks them on to a scoring grid (see fig1) or they write them on the grid themselves. This approach helps stop people from being influenced by what the others are scoring. In this and in the writing down of criteria (stage 2) the evaluator encourages the participants not to think too hard, but to write down what comes easily to mind. Whilst the workshop members are having a tea break the evaluator totals the scores both by criteria and participant.

This element of the workshop can be seen as producing a quantifiable scoring of performance that over a longitudinal evaluation can be used to measure progress. However caution should be exercised in doing this, as it should with all similar surveys and as is made clear in Stage 4 of the evaluation. It is the experience of East Oxford Action that programmes which include capacity and social capital building often do not score as might be expected – i.e. approval ratings do not rise as problems are tackled. The reason for this is that in disadvantaged communities there are often low expectations. If through the programme one raises expectations then the scoring of performance may be affected adversely. For example in East Oxford there was at the beginning of the regeneration programme an important piece of open space which was used as a dumping ground for rubbish, and as an area for drug abuse. The regeneration programme transformed the open space into a pleasant park. However, with their expectations raised, local people would still complain about the occasional dropping of litter.

Stage 2 - Prioritising criteria for success

In empowerment evaluation this is done through a voting system; equal weight is given to all participants. Each participant is given a set number of sticky dots (number dependent on the number of criteria to vote on, but usually 5 – 10) and each participant is asked to stick their dots next to their priority criteria – they can distribute the dots across the criteria or if they so feel put all the dots against just one. The participants come up to the flip chart and vote together, this again allows participants to think about the views of others on the subject. The evaluator then counts the dots and orders the criteria by numbers of vote. There is often a discussion at this point.

It is interesting in a longitudinal evaluation to see the variation between the results on stage 2 . This can show that concerns that featured highly at the beginning of the project have now been addressed and are no longer a consideration. For example in a training project evaluation that East Oxford Action undertook, for the first evaluation “recruitment of people onto the training course” ranked very highly, this had totally disappeared by the end evaluation workshop. Through the evaluation process however, the importance of recruitment at the beginning had been captured and so could be learnt from for future project planning.

Stage 2 - Criteria for success

In this stage the workshop focuses on the criteria for success in the evaluation. This is couched in terms of a question: “What needs to happen in order to achieve the agreed aim?” It is important to use accessible language and questions, hence “what needs to happen” instead of “what are the criteria of success”. It is important also for the evaluator to ensure that the language in the discussions is inclusive. This usually means asking for explanations and even getting the group to agree the banning of acronyms. Each group member is given five post-it notes and the group is asked individually to choose five things that need to happen and write one on each of the post-it notes. The participants then place their post-its on a flipchart. They are encouraged to bunch together notes which have the same criteria. The process allows the participants to see what others have written and to think. By writing the notes individually everyone's views are given equal weight. The evaluator then talks through the notes with the participants, seeking clarification as appropriate and the evaluator facilitates the group to agree the top things (Fetterman suggests ten to twenty) which need to happen. These are written up on a flipchart.

First Stage - Define the Aim

Stage 1

The single most important element is the definition of the aim of the programme or project. If an evaluation’s function is to measure the programme’s success, it is necessary to know what the programme aims to do. It is East Oxford Action's experience that this is where a lot of problems arise, both in terms of power - with different groups vying to define the aim, and in terms of lack of clarity – with different groups having different interpretations of the aim and thus different views on whether the project has been a success.

The function of the evaluator in this process is to mediate between those with differing views on the aim of the project, and to enable them to resolve their differences and to come if possible to a consensus on the aim. At the end of the stage the group should have negotiated and agreed one aim. Where the evaluation has not started at the beginning of the programme the differences over the aim between those involved can often explain the problems the project is encountering and part of this stage of the workshop much time may need to be devoted to exploring this. Workshops undertaken at the middle and end of the programme repeat the question. This reveals whether the aim has changed or perhaps even needs to be. In cases where there has been disagreement over the aims in the first evaluation it is useful to see whether these differences have been resolved.

Scene Setting for an Empowerment Workshop

Scene setting for the evaluation workshop is crucial. Fetterman suggests that this take place in “an open session with as many staff members and participants as possible.” Fetterman, 2001. East Oxford Action's experience of facilitated workshops such as these are they are best when the numbers are limited to about fifteen. This raises problems for large programmes. Our experience is also that the process of inviting people to the workshop offers an opportunity for control – ie only inviting people who will support your point of view. Whilst all measures are taken to prevent this – through explaining the evaluation process to those doing the invitations – it is also useful to keep an eye open for this during the workshop and perhaps do some follow-up work if appropriate.

An agreement by all those involved to subscribe to openness and mutual respect needs to be established at the beginning – the evaluator making clear that everyone is equal. East Oxford Action has experienced problems with this, for example where cultural barriers prevent one group from speaking in front of another. Another example is that of a project evaluation taken with people with mental health problems, where it was felt that a fully mixed workshop would be too stressful for those involved. In this circumstance we ran a series of workshops with the different groups (if the participants are likely to be intimidated by the process), whilst ensuring that the different groups heard what each other are saying and eventually run a mixed group workshop once trust had been built up. With regeneration programmes made up of several projects delivering different elements of the programme, it is necessary to undertake several workshops looking at the overall aim and the projects' aims.

The choice of time and venue for the evaluation workshop is also important – where possible it should be designed to suit the needs of everyone, but where this is not possible to suit the needs of the most vulnerable in the group.

The stages of an evaluation workshop

East Oxford Action (EOA) , has been carrying out empowerment evaluations for internal projects and for clients for the last three years. During this time we have adapted the approach to the circumstances of British regeneration programmes and project delivery. A team of empowerment evaluators drawn from groups and individuals who have worked with East Oxford Action has been trained.

At East Oxford Action we break the process down into five stages (expanded from Fetterman's three) :

• The first stage of an empowerment evaluation is to establish the aim of the project

• The second stage is to agree criteria for success

• The third stage is to score current performance against the criteria

• The fourth stage is a facilitated discussion around the scoring on the criteria, to get beneath the skin of the scoring

• The fifth stage is to identify actions to address the issues coming out of the workshop and to agree what actions are needed to a) address the issues and b) to evaluate performance.

Coming out of the evaluation workshop therefore are

• Agreement on project aim, criteria for success, ways of measuring success

• Agreement on priorities for future action in the project and evaluation

• A written record of the evaluation workshop.

It should start at the beginning but...

The first point of note is that while ideally these sort of evaluations would start at the beginning of the programme, East Oxford Action has yet to experience one that has done so. The reality of funding programme rounds with the short notice between the funding being announced and the project or programme being submitted virtually prohibits this. Another inhibitor is the need to fit the programme to the funder's specification, to balance the participants' objectives and those of the funder – power dynamics in play before the project/programme even starts. However where participatory and empowerment evaluations are employed albeit after the start of the programme, it is possible to use them as the starting point for the next programme, hence the end-of-programme evaluation session is also the opening session of the next programme.

Introduction to Empowerment Evaluation

Empowerment evaluation is arguably a form of participatory evaluation developed by David M. Fetterman of Stanford University and now used extensively (particularly in the USA) to evaluate projects and programmes. Like participatory evaluation “Empowerment evaluation has an unambiguous value orientation -- it is designed to help people help themselves and improve their programs using a form of self-evaluation and reflection “ Fetterman, 2001.

Empowerment evaluation is structured around a workshop. The workshop brings together stakeholders—all those interested in the project including the beneficiaries and lower level staff as well as the managers and funders. In an ideal world the evaluation will start at the beginning of the project with subsequent workshops held at the mid-term and end.

Fetterman identifies three steps in an empowerment evaluation:

“ a) developing a mission,vision or unifying purpose, b) taking stock or determining where the programme stands, including strengths and weaknesses; and c) planning for the future by establishing goals and helping participants determine their own strategies to accomplish goals and objectives.” Fetterman, 2001

Extract from paper Zoe Brooks 2007

Introduction to Participatory and Empowerment Evaluation

“Participatory monitoring and evaluation is not just a matter of using participatory techniques within a conventional monitoring and evaluation setting. It is about radically rethinking who initiates and undertakes the process, and who learns or benefits from the findings.” Institute of Development Studies, 1998

Participatory and empowerment evaluation places all stakeholders at its heart, especially those who will live with the results of the programme. In such an approach the evaluation is not seen as a final judgement on whether the project or programme has been a success or not, but as a means to help people help themselves and improve their programmes. In this approach the evaluator is not an external judge but a critical friend, holding up a mirror to those involved in delivering the project, and facilitating the generation of solutions to problems and a more accurate self-image.

Participatory approaches to evaluation are used extensively by organisations working in the Third World, such as OXFAM and the United Nations Development Programme. The characteristics of participatory evaluation as opposed to more traditional evaluations are:

• the participation if a broad range of stakeholders rather than a limited number in a traditional evaluation

• the focus on participants: residents, project staff and stakeholders, rather than funders and programme managers

• the evaluator as facilitator and critical friend, rather than as expert

• a focus on learning through the process rather than on accountability/judgement

• the involvement of the participants in collecting the data, with the support of the evaluator, rather than the outside experts doing so.

There are many ways of delivering participatory evaluation and many tools for the job, especially rapid appraisal techniques, which will be familiar to the world of community development and regeneration.

Extract from paper by Zoe Brooks CEO East Oxford Action, 2007

Introduction

This blog has been created to talk about and discuss participatory and empowerment methods of evaluation.