|||
Very often, evaluation TORs are so comprehensive that it is not really possible in the time available to do proper independent data gathering for many or most of the research questions and sub-questions. For example a TOR might contain eight major headings with say five sub-heads each, that makes say around 40 sub-questions. Often, just one of these questions, e.g. “achievement of planned objectives” can break down further into dozens of additional sub-questions, e.g. according to different planned outcomes.
So a top-level question might be, say: Sustainability (1 of 8 top-level headings)
A next-level question might be, say: Institutional Sustainability (1 of say 5 kinds of sustainability: 8*5=40 questions at this level)
A level-3 question might be, say: / Institutional sustainability of Outcome A1 (One of say three outcomes: 853=120 questions at this level)
What is more, the ToR may mandate or imply that answers to these questions may in turn have to be broken down by key categories for example gender, region, ethnicity.
Finally, frequently there are additional lists of “key additional questions” or “cross-cutting issues” and so on which also have to be addressed alongside the standard questions mentioned already.
So this understandable desire to have a large quantity of evaluation questions answered means that in practice the evaluator is limited in the quality and depth of independent data collection and triangulation of data sources which they are able to carry out.
In this kind of case, in order to answer so many questions, evaluators are usually limited to balancing different opinions and evidence given directly by some subset of stakeholders and key informants.
Typically, an evaluator will ask these stakeholders one-by-one (or in small focus groups) a series of questions which are not directly congruent to the findings and recommendations sections of the evaluation report but which can somehow subsequently be synthesised into those headings.
The “evaluation matrix” traditionally shows how this mapping and re-mapping from evaluation questions to pieces of evidence, and back again, is carried out; how each of the evaluation questions is answered by gathering and analysing different data from different sources. So at the start of the evaluation, the evaluation questions get translated into applications of specific tools with specific sources, then at the end of the evaluation process all this data is re-combined and translated back into the structure of the evaluation report Findings, Conclusions and Recommendation sections.
This is a good approach and ultimately it is in the background of Focus Doc processes too. But it is pretty tricky to manage.
More time for additional primary evidence, whether brought in by contributors or freshly gathered for the evaluation, because less time is necessary to balance secondary opinions from contributors.
This procedure would certainly not replace traditional evaluation procedures; the evaluator will still be conducting normal interviews face-to-face and remotely, but he or she will not need to do so many, and can target interviews where the ongoing online discussion reveals a particular need.
The Focus Doc process does not suggest that the right way to answer to evaluation questions is to take some kind of average of expert opinion on each question. Quite the contrary.
Focus Docs is not about finding average or consensus. The Focus Docs process aims to enable rich discussion and to allow different opinions to evolve around the evaluation questions like a kind of structured focus group discussion which evolves over a few days.
The Focus Docs is also not about relying only on opinion. Instead, it encourages contributors to provide, weight up and identify gaps in different sources of external evidence, the bedrock on which their opinions are, hopefully, built.
So, in this case, why not ask these stakeholders directly to contribute of the report?
Why not formulate the main findings and recommendations report headings as questions and ask key stakeholders to collaboratively provide answers to them directly (as well as improving the questions)?
… backups, etc ….
The evaluator’s role is similar to, but not quite the same as, that of evaluators in traditional evaluations.
The Focus Docs approach might seem to some to undermine the traditional (rather revered) role and status of the evaluator. But first and foremost, Focus Docs does require the more or less full-time involvement of an experienced evaluator.
to make judgements about the value and quality of execution of a declared intention such as a to initiate a social project
to specifically address the different ways in which contributors’ answers might not fully represent the truth. These issues arise in any evaluation.
In a traditional interview, it is easier to let the evaluator know at least that you feel uncomfortable about something ….
To be eligible for the prize(s), you have to make at least 20 contributions in the first two days.
After 2-3 days, I analyse the contribution rate to see if there are any subgroups e.g. men, or older people, or people located more remotely, who are responding proportionately less often. I can send round a general e-mail with this information, saying “do you want these groups to remain less represented?”
There are always topics which some or all stakeholders would rather not talk about.
I can address these during individual talks with individuals and individual stakeholder groups as well as in the discussion on evalcrowd. On evalcrowd, I can address these themes ad-hoc but can also place them as questions:
Place for anonymous questions?