|||

Does it make sense to try to measure progress on the highest levels of a logframe?

Another interesting discussion on the M&ENews mailing list - does it make sense to try to measure progress on the highest levels of a logframe?

A couple of opinions -

True, it is often hard to attribute changes in higher-level” items to our project. But we will often still want to monitor changes if we can. So if a school has introduced a program to reduce truancy, the staff will surely want to monitor truancy level, even if they know it is only to a certain extent within their power to influence. In other words, indicators can be for actual monitoring (in the ordinary sense of the word) as well as for attribution. I guess this is was the original idea behind the M of M&E, which the pseudoscience of logframes sometimes obscures - the idea of keeping an eye on progress towards an important outcome. This keeping an eye on progress towards the outcome” is about a lot more than just comparing baseline and endline scores and is, I think, one of the marks of real, effective management. It is something we do instinctively when we really care about reaching the outcome. And we do it just the same whether we have 100% control or only 5% control over the outcome, i.e. regardless of whether success or failure can be totally attributed to our efforts. Ordinary life outside the world of projects and logframes are full of this natural monitoring - just think of what we have to keep an eye on when we are bringing up children, or running a school or a business, or managing a national bank. So if a program plan or logframe is a living, day-to-day tool to help our project succeed, it will help us visualise what we are trying to reach and give us a handful of key, usually low-tech ways to regularly test what progress we are making and get warning signs if we are getting off track. Anything else in our plan is dead weight or logframe bloat”.

What does a parent keep an eye on to tell if a child is getting too tired to finish a task? Now those are indicators and that is monitoring. Who said you need a PhD for M&E? Parents don’t usually even have program plans and yet are experts at using complex, sensory data to monitor and manage progress towards outcomes.

So - I would say that the primary function of a logframe or program plan should be to help us with real-life, daily monitoring of progress, - and the attribution function is secondary and is derived from the primary function.

In practice, we try to judge performance even where attribution is difficult because control over the outcome is less than 100%. We are always judging the performance of politicians, school directors, managers and so on, though we are quite well aware that their successes and failures are only partly to be attributed to their efforts. We take note of the baseline and endline data but we hopefully don’t take too much notice of it. And interestingly, we rarely complain that we only have a sample of one or that the counterfactuals are insufficient.

One more thing - it is true that changes in higher levels” sometimes involve changes in attitudes, beliefs etc (which are supposed to be hard to measure objectively, giving us a reason not to monitor progress on them). But it is a myth that they always do. To repeat the same example, reducing truancy is an important behavioural outcome which might have several layers below it in a school action plan. But it is not hard to measure. The MDGs are, I guess, high-level outcomes” (to continue with this not very helpful language of hierarchical levels) but they are very concrete and not particularly hard to measure.

Up next Omni test for statistical significance In survey research, our datasets nearly always comprise variables with mixed measurement levels - in particular, nominal, ordinal and continuous, or Social Capital article published at last International Journal of Internet Science Volume 8, Issue 1 http://www.ijis.net/ijis8_1/ijis8_1_bosancianu_et_al.html Social Capital and Pro-Social
Latest posts Causal Mapping - an earlier guide The walk to school in Sarajevo Glitches Draft blog post for AEA365 Theory Maker! Inventory & analysis of small conservation grants, C&W Africa - Powell & Mesbach! Lots of charts! Answering the “why” question: piecing together multiple pieces of causal information rbind.fill for 1-dimensional tables in r yED graph editor Examples of trivial graph format Using attr labels for ggplot An evaluation puzzle: “Talent show” An evaluation puzzle: “Mobile first” An evaluation puzzle: “Many hands” An evaluation puzzle: Loaves and fishes An evaluation puzzle: “Freak weather” An evaluation puzzle: “Billionaire” Publications Using Dropbox for syncing Shiny app data on Amazon EC2 Progress on the Causal Map app Articles and presentations related to Causal Maps and Theorymaker Better ways to present country-level data on a world map: equal-area cartograms A starter kit for reproducible research with R A reproducible workflow for evaluation reports Welcome to the Wiggle Room Realtime comments on a Theory of Change Responses to open questions shown as tooltips in a chart A panel on visualising Theories of Change for EES 2018? Peer mentoring for evaluators How do you explain reproducible research to clients? Links for my AEA eval2017 presentation, Washington DC