|||

Draft blog post for AEA365

This is a blog post I’m working on for the AEA365 blog, we’re going to do a series on the Theories of Evaluation TIG.


Hi, I’m Steve Powell, a freelance evaluator. I’m really interested in theories of, about, and within evaluation, and the conceptual headaches they can bring with them. Here I’m posting a couple of evaluation puzzles which may or may not give you a headache, make you smile, or even perhaps have a light-bulb moment.

Sometimes as evaluators we are so busy with practical challenges that we don’t have much time to worry if there is sufficient agreement about the meaning of the words we use, even when we use difficult words like attribution”, impact” or intention”.

Puzzles and paradoxes have often been used in philosophy, from Zeno and Zen to the Sufi mystics, to help us question and sharpen up our concepts. I’ve written a few evaluation puzzles (http://www.pogol.net/tagged/puzzles) which I have found useful in workshops to get people realising that some of the ideas we use when doing evaluation stand on quite shaky ground.

Below are two of them. If you know of any similar evaluation puzzles or paradoxes, I’d be really interested in hearing about them.


An evaluation puzzle: Freak weather”

Two NGOs prepare ships to help people stranded after a tsunami on a remote island.

NGO A is well-prepared and sets sail in time.

NGO B has barely enough equipment and leaves too late to do any good.

A freak hurricane sends ship A back to base but slings ship B miles ahead.

Ship B reaches the island and just manages to save many lives.

Ship A still manages to arrive on time but its help is no longer needed.

Which NGO was most effective? Which had the most impact? Which should be praised the most? Did NGO A fail?


An evaluation puzzle: Billionaire”

A billionaire left 10 million EUR in his will to establish a trust, with instructions that it should be used to just do good”.

During the ten years since then, support for same-sex marriage has shifted from 10% to 80% public approval, and almost all liberals are now in favour.

In the ninth year, the trust gives 1 million to a campaign for a law on marriage equality, which substantially contributes to the passing of the law in the tenth year.

We don’t know for sure, but most likely the billionaire, like most of his peers and friends, did not support marriage equality when he was alive. But most of his peers and friends now support it.

Now, in the 11th year, you are asked to evaluate whether the trust was used effectively and whether the activities were relevant to the intentions of the billionaire.

Up next Theory Maker! “A theory of change for our project!” “Great, but so fiddly to actually draw.” Theory Maker is designed to meet this need. Theory Maker is a Glitches This blog is hosted at blot.im which is a wonderfully simple service. Yesterday I added another blog on the same account and the dates of some of
Latest posts causal-map Causal Map intro Causal Mapping - an earlier guide The walk to school in Sarajevo Glitches Draft blog post for AEA365 Theory Maker! Inventory & analysis of small conservation grants, C&W Africa - Powell & Mesbach! Lots of charts! Answering the “why” question: piecing together multiple pieces of causal information rbind.fill for 1-dimensional tables in r yED graph editor Examples of trivial graph format Using attr labels for ggplot An evaluation puzzle: “Talent show” An evaluation puzzle: “Mobile first” An evaluation puzzle: “Many hands” An evaluation puzzle: Loaves and fishes An evaluation puzzle: “Freak weather” An evaluation puzzle: “Billionaire” Publications Using Dropbox for syncing Shiny app data on Amazon EC2 Progress on the Causal Map app Articles and presentations related to Causal Maps and Theorymaker Better ways to present country-level data on a world map: equal-area cartograms A starter kit for reproducible research with R A reproducible workflow for evaluation reports Welcome to the Wiggle Room Realtime comments on a Theory of Change Responses to open questions shown as tooltips in a chart A panel on visualising Theories of Change for EES 2018? Peer mentoring for evaluators