|||

Value and measurability in development

There is an interesting discussion on the xceval mailing list right now about value and measurability in development.

Seems to me there are four different claims here. Two are reasonable, two are shocking.

At the start there is what we would now see as an extremist positivist stance in the philosophy of science: the extreme pro-measurement claim that nothing which is not measurable exists or has value. Nowadays I don’t think there are many people who after reflection would agree with that stance. But nevertheless too many years of working with a measurement technology, whether in the hard sciences or development, can lead one to passively acquiesce to it (hence, you become what you measure”). Sometimes the World Bank might seem to be doing this.

Frustration with this stance might lead one to the equally absurd opposite claim that those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable”. This is the claim that started the discussion and which was attributed to Andrew Natsios. You could call this the extreme anti-measurement claim, that measurable is the opposite of valuable, at least in development. One black-and-white claim begets another. Isn’t this absurd? Frustration with the original extremist stance is understandable but to want to deny it by turning it on its head is just adolescent. After all, there are libraries full of fairly well-established interventions which are both transformational and measurable, like intelligent investment in teacher in-service training. Unfortunately I fear there are legions of people working in development who out of frustration with the first claim give up and blindly opt for the second.

You could call Einstein’s Not everything that counts can be counted and not everything that can be counted, counts.” the moderate claim. (This was brought into the discussion by Kita Louisa from Nairobi). This is surely a more mature response to the first extremist claim without falling into the trap of the second: measurable does not equal valuable, but nor are they opposites.

But in the end I agree with Kita Louisa who makes a more modest but more useful fourth claim: ease of measurement bears no relation to value. (This is closer to what Natsios actually said, http://www.cgdev.org/content/publications/detail/1424271, though his more moderate claims get quoted in a more extreme form, which is interesting.) Our attempts to measure value inevitably end up focusing on whatever is easier to measure, and those things tend to be at what we in development often call a lower level”, which means we often give up on the things at a higher level which are also valuable.

I wouldn’t even say that higher level things tend to be more valuable. A vaccination is quite low level, pretty easy to measure. Its ultimate value depends on the rest of the system, but that is true for higher-level things too. But higher-level things do tend to be harder to measure.

In other words, we need to invest more time and courage into trying to conceptualise and report higher-level outcomes. (This is what most mainstream social scientists have been involved with in one way or another for at least the last few decades, fallibly, but not without some success).

Whereas both extremist positions advocate not even trying.

Up next Which parents are satisfied with their child’s education? Those who know their class representative well. Especially in poor schools. Another result from our OSI / ESP survey of nearly 11000 parents in ten countries. Dots are individual parents. The y-axis is individual parents’ State of the World’s Volunteerism - neatly balanced, maybe? Just got back from Istanbul (actually an industrial estate outside Istanbul) to take part in a regional meeting to contribute to the State of the
Latest posts Causal Map intro Causal Mapping - an earlier guide The walk to school in Sarajevo Glitches Draft blog post for AEA365 Theory Maker! Inventory & analysis of small conservation grants, C&W Africa - Powell & Mesbach! Lots of charts! Answering the “why” question: piecing together multiple pieces of causal information rbind.fill for 1-dimensional tables in r yED graph editor Examples of trivial graph format Using attr labels for ggplot An evaluation puzzle: “Talent show” An evaluation puzzle: “Mobile first” An evaluation puzzle: “Many hands” An evaluation puzzle: Loaves and fishes An evaluation puzzle: “Freak weather” An evaluation puzzle: “Billionaire” Publications Using Dropbox for syncing Shiny app data on Amazon EC2 Progress on the Causal Map app Articles and presentations related to Causal Maps and Theorymaker Better ways to present country-level data on a world map: equal-area cartograms A starter kit for reproducible research with R A reproducible workflow for evaluation reports Welcome to the Wiggle Room Realtime comments on a Theory of Change Responses to open questions shown as tooltips in a chart A panel on visualising Theories of Change for EES 2018? Peer mentoring for evaluators How do you explain reproducible research to clients?