More about Value
Single source of Value?
One can argue that marking individual Variables as Valuable …
Outcome 1 !Value;Outcome 2 !Value Some Intervention
… is just an abbreviation for this:
Value, e.g. to donor !Value Outcome 1;Outcome 2 Some Intervention
This way we can distinguish different Values:
Value to donor 1 !Value; Value to donor 2 !Value Outcome 1;Outcome 2 Some Intervention Value to donor 3 (only interested in Outcome 2) !Value Outcome 2
… and, very useful, allow different Variables to have different relative importance:
Value, e.g. to donor !Value (width=7)Outcome 1 (width=3)Outcome 2 Some Intervention
Example: More than one valued Variable, possibly in causal relationships to one another
This next diagram shows a Theory of Change. In this case, there is more than one !do Variable, and more than one valued Variable.
*improved* Student academic performance !valued *improved* Student problem-solving skills !valued !do Our intervention *improved* Teachers skills to support problem-solving !valued !do Our intervention
There might even be, as here, further Variables downstream marked as “interesting but not valuable (further) consequences”.
A consequence we don't particularly value A valued outcome !valued A valued intermediate outcome !valued An intermediate Variable !do Something we can control; !do Something else we can control; Something we cannot control Another valued outcome !valued An intermediate Variable wrap=9
In this example, note there are two valued Variables, one of which is a consequence of the other. The implementing agency cares about both the improved skills (which may sustain for several years) and the better harvest (which is important now); but it is not so interested in the tools, which it believes will probably be dispersed quite soon and are anyway cheap to buy or make. Note also the monitoring visits, which may or may not represent a large cost or be considered an “input”, but are added one step further “downstream” just to make sure. (There is probably also at least one feedback loop involved, from an assessment of the farmer’s post-training skills and motivation which influences how much effort is put into the monitoring visits.)
We have often been told that the value a project produces can only be associated with leaf variables, i.e. ones at the downstream end of your project which do not contribute to any more variables; classically these are sometimes called “Goals”.
There is a kind of “fake science” around logframe theory which tries to argue that for some higher reason, only leaf variables can be genuinely valuable. But think about the multiple effects and “side-effects” which a project can have and you will realise lots of ways in which different factors within and across causal chains can be valuable in their own right.
Some old-fashioned Logframe formats don’t even allow for more than one valued Variable in a Theory of Change. If this is just explained as being due to inflexible templates, that is just a pity; but if it is justified by some appeal like “well, a project just can only have one Goal”, this would be an example of the “false science of Logframes”.
(Sometimes also there is a discussion about how a project produces valuable top-level results but that these themselves are only valuable insofar as these contribute to further, ineffable donor goals, sometimes called “strategic goals” or similar.)
Quality of harvest !valued New seed varieties available !do Seeds delivered Farmers apply their new skills !do Monitoring visits - remind and encourage farmers Farmers have appropriate skills !valued !do Training workshops held Tools available !do Tools distributed Good rains wrap=8
Later in the book we will look at how to include more than one Actor in a Theory of Change as well as how to include Variables which are valued differently.
Example: Another example
Another example in which a valued Variable might influence another valued Variable, like this:
Children feel happy the next day when they remember the party !valued Children have fun at the party !valued !do Clown comes to children's party in displaced persons' camp
Theorymaker allows for these perfectly ordinary possibilities. So a clown at a children’s party might be trying to make sure the children have fun now and at the same time hoping that the fun they have makes a more lasting impression and makes them smile the next day too.
Example: Valuing the middle
Of course, not everything is best when it is maximally large. In this case it is easy to define another Variable on the basis of existing Variable(s) for which more does mean better.
!Rule Optimal use of resources (best at about 100 visitors/week) !valued Number of visitors to youth centre
In the above example, the definition Rule could be something like this:
Example: Cumulative value
Surprisingly often, project success is measured and even defined at endline only, even when value has been delivered over the whole life of the project.
On the other hand, it is not necessarily true that a simple cumulative score is the right thing to measure either.
Generally, a defined Variable intended to record the Value accumulated over the life of a project could weight according to time, e.g. perhaps the rapidity of an intervention after a disaster is particularly important, in which case we might want to weight the recorded Value in order to take account of this, giving more weight to the first few days and weeks.
There are also important biases and illusions here, see the chapter on Kahneman.
As evaluators, sometimes we are told that certain things “just are” valuable to the client, and we are told how to calculate their Value, according to a pre-defined rule, usually given in an evaluation Terms of Reference.
But Value is a challenge to evaluators above and beyond these demands.
First, we might need to make intermediate calculations about the value or quality of something, for example the quality of some teaching, even though this is not of direct interest to the client and no specific criteria are given. If the quality of the teaching is relevant to understanding why some part of a project was not effective, the client will expect the evaluator to report on it.
Second, in some cases we may need to make assessments of quality or value even in connection with Variables which have been explicitly defined by the client as “what counts”, even where this involves disagreeing with the client. This might sound like an unwelcome moral or political obligation which I want to push for moral or political reasons. But no, it follows from the nature of our work as evaluators.
- There are myriad concepts involving like “quality” and “value” which we come across doing evaluation, which have no central defining characteristic but bear only family relationships to one another; and these myriad concepts fade gradually off into myriad others which are not obviously about worth or value: there is no clear-cut distinction between “facts” and “values” in evaluation.
- Disputes about value and quality in evaluation are disputes about whether evaluation Statements correctly reflect what they are meant to reflect (e.g. are they reliable and valid) and as such are no different from disputes about other terms and concepts. Our duty as evaluators includes ensuring that our evaluation Statements (especially but not only those involving and implying value and quality) are correctly understood.