Making a Difference

In ordinary language we frequently understand and illustrate the idea of (causal) influence by appealing to examples involving Variables “stretched” across time. So we tend to think that (causal) influence is necessarily about change in a temporal sense. We talk about Theories of Change because we think we are trying to change things in a temporal sense, i.e. we want to see the Levels of a series of Variables go, say, continuously up, if we make a graph of it against time. For example, we want to see the percentage of young people registered to vote go up from week to week.

But in fact it doesn’t always make sense to talk about trying to change a Variable. This is especially true for Variables which exist only once, like “the number of people attending the opening of the 2024 Olympics”. We can’t try to change this Variable in the sense of making it go up, day by day or second by second, because it only happens once.

So it isn’t very helpful if we try to claim that in general our projects and programmes aim to change things in the sense of going up or down over time.

Instead, Theorymaker native speakers talk about making a Difference, which is a much more satisfying and general way to talk about what we are trying to do with our Theories of Change.

Suppose we are running a campaign to improve access to the 2024 Olympics for people with restricted mobility. We think without the campaign, only about 5,000 such people would attend and we want to increase this number to 40,000.

In Theorymaker, a Difference is written like this:
The number of people with restricted mobility attending the 2024 Olympics *40 000 - 5 000* ((positive counting numbers))))

You can think of the Level noted after the double-hyphen as a contrasting or even “counterfactual” Level.

As usual, the possible Levels of the Variable are noted in the double brackets.

It only makes sense to talk about “changing” a Variable if a) this is understood to refer to a set of “for time” Variables and b) if the counterfactual Level is fairly constant.

Let’s look at a case where (b) is not true. So for example, a third-grader’s reading skills stretch out over the whole year, so it is a for-each Variable, i.e a set of potential measurements, so criterion (a) is fulfilled. But if we introduce a new reading method for kids we don’t say we are trying to change their reading skills because at third-grade they are changing (improving) all the time anyway, even without our new method; the counterfactual is not constant, so (b) is not fulfilled. Instead of talking about change, we perhaps say we are trying to accelerate their reading.

“Making a difference” is more general than “changing things”

What are we trying to do with out project? In some circumstances it is quite reasonable to say we are trying to change things, e.g. to change the Levels of this and that Variable. But sometimes this is confusing. It is always better to say we are trying to influence the Variable or to make a difference.

A related example where neither (a) nor (b) are true: It would also be possible that we only care about the score at the end of the academic year, it is only then that we will measure and decide if we succeeded. In that case, criterion (a) isn’t fulfilled either. The Variable in question belongs to just one fixed time-point and for this reason too, we would not usually say we are not trying to make a “change” in the temporal sense at all.

In fact, the very word “Variable” suffers from the same ambiguity - it makes us think first of all of variability across time, like temperature going up and down, rather than the what it means in Theorymaker: variability across a set of possibilities.

The idea of change is much less general than the idea of making a difference, and depends on it. We really should be talking about “Theories of making a Difference”, not “Theories of Change”.

Of course, in the world of projects and M&E we are often very keen to identify and use Variables which do extend at least over the life of the project, i.e. “for time” Variables across the life of the project. Sometimes this leads to the misunderstanding that Variables have to be of that nature (this incorrect assumption can lead to an M&E office insisting on measuring a baseline for things that don’t even exist yet, such as the number of people using a centre which hasn’t yet been built).

Factual and Counterfactual

Any Statement about a Factual Level of a Variable can be extended by writing another, different Level (or set of Levels) of the same Variable after the --

So if …

The law was *passed--* ((no, yes))

… is a valid Statement, so is

The law was *passed--not passed* ((no, yes))
    

… note that the Factual and Counterfactual need not exhaust all the Levels of the Variable:

If …

1) The project was funded with *1--* million USD ((continuous number))

… is a valid Statement, so are

2) The project was funded with *1--less than 1* million USD, ((continuous number))

… and …

3) The project was funded with *1--0* million USD ((continuous number))

… and even …

4) The project was funded with *1--0.5* million USD ((continuous number))

English is often ambiguous about what counts as not funding the project with 1 million USD. Does that mean “no funding at all”, or just “less than 1 million”, or some other contrast like “not .5 million”, which was perhaps last year’s amount? Theorymaker makes an intended contrast clear.

So what is a counterfactual?

In the case of no/yes Variables, the counterfactual Level is just the negation of the Factual. Unfortunately, some discussions of the logic of evaluation fall into the trap of thinking that there is nothing more to say (xx DFID). However, there are plenty of Variables which are not in no/yes format.

For example, we might assess that without our programme, only about 5% of dropouts will return to school, whereas with the help of our intervention, we aim to improve the rate to 50%. 5% is not the opposite of 50%.