Summary

This is a summary of some of the key definitions, ideas etc in the book, in the order in which they appear, with links to the relevant chapters.

(There is a bug which is eating some of the apostrophes and formatting in this summary - sorry.)

The book is disguised, in a light-hearted way, as an introduction to Theorymaker, the fictional dialect of a fictional island, Theorymaker Island. Learning Theorymaker will help you express evaluation ideas and concepts precisely. Written Theorymaker looks, and is, a lot like English, though it is structured in a special way. But mostly inhabitants of Theorymaker Island, who call themselves Theorymaker native speakers, communicate using a special kind of diagram. This is especially popular with the younger ones who, of course, have an app for that. The diagrams correspond exactly to the written language but they are more fun and easier to understand.

Theorymaker.info is a free website which converts written Theorymaker to diagrams.


English speakers can just use bits of Theorymaker without learning the whole dialect.

You can pick and choose which Theorymaker phrases you want to use.


Theorymaker native speakers are not trying to sell you new evaluation methods or templates or tools. We really have enough of that already. Its much more about understanding and discussing and getting clear about some of the basic concepts we use when we do evaluation.


There are just a couple of Theorymaker translation rules for translating Theorymaker to and from English. They are printed in light purple boxes.

Whereas Theorymaker slang gives you some ways to simplify Theorymaker. So slang rules translate Theorymaker into Theorymaker. Rules for using Theorymaker slang are printed in pink boxes (you can see some below). Theorymaker Slang allows us to introduce new, alternative phrases into Theorymaker, defined in terms of existing phrases.


Words for talking about Theorymaker, aka meta-Theorymaker, are presented in blue boxes like this. We use meta-Theorymaker for example to specify what counts as a Theory or a Statement in Theorymaker. Unlike Slang, meta-Theorymaker is just English and isnt really part of Theorymaker.

Usually in every-day English we dont need to bother with specifying which alternatives are being excluded when we make a statement about something. But Statements in Theorymaker are a bit more precise: we are required to list or somehow mention the alternatives. The point of this isnt added precision. The point is to be able to construct Theories, to show how something being one way (rather than another) contributed to another thing being one way (rather than another).

The most affected area was  *East--* ((North, South, East, West)) Province.

This Statement says that the most affected area was East Province rather than any of the (three other) alternatives. The corresponding diagram has a black symbol to the left, one of various different shapes, which reminds that this is a real Variable we can observe or go out and measure or check; in this case the black symbol has a jumbled brick shape tells us that this is a Statement about an unordered or nominal Variable.

If you hover your mouse over the word TEXT to the top right of the diagram, you will see the equivalent written Theorymaker.


A Statement is just like an English statement but with two tweaks. First, one part (the factual Level) is followed by a double hyphen and enclosed with stars (*); second, it is followed by a specification in double brackets of the alternatives (the Levels) which could go in place of the factual Level.

Example:

The traffic light is *orange--* ((red, orange, green))

The specification of alternatives can be explicit, e.g. red, orange, green or implicit e.g. 0-100, or even vaguer - see xx.


The traffic light is *orange--* ((red, orange, green))

is equivalent to the English statement

The traffic light is orange

but the alternatives are made more explicit than in English. You can also think of it as:

The traffic light is orange (possibilities are red, orange or green)

If more than one element is marked, e.g.

The traffic light is *orange,green--* ((red, orange, green))

this is equivalent to the English statement

The traffic light is orange or green. 

Similarly,

The child's height is *165cm--* ((positive continuous numbers))

is equivalent to the English sentence

The child's height is 165cm.

whereas

The child's height is *165 to 175cm--* ((positive continuous numbers))

is equivalent to the English sentence

The child's height is between 165cm and 175cm.

The corresponding Theorymaker diagrams look like, for example, this:

The traffic light is *orange--* ((red, orange, green))

The set of alternatives within double brackets in a Theorymaker Statement are called the Levels of the Statement.

We can also say that the set of all the Statements associated with one Variable is the set of Levels of the Variable.

Sometimes each member of the set is listed explicitly, sometimes it is indicated in other ways e.g.by specifying two extremes, e.g. 1-10.

There may be an ordering of the Levels, specified implicitly or explicitly, e.g.:

The national security alert status is low ((low < medium < high))

The height of the child is 125 ((continuous number)) cm. 

In an expression like *X--Y*, X is the factual Level (or Levels) of the Statement (or just the Factual)

So in the Statement I scored *20%--* ((0-100%)), 20% is the factual Level.

In the Statement I don't know my exact percentage but I know I scored *15 to 20%--* ((0-100%)), 15 to 20% is also the factual Level - even though this is a range of Levels rather than a single Level.


A Fact is whatever makes a Statement true.


By convention, sometimes Theorymaker native speakers dont bother with is, are etc when writing out their Statements. So this:

The colour of the traffic light is  ((red, orange, green))

is equivalent to this

The colour of the traffic light  ((red, orange, green))    

Also, Theorymaker native speakers have an alternative form for writing out so-called no/yes Statements, those which can only be true or false. So this:

The law on the new flag is ((not passed, passed))

is equivalent to this

The law on the new flag is passed ((no, yes))

Some Theorymaker native speakers also tolerate nominal or interrogative forms such as these:

Whether the law was passed ((true, false))

Was the law  passed ((no, yes))?

The passing of the law ((not passed, passed))

Theories and Theories of Change are written down in Theorymaker as connections between Variables; so first we need to learn how to write down Variables.

Variables have labels like this

The security alert level was ((green < orange < red))

which we call, unsurprisingly, Variable-labels. (In this case, the three options are arranged into some kind of order of importance or severity, as shown by the < symbol.)

This Variable-label is a kind of template which can be used in any of the following Statements:

The security alert level was *green--* ((green < orange < red))
The security alert level was *orange--* ((green < orange < red))
The security alert level was *red--* ((green < orange < red))

we call this set of possible Statements the corresponding language-Variable.

Loosely, Theorymaker native speakers also use the word Variable for whatever the written language-Variable refers to, or, as Theorymaker native speakers say, whatever the written language-Variable reports.


This security alert level is ((red, orange, green))

Any line like of text like the one above is called a Variable-label. It is like a Statement but without any particular Level being marked as factual.


The set of all the Statements associated with a Variable-label is called a language-Variable.

In ordinary language we frequently understand and illustrate the idea of (causal) influence by appealing to examples involving Variables stretched across time. So we tend to think that (causal) influence is necessarily about change in a temporal sense. We talk about Theories of Change because we think we are trying to change things in a temporal sense, i.e.we want to see the Levels of a series of Variables go, say, continuously up, if we make a graph of it against time. For example, we want to see the percentage of young people registered to vote go up from week to week.

But in fact it doesnt always make sense to talk about trying to change a Variable. This is especially true for Variables which exist only once, like the number of people attending the opening of the 2024 Olympics. We cant try to change this Variable in the sense of making it go up, day by day or second by second, because it only happens once.

So it isnt very helpful if we try to claim that in general our projects and programmes aim to change things in the sense of going up or down over time.

Instead, Theorymaker native speakers talk about making a Difference, which is a much more satisfying and general way to talk about what we are trying to do with our Theories of Change.

Suppose we are running a campaign to improve access to the 2024 Olympics for people with restricted mobility. We think without the campaign, only about 5,000 such people would attend and we want to increase this number to 40,000.

In Theorymaker, a Difference is written like this:
The number of people with restricted mobility attending the 2024 Olympics *40 000 - 5 000* ((positive counting numbers))

You can think of the Level noted after the double-hyphen as a contrasting or even counterfactual Level.

As usual, the possible Levels of the Variable are noted in the double brackets.


Any Statement about a Factual Level (or set of Levels) of a Variable can be extended by writing another, different Level (or set of Levels) of the same Variable after the --

So if

The law was *passed--* ((no, yes))

is a valid Statement, so is

The law was *passed--not passed* ((no, yes))
    

note that the Factual and Counterfactual need not exhaust all the Levels of the Variable:

If

1) The project was funded with *1--* million USD ((continuous number))

is a valid Statement, so are

2) The project was funded with *1--less than 1* million USD, ((continuous number))

and

3) The project was funded with *1--0* million USD ((continuous number))

and even

4) The project was funded with *1--0.5* million USD ((continuous number))

English is often ambiguous about what counts as not funding the project with 1 million USD. Does that mean no funding at all, or just less than 1 million, or some other contrast like not .5 million, which was perhaps last years amount? Theorymaker makes an intended contrast clear.


If a Statement includes something of the form XY, this can be read as X, rather than Y.

This chapter begins the attempt to classify Variables (and language-Variables) according to their Levels and the relationships between them, looking first at discrete Variables.

The alert status is ((green < orange  < red)).

The law was passed ((no,yes))

The most affected area was ((North, South, East, West)) Province.

The black on/off symbol next to the Variable The law was passed shows at a glance that this is a so-called no,yes Variable.

The black symbol with a block going up to the right, next to the Variable The alert status, shows at a glance that this is a so-called ordinal Variable - a discrete Variable whose Levels can be compared with one another, which allows the Variable to express a sense of more or less.

In this chapter we also look at some more unusual examples of discrete Variables such as hierarchical and circular.


A discrete Variable has Levels which can be listed in advance.


A Variable is ordered if it is possible to say, for each of its Levels, whether it is in some sense bigger or smaller than any other Level.

By convention, Theorymaker native speakers also treat no/yes Variables as ordered too, with the yes Level being greater than the no Level.

In this chapter we look at Variables with indefinitely large sets of Levels, such as anything we can count or measure.

The child's weight is ((continuous))kg.

The class size is ((counting numbers)).

Numerical Variables include integer Variables, counting Variables, percentages and so on.

In a later Section, we look at what happens at the extremes. Can you have a reading age of 1000, and if not, why not?

This chapter looks at Features, things which are common between more than one Variable, like for example belonging to a specific person.

The height of child A, person=child A

The weight of child A, person=child A

The height of child B, person=child B

The weight of child B, person=child B

wrap=99

So in this case, there are four Variables but two are associated with child A and two are associated with child B. Theorymaker diagrams shown this Feature in a separate line within the Variable, with the name of the Feature in coloured type.


Features common to more than one Variable, spelt out in ordinary English, can be rewritten in the form of text following the name of the Variable,

This

The height of child A 

The weight of child A

The height of child B 

The weight of child B

can be rewritten like this:

The height of child A, person=child A

The weight of child A, person=child A

The height of child B, person=child B

The weight of child B, person=child B

(The commas are optional.)

Decorations are purely a way to change the appearance of Theory of Change diagrams, such as adding colours, without changing their meaning.

Some Variable with dazzling decorations; colour=blue7; shape=oval; fillcolor=blue3:green1

Lots of Theory of Change diagrams use different kinds of decorations to embellish the Theory they present. We see all kinds of things from clouds and stars to the size of the lettering or the position of a Variable within a diagram which maybe are supposed to have a clear logical meaning or maybe not.

We suggest when you make a Theory of Change, you make a clear distinction, as in Theorymaker, between things like symbols and colours which have a defined meaning and those which are just suggestive or literally only there to make the diagram look more attractive.

If you want to merely decorate your diagram, Theorymaker provides a whole bunch (see xx) of ways to do that. They are expressed in a way similar to Features, see above, but they are preceded by a semi-colon.

They make no difference to the Theorymaker idea expressed, only the diagrams look different.

So for example this diagram

Some Variable

has the same meaning as this diagram

Some Variable; colour=cornsilk

only the appearance has changed.

Before we look at Theories of Change in particular, we look at Theories in general. All Theories are built up of simple Theories, like this:

Student satisfaction with school

 Student feels they live up to expectations

 Student feels fulfilled

 Student feels supported and liked

A single child Variable is predicted by one or more parent Variables.

Here we can see a simple Theory predicting Student satisfaction with school.


A Variable label, followed on successive lines by one or more (usually different) Variable labels, each indented by one space, is called a simple Theory.

Usually we will just refer to these Variable labels as Variables.

The indented Variables are called the parent or influence Variables and the first Variable is called the child or consequence Variable. We also say that a parent Variable is upstream of its children, which are downstream of its parent(s).

The consequence Variable

 An influence Variable

 Another influence Variable

The consequence Variable may include a phrase beginning !Rule, which describes how the Consequence is influenced by the influence Variables, see chapter xx.

In the corresponding diagram, arrows go from the Influence(s) to the Consequence.


A simple Theory claims that the consequence Variable is directly influenced by the influence Variables. Any Rule, if specified, says how it is influenced on the basis of the Levels of the influence Variables. Concretely, (Pearl 2000) it means if the influence Variables are forced, manipulated, to take such-and-such Levels, the consequence Variable will take whatever Level is specified for it by the Rule.


Theorymaker shows how causal claims can be represented in terms of relationships between Variables governed by Rules. But behind this is a strong thesis, namely that all causal and even weakly causal claims can be represented as relationships between Variables. With weakly causal, we explicitly cover also relationships in which the Influencers make some kind of difference, even when they dont completely determine the Consequence.

Sometimes it is more natural to say

The Mechanism by which unconditional cash transfers are supposed to benefit women is not really clear

rather than something like:

There is no clear and agreed Theory which leads from unconditional cash transfers to benefits for women

after all, it is not quite the same to talk about what Theories people might have; sometimes we just want to talk about the Mechanisms without worrying about who believes what or says what.

So we introduce another Word in Capitals: Mechanism.


A Mechanism is just whatever the corresponding Theory is about. You could also say A Theory is the assertion of a Mechanism.

We look at how to join together simple Theories.

So this Theory says that the a students satisfaction with school depends on how much they feel fulfilled, supported and liked, and that they live up to expectations

Student satisfaction with school

 Student feels they live up to expectations

 Student feels fulfilled

 Student feels supported and liked

and this says that a students feeling that they live up to expectations depends on their performance

Student feels they live up to expectations

 Student performance

As there is one Variable which appears in both Theories, we can combine them like this:

Student satisfaction with school

 Student feels they live up to expectations

 Student feels fulfilled

 Student feels supported and liked



Student feels they live up to expectations

 Student performance

Here we can see a composite Theory built up out of two simple Theories, one predicting Student satisfaction with school and one predicting Student feels they live up to expectations.


If a Variable occurs in two different simple Theories, (but not both times as a consequence Variable11), the result of appending one beneath the other is called a composite Theory.


In a composite Theory in which a Variable V appears both as influence and consequence Variable, one of the places where the Variable appears as an Influence (this might be within a simple Theory or a composite Theory) can be replaced by the entire other Theory in which it appears as a consequence Variable. The replacement respects indents, so that the old simple Theory now appears with an additional indent.


A Theory is any Theorymaker text or diagram which is either a simple Theory or a composite Theory.


In a Theory of Change, any Variable with no Influence Variable (i.e., with no arrows coming into it) is called a Root Variable.


In a Theory of Change, any Variable with no Consequence Variable (i.e., with no arrows coming out of it) is called a Leaf Variable12.


This piece of slang is just a convenience if you want to list several Variables all on one line.

In any Theory in Theorymaker, you can rewrite this

A

 X

A

 Y

like this

A

 X;Y

and you can rewrite this

X

 A

Y

 A

like this

X;Y

 A

(And in these examples, X can be a single Variable or may already be itself a list of Variables.)

A Rule is any set of instructions which given the Levels of one or more parent Variables specifies a result in terms of the Levels of a child Variable. It is an algorithm which humans, and/or possibly machines, can follow in order to calculate some outcome on a child Variable in terms of the Level(s) of one or more parent Variables. We use Rules inside our Theories about Mechanisms to make more or less precise predictions about how things will turn out. Within Theories, Rules encode our more or less limited (causal) knowledge of how the world works.

Student satisfaction with school 

 Student feels they live up to expectations

 Student feels fulfilled

 Student feels supported and liked

The two red symbols on the predicted Variables show that we have some kind of Rule or principle, however vague, which tells us how the expected Level of the child Variable depends, at least to some extent, on the Level of the parent Variable(s). In particular, the rising red arrows (which are the default for simple Theories) show that we believe that in each case the influence of the parent Variable(s) are in some sense positive, i.e. the child Variable increases as the parent Variable(s) increase.

The upward-sloping black triangle shows that each Variable in this Theory is ordered - see xx.

We will look at the red half-circle symbol later.

We can think of the black symbol and the red arrow symbol as representing two different aspects of the same Variable: the data Variable represented by the black symbol and the Rule Variable represented by the red symbol. In a specific instance, the Rule will most often predict a Level which is more or less different from the Level recorded for the actual data Variable. So the two are in a kind of tension.


We already saw that a simple Theory may include the specification of a Rule, written as part of the Variable and preceded by !Rule.

A Rule is any set of instructions which given the Levels of one or more parent Variables specifies a result in terms of the Levels of a consequence Variable.


Theorymaker native speakers say they are zooming in when they replace a simple Theory with a composite Theory which reproduces the same Rule; and they say they are zooming out when they do the opposite; replacing a composite Theory with a simple Theory (with all the same exogenous Variables) with the equivalent Rule, i.e.a Rule which produces the same results on the output Variable.

Theorymaker is useful in exposing how things work in our Theories and Theories of Change because it provides us with some rough templates for the most common kinds of Rule, the ways in which Variables can influence one another.

In the model below, rather than making individual and independent positive contributions, motivation and ability multiply their effect on performance. (According to soft arithmetic for multiplication, which is defined even for non-numeric Variables.)

Student exam performance !Rule:multiply

 Motivation

 Ability

In the next model, the effect of external disturbances on performance is marked as negative.

Student exam performance

 Motivation

 Ability

 (!-)External disturbances

In the case below, the influence of motivation is marked as very weak, whereas the wide arrow shows the substantial influence of ability.

Student exam performance

 (Influence=very weak;width=2)Motivation

 (width=9)Ability

 (!-)External disturbances

In this next example, one rich Variable influences another via a rich Rule (marked with a ?).

Student mood ((Rich))  !Rule:Rich

 Type of activities ((Rich))  

Next, the performance is hypothesised to be optimal under a medium level of stress, falling off again as stress increases too far.

Student exam performance !Rule:updown

 Level of stress 

Below, a persons feeling of satisfaction with some task is highest when the level of challenge is balanced by motivation and resources.

Satisfaction !Rule:balanced

 Challenge;Motivation and resources

Here, a persons feeling of satisfaction with some task is conceived of as a ((no,yes)) Variable which is yes if and only if the persons motivation and resources are adequate for the task. So only if the motivation and resources are greater than or equal to some Level determined by the challenge, the person feels satisfied. This kind of configuration is quite common in project design.

Satisfaction ((no,yes)) !Rule:adequate<U+2507><U+2507>

 Challenge;Motivation and resources

In this final example, more details about the Rule are given in the text.

Student mood ((Rich))  !Rule:Rich effect of activity type lasts a long time 

 Type of activities ((Rich))  

The nature of the various links between parent and consequence Variables in typical logic models or Theories of Change is rarely explicitly specified. Sometimes this is not a problem, but in other cases serious confusion can arise.


Variable1 !Rule Variable2 has strong influence on Variable1 

 Variable2

can be rewritten like this; and in the corresponding diagram, the individual influence of the Variable is marked at the head of the arrow.

Variable1

 (Influence=strong)Variable2

 A !Rule B has an overall positive influence on A
  B

can be rewritten like this:

A
 (Influence=+)B

and similarly with negative influences.

In Theorymaker, marking a + at the head of an arrow suggests that its Influence is overall positive, and marking a - suggests the opposite.


A Rule between two ordered Variables is overall positive if the Effect of any positive Difference on the upstream Variable is also a positive Difference.


By convention, when confronted with a simple Theory in which there is no information about the Rule or even the direction of the influences, but in which the Variables are ordered Variables see above, Theorymaker native speakers make the following assumption:

Assume the influence of each Influence Variable on its Consequence(s) is roughly positive, unless otherwise stated.

Intervention Variables are those which an Actor plans to manipulate, to change.

The intervention Variable is marked with a green arrow in Diagrams. In most cases, like this, the Intervention is on a ((no,yes)) Variable, with the Intervention making the yes Level factual and the no Level counterfactual. So the default appearance of an Intervention is this:

A consequence

 !do An Intervention

We can designate any Variable as an Intervention Variable by using the !do mark and, generally, specification of an ordered pair of Levels of the Variable called the Factual and Counterfactual.

If the factual Level of an Intervention Variable is something like yes and the counterfactual Level is something like no, they can be omitted, as can the designation of the Levels (((no,yes))). So the above diagram is equivalent to this:

A consequence

 !do An Intervention

We can abbreviate given that the Intervention Variable takes the factual Level with just given the Intervention.

We learn to express Difference, Effect and Effectiveness in Theorymaker based on the concepts of Factual and Counterfactual.

*250--0* teachers were trained; *increased* ministry engagement 

 !do The project was funded with *1--0* million USD ((continuous number))

In this example, the Intervention is to fund the project with 1 million USD (rather than none). 250 teachers are trained (rather than none) and ministry engagement is increased (but by some unknown amount).

We can say for example that the Effectiveness of the Intervention, the Difference it makes, on the number of teachers trained is 2500, i.e.250 rather than zero. This case is typical in that the funding of the project only incompletely determines the number of teachers trained. Nevertheless, 2500 is judged the most likely Difference, other things being equal. Calculating the Difference is a task for Soft Arithmetic. In some cases it will not be able to consolidate the various possible outcomes (depending on the Levels of other important Variables) into a single Difference like 2500 and the expression of the Difference will be a (possibly nested) serious of conditional statements.

When an Intervention also includes one or more noise Variables, and the Levels of those noise Variables are fixed, usually because the Intervention has already happened and we have recorded the factual Levels: we can calculate the Effectiveness of the Intervention given that those noise Variables take a specific set of Levels; we can call this the Effect of the Intervention.

However the Effect is not so interesting in evaluation because it depends too much on coincidence. Effectiveness is immune to coincidence.


This

Current weight is *80--85* kg ((lo-hi))

can be replaced with this

The Difference across the Variable "Current weight" ((lo-hi)) is *80--85* kg

So a Difference across a Variable is essentially an ordered pair of its Levels, written between asterisks and separated by a double minus sign, like this:

*F--C*

where C is the Counterfactual and F is the Factual.

translation:

We can think of Difference as a kind of generalised subtraction which contrasts the Level under an Intervention with the counterfactual Level.


The Image of a Level(s) of an upstream Variable on a downstream Variable given some Theory which links them is the Level(s) of the downstream Variable dictated by the intervening Rules.

The Image of a Difference across an upstream Variable on a downstream Variable given some Theory which links them is: the Difference on the downstream Variable between the Image of the Factual and the Image of the Counterfactual: but is it the Difference between the Images of the corresponding Levels, or the Image of each of the Differences?

In general, when there is more than one parent Variable, the Image of a Level is a set of conditional statements, one for each possible combination of the levels of the other influence Variables, each presenting the effects of this difference on the consequence Variable, given the combination; each statement is weighted, where available, by the probability of each combination. So in some cases, where Variables are numerical and probabilities are known, the Contribution can be collapsed right down to a single quantity, using ordinary statistical calculations, but see the Chapter on Soft Arithmetic. In many other cases it can be partially collapsed but (again using Soft Arithmetic) will remain as a set of Differences.

So as the Image of a Difference is the Difference between the Images of the corresponding Levels, in the general case the component Differences will also be sets of conditional Statements.


In a Theory, any root Variable which is not marked as under the control of an Actor is called a noise Variable. Noise Variables may or may not have a probability distribution associated with them.


The Image of a Difference across a Variable V on a Variable U downstream of it, given a certain Theory, is called the Effect of V on U.


More generally, any global expression of a Difference can be highlighted with enclosing asterisks even if it is not written with the soft subtraction symbol --.

*Improved* teaching quality

 *Improved* teaching skills

  *More* teacher training



wrap=5

The currency of evaluation is Differences.


Given that the world is just exactly as it is, what holds open the idea of counterfactuality? How do we come to say more plausibly that the train might have arrived late, in which case we would have missed the plane, rather than that the solar system wasnt suddenly full of cheese? Is it the similarity of nearby possible worlds to this one (Lewis 1979)? No: Pearl suggests that it is the relative autonomy of the constituent Variables and Mechanisms which provides the hinges of counterfactuality which make explanation possible.

We already saw that we can use Features as a way of highlighting parts of Variable labels which are common between different Variables, e.g.

Student-teacher interaction is more solution-oriented

 Teachers attend training, Stakeholder=Teachers

 Teachers attend workshops, Stakeholder=Teachers

 Students attend workshops, Stakeholder=Students

 Students take part in discussion on social media, Stakeholder=Students

A more elegant solution is often to use grouping boxes like this:

Student-teacher interaction is more solution-oriented





-Activities, stakeholder=teachers

 Teachers attend training

 Teachers attend workshops



-Activities, stakeholder=students

 Students attend workshops

 Students take part in discussion on social media 

We can also use grouping boxes simply to highlight or separate parts of a Theory. Used like this, they mean virtually nothing. They just make the Theory easier to understand and, crucially, help us resist the temptation to use causal arrows for this purpose

Some outcome

-First part

 Some activity

  Some input activity



-Second part

 A further activity

  One more input

Theories often include a large number of similar Variables, for example similar Variables for many similar persons (e.g.the height of many children in a class). So in practice we often see different ways to group together these kinds of repeated Variables and often represent them as if they were just a single Variable. We often do this without thinking - on our Theory of Change, we put a box for one Variable for peoples attitudes to recycling and another for the law is passed without thinking that the first is probably conceived as a whole column of data, one for each person, whereas the second is just a single datum, yes or no.

Any set of Variables which differ only by one Feature can be rewritten as one for-each Variable.

For example, each of these Variables represents the temperature outside this house at a different time point.

Temperature outside this house,  time=yesterday

Temperature outside this house,  time=today

Temperature outside this house,  time=tomorrow

These can be rewritten like this:

Temperature outside this house    !for-each time (yesterday, today, tomorrow) 
- !for-each teacher 



Teacher ability

!Rule (mean)  Average Student achievement;Teacher feeling of work satisfaction, Rule ??



-- !for-each student 

 Student achievement

  Teacher ability

  Student motivation

   Teacher ability

-





Teacher ability

 School-level support to teacher development

Some Variables might be defined just at one particular point in time, such as the number of people attending the Olympics 2016 closing ceremony on July 1 or the performance of a student at a particular exam. Fixed time can be specified, usually with some kind of date, like this:

Number of people attending Olympics 2016 closing ceremony,  Time=July 1

otherwise we assume the time is relative rather than fixed.

Sometimes we have general time Rules which are valid for quite a range of time but do not apply to any particular time; nevertheless a delay is implied or can be specified like this:

Student exam performance

 (A bit later)Revision behaviour



wrap=8

As this vague delay of a bit later is the default, it can usually be left off.

This general time Rule can be applied in particular to fixed time-points:

Student exam performance, Time=exam in early summer 2018

 Performance-boosting workshop, Time=February 2018



wrap=8

We usually assume that the delay is implied to fit the gap between the time points, but we can specify it too:

Student exam performance, Time=exam in early summer 2018

 (3-4 months later)Performance-boosting workshop, Time=February 2018



wrap=8

When several Variables share a fixed time-point (e.g. baseline, or year_1), it can be very convenient to group them with grouping boxes.

-Endline

Student achievement (endline)

-Baseline

 Student achievement (baseline)

 Student ability

 Teacher skills

Our Theories about specific projects and programmes are usually situated in some fixed time - today, next month, next year etc. Whereas the kind of more universal social science Theories we (hopefully) draw on to get evidence for our Theories are usually situated in relative time.

In this chapter we look at some more details about how Variables can be positioned in time, relatively or absolutely.

Number of helpline calls answered   !for-each night

 Number of helpline staff available   !for-each night

The above Theory explicitly mentions two sets of Variables containing data for each night over many nights. We have already met for-each Variables; Theorymaker native speakers sometimes refer casually to for-each Variables as if they were individual Variables, even though they are really sets of similar Variables, differing only in one feature. For-each time Variables are sets of similar Variables which differ only in their time point.

If we were going to try to record this little Mechanism, it would generate a lot of data.

The above example is subtly different from the kinds of timeless examples we have met so far in this book:

Number of helpline calls answered in any evening

 Number of helpline staff available in any evening

That simpler form is more like a general rule which applies at any particular time. It doesnt matter if it applies many times or even only once.

Projects and programmes often mix one-off Variables (like a launch event or the parliamentary debate on a law), Variables which are repeated at discrete intervals, such as a series of training events, and Variables which range over continuous time. Often, these are not clearly distinguished in a Theory of Change. Theorymaker has some simple abbreviations for these:

Number of people attending opening event of Olympics 2024, Time=__^

 Initial media events, Time=^^_

 Social media buzz about the Olympics, Time=~~~

  Number and visibility of celebrities attending pre-launch event in 2023, Time=^__

  Initial social media activities, Time=~~_

  Initial media events, Time=^^_

Explanation:

  • Time=__^ says that this is a single one-off event (^) which comes around the end of the project and which is not defined in the preceding period __.
  • Time=^__ says that this is a single one-off event (^) which comes around the beginning of the project and which is not defined in the period __ which follows it.
  • Time=~~~ says that this is a for-each time Variable which stretches continuously in Time through the entire period.
  • Time=~~_ says that this is a for-each time Variable which stretches continuously in Time for most of the project but not in the final period.
  • Time=^^_ says that this is a for-each time Variable which is defined a small number of discrete times in the first part of the project but not in the final period.

These symbols ~ ^ and - are easily typed and meant to give a quick impression, at a glance, of what happens when in a project or programme. For a more complicated project with many phases, Theorymaker native speakers sometimes use longer chains of symbols. So for a six-year project might show the input of an annual workshop like this: ^_^_^_^_^_^_.

So far, we have considered only quite ephemeral for-time Variables whose Level at any point in time is dictated entirely by other Variables:

Average temperature of Irish Sea ~~~

 Incoming solar radiation on to Irish Sea ~~~

but this suggests that as soon as the sun stops shining, the temperature drops correspondingly. But a for-time Variable like this is called in System Dynamics23 a stock Variable, which means it behaves something like this:

Average temperature of Irish Sea ~~~

 Average temperature of Irish Sea ~~~

 Incoming solar radiation on to Irish Sea ~~~

This arrow from the Variable to itself is assumed by default to maintain the Level in the previous instant.

An eternal Variable whose Level never changes ~~~

 An eternal Variable whose Level never changes ~~~

Whereas in this case, a teachers skill is supposed to naturally decline from instant to instant, which might be a good reason to have some top-up training from time to time:

Teacher skill level ~~~

 (!-)Teacher skill level ~~~

 Top-up training ^^^

In this final example, we learn that the clients Level of animal phobia is likely to increase over time untreated, but if guided exposure to the feared animal is carried out (say, each week), then the Level of phobia will decrease from the previous time-point.

Client's animal phobia  ~~~

 (+)Client's animal phobia  ~~~

 (-)Weekly guided exposure to feared animal ^^^^

Theorymaker native speakers often find it convenient within their Theories to include calculations on other Variables. So they can apply a Rule to one or more Variables but this Rule is not a prediction of the Level of some other measurable Variable; rather it is just a Rule. Theorymaker native speakers sometimes call it a bare Rule. This Rule still, in a sense, creates a Variable - a bare Rule Variable, one that is defined solely in terms of the parent Variable(s).

So there is no text before the !Rule mark in the written Theorymaker; if there was, it would name a measurable Variable, one which could agree with the prediction or not. In the diagram, there is just a blank line at the top of the Variable.

So if we have a set of Variables for the number of health centres in each region, we can also define a Variable total number of centres.

!Rule Total number of health centres opened  

 Total number of health centres opened in North Region

 Total number of health centres opened in South Region 

If the parent Variables in a simple Theory are joined to the child Variable by definition, we just make sure there is no text before the : !Rule.


Any block of text means the same as the same block of text with the addition of any Definition, providing the defining Variables are already mentioned in the block.

!Rule Defined Variable   

 Defining Variable 1

 Defining Variable 2 

Every Theory is true in some particular context. We look a little more at what that means. We also look at how Theorymaker native speakers think about assumptions.

A Context for a Statement S is any Theorymaker Statement C such that S is true given C. So as a Statement may contain (see xx) a set of relevant Variables possibly linked into some Theory. Some of the Variables may be fixed to specific Levels.

In this case, we know that without soundproofing, I notice the children playing downstairs if they are loud. But specifying a context doesnt tell us anything about what would have happened outside the context, so we dont know how this would work out if we did have soundproofing.

-Context: Soundproofing *no--yes* 

Whether I notice the children downstairs ((no,yes))

 How loudly the children are playing ((lo-hi))

A Theory T is true in the context where Y=x just means that there is a larger Theory containing T, in which the Variable Y is fixed to Level x, in which T is true.

More generally, the context may consist of a Variable being fixed to one of a subset or disjunction of Levels; and it may consist of a set or conjunction of several Variables being fixed to single Levels and/or to disjunctions of Levels; and it might consist of a conditional constraint, e.g. the Levels of Variable A and Variable B combined must not add up to more than 10.


A Theory T is true under the assumption that Y is just another way of saying T is true in the context where Y, where Y is some Theorymaker Statement.


The difference between general and specific Theories is only one of degree; also between laws of social science and specific instances of their application. Every Theory applies in some more or less general context. Theories which apply to some more general (more vague) context can be adapted also to some more specific context, i.e.a context in which more Variables (and their Levels) are specified.

Whats an indicator? For Theorymaker native speakers, it is just another Variable.

The fake science of logframes has led many to believe that every item in a Theory needs to have additional things called indicators - which have to be easily operationalised and measured. But:

  • any such indicator is also a Variable. So if every Variable also needs indicators, do the indicators need indicators, which in turn need indicators? Of course not. That would be turtles all the way down.
  • any ordinary Variable may well also be clearly measurable; additional information may not be necessary. There are plenty of examples such as the passing of a particular law.
  • not every Variable needs measuring numerically; or at least, reducing it to numbers may not give it its best expression.

When we do find it difficult to record a Variable with sufficient accuracy, we may consider using a proxy Variable is used which is easier to measure and gives some information, but not perfect information, about the Variable of interest: and Indicator.

Average score on appropriate questionnaire ('indicator')

 Tolerant attitudes to other ethnic groups amongst young people (not measured directly)

In most cases we wont have space to display these indicators as separate Variables, so we can list them under the Variable of interest. But doing so doesnt change their status and doesnt mean that every Variable needs to have indicators.

Tolerant attitudes to other ethnic groups amongst young people (not measured directly), Indicator=average score on appropriate questionnaire

By and large were treating Theorymaker as adding some new expressions into English. But if you just want to use the theorymaker.info web app, youll want to know what you can type and what you cant.

Into the web app you can only type valid Theorymaker on its own, with no ordinary English. Valid Theorymaker is just one or more valid Statements and/or Theories (as defined xx and xx), with each one listed below the preceding one.

Perhaps surprisingly, the definitions of Theory of Change, Intervention and Valued Variable are inter-related. You cant really understand one without understanding the other two.

Informally, we can say:

If someone has a Theory how doing one thing can influence something else they value, they have a Theory of Change.

We mark a Variable we value with !valued (which is printed as a red smiley). So if this is our Theory, that means we value the outcome it contains and therefore presumably we are planning to !do the Intervention.

An outcome we value !valued 

 !do Something we can do

The Mechanism which a Theory of Change reports on is called a Project.


A Theory about how doing X can influence Y can be called someones Theory of Change if a) they can do X, b) they value Y and c) they believe the Theory.

Any Variable marked with !do is to be manipulated by some Agent, and this same Agent believes the Theory. We could call this Agent the Author of the Theory of Change. Usually the Author is us. Under the planned intervention, the Author intervenes to force the Variable to the factual Level rather than the counterfactual Level which it would have had without the intervention.


Any ordered Variable (an ordinal, numerical or intensity Variable) which is maximised by a person believing a particular Theory of Change is called a valued Variable.

Translation: If the Author knows and believes the Theory of Change and intervenes with such-and-such effects, we can assume that they desire the effect of the intervention according to the Theory of Change.


A valued Variable can be marked with !valued.


A Project is what a Theory of Change reports on. It is a Mechanism which has an implementing Actor who values certain Variables.

Theorymaker provides various ways of dealing with costs. Just as Actors try to maximise Value, they will try to avoid or minimise Costs. Costs are marked with a purple frown symbol in Theorymaker.

Something valued, influenced by the input !valued

 v: Some input, which costs !cost

 c: Some other input, which also costs !cost

We can think of cost Variables as just being a kind of Variable which makes a negative contribution to a Value Variable.

Costs can be directly summarised as negative Value using a Rule. This is the approach taken in Cost-Benefit Analysis (King 2016). (This is sometimes expressed as saying that cost and benefit are recorded in the same units, though this isnt really the point - height and width are expressed in the same units but they arent the same Variable.) The point of CBA is actually to combine Value and Cost into one Variable, e.g.as a difference or a ratio.

!Rule Overall value (v-c) !valued

 Some outcome, influenced by the input 

 v: Some input, which costs !cost

 c: Some other input, which also costs !cost

Limited resources, for example within a Project once it has started, can be modelled using constraints, e.g.

-Constraint: sum of i1 and i2 must not exceed 400 USD



Some outcome !valued

 i1: some input ((positive USD))

 i2: some input ((positive USD))

As evaluators, sometimes we are told that certain things just are valuable to the client, and we are told how to calculate their Value, according to a pre-defined rule, usually given in an evaluation Terms of Reference.

But Value is a challenge to evaluators above and beyond these demands.

First, we might need to make intermediate calculations about the value or quality of something, for example the quality of some teaching, even though this is not of direct interest to the client and no specific criteria are given. If the quality of the teaching is relevant to understanding why some part of a project was not effective, the client will expect the evaluator to report on it.

Second, in some cases we may need to make assessments of quality or value even in connection with Variables which have been explicitly defined by the client as what counts, even where this involves disagreeing with the client. This might sound like an unwelcome moral or political obligation which I want to push for moral or political reasons. But no, it follows from the nature of our work as evaluators.

  • There are myriad concepts involving like quality and value which we come across doing evaluation, which have no central defining characteristic but bear only family relationships to one another; and these myriad concepts fade gradually off into myriad others which are not obviously about worth or value: there is no clear-cut distinction between facts and values in evaluation.
  • Disputes about value and quality in evaluation are disputes about whether evaluation Statements correctly reflect what they are meant to reflect (e.g.are they reliable and valid) and as such are no different from disputes about other terms and concepts. Our duty as evaluators includes ensuring that our evaluation Statements (especially but not only those involving and implying value and quality) are correctly understood.

As evaluators, sometimes we are told that certain things just are valuable to the client, and we are told how to calculate their Value, according to a pre-defined rule, usually given in an evaluation Terms of Reference.

But Value is a challenge to evaluators above and beyond these demands.

First, we might need to make intermediate calculations about the value or quality of something, for example the quality of some teaching, even though this is not of direct interest to the client and no specific criteria are given. If the quality of the teaching is relevant to understanding why some part of a project was not effective, the client will expect the evaluator to report on it.

Second, in some cases we may need to make assessments of quality or value even in connection with Variables which have been explicitly defined by the client as what counts, even where this involves disagreeing with the client. This might sound like an unwelcome moral or political obligation which I want to push for moral or political reasons. But no, it follows from the nature of our work as evaluators.

  • There are myriad concepts involving like quality and value which we come across doing evaluation, which have no central defining characteristic but bear only family relationships to one another; and these myriad concepts fade gradually off into myriad others which are not obviously about worth or value: there is no clear-cut distinction between facts and values in evaluation.
  • Disputes about value and quality in evaluation are disputes about whether evaluation Statements correctly reflect what they are meant to reflect (e.g.are they reliable and valid) and as such are no different from disputes about other terms and concepts. Our duty as evaluators includes ensuring that our evaluation Statements (especially but not only those involving and implying value and quality) are correctly understood.

We look at three commonly used generic factors for understanding human behaviour - know-how, resources and motivation - and rediscover them as inherent aspects of Theories of Change as understood by Theorymaker native speakers.

Theorymaker native speakers use the word Variable to talk about both language-Variables and the Variables they refer to. So you can say number of visitors to the new centre and mean either the language Variable which is part of your plan, your shared model of how things are going to work; you can think of this as a social phenomenon or as the paper-and-pencil (or more likely electronic) manifestation of this shared model. Alternatively you can mean the actual number of visitors, a Variable which you can observe or measure e.g.by going to the centre, without any knowledge of any plan or Theory.

If you are busy developing a Theory of Change, you wont have time to worry about the difference. But when we come to look at how our evaluation Report is supposed to reflect various aspects of the project being evaluated, we will need to look at the way language-Variables refer to or report Variables in the rest of the world beyond the plan, and that is what we will do in this chapter. In this Chapter we will also explore how language-Variables are themselves Variables.

The report says 'The height of child X is ...' ((positive continuous numbers)) !Rule: some reporting or measurement Rule ensuring the report of the height reflects the actual height

 The height of child X is ((positive continuous numbers)) 



direction=TB 

So this diagram shows a causal link between the actual height of the child and the report of its height, say on a school health record. This causal link will normally be a relatively involved one which we could probably zoom in on to identify other intervening Variables: we can call the whole thing the reporting Mechanism.

Note - a Variable-label is about the Variable. But its the language-Variable which reports on the Variable. A Variable-label is a series of marks or sounds whereas a language-Variable is an actual Variable, a set of Variable-labels.

We define Evaluations as Reports on the value of Theories of Change, i.e.Assessments of the valuable Difference made by a Project.

So Evaluation in this sense necessarily involves assessing to what extent an Intervention can actually add Value, i.e.maximise valued Variables.

So designing an evaluation process involves essentially designing a complex Mechanism which takes a Project as its input and outputs a Report on the quality of the Project; did it do something valuable?


A key task of the evaluation, and/or of the evaluation Terms of Reference, is to agree on a Theory, the Evaluation Theory, which is usually composite, to generate a Statement of the reporting Variable(s), the Evaluation Statement, which may also be arbitrarily composite.

The Evaluation Theory often forms a big part of the evaluation ToR and the evaluation inception report.

(The same applies to any kind of Report, not just evaluation Reports.)

Establish the best Theory of Change using the best evidence you have.

Combine multiple sources of evidence.

The scope of an evaluation isnt just the extent of the sample. Its the range (geography, stakeholders, time, etc) to which we are prepared to generalise findings from the sample, i.e.the most general (inclusive) population for which our sample is representative.

Understand and refine the evaluation ToR aka the Evaluation Theory which takes the Project with all its Variables as input and discusses how to calculate the ultimate evaluative Variables like Sustainability, Relevance etc

Ordinary arithmetic takes numbers as input, and produces other numbers, usually fewer of them or expressed more simply.

We need soft arithmetic to be able to deal with data which is not clearly defined.

Most often, soft arithmetic takes comparisons as input, and produces other comparisons, usually fewer of them or expressed more simply.

Ordinary arithmetic is often a very important part of evaluation. But Soft Arithmetic is even more important and used even more frequently.

We use Soft Arithmetic all the time for example when we make comparative judgements in evaluation. Comparative judgements - saying that one thing is better (or bigger or more long-lasting) than another - are central to evaluation, both as its raw material and as its output.

For example:

If

the situation in province A is better than the situation in province B,

and

the situation in province B is better than the situation in province C,

then

the situation in province A is better than the situation in province C.

This is a kind of calculation, a kind of syllogism, which we can do without numbers. Its ancient, and we do it all the time. Some calculations are possible with this Soft Arithmetic without numbers; others are not. So without numbers,

If

the situation in province A is better than the situation in province B,

and

the situation in province A is better than the situation in province C,

then

without a numerical score, we cant compare B with C.

So just as with numbers, calculations with comparisons can help us some of the time, but not all of the time.

Just look at the table of contents for this Chapter (in the web version, this is in the menu at the left) to see some of the wide range of problems for which we can and do utilise Soft Arithmetic.

We show how some basic evaluation Variables like Relevance can be calculated (using Soft Arithmetic) according to certain Definition Rules.

If required, pre-calculate the likely theoretical Levels of the evaluation Variables: what is the likely Effectiveness, Impact, etc., of this project, given what we already know? This might involve information you calculate yourself using Soft Arithmetic (e.g.putting existing information on outcomes in relation to existing information on inputs in order to assess cost-effectiveness), and info you get directly, e.g.from a previous study.

The Contribution of d on V is the Effect of d on V relative to V. We can write this in Soft Arithmetic as Effect of d on V // V, whereby the // in Soft Arithmetic is only like normal division in special, numerical cases, but serves as a fuzzy analogue of division in more general, i.e.non-numerical cases.


If the effect of U on V is c, c can be attributed to U.

Assess quality and coverage of existing information, particularly if the Project has now been completed - not just the factual Levels of key Variables but any updated information on the strength and nature of the hypothesised links between them.

Decide where the biggest information gaps are;

Gather additional data, i.e, conduct the evaluation fieldwork;

Use Soft Arithmetic to update the model with the new evidence and recalculate the key evaluative Variables.

Ensure that evaluation Statements provide useful and actionable information, in the context in which the Report will be read. For example, simply accepting or rejecting hypotheses is not as useful as a best guess, with an indication of how confident we are in the Statement, about what to do next.

In this chapter, we look at what happens when Variables arent clearly defined but still have too much going on to be thrown away.

Most of our actual Theories of Change are full of Variables like this (even though we might have to squash them into a more precise format if we want to use them to manage a project.)

It is quite easy to say and understand:

The child's weight is  *25kg--* ((continuous)).

even though we would find it hard to agree on a precise lower or upper bound to this range, and even though the only weighing machine we have is rather inaccurate.

It is remarkable how often we have to deal with Variables for which we are quite confident in some middle part of the Variables range, but we could never say what are the absolute maximum or minimum. Obviously it makes no sense to say that a child weighs 0g, and in a different way it hardly makes sense to say a child weighs 100000000kg.

Any kind of Variable can be more or less vague. And often we arent even sure exactly which kind a Variable is, but still it might be useful to us.

We already looked at some examples of Variables with discrete, limited sets of Levels. Now we will look at another kind: intensity Variables. Different instances of intensity Variables can be easily compared with one another, but we find it hard to assign numbers to them.

My interest in this assignment is *moderate--* ((lo-hi))

Here, someone is making an expression of intensity without reference to any kind of number. We might find it hard to specify all the possible ways to complete the sentence meaningfully but nevertheless we can make comparisons: I might know that my interest is lower than the previous assignment - or even if I am not sure any more, it certainly makes sense to say that this level of interest is lower than last time.

The black symbol like a rising graph shows at a glance that the Levels of this Variable can be compared with one another.

Mostly, Theorymaker native speakers just use the generic expression ((lo-hi)) and variants such as ((lo-0-hi)), see below, for these kinds of Variables.

In fact, ((lo-hi)) is the default for all Variables in Theorymaker. If you dont specify any Levels, Theorymaker native speakers assume you are talking about a ((lo-hi)) Variable, and even if you do, the ((lo-hi)) text is not shown in the Diagram.


An intensity Variable is one for which the Levels are specified not explicitly but in one of the following forms: ((lo-hi)) ((0-hi)) ((lo-0-hi)).


Some Variable ((lo-hi))

can be rewritten just like this:

Some Variable

Some Variable ((no, yes))

can be rewritten just like this:

Some Variable

We already looked at Variables which are vaguely defined, in the sense that we cant list or even precisely specify all their Levels. An example was reading age, which quite clearly covers the ages, say, 4 to 18 but it is not clear (or very important) if there is a reading age of 0, or 90, or 900.

In this chapter, we look at what happens when Statements are more or less vague.

Much of our data we get as evaluators consists of vague Statements.

The child's weight is around *40kg--* ((continuous))


The country with the most biodiversity in the world is *one of the poorest countries* ((list of all the world's countries))

Although in this example, the Variable is not vague (except: is Kosovo a country? Palestine?), but the factual Level is quite vague. What counts as poor? What does one of the poorest mean? One of the 100 poorest? 50?

The important thing about vague Statements is that we can still sometimes do important work with them, thanks to the magic of Soft Arithmetic.

In most real-life cases, the child Variable is not completely determined by its parents. In fact in many projects we only have the roughest of ideas how much of a contribution our intervention can really make, and it may not be very big.

So we can say a Rule is complete if it more or less totally determines the Levels of the parent Variables, slight if there is only just a discernible connection and incomplete otherwise.

Family spending power !Rule:incomplete 

 (Influence=-)Income 

 (Influence=-)Debts 

 (Influence=+)Savings  

The circle which follows the arrow it shows roughly how much of an influence the parent Variable(s) have. In this case, the half-filled circle, which is also the default, shows we think there is some noteworthy influence but not total control.


In the specification of a Rule, the signs !Rule:slight !Rule:incomplete and !Rule:complete can be added to show that the contribution of all the influence Variables taken together is just slight, incomplete or effectively total, respectively. The corresponding diagrams show empty, half-full and full circles, respectively.


If not otherwise specified, Theorymaker assumes the combined influences on a Consequence only partially determine its Level. So given a Simple Theorem with no Rule and no circle, or whose Rule does not specify how completely the Consequence is determined by the Influence(s), we can assume that the influence is incomplete - neither slight nor total.

So this

A

 B

is equivalent to this:

A !Rule:incomplete

 B

In this chapter, we look at rich Variables.

The first Variable in this sentence is a rich Variable:

Because of the contents of the report, the Board increased project funding.

We can imagine various versions of the report, and tiny variations on those different versions, all of which would pretty certainly have led to the Board making the same or a similar decision; and we can imagine many other versions of the report which would have led to a different decision. So the contents of the report" is behaving just like any other Variable - it can take different Levels, with corresponding consequences.

But we can hardly ever specify in advance all the Levels of these kinds of Variables. There is a bewildering variety of possible versions of the report, and we wouldnt even know how to start listing or categorising them all, and they certainly dont have a single obvious higher or lower, better or worse. You can always try to break down a rich Variable into many subsidiary Variables representing for example the length of the report, how positive the language is, and so forth. But these are not the same as, or better than, the original rich Variable, and they would seem to lose almost all of the relevant information in the report.

Nevertheless, we use rich Variables all the time to help tell causal stories.

Rich Variables are packed with information: they have high information entropy.

Rich Variables can be treated as Open or Closed. For example,

Variables like my sons mood and the findings of the report are real, fully-fledged Variables just like Germanys GDP. They are not waiting to become Variables when some scientists get around to inventing a way of measuring them objectively. Theorymaker recognises that, beyond a few islands where hard, valid and reliable numbers are available, most of the time people, including evaluators, have to make judgements on the basis of vague, fuzzy and poorly defined data which may be nothing more than information like most stakeholders thought the project was more successful than last year.

If you want to evaluate a project, you have to be able to formulate ideas about how it works, and usually that means getting down and dirty with fuzzy Variables and fuzzy Rules. Using Theorymaker helps evaluators write down and deal with Variables which are hard to even define, let alone measure. Theorymaker is flexible and generous enough to be useful in actual evaluation contexts with this kind of data.

Example of a fuzzy Rule between two fuzzy Variables: The child copies the teachers movements.

child's movements ((fuzzy)) !Rule:fuzzy !Rule copies

 teacher's movements ((fuzzy))

If a downstream Variable has an incomplete Definition we can also call it an open Definition. A complete Definition we can also call a closed Definition.


Evaluators have a moral and/or professional duty to ensure that, the Labels of defined Variables are reasonable and do not, for example, promise more than they can really deliver.

These are the most common kinds of Variables Theorymaker native speakers use.

  • If you want to indicate that there is an ordered list of alternatives, write a < in it somewhere.
  • If you want to indicate that there is a list of alternatives, write a comma in it somewhere, even if you arent explicitly listing the alternatives.
  • Other kinds can be marked with ((?)).
The alert status is ((green < orange  < red)).

The law was passed ((no,yes))

Hottest country this year ((list of all countries,))

The most affected area was ((North, South, East, West)) Province.

The child's weight is ((continuous))kg.

The class size is ((counting numbers))

My interest in this assignment is *moderate--* ((lo-hi))

Brother's mood ((fuzzy))   

The contents of the report ((rich))   

The bird's species ((?))     

Whenever there is no clear inter-subjective agreement about something or agreed algorithm about key material, the evaluator may have to make a Judgement. See Scriven on probative inference, etc.

The aim is to make a suggestion capable of post-hoc consensus on this suggestion. But even when there is, the evaluators making of the suggestion counts as a Judgement.

Evaluation constantly involves making judgements about the meaning of items within reports, plans, contracts etc.

  • In the vast majority of cases there is no significant disagreement about the meanings of words used, e.g. questionnaire, police commissioner, etc.
  • In some other, important, cases, project documentation and practice gives us ways to agree, algorithms, about other concepts which would otherwise be unclear such as project success, beneficiary satisfaction, etc.
  • In a final set of cases, there might be important disagreement about key concepts such as negative effect, gender-appropriate, valuable with no obvious way to adjudicate them. This falls under making a Judgement. One way the evaluator can help clarify is to make suggestions for plausible existing definitions, agreements etc. So the evaluator might say, the most widely accepted definition of abuse in this field is the xyz definition, and they say . Even there might be post-hoc agreement to take this suggestion, the evaluators making of the suggestion counts as a Judgement.

A Statement should accurately report what it is intended to report and understood to be reporting.


If the readers of (or listeners to) the Statement understand the correct Variable, we can say the Statement and the reporting Mechanism are valid. So Validity is the extent to which a report Variable actually reports the Variable its audience understand it to be reporting, and not some other Variable.

Here we look again at what we mean by one Variable reporting another. In Theorymaker, a Report can have various different formal properties: sub-types are Judgements, Descriptions, Measurements etc. We are particularly interested in reporting on the quality or value of something. Ultimately we will see that evaluation involves Reporting on the Value of Projects.

Theorymaker native speakers use the word Variable to talk about both language-Variables and the Variables they refer to. So you can say number of visitors to the new centre and mean either the language Variable which is part of your plan, your shared model of how things are going to work; you can think of this as a social phenomenon or as the paper-and-pencil (or more likely electronic) manifestation of this shared model. Alternatively you can mean the actual number of visitors, a Variable which you can observe or measure e.g.by going to the centre, without any knowledge of any plan or Theory.
If you are busy developing a Theory of Change, you wont have time to worry about the difference. But when we come to look at how our evaluation Report is supposed to reflect various aspects of the project being evaluated, we will need to look at the way language-Variables refer to or report Variables in the rest of the world beyond the plan, and that is what we will do in this chapter. In this Chapter we will also explore how language-Variables are themselves Variables.

The report says 'The height of child X is ...' ((positive continuous numbers)) !Rule: some reporting or measurement Rule ensuring the report of the height reflects the actual height

 The height of child X is ((positive continuous numbers)) 



direction=TB 

So this diagram shows a causal link between the actual height of the child and the report of its height, say on a school health record. This causal link will normally be a relatively involved one which we could probably zoom in on to identify other intervening Variables: we could call the whole thing the reporting Mechanism, following a reporting Rule.


In Theorymaker, a Variable which says something about another Variable is called a Report Variable. Its Rule is then called a reporting Rule. (We will go into this phrase says something about in more detail later xx). Its Levels correspond exactly to the Levels of the reported Variable:

'Variable B has the Level ((x))'

where x ranges over the Levels of B. The only way to distinguish a reporting Variable from the Variable it reports on is the quotes.