Composite Theories, valid Theorymaker

We look at how to join together simple Theories into composite Theories.

Student satisfaction with school

 Student feels they live up to expectations

 Student feels fulfilled

 Student feels supported and liked



Student feels they live up to expectations

 Student performance

Here we can see a composite Theory built up out of two “simple Theories”, one predicting Student satisfaction with school and one predicting Student feels they live up to expectations.

Simple Theories form the basic units which we can join together to form composite Theories, and in particular to build Theories of Change about our projects and programmes. In part 3 of this book we will see how we can also use these simple Theories to construct designs for evaluating those projects and programmes. These simple Theories are really all we need for these tasks. And they mirror the basic units of a fairly new scientific paradigm based around the work of Judea Pearl (Pearl 2000), which links the concepts of causality, probability and reasoning and puts them on a sound theoretical (& mathematical) footing. Algorithms based on this way of building a theory are being used right now to help artificial intelligent systems form working theories of the world around them.

So if we can show how to build up Theories of Change and evaluation designs using just these basic units, we have gone a long way to also putting Theories of Change and evaluation designs on the same, sound, theoretical footing.

The Law of valid Theorymaker

This single Law - with just three parts - tells you not only what counts as valid Theorymaker but also how to translate any Theorymaker sentence into English - i.e. it gives what logicians would call a syntax and a semantics for Theorymaker.

Basically it says that Statements and simple Theories and combinations of them are all valid Theorymaker.

A Statement is valid Theorymaker. Its translation is trivial - any Statement in Theorymaker is by definition equivalent to an ordinary English sentence.

A simple Theory is valid Theorymaker. Its translation is also simple: if the influence Variables are manipulated to take such-and-such Levels, expect to see such-and-such Levels of the consequence Variable, depending on any details set out in the Rule.

If two blocks of text are valid Theorymaker, so is the block of text formed by appending one beneath the other. The translation of the combined block of text is just the English translation of the first block and the translation of the second block.

There are quite a few caveats, see xx.

Note we say “if the influence Variables are manipulated to take such-and-such Levels” rather than “if the influence Variables are observed to take such-and-such Levels”.

Theorymaker is constructed from a very limited number of primitive parts: just Statements and Theories built up out of Variables. On the basis of these we can also define many further concepts like Impact, Assumption, Efficiency, Sustainability, Contribution, and so on.

These definitions (“Contribution”, etc), called Theorymaker slang are supposed to merely tweak the ordinary English we use to talk about evaluation.

Theorymaker is comprehensive

Learning Theorymaker will provide you with tools comprehensive enough to model many different kinds of real-life projects whether they are structured like a traditional logframe or include adaptive elements, diverging scenarios, feedback loops, chaotic elements, vague and emergent goals, as well as different agencies which value different outcomes differently.

Not only individual projects but also big parts of other evaluation approaches can be expressed very nicely within Theorymaker. So if you can have a serious discussion within some other framework, you can have the same discussion, just as rich, and perhaps a bit more clearly, in Theorymaker.

This means that proponents of, say, Outcome Mapping and Randomised Controlled Trials can actually have a substantive conversation in Theorymaker without spending most of it arguing about definitions - or misunderstanding each other because of them.

How is a Statement in Theorymaker equivalent to an English statement?

If the Variable is this:

The traffic light is ((green, orange, red))

… the Statements look like this:

The traffic light is *green--* ((green, orange, red))

The traffic light is *orange--* ((green, orange, red))

The traffic light is *red--* ((green, orange, red))

… which are equivalent to these English sentences:

The traffic light is green 

The traffic light is orange 

The traffic light is red 

Notice that the only difference is that in Theorymaker, every Statement indicates also all the possible alternatives.

Why isn’t one Variable on its own valid Theorymaker?

The law was ((not passed, passed))

The Ambassador ((didn't support, supported)) the campaign

Note neither of the two lines is a Statement, and cannot be true or false, but the combination can certainly be true or false. For example, the Ambassador’s support might be quite immaterial to the passing of the law, in which case this Theory would be false:

The law was ((not passed, passed))

 The Ambassador ((didn't support, supported)) the campaign

Combining simple Theories

If a Variable occurs in two different simple Theories, (but not both times as a consequence Variable12), the result of appending one beneath the other is called a composite Theory.

We already saw this:

C !Rule: some Rule

 A 

 B

Suppose we add another simple Theory which shows what, in turn, influences A.

A  !Rule some other rule 

 X 

 Y

 Z

We already saw that these two valid blocks of Theorymaker can be combined into one.

Now, if we combine these simple Theories into one block of Theorymaker, we have also got a composite Theory because it has more than one stage: one Consequence Variable (A) is also an influence Variable. In Theorymaker, it is called a “composite Theory”. Real-life Theories of Change are nearly always multi-stage Theories like this.

We will leave off the “!Rule” parts as they make no difference at the moment.

C 

 A

 B

A

 X

 Y

 Z 

Theorymaker automatically displays any composite Theory in a special way: it combines all the information about each individual Variable to produce a composite diagram.

So you can see that in Theorymaker, it is enough just to paste the two written Theories together to get the corresponding composite diagram. And the written Theorymaker above produces just the same results as the equivalent “slang” expression below, in which the second simple Theory about the Influences on A is substituted into the first simple Theory in which A is itself an Influence on C:

C

 A

  X

  Y

  Z 

 B

We can summarise this with another piece of Theorymaker slang:

In a composite Theory in which a Variable V appears both as influence and consequence Variable, one of the places where the Variable appears as an Influence (this might be within a simple Theory or a composite Theory) can be replaced by the entire other Theory in which it appears as a consequence Variable. The replacement respects indents, so that the old simple Theory now appears with an additional indent.

So in Theorymaker, the “influencing Variable(s)” are placed beneath, indented by one more space than the “consequence Variable” above, regardless of how far they are already indented. So if the first Variable is not indented at all, then the following Variables will be indented by one space.

The Law of valid Theorymaker also allows cases like this, in which there is more than one leaf Variable.

C 

 A

 B

D

 B

Finally, a definition of “Theory”:

A Theory is any block of Theorymaker which is either a simple Theory or a composite Theory.

But I am not sure if “composite” is the right word, as it suggest some kind of fixed architecture with everything having a defined place. Whereas in fact Pearl’s approach, and a Realistic Evaluation approach, see these simple Theories as in a sense self-organising into a system which may be to some degree resistant even to modification of some of the elements (Variables and Theories). (It is strange to say this, as the robustness comes from the notation, i.e. from Theorymaker or from some other DAG-based notation; we are not describing a physical process.) So one might prefer a phrase like “networked Theory” or “Theory patchwork” rather than “composite Theory”.

Why not have multiple child Variables in a Theory?

This would be possible too. It just seems simplest to keep it like this.

Some definitions: Root and Leaf Variables

Here are some useful ways of talking about Theories.

In a Theory of Change, any Variable with no Influence Variable (i.e., with no arrows coming into it) is called a Root Variable.

In a Theory of Change, any Variable with no Consequence Variable (i.e., with no arrows coming out of it) is called a Leaf Variable13.

In a Theory of Change, if a Variable B is directly or indirectly a Consequence of another a Variable A, we say B is downstream of A and also that A is upstream of B.

A leaf Variable B,   downstream of A

 Some intermediate Variable,   neither a root nor a leaf

  A root Variable A,   upstream of B

Composite Mechanisms

Just as we can see a larger Theory as being made up of simple Theories (each of which just has a single Consequence Variable), so we can see the corresponding Mechanism as a composite made up of corresponding simple Mechanisms.

To an evaluator, a project or programme is many things but most importantly it is a Mechanism - a route, however tortuous and complex and self-regulating - from intention to achievement, from upstream Variables to downstream Variables; and one of the evaluator’s key tasks is to capture this Mechanism in an adequate Theory.

Statements and Facts

xxxxx the factual state of a Mechanism.

By extension, we can also say that the Factual state of a (simple or complex) Mechanism (see xx) is the result of appending all the Statements about the Levels of the component Variables14 .

Slang: lists of Variables with semi-colons

In any Theory in Theorymaker, you can rewrite this

A

 X

A

 Y

… like this …

A

 X;Y

and you can rewrite this

X

 A

Y

 A

… like this …

X;Y

 A

(And in these examples, X can be a single Variable or may already be itself a list of Variables.)

The different formulations produce the same diagram:

B;C

 X;Y

  A
B

 X

  A

B

 Y

  A

C

 X

  A

C

 Y

  A      

What is not a Variable, what is not a Theory?

Let’s answer the question by looking at some diagrams which do not correspond to any Statements in Theorymaker.

In this example, groups and institutions are connected with one another, showing relationships perhaps of power or communications between different parts of a system. But these parts are people, things or institutions, not Variables. In particular, the lines below, e.g. “Farmers” are not Statements.

title=This is not a Theory of Change and these are not Variables!



Government 

 Farmers;Students

  Government

You can use Theory Maker for making any kind of directed or even undirected graph, you can even make organigrams with it, but that is not its primary purpose.

The stability / autonomy of simple Mechanisms

Pearl: The rule-based system, scientifically speaking, was on the wrong track. They modeled the experts instead of modeling the disease. The problems were that the rules created by the programmers did not combine properly. When you added more rules, you had to undo the old ones. It was a very brittle system.

One key idea is that we can rely on the existence of enough relatively stable mechanisms in the world, and that we can identify them correctly enough with limited cognitive resources: in a nutshell, NOT the hippy slogan of “everything is linked to everything”.

It is trendy to say “everything is connected to everything else”; in fact, of course, it isn’t. If that were the case it would never be possible to describe or know anything. The description of a single event would require recording also every other piece of information in the universe.

For example:

acceleration of object   !Rule force * mass

 mass of object

 force on object

Here, the accuracy and applicability of the Rule has been verified millions of times. Yet of course in any particular application, the acceleration of any particular object won’t be exactly what is predicted just from mass and force, because there will always be other influences from friction to solar wind. Nevertheless we can and do put people on the moon using a kind of Lego of physical formulae.

If the universe is “really” a simulation in some Matrix (or at least, its laws of physics are compatible with this “hypothesis”) then relatively stable Mechanisms would reduce astronomically the already astronomical computing requirements for the Matrix.

Just because your notation looks like a basket of laughably simplistic heuristic-like rules doesn’t mean the world is like that.

Absolutely. Does the fact that a particular notation is particularly useful tell us something about the world? Who knows.15 But you have to allow that it tells us something.

We could certainly discuss the question of what our basic knowledge really consists in - a bunch of heterogeneous rules-of-thumb, or a few poorly-understood fundamental principles from which everything can be derived? Pearl points out in this interview, that the latter approach was the basis of “expert systems” in the 70s and 80s, which consisted essentially of just a single compound Rule. This turned out to be much too fragile and breaks as soon as something changes, becoming actually meaningless. Purists could consider to say this is really the right approach but meanwhile children and Googles’ bots have long ago overtaken us and are running the world on the basis of rules-of-thumb.

Why explicitly causal approaches, like Theorymaker, are better than correlational approaches

… even in the physical sciences.

The above is an interesting example, because in physics textbooks the formulae are reversible, so this would be equally possible:

mass of object   !Rule force / acceleration

 acceleration of object

 force on object

… but we don’t in fact think of mass as being caused by force and acceleration. Pearl (Pearl 2000) uses this as an example of the problems of trying to capture the contents of scientific discovery in equations rather than directly in terms of causal statements .

References

Pearl, Judea. 2000. Causality: Models, reasoning and inference. Cambridge Univ Press. http://journals.cambridge.org/production/action/cjoGetFulltext?fulltextid=153246.


  1. why not? because we wouldn’t know what Rule to apply, except in the special case in which we can assume linear superposition, see xx

  2. Often called a “sink”.

  3. In possible-worlds terminology xx, this is a single point xx

  4. Some Wittgenstein reference