Who is this book for?
What kind of M&E?
What’s a project, what’s a programme?
Why I am writing this book
Motivation: what problems does Theorymaker address?
Frequently asked questions
But can you really apply all this stuff in practice?
Doesn’t “Learning Theorymaker” skirt all the political issues around evaluation?
Is there a lot of maths?
But isn’t it complicated?
But why is this approach so formalistic? Isn’t life really much messier?
Scepticism: is this approach too optimistic about plan-ability?
Thanks to …
Overview of the book
What makes a good Theory: A check-list
Theorymaker speech, writing and diagrams
Is Theorymaker a “framework” for evaluation?
Does Theorymaker give us a template for any programme theory or evaluation?
Theorymaker, a website which understands Theorymaker
theorymaker.info speaks Theorymaker!
What else you can do at Theorymaker.info
About Theorymaker diagrams
Translating between Theorymaker and English; Theorymaker slang; meta-Theorymaker
meta-Theorymaker: talking about Theorymaker
Variable-labels and Variables
Classifying Variables according to their Levels
Example: multiple ordinal variables
Quality & Quantity (Q&Q)
Note for Nerds: Why not just define Variables in terms of numbers and equality?
Variables: the acts of people and Gods?
Levels of a Variable: collectively exhaustive and mutually exclusive?
Making a Difference
“Making a difference” is more general than “changing things”
Factual and Counterfactual
A Counterfactual isn’t always just a negation
Different kinds of Variables: discrete Variables
Reminder: we normally don’t distinguish between Variables and language-Variables
Dichotomised continuous Variables
Intensity Variables with just two Levels (“dichotomised”)
Nominal variables: Statements with an implicit list
Ordinal Variables (step-by-step ordering)
Hierarchical Variables (classification)
Different kinds of Variables: Numerical Variables
Count (natural number) Variables
Continuous (rational) Variables
Percent and proportion Variables
Theories and Mechanisms
Mechanisms in Realistic Evaluation
Combining simple Theories
Some definitions: Root and Leaf Variables
Composite Statements and Facts
Slang: lists of Variables with semi-colons
a Variable, what is
The stability / autonomy of simple Mechanisms
Why explicitly causal approaches, like Theorymaker, are better than correlational approaches
Why isn’t one Variable on its own valid in Theorymaker?
Specifying a Rule
What does it mean to say one Variable influences another?
What does it mean to say
Variables influence another?
Combining more than one simple Theory: how do the Rules combine?
Rules and functions
Kinds of Rule
Independence of influence Variables
What do the arrows
? Causation, the mysterious force?
How is the algorithm to be specified?
Specific and general Mechanisms and Theories
Reality of zoomed-out Mechanisms
Rules can themselves be composite
Rules - different types
Saying whether an influence is positive or negative
Individual influences marked on arrows
Individual influences: - and + signs
Slang: influences are “overall positive”
(Multiply) overall positive
Binary Variables (“sufficient”)
Binary Variables (“necessary”)
Count-to-nominal: Choosing the winning candidate
More binary examples
Parallel contribution; overdetermination; sufficient not necessary conditions
Necessary but not sufficient
Necessary and sufficient
Multiple Variables - Q&Q
Multiple Intervention Variables behave like just one
Intervention Variables are effectively root Variables
Image, Counterfactual, Difference and Effect
Meaning of Factual and Counterfactual are conditional statements already given by the Rules
Calculating the Image
Causal, not statistical
Updating the Theory
Dealing with the contribution of other upstream Variables
Differences as raw data
Difference and subtraction
Soft arithmetic with Differences
Expressing difference with asterisks helps to focus on the actual Variables
Differences and “targets”
The currency of evaluation is Differences
Where do alternative possibilities come from?
Evaluators can’t escape thinking in Differences
Yes, Difference is being used in a technical sense
Variables grouped by stakeholder
Variables grouped by narrative
Variables grouped by organisational unit
Variables grouped by “Pillars”
Which is the superprocedure, the final Rule?
Limitations of grouping boxes
One Variable, one data point
Rules with for-each Variables
Rule is same across all cases
Rule differs from case to case
Aggregating for-each Variables
Using grouping boxes with for-each Variables
More to come!
Repeated and selected Variables
Fixed and Relative Time
A general time Rule
“For-each Time” Variables
More to come!
Rules with “for-each time” Variables
Rules with “for-each time” Variables which change over time
“Stock” or “Memory” Variables
More to come!
Definitions with arithmetical Variables
Definitions with lo-hi Variables
Definitions with thresholds
Reality of “defined Variables”
Strange duality of Theories, Mechanisms and Definitions: Interchangeability of perspectives
Tricks with Definitions: collapsing and factorising, etc
Combining several Variables into one with fewer Levels
Aggregating “for” Variables
And that is it!
Contexts and Assumptions
Context as a reduction in possibilities, in which certain Variables are set to certain Levels
Context as a particular set of Variables
Variables which only exist within certain Contexts
Assumptions, understood as equivalent to context
Assumptions that the Theory is adequate
Assumptions that the Mechanism will not be manipulated from outside
Different ways to show Assumptions/Context in a diagram
… As additional Variables
… Using a grouping box
… Labelling the arrow
… Using a Feature
Specific and general Theories and Mechanisms
Logical independence of Variables within a Context
Downstream proxy Variables as “indicators”
More to come!
Using an existing proxy
Tighter specification / means of verification
The formal syntax of Theorymaker for the web app
II From Theories to Theories of Change
Checklist: what makes a Theory of Change?
Value, Intervention, belief
Inter-related: Value, Intervention, belief
Valued Variables and intervention Variables in Theorymaker
A Theory of Change is a Theory about a Project!
Cost and resources
Resources and constraints
Cost and negative value
Valuing process too
Valuing a process
Other names for valued Variables?
Single way to cope with both project “outcomes” and other values
More about Value
Single source of Value?
Example: More than one valued Variable, possibly in causal relationships to one another
Example: Another example
Example: Valuing the middle
Example: Cumulative value
Value cannot be separated from facts in evaluation
Nothing special about value
Nothing trivial about closed Value
The Value of an Intervention
Know-how, Resources, Motivation
Understanding the motivation of others using
Theory of Change
III Doing Evaluation
Reporting: Variables and language-Variables
Evaluation: Appraisal of projects and programmes
Reporting and value
The evaluation Rule
The reporting Statement
Evidence for a Theory of Change
An Evaluation design requires a Best Adequate Theory of Change (“BAT”)
This is hard work
Adequate Theories; the 30-30 principle
Adequate theories: missing Variables?
Is there actually such a thing as an incomplete Mechanism, or are there only incomplete Theories?
Is evidence essentially correlational?
Measurement, data, evidence
Sources of evidence
Updating a Theory of Change based on what actually happened
Scope and generalisation
Constructing the evaluation Theory
Soft arithmetic in evaluation
Where do comparisons come from?
Another kind of Variable?
Let’s do some Soft Arithmetic
Judgements with equality
Danger: we can’t assume normal arithmetic principles
Aggregation conditioned by a Variable with known probabilities and numerical Levels
Aggregation conditioned by a Variable with known probabilities and soft Levels
Aggregation conditioned by a Variable with soft probabilities
Aggregation conditioned by a Variable with probabilities not defined, e.g. an Intervention
Comparison with standards
Comparison with rubrics
“Mere connections” in Judea Pearl’s causal diagrams
Can influences die out over the length of a chain?
More examples of Soft Arithmetic with numbers
Example: lives plus dollars.
Example: lives divided by dollars.
Example: lives minus lives.
Example: calculating the “mean” of a directed Variable
Example: deciding if the influence of a Variable is “roughly monotonic”
Multiplication by numbers-less-than-one as a way of expressing lack of certainty
Soft arithmetic for comparing costs and benefit Variables
Soft arithmetic for adjusting for the reliability of a source
Soft arithmetic for combining evaluation data from different kinds of sources
Soft arithmetic for combining intermediate evaluation findings
Soft arithmetic for combining evaluating intermediate evaluation findings
Monotonicity and Chaining
Soft arithmetic and “balanced” dashboards
Key Ratios and Evaluation Theories
DfID says “cost-effectiveness”. I say “impacts per input”. Here’s why.
Attribution and contribution
Total versus partial
Contribution excuses you from estimating the counterfactual
Attribution is the whole process
Contribution is not necessarily quantifiable
Attribution: the reverse direction, and not necessarily partial.
Contribution described for different kinds of Rules
Rules with only propositional (Boolean) Variables
Words in Capitals: Thin contributions
Planning evaluation research
Updating the model, after the research
Formal criteria for the quality of a Theory? Of an evaluation Mechanism?
Formal quality of a Theory?
Formal quality of an evaluation Mechanism?
What do we need to know? Utility.
IV Wickedness: vague, rich and open Variables, Statements, Rules and Theories
Vagueness & Gaps
Different kinds of Variables: vague Variables
Example: reading age
Vague underlying definitions
Example: smartphone ownership
We can still make some serious statements about vague Variables.
More to come …
Rules which are not explicit listings can even tolerate modifications to the Variables
Different kinds of Variables: Intensity Variables, made for comparison
Zero & negative Intensity Variables
Middle sweet-spot Variables
Standardised intensity Variables
Generic valence or V-Variables
Evaluation is about comparisons anyway
Statements with intensity Variables:
More to come
Clear and Vague Rules
Expressing the completeness of Rules using full, half-full and empty circles
Slang: assume influences are incomplete
Structure for narratives
Different kinds of Variables: rich Variables
Mechanisms as Rich Variables
Why collapsing and factorising might not be valid
Living with fuzziness
Open aka Incomplete Definition Rules - simple version
Conceptual parts: “falls under” or “is an example of”
Professional duty of interpretation
Convention: assume definitions are total
Kinds of Variables: summary
V The consequences of wicked Statements, Variables, Rules …
Making judgements about language within written and spoken material
Making judgements about language within written and spoken material
Quality of evaluation evidence
Reliability and Validity
Bias and illusion
Reporting : Variables and language-Variables
Language-Variables are Variables too
Reports are about Variables, not things
Theorymaker and communication theory
A Theorymaker theory of meaning?
Reporting that intervenes
Reporting on more than one Variable
Reporting on Mechanisms
The reported and reporting Variables: same type?
VII Some applications
Rubrics in evaluation
Variables and targets: An easy trick for writing the names of your Theory of Change or Logframe items which helps to focus on the actual Variables
Outcome Harvesting is not harvesting but hunter-gathering
What’s your ToC? Theories of Change are the perfect way to understand others
What is a Theory of Change?
What is a Theory?
Adding in !do Variables and !valued Variables
Adding more details to the diagram
Conscious and unconscious Theories
A Theorymaker theory of motivation and behaviour change
The dual nature
A simpler but deceptive alternative
Updating an Agent’s Theory of Change
Updating with information
Updating in other ways
Solubility of Theories
“A leads to B because C”: how do I put this into a Theory of Change?
Beaufort and Rubrics
Does the UK election tell us what the electorate wants on several issues simultaneously?
Published with bookdown
comments powered by Disqus.