Theorymaker
Preamble
Warnings!
Who is this book for?
What kind of M&E?
What’s a project, what’s a programme?
Why I am writing this book
Motivation: what problems does Theorymaker address?
Theoretical background
Frequently asked questions
But can you really apply all this stuff in practice?
Doesn’t “Learning Theorymaker” skirt all the political issues around evaluation?
Is there a lot of maths?
But isn’t it complicated?
But why is this approach so formalistic? Isn’t life really much messier?
Scepticism: is this approach too optimistic about plan-ability?
Thanks to …
Overview of the book
I Theories
Section Overview
What makes a good Theory: A check-list
About Theorymaker
What’s Theorymaker?
Theorymaker speech, writing and diagrams
Beginners welcome
Is Theorymaker a “framework” for evaluation?
Does Theorymaker give us a template for any programme theory or evaluation?
Theorymaker, a website which understands Theorymaker
theorymaker.info speaks Theorymaker!
What else you can do at Theorymaker.info
About Theorymaker diagrams
Theorymaker slang and meta-Theorymaker
Statements
meta-Theorymaker: talking about Theorymaker
Statements
Levels
Facts
Why bother?
Conventions
Variable-labels and Variables
Classifying Variables according to their Levels
Multiple Variables
Example: personality
Example: location
Example: multiple ordinal variables
Quality & Quantity (Q&Q)
Unusual dimensionality
Note for Nerds: Why not just define Variables in terms of numbers and equality?
Variables: the acts of people and Gods?
Levels of a Variable: collectively exhaustive and mutually exclusive?
Making a Difference
“Making a difference” is more general than “changing things”
Factual and Counterfactual
So what is a counterfactual?
Different kinds of Variables: discrete Variables
Discrete Variables
Reminder: we normally don’t distinguish between Variables and language-Variables
Binary Variables
Dichotomised continuous Variables
Intensity Variables with just two Levels (“dichotomised”)
Nominal Variables
Nominal variables: Statements with an implicit list
Ordered Variables
Ordinal Variables (step-by-step ordering)
Hierarchical Variables (classification)
Circular
Different kinds of Variables: Intensity Variables, made for comparison
Zero & negative Intensity Variables
Middle sweet-spot Variables
Standardised intensity Variables
Generic valence or V-Variables
Evaluation is about comparisons anyway
Statements with intensity Variables:
Different kinds of Variables: Numerical Variables
Count (natural number) Variables
Integer Variables
Continuous (rational) Variables
Percent and proportion Variables
Different kinds of Variables: rich Variables
Example
Rich Variables (different from other rich Variables?)
Why collapsing and factorising might not be valid
Qualitative?
iterative
Different kinds of Variables: vague Variables
Vague extremes
Example: reading age
Vague underlying definitions
Example: smartphone ownership
We can still make some serious statements about vague Variables.
More to come …
Rules which are not explicit listings can even tolerate modifications to the Variables
Kinds of Variables: summary
Vague Statements
More to come
Zadeh
Features
Decoration
Simple Theories
Theories and Mechanisms
Mechanisms in Realistic Evaluation
Composite Theories, valid Theorymaker
The Law of valid Theorymaker
Why isn’t one Variable on its own valid Theorymaker?
Combining simple Theories
Why not have multiple child Variables in a Theory?
Some definitions: Root and Leaf Variables
Composite Mechanisms
Statements and Facts
Slang: lists of Variables with semi-colons
What is not a Variable, what is not a Theory?
The stability / autonomy of simple Mechanisms
Why explicitly causal approaches, like Theorymaker, are better than correlational approaches
Rules
What does it mean to say one Variable influences another?
What does it mean to say
several
Variables influence another?
Combining more than one simple Theory
Rules and functions
Kinds of Rule
Independence of influence Variables
Incomplete Rules
What do the arrows
mean
? Causation, the mysterious force?
How are is the algorithm to be specified?
Specific and general Mechanisms and Theories
Reality of zoomed-out Mechanisms
Rules can themselves be composite
Rules - different types
Saying whether an influence is positive or negative
Individual influences marked on arrows
Individual influences: - and + signs
“Overall positive”
Slang: influences are “overall positive”
(Multiply) overall positive
Binary Variables (“sufficient”)
Binary Variables (“necessary”)
Count-to-nominal: Choosing the winning candidate
Moderators
More binary examples
Parallel contribution; overdetermination; sufficient not necessary conditions
Necessary but not sufficient
Necessary and sufficient
No influence
Multiple Variables - Q&Q
Rich Rules
Painting competition
Sub-symbolic Rules?
Structure for narratives
Complete and incomplete Rules
Expressing the completeness of Rules using full, half-full and empty circles
Slang: assume influences are incomplete
Living with fuzziness
Intervention Variables
!do
Multiple Intervention Variables behave like just one
Intervention Variables are effectively root Variables
Image, Counterfactual, Difference and Effect
Meaning of Factual and Counterfactual are conditional statements already given by the Rules
Difference
Image
Noise Variables
Effect
Calculating the Image
Causal, not statistical
Aggregation
Updating the Theory
Dealing with the contribution of other upstream Variables
Differences as raw data
Difference and subtraction
Soft arithmetic with Differences
Expressing difference with asterisks helps to focus on the actual Variables
Differences and “targets”
The currency of evaluation is Differences
Where do alternative possibilities come from?
Evaluators can’t escape thinking in Differences
Yes, Difference is being used in a technical sense
Grouping boxes
Variables grouped by stakeholder
Variables grouped by narrative
Variables grouped by organisational unit
Variables grouped by “Pillars”
Scenarios
Which is the superprocedure, the final Rule?
Limitations of grouping boxes
For-each Variables
One Variable, one data point
Rules with for-each Variables
Rule is same across all cases
Rule differs from case to case
Aggregating for-each Variables
Using grouping boxes with for-each Variables
Continuous for-each
More to come!
Calendar bars
Repeated and selected Variables
Fixed and Relative Time
Delays
A general time Rule
“For-each Time” Variables
More to come!
Rules with “for-each time” Variables
Rules with “for-each time” Variables which change over time
“Stock” or “Memory” Variables
More to come!
Definitions
Definitions with arithmetical Variables
Definitions with lo-hi Variables
Definitions with thresholds
Reality of “defined Variables”
Strange duality of Theories, Mechanisms and Definitions: Interchangeability of perspectives
Incomplete Definitions
Tricks with Definitions: collapsing and factorising, etc
Combining several Variables into one with fewer Levels
Aggregating “for” Variables
And that is it!
Contexts and Assumptions
Context as a reduction in possibilities, in which certain Variables are set to certain Levels
Context as a particular set of Variables
Variables which only exist within certain Contexts
Assumptions
Assumptions, understood as equivalent to context
Assumptions that the Theory is adequate
Assumptions that the Mechanism will not be manipulated from outside
Different ways to show Assumptions/Context in a diagram
… As additional Variables
… Using a grouping box
… Labelling the arrow
… Using a Feature
Specific and general Theories and Mechanisms
Combining contexts
Logical independence of Variables within a Context
Soft arithmetic in evaluation
Goodbye Galileo?
Where do comparisons come from?
Another kind of Variable?
Let’s do some Soft Arithmetic
Transitive rule
Differences
Judgements with equality
Soft addition
Soft division
Danger: we can’t assume normal arithmetic principles
Aggregation
Aggregation conditioned by a Variable with known probabilities and numerical Levels
Aggregation conditioned by a Variable with known probabilities and soft Levels
Aggregation conditioned by a Variable with soft probabilities
Aggregation conditioned by a Variable with probabilities not defined, e.g. an Intervention
Comparing ratios
Comparison with standards
Comparison with rubrics
“Mere connections” in Judea Pearl’s causal diagrams
Can influences die out over the length of a chain?
More examples of Soft Arithmetic with numbers
Example: lives plus dollars.
Example: lives divided by dollars.
Example: lives minus lives.
Example: calculating the “mean” of a directed Variable
Other examples
Example: deciding if the influence of a Variable is “roughly monotonic”
Multiplication by numbers-less-than-one as a way of expressing lack of certainty
Rounding errors
Soft arithmetic for comparing costs and benefit Variables
Soft arithmetic for adjusting for the reliability of a source
Soft arithmetic for combining evaluation data from different kinds of sources
Soft arithmetic for combining intermediate evaluation findings
Soft arithmetic for combining evaluating intermediate evaluation findings
Monotonicity and Chaining
Indicators
Downstream proxy Variables as “indicators”
More to come!
Combining subscales
Using an existing proxy
Tighter specification / means of verification
Partial Definitions
II From Theories to Theories of Change
Section Overview
Checklist: what makes a Theory of Change?
Value, Intervention, belief
Inter-related: Value, Intervention, belief
Valued Variables and intervention Variables in Theorymaker
A Theory of Change is a Theory about a Project!
Cost and resources
Resources and constraints
Cost and negative value
Valuing process too
Valuing a process
Other names for valued Variables?
Single way to cope with both project “outcomes” and other values
More about Value
Single source of Value?
Example: More than one valued Variable, possibly in causal relationships to one another
Example: Another example
Example: Valuing the middle
Example: Cumulative value
Intrinsic Value
Value cannot be separated from facts in evaluation
Nothing special about value
Nothing trivial about closed Value
The Value of an Intervention
Know-how, Resources, Motivation
Signalling valuation
Understanding the motivation of others using
their
Theory of Change
III Doing Evaluation
Section Overview
Reporting: Variables and language-Variables
Reporting : Variables and language-Variables
Reporting
Language-Variables are Variables too
Reports are about Variables, not things
Theorymaker and communication theory
A Theorymaker theory of meaning?
Kinds of Report
Reporting that intervenes
Reporting on more than one Variable
Reporting on Mechanisms
Diagram
The reported and reporting Variables: same type?
Emergent reported Variables?
Do we construct Variables or just observe them?
Evaluation: Appraisal of projects and programmes
Reporting and value
The evaluation Rule
The reporting Statement
Open aka Incomplete Definition Rules - simple version
Parts
Physical parts
Conceptual parts: “falls under” or “is an example of”
Professional duty of interpretation
Convention: assume definitions are total
Evidence for a Theory of Change
An Evaluation design requires a Best Adequate Theory of Change (“BAT”)
This is hard work
Adequate Theories; the 30-30 principle
Adequate theories: missing Variables?
Is there actually such a thing as an incomplete Mechanism, or are there only incomplete Theories?
Noise Variables
Is evidence essentially correlational?
Measurement, data, evidence
Sources of evidence
Updating a Theory of Change based on what actually happened
Making judgements about language within written and spoken material
Making judgements about language within written and spoken material
Quality of evaluation evidence
Reliability and Validity
Validity
Reliability
Bias and illusion
Triangulation
Scope and generalisation
Constructing the evaluation Theory
Key Ratios and Evaluation Theories
Key Ratios
Evaluation Theories
DfID says “cost-effectiveness”. I say “impacts per input”. Here’s why.
Attribution and contribution
Definitions
Total versus partial
Different directions
Contribution excuses you from estimating the counterfactual
Attribution is the whole process
Contribution is not necessarily quantifiable
Solution
Attribution: the reverse direction, and not necessarily partial.
Contribution described for different kinds of Rules
Rules with only propositional (Boolean) Variables
Words in Capitals: Thin contributions
Planning evaluation research
Updating the model, after the research
Formal criteria for the quality of a Theory? Of an evaluation Mechanism?
Formal quality of a Theory?
Formal quality of an evaluation Mechanism?
What do we need to know? Utility.
IV Wicked, open, fuzzy systems
Section Overview
V Some applications
Section Overview
Rubrics in evaluation
Variables and targets: An easy trick for writing the names of your Theory of Change or Logframe items which helps to focus on the actual Variables
Outcome Harvesting is not harvesting but hunter-gathering
What’s your ToC? Theories of Change are the perfect way to understand others
Intro
What is a Theory of Change?
What is a Theory?
Adding in !do Variables and !valued Variables
Adding more details to the diagram
Conscious and unconscious Theories
Reporting
A Theorymaker theory of motivation and behaviour change
The dual nature
A simpler but deceptive alternative
Updating an Agent’s Theory of Change
Updating with information
Updating in other ways
Solubility of Theories
“A leads to B because C”: how do I put this into a Theory of Change?
Beaufort and Rubrics
Does the UK election tell us what the electorate wants on several issues simultaneously?
References
Published with bookdown
Learn Theorymaker
The Value of an Intervention
The Value of an Intervention is a Difference on the Value of the Effect.
Please enable JavaScript to view the
comments powered by Disqus.