Reporting : Variables and language-Variables
Here we look again at what we mean by one Variable “reporting” another. In Theorymaker, a Report can have various different formal properties: sub-types are Judgements, Descriptions, Measurements etc. We are particularly interested in reporting on the quality or value of something. Ultimately we will see that evaluation involves Reporting on the Value of Projects.
Theorymaker native speakers use the word “Variable” to talk about both language-Variables and the Variables they refer to. So you can say “number of visitors to the new centre” and mean either the language Variable which is part of your plan, your shared model of how things are going to work; you can think of this as a social phenomenon or as the paper-and-pencil (or more likely electronic) manifestation of this shared model. Alternatively you can mean the actual number of visitors, a Variable which you can observe or measure e.g. by going to the centre, without any knowledge of any plan or Theory.
If you are busy developing a Theory of Change, you won’t have time to worry about the difference. But when we come to look at how our evaluation Report is supposed to reflect various aspects of the project being evaluated, we will need to look at the way language-Variables refer to or “report” Variables in the rest of the world beyond the plan, and that is what we will do in this chapter. In this Chapter we will also explore how language-Variables are themselves Variables.
The report says 'The height of child X is ...' ((positive continuous numbers)) !Rule: some reporting or measurement Rule ensuring the report of the height reflects the actual height The height of child X is ((positive continuous numbers)) direction=TB
So this diagram shows a causal link between the actual height of the child and the report of its height, say on a school health record. This causal link will normally be a relatively involved one which we could probably “zoom in” on to identify other intervening Variables: we could call the whole thing “the reporting Mechanism”, following a “reporting Rule”.
So the Levels of a simple Variable reporting on the state of a traffic light are something like these:
'The traffic light is red' 'The traffic light is orange' 'The traffic light is green'
… where the traffic light Variable is something like this:
The traffic light is red The traffic light is orange The traffic light is green
Language-Variables are Variables too
We already saw that language-Variables report or refer to Variables, Statements report Facts, and that Theories (consisting of language-Variables and Statements) report Mechanisms. So in this sense Variables and language-Variables are two very different kinds of things.
But on the other hand, non-Theorymaker native speakers might be surprised to hear the following claim.
A language-Variable not only reports a Variable but is also itself a perfectly ordinary Variable.
When you think about it, of course it is.
Imagine I am measuring a child’s height and fill the number in on the child’s health form in the box marked “height”.
Each possible response is a Statement, one Level of a language-Variable. Loosely, we can call this language-Variable “the height as reported on the form” and say it reports the actual Variable, the height of the child. We call it a linguistic Variable simply because it actually uses language - as always, in a particular context; in this case an important part of the context is that the other people around me speak the same language and understand the health form, etc.
When we talk about the language-Variable, rather than using it, we can recognise it as a Variable just like the actual height is. So when the Level of the Variable
child's height is 1m, the Level of the language-Variable
report of child's height should be
The critical thing to understand is that under the right conditions, language-Variables can influence and be influenced by other Variables.
Reports are about Variables, not things
In English, we use words like “appraisal”, “judgement” etc. in different ways. We can say “we are measuring the child” interchangeably with “we are measuring the child’s height”, which names the specific reported Variable which the Report will announce. Theorymaker only uses the second form. So in English we can say “go and appraise that restaurant” - but if the actual Appraisal is about the campaign’s overall cuisine quality, on a 1-5 scale, then what we have reported is just that single Variable, which has a 1-5 scale, and not the myriad other aspects of the restaurant. The appraisal Mechanism in this case has to home in on just the one aspect of the restaurant which interests us most.
'Overall quality of the restaurant ((1-5))' -Various aspects of the restaurant (some convoluted causal link)Overall quality of the restaurant ((1-5)) Weight of the restaurant Length of the restaurant Myriad other aspects ...
Theorymaker and communication theory
In order to write this chapter, we need to first have a theory of meaning in a Variable-based world. This ties in closely with Shannon’s communications theory, which is Variable-based.
This reminds us of Shannon’s transmitter-channel-receiver paradigm (Shannon 2001) and also of the stimulus-response paradigm in psychology: we already saw that causation and explanation are essentially contrastive; also information is essentially contrastive, and the objective measure of information content depends on the number of alternative messages.
The Appraisal has the same information content as the Appraised Variable.
A Theorymaker theory of meaning?
Our job as evaluators (or as anyone doing any kind of Reporting) is to make sure that the Report Variable is caused by the reported Variable so that the Report Variable always takes the Level “Variable B has the Level b” just when Variable B just does in fact take the Level b, for all Levels of B; in this case the Report Variable will always be correct (and there we have a very quick and dirty Theorymaker theory of meaning, truth and reference …).
You can see this results in what may seem like a primitively realist analysis of meaning: A Statement X is true if and only if there is an actual state of affairs which corresponds to it and which has the same logical shape as it; it is also in a sense a counterfactual or contrastive theory of meaning.
You could say this makes Theorymaker constructivist, because the Variables which our language refers to are defined via language Variables; and yet Theorymaker is realist because the Variables we are communicating about are “really out there”; they actually cause the truth and falsehood of our Statements.
Kinds of Report
It is useful to distinguish between different kinds of Report.
|Not valued (Descriptions)||Valued (Appraisals)|
Elsewhere xx we look at “valued” Variables in more detail.
Reporting that intervenes
The meta-Theorymaker word “Reporting” does sound a bit like a purely passive, receptive procedure. But of course the reporting Rule often involves to a greater or lesser extent intervening.
We should note there is nothing stopping Variables in the reporting Mechanism intervening on other Variables (although not actually on the reported Variable itself or on any Variable which affects it).
Everything we have said (and will still say, under Wicked Systems) about different kinds of Mechanism applies to reporting Mechanisms too. So for example, a reporting Mechanisms might adapt its intervention according xxxx
Reporting on more than one Variable
We already saw that Statements may be composite, not just simple. So Reports can be composite too.
Remember the isomorphic principle: xx - when we say “Rule” this is essentially unimportant whether we are thinking of a single Rule with multiple stages or a composite Rule. Remember also the principles of collapsing and factorising, which essentially mean we don’t have to keep worrying whether the reported Variable is several Variables or one.
Any Variable, or set of Variables, or a Mechanism, can be the subject of a Report.
Reporting on Mechanisms
(Need to say more about this)
You might have two reporting Variables, reporting on the upstream and downstream Variable in a simple Mechanism. But these two Variables won’t normally constitute a Mechanism. The Level of the bottom-right Variable in this diagram is controlled by the Variable it reports on, not by the “Report on classroom noise”.
-Reported Mechanism Teacher irritation Classroom noise -Reporting Variables Report on teacher irritation Teacher irritation Report on classroom noise Classroom noise
Nevertheless it is quite common to report on some aspect of a larger Mechanism, but the reporting Variable is still a Variable, not somehow a whole Mechanism which does reporting.
For example, we might want to report on a project’s cost-effectiveness (more precisely, on its outcomes/inputs ratio, see xx).
-Reported Mechanism, e.g. a project to be evaluated Project outcomes Project inputs -Reporting Variable Project outcomes/inputs ratio !Rule: divide outcomes by inputs using Soft Arithmetic Project outcomes Project inputs
This diagram is a possible fragment of an Evaluation Theory.
-Reported Mechanism e.g. a Project a b -Reporting e.g. on aspects of the Project Report on a a Report on b b Report on relationship between a and b !Rule: e.g. soft division a;b Report which summarises Reports on a and b Report on a;Report on b title=Reporting understood as a Mechanism. Note all links are causal diagram;fontsize=10
some consequence used as a proxy ('indicator') -Reported Mechanism e.g. a Project a - -Reporting e.g. on aspects of the Project Report on a (via proxy) - some consequence used as a proxy ('indicator') title=Reporting understood as a Mechanism: use of proxy Variable. Note all links are causal diagram;fontsize=10 rank=same;a;Report on a (via proxy)
The reported and reporting Variables: same type?
As we already saw, we call a Report “numerical” if its Levels include the actual symbols for numbers; and by extension we call the Variable it refers to “numerical” too.
We’d like to say, “the reporting Variable is always of the same type as the Variable it reports”. But this can be problematic when we turn later to xx emergent reporting Variables.
Emergent reported Variables?
- if the reported Variable is, say, binary, the report Variable is binary too.
- if the current national alert level is ((green < orange < red)), then a Variable reporting this level will also have three ordinal Levels.
But: what if the reported Variable is an old oil painting and want to just, say, report on its monetary value in USD. We can claim that the result of the reporting process (a numerical value above zero) actually reflects some aspect of the painting itself, namely its monetary value above zero. But in practice the valuation is a process which begins with something more complex (multiple aspects of the painting) and only then arrives at the monetary value, a simpler numerical Variable.
In some cases, the reported Variable seems to emerge as the result of a reporting process rather than just “being there already”. This is just an example of the kind of emergence we look at in our later section on “wicked” phenomena.
In these cases it seems as if the reporting algorithm begins with a more primitive substrate (“raw data”) and then constructs a Report, which is perhaps of a simpler structure e.g. an ordinal Variable, which only then gives us the idea that this Variable actually reports a new, constructed and emergent Variable which is somehow inherent in the original substrate. In some cases, people working with this kind of algorithm can develop the skill of being able at least to estimate the final Level of the reporting Variable before actually constructing the Report. In other cases, there is no way to predict it without actually going through the reporting algorithm - a classic sign of irreducible complexity (Wolfram xx).
Specifically in evaluation, we often have reported Variables (“Evaluands”) which seem to be emergent, for example we often start with “everything that happened in the life of the project” in order to make conclusions about, say, management effectiveness.
Do we construct Variables or just observe them?
Compare these two.
Doctor's report on child's height ((positive continuous)) !Rule: well established measurement procedure Child's height as measured by doctor((positive continuous))
In the case above, we can plausibly say that the child’s height is an unproblematic continuous Variable just waiting to be measured. But in the case below, can we say this?
Focus group report estimation of level of participants' agreement with suggested policy ((lo-hi)) !Rule: quite well established procedure for capturing emerging themes, etc Focus group as observed by expert((fuzzy))
We can certainly imagine a whole series of focus groups, perhaps recorded on video, which Theorymaker native speakers would understand as a “for Variable”, i.e. a set of several Variables. We can understand each actual focus group, with its special atmosphere, perhaps a fit of coughing, some jokes, some heated arguments, as the value or Level of some complex kind of Variable.
Theorymaker native speakers are quite relaxed about the fact that most of the Variables we use in everyday life are fuzzily defined. Firstly, often no-one is quite clear exactly what Levels are in the domain. Secondly, even when they are, we are often quite uncertain about which Level a Variable has and we aren’t even sure how we ever could find out for certain. And nevertheless, we manage to communicate.
Shannon, Claude Elwood. 2001. “A mathematical theory of communication.” ACM SIGMOBILE Mobile Computing and Communications Review 5 (1). ACM: 3–55.