Different kinds of Variables: Intensity Variables, made for comparison

We already looked at some examples of Variables with discrete, limited sets of Levels. Now we will look at another kind: intensity Variables. Different instances of intensity Variables can be easily compared with one another, but we find it hard to assign numbers to them.

My interest in this assignment is *moderate--* ((lo-hi))

Here, someone is making an expression of intensity without reference to any kind of number. We might find it hard to specify all the possible ways to complete the sentence meaningfully but nevertheless we can make comparisons: I might know that my interest is lower than the previous assignment - or even if I am not sure any more, it certainly makes sense to say that this level of interest is lower than last time.

The black symbol like a rising graph shows at a glance that the Levels of this Variable can be compared with one another.

Mostly, Theorymaker native speakers just use the generic expression ((lo-hi)) and variants such as ((lo-0-hi)), see below, for these kinds of Variables.

In fact, ((lo-hi)) is the default for all Variables in Theorymaker. If you don’t specify any Levels, Theorymaker native speakers assume you are talking about a ((lo-hi)) Variable, and even if you do, the ((lo-hi)) text is not shown in the Diagram.

We must extend our logic to handle the new imprecision (Scriven 1981b)

The most important and common kind of Variable in Theorymaker is one which is informally just as common in the world outside Theorymaker Island too; and yet, the rest of us tend to look down on them and don’t even really have a name for them. We are not able to specify the Levels of these Variables and yet we can still have information about them in the form of comparisons.

If that surprises you, read on.

An intensity Variable is one for which the Levels are specified not explicitly but in one of the following forms: ((lo-hi)) ((0-hi)) ((lo-0-hi)).

Often Theorymaker native speakers will present a Statement like this:

This morning I feel  *OK--* ((worst possible mood - best possible mood))

but the completion part (“OK”) is not really taken seriously. The point is that we are in a position to make comparisons with this Variable. So the real data for intensity Variables is like this:

My mood today is better than my mood yesterday 

My mood today ((lo-hi)) > My mood yesterday ((lo-hi))

This is why our original definition of Variables didn’t insist on our really being able to list all the Levels.

Lo-hi Variables are so common in evaluation that by convention, Theorymaker native speakers assume that Variables are lo-hi unless otherwise stated. This means we don’t have to bother writing “((lo-hi))” next to every Variable unless we really need to.

Some Variable ((lo-hi))

… can be rewritten just like this:

Some Variable

If there is no possibility of confusion, Theorymaker native speakers may also leave off “((no, yes))” after no/yes Variables.

Some Variable ((no, yes))

… can be rewritten just like this:

Some Variable

Many or most Theories of Change include Variables which intuitively express some kind of scale or intensity, without being obviously associated with any actual numbers. Examples include well-being, resilience, approval, competence, skill, satisfaction. Usually in Theories of Change these kinds of words are just written into a box and not taken seriously until some numerical “indicators” are assigned to them, for example score on questionnaire. In this book, we take these Variables seriously and at face value; we call them intensity Variables. Their key feature is that, given two Levels of an intensity Variable, either they are equal or one is greater than the other; yet we find it difficult to assign numbers to these Levels.

So if someone says “what kind of number would you assign to the quality of that civic education curriculum”, this number is usually arbitrary. To say “oh, it gets a 7.5” is to give too much information, because we might not really be able to measure quality with such accuracy; and on the other hand it gives no information at all, because we need some kind of key to tell us what 7.5 means. What we do have, is information like “this curriculum is a helluvalot better than that other one we reviewed last year”. So, the most useful kind of data for intensity Variables, which are often key Variables in evaluation, is in the form of comparisons, not numbers.

This is OK, but …

My mood today *quite good--* ((lo-hi))

… this kind of data is more useful

My mood today ((lo-hi)) > My mood yesterday ((lo-hi))

We can even use this kind of Statement, if we have enough of them, to put a whole host of Variables into order according to their Levels, without any numbers.

In fact, the usual practice of writing down the name of some Variable and then trying to capture it through numerical “indicators” rather begs the question - how would we know if a suggested numerical indicator is adequate to capture the actual Variable if we didn’t already have some kind of implicit and shared prior understanding of it?

One reason for the pre-eminence of the familiar kinds of quantitative models we usually see in evaluation is that they allow for comparison. You can have as much detail as you like about the culture which a project created in a school, but it won’t help you decide if this school is better than that one, or better than before; unless you can use it to make comparisons. However comparisons don’t in fact require numerical Variables. It is the quintessential characteristic of intensity Variables, see xx, that they can be compared. With intensity Variables, we can say that A is bigger than B but not necessarily by how much, i.e. we can’t compare the difference A–B with the difference C–D.

As an example, evaluators and researchers often attempt to “measure”, say, the disaster preparedness of a given community - to put numbers on it - and no agreement has been reached. But behind this, we do have a rough idea that we could say of two different communities that one was more prepared than the other, and relevant experts and stakeholders might even agree in a majority of cases, without there being any agreement at all about what kind of numbers were involved. In this kind of case, we could say there really is a Variable involved, ranging somehow from low to high disaster preparedness, so we could write it like this:

Disaster preparedness of this community ((lo-hi))

This way of thinking about Variables might seem terribly vague and informal. But it is important to understand that we can do serious reasoning with these kinds of Variables, in fact this is one of the key tasks of the evaluator.

For example, if we know

Village A is more resilient than Village B

and

Village B is more resilient than Village C

… we can quite rigorously conclude:

Village A is more resilient than Village C

This is just one example of the kind of non-numerical yet formal reasoning which I call “Soft Arithmetic”: see xx.

There is a kind of prejudice amongst certain Earthling social scientists that these kinds of Variables are really just numerical Variables, poorly formulated or understood. But they are wrong.

Another example: It should be easy to understand the model below - the children can whine up to a certain point, but their father will explode if they exceed it. We can understand that there is such a thing as the children not whining at all, and that the amount of whining can increase, and even exceed a certain threshold - all of this without any kind of agreement about any numbers which might be involved.

Father explodes ((no,yes)) !Rule beyond a certain threshold of children whining,   father flips from 'not exploded' to 'exploded'

 Children whine ((lo-hi))

Even if we could agree on a way of assigning numbers to moods, for example by using a questionnaire, we have to accept that the numbers assigned by any such method are arbitrary. An intensity Variable is what lies behind, is common to, all the different possible ways to “operationalise” it into numbers.

Another interesting thing about Q Variables is that lo and hi are not really parallels. The way increasingly meaningless extremes crumple together in the lo zone is different from the way they crumple together in the hi zone.

If you don’t know exactly what Levels your Variables range over, you aren’t making sense.

I would rather say the contrary: If your philosophy of science tells you that a Variable isn’t a Variable without a clearly defined set of Levels, preferably numbers, it can range over, then you should dump your philosophy, because the following is undeniably true even though we don’t know how to properly specify the Levels of these Variables:

From this

Approval rating increases as positive media exposure increases

and this

Positive media exposure has increased

we can conclude

approval rating should go up at least a little, other things being equal

You could call it a syllogism with intensity Variables.

I call this kind of reasoning Soft Arithmetic and you can read more about it later.

Zero & negative Intensity Variables

Sometimes we have a fairly clear agreement that there is a zero Level, below which the scale can’t move. We can express these kinds of Variables like this:

Amount of rain ((0-hi))

Quality of harvest ((0-hi))

Perceived distance from city centre ((0-hi))

Sometimes we can conceive some intensity Variables as ranging from somehow negative, through zero to somehow positive, i.e. the “lo” Level is below zero:

Wellbeing ((lo-0-hi))

Child’s feelings towards a school subject ((lo-0-hi))

So we have met these three kinds of intensity Variables:

  • ((lo-hi))
  • ((lo-0-hi))
  • ((0-hi))

Here one is quite free to argue that, say, Wellbeing can only be conceived of as positive, and we should use something else to capture the negative or aversive part of this idea. The point is that it is usually quite easy to agree that a Variable of interest can be expressed as one of these three kinds, even though there might be some split opinion about which kind - for example, whether there is also a zero Value. For example, could there really be a harvest with zero quality?

It also seems quite natural to distinguish between intensity scales which start at some kind of zero and ones which can also be negative. We could assess the first via a questionnaire with these kinds of questions:

I approve of the President *quite a bit--* (from 0 to hi)

We could assess this kind of approval with this familiar kind of four-point question:

I approve of the President *a little--* (not at all < a little < a lot < completely)

But sometimes we conceive of “approval” of something which can also be negative:

I approve of the President *a little--* (from strongly disapprove to strongly approve)

… this captures antagonism as well as mere lack of approval. We can perhaps imagine being able to answer a question like “do the negative ratings amongst Republicans cancel out the positive ratings amongst Democrats?”

Middle sweet-spot Variables

Variables for which the sweet spot is in the middle, yet have two or more arms.

How about this: (Evergreen 2017)

by all means we can model this with other concepts:

!Rule distance from middle ((lo-hi))

 Level on sweet-spot scale ((?)) 

but whether to model something or leave it as a primitive idea is a decision that has to be taken.

Standardised intensity Variables

We can claim the Beatles were better than Wings, and that the Beatles have to be up there near the top end of a scale of bands. But could there be a band better than the Beatles by a hundred miles? Most of the bands there have ever been are strewn out on the range between lo and hi and there are certainly a few dozen right up there at the top end; and some might claim there is just one band which nevertheless stands out as a so-called outlier beyond the others. The point is that we have successful conversations and arguments on this kind of topic, where we implicitly or explicitly compare the quality of things, without worrying if there could be an end to all comparisons or whether, for each and every band, there could always be another which was better.

The conventions “lo” and “hi” neatly sidestep these kinds of problems and play a similar role to the idea of infinity on numerical scales. But whereas the idea of infinity threatens to take us off into realms of unreachability (could we ever go so far?), hi just fades away into realms of meaninglessness (a Sixties band way, way, way, better than the Beatles? really?).

So there is one view of (or “version of”) intensity Variables which, as Kahneman claims, have a standarised anchoring so that the top range on one Variable is comparable with the top range on another. Kahneman claims that people have no difficulty in answering the question “Name me an English Football League club which is as good, compared to the other clubs, as Episode 7 of the Big Bang Theory compared to other episodes”.

This goes back to (Thurstone 1928): “The center of the whole problem lies in the definition of a unit of measurement for the base line.”

This would also suggest that we can at least to an extent say, for these kinds of Variables, whether the difference between x and y is less or more than the difference between j and k.

Evaluation Terms of Reference often make use of these kinds of Variable, perhaps expressing them with some kind of Rubric, covering a range something like this:

  1. exceptional performance
  2. good performance
  3. adequate performance
  4. poor performance
  5. unacceptable performance

The trouble is that we don’t really feel that this scale has to be broken down into exactly five steps. Four or seven or eleven might be possible too.

Generic valence or V-Variables

Not only are most of the interesting Variables in M&E standardised intensity Variables, but most of these involve what we might call “generic valence”.

This just means that they cover a range of quality or desirability from " this is what we really don’t want" to “this is what we really do want”. By convention, Theorymaker native speakers express this with just this mark: !V. Like this, we even see some examples where the usually verbose Theorymaker can even be a little more concise than English.

So rather than this:

This morning my mood is *OK--* ((worst possible mood -  best possible mood)).

The President's approval ratings were *quite negative--* ((worst possible - best possible)).

… we can say this:

This morning my mood is *OK--* ((lo  - hi)) !V

The President's ratings were  *quite negative--* ((lo - hi)) !V

We will call these kinds of generic intensity variables, which are not associated with any numerical scale but which are meant for asserting comparison of quality or desirability V variables. - V for “value”, with a tip of the hat to Michael Scriven9.

But perhaps what Scriven meant wasn’t this; perhaps he is talking about transcendental value. It wouldn’t be difficult to model this in Theorymaker.

Evaluation is about comparisons anyway

The importance of these generic intensity Variables on Theorymaker Island seems to run contrary to the strongly-held opinion of some evaluators outside Theorymaker Island that only numerical Variables are really important in evaluation. Well: consider that the minimum we require from an evaluation report is to make comparative statements. Comparative statements need only to be of intensity type; they don’t need to be numerical. That’s it. Variables of any kind can feed into such a comparison.

Another way of putting it: if the M in SMART means “measurable”, and measurement is about assigning a number to something, then no, SMARTness is not necessary for useful and precise, evidence-based M&E; comparability is often enough.

You could say, the quintessential task of evaluation is to make V-Statements; to be able to say, with evidence, that one thing is better or worse than another.

Theorymaker native speakers don’t think for a moment that there always has to be an absolute truth about these kinds of comparisons, any more than we do. What we are looking at here is the language they (and we) use to express comparisons whether or not we are right, and whether or not there is universal agreement.

Similarly, there can be plenty of disagreement about what constitutes “quality” in any particular case. So you might have two music buffs who actually agree about the Beatles and, say, the Rolling Stones, and their qualities, but can’t agree about which was overall the best band of the Sixties because they explicitly favour different aspects of quality - innovation, say, or stylistic impact.

Statements with intensity Variables:

References

Scriven, Michael. 1981b. The logic of evaluation. Edgepress Inverness, CA. http://www.coris.uniroma1.it/news/files/Scriven{\_}Logic{\_}evaluation.pdf.

Evergreen, Stephanie. 2017. “The Gauge Diagram Scenario One : Visualizing the sweet spot.” stephanieevergreen.com/gauge-diagram/.

Thurstone, L. L. 1928. “Attitudes Can Be Measured.” American Journal of Sociology 33 (4): 529–54. doi:10.1086/214483.


  1. xx