Dear OECD; you only need two to triangulate

I was very surprised today to find that the OECD DAC definition of triangulation says that you have to have at least three or more sources to do triangulation. I think this is wrong and here I will say why.

I haven’t been able to find any other authorities who mention this insistence on three sources. For example, Michael Scriven does not mention it. Our good friend Wikipedia talks about two or more sources. Unfortunately, plenty of secondary sources quote the DAC definition as gospel.

This definition can have important negative consequences. For example, suppose an evaluation needs to gather information to answer two evaluation questions and has time to collect four sets of data. Other things being equal, I would suggest collecting two sets of data for the first question and two sets for the second, triangulating within each pair. However if you were following the DAC definition you would have to collect three sets for one question and would have only time left to collect one set for the second evaluation question; so for the second question there could be no room for triangulation between sources. Or, suppose an evaluator had one source for one evaluation question and was offered the opportunity to have another source for free - she or he might say “sorry but no, without a third source I can’t triangulate. You can keep your data”. That would be a real shame.

So what is triangulation?

Suppose a plane explodes in the air in a remote region and we are sent as investigators to try to locate it. We have no other information so we have to ask the local people who live in small farms spread across the tundra. Many say they saw the explosion. We ask one person and she says: “oh yes, I saw a big light in the sky, it was over there” and points to some distant hills. This information is a bit of a help but as we don’t know if the explosion was 5 km away or 500, we would have to trudge down the informant’s line of sight. So what we do is set off, roughly at right angles to the informant’s line of sight, to find another eye-witness some distance away. We find one, and he points in the direction he saw the explosion. Of course now we can draw these two lines on a map and where they cross is a good guess at the actual place. The second source fills in a crucial uncertainty, or “degree of freedom”, or bias, or gap, or blind spot, in the first.

Sure, we could look for a third and fourth source too, and they would successively help us improve our confidence in the estimate - but they wouldn’t add anything like as much as was added by the second compared to the first.

This little parable is fun because on the one hand it illustrates the actual, geometrical use of triangulation from which the social science approach gets its name, and at the same time it is a little piece of actual social science research too, in which the limitations of the individual sources are due to geometry rather than, say, social class or caste or fears of job losses. Yes, the “tri” in “triangulation” comes from the triangles you draw when fixing a position, not directly from the Greek word for “three”.

I’d like to add something more. Neither the OECD or Scriven definitions are very enlightening about what exactly a second or additional source can add in order to facilitate triangulation. More fundamentally, what counts as an additional source? Usually, if we had interviewed a staff member about their views on a programme, we wouldn’t consider a second interview with their colleague to be useful for triangulation. It isn’t relevantly different as a source. But what is relevantly different? I would suggest that a second source is relevantly different, and therefore can be used for triangulation, if it can be expected to fill in or address systematic blind spots or biases in the first source (and vice versa). If you already have one source and have the opportunity to collect just one more, the wise evaluator will look for a source best able to fill in or address the most important systematic biases or blind spots in the first source (and vice versa). So if my biggest concern about the project staff’s opinion on project effectiveness is that their jobs depend on it, I could look for the opinion of a disinterested source. And although that disinterested source might well be otherwise a bit less informed about the details of project implementation, I don’t mind so much because the first source can fill in the second source’s weaknesses on this score.

Triangulation is essentially not an accumulation of more data but an opportunity to reflect on the reasons for, and then correct, gaps, blind spots and biases in data.

Footnote: Patton (1999) mentions four kinds of triangulation - methods, sources, analysts and theories/perspectives. Here I have only talked about “sources” but I am using this to cover any of the four - methods, sources, analysts, and theories. What Patton does emphasise is that the point in triangulation is not to reach consensus or, worse, some kind of average, but to understand and reflect on the reasons for any discrepancies, in order to deepen our understanding.