Follow Simon Caulkin on Twitter



Article Archive
2020
July
June
May
March
February
2019
December
November
October
September
August
July
June
May
April
March
February
January
2018
November
October
September
August
July
June
May
April
March
February
January
2017
December
November
October
September
August
July
June
May
April
March
February
January
2016
December
November
October
September
August
July
June
May
April
March
February
January
2015
December
November
October
September
August
July
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
May
April
March
February
January
2013
December
November
October
September
August
July
June
May
April
March
February
January
2012
December
November
October
September
August
July
June
May
April
March
February
January
2011
December
November
October
September
August
July
June
May
April
March
February
January
2010
December
October
September
2009
November
October
June
May
April
March
February
January
2008
December
November
October
September
August
July
June
May
April
March
February
January
2007
December
November
October
September
August
July
June
May
April
March
February
January
2006
December
November
October
September
August
July
June
May
April
March
February
January
2005
December
November
October
September
August
July
June
May
April
March
February
January
2004
December
November
October
September
August
July
June
May
April
March
February
January
2003
December
November
1998
January
1997
January
1996
February
1994
May


Hitting the target and missing the point

Thu, 14th May 2020

Targets. Stretch targets. 100,000 coronavirus tests a day by the end of April. That turned out well, didn’t it?
When on 2 April health secretary Matt Hancock announced his goal of carrying out the famous 100,000 tests a day by the end of April, the result was predictable.
Given that at the time the daily testing rate was around 11,000, attention naturally focused on the number, and whether it would be achieved. And that's where the debate stuck for the month. Not on why 100,000 or the purpose of the testing – the number. 
On 1 May Hancock used the daily coronavirus briefing to declare that testing numbers had hit 122,347: the pledge had been met. Again, the number hogged the attention. Was it true? Had it really been hit? How?
Well, yes and no. It transpired that between the announcement of the target and the declaration of victory, the definition of ‘completed tests’, which previously meant ‘completed tests’, had quietly changed to ‘completed tests plus test kits in the post’. Subtracting the latter category left a ‘real’ figure of 82,000 actually carried out. Cue a new furore – again about the numbers.
What happened is a textbook illustration of the unintended effects of targets and their faithful sidekick, Goodhart’s Law.
To paraphrase W. Edwards Deming: in the case of a stable system there’s no point in setting a target, because you’ll get what it delivers. But with a non-stable system, there’s no point in setting a target either, because you have no idea what it will deliver. A numerical target in such circumstances is a finger stuck up in the air. Unless you know how to improve system capability permanently (I don’t think so), to hit it you have to be either incredibly lucky (in which case you’ll have to be even luckier to do it again tomorrow); or alter the parameters to make the target attainable.
Hancock did what everyone does when faced with the imperative to hit an arbitrary target: he managed the thing that he could – in this case, the definition of success. 
But this is not a harmless bit of jugglery. Deming again: ‘What do “targets” accomplish? Nothing. Wrong: their accomplishment is negative.’ There is a high cost to his action – which is where Goodhart comes in. 
As economic adviser at the Bank of England, Charles Goodhart noted that attempts to manage monetary policy by using any definition of the money supply was constantly subverted by actors finding novel ways to circumvent the definition. Hence his law, usually formulated as, ‘when a measure becomes a target, it ceases to be useful as a measure.’ A metric can be either a target or a measure. It can’t be both.
Take Hancock and his tests. To meet his target, he included in his count for 30 April around 40,000 test kits mailed out to the public and to hospitals. Of this number (pay attention here), while the Department of Health and Social Care counts the number of people that test positive, it doesn’t collect figures for tests completed. 
What’s worse, since mid-April the government figures include on the same basis (ie people testing positive but not tests completed) 17,500 variegated tests consisting of both diagnostic and antibody tests, thus adding oranges to uneaten, partially eaten and completely eaten apples. As Tim Harford declared incredulously on his latest ‘More or Less‘ show: ‘It’s almost as if they don’t care if the number of tests is consistent or indeed accurate, as long as it’s big.’ 
At any rate, the upshot of this piece of target-setting is exactly as Deming and Goodhart predicted: the system is beyond comprehension and the figures such a dog’s breakfast that no one can tell what they mean. It seems highly unlikely that Hancock’s original target has been met at all since 30 April, but how can anyone know for sure, including the government? The only certainty about the figures is that they are bogus. You might think that when the subject is life or death, this matters, no?
Yet the damage done by targets doesn’t stop there. What most people don’t get (including a ‘science writer’ on a previous edition of ‘More or Less’) is that the problem with targets isn’t that they don’t work. It’s that they do. 
A target is typically a one-club solution to a problem with many moving parts. But the first law of systems is that you can’t optimise one part of a multipart system without sub-optimising others. Any benefits are outweighed by unintended consequences elsewhere in the system. Focusing attention (often with added incentives) on the target rather than the purpose ensures that even if the target is hit, the point is missed.
Targets displace purpose. Tests are a means, not an end. But reporting 100,000 of them became the purpose, both for Hancock and his critics. Yet why 100,000 a day, rather than 75,000 or 250,000? What are we testing for in the first place? Deming once more: ‘Focus on outcome is not an effective way to improve a process or an activity…[M]anagement by numerical goal is an attempt to manage without knowledge of what to do’. Another finger in the air. Or, more tersely: ‘Having lost sight of our objectives, we redoubled our efforts.’
Consistent failure to meet the daily target underlines the point: it bears no relation to purpose, or any other kind of reality, really. Not production capacity, as we have seen. Even more serious, not with demand either – shortage of which, or shortage of which in the right place, has been put forward as a reason for the target debacle.
To be effective, a system needs to be designed against demand. And demand is determined locally. Testing is the first step in the ‘test, trace, isolate’ strategy that the government first initiated and then discontinued in March, and has now resurrected. By definition, that strategy has to play out out locally, where the infection occurs, tracing begins and treatment takes place. But bypassing hospitals and 400 or so existing small labs dotted around the country, all tightly linked to local primary care, the government, as with the Nightingale hospitals, is relying on giant regional testing factories, set up from scratch and remote from their users in every sense. A lurch backward to early 20th century industrial thinking, these in the view of many observers are the exact opposite of what is needed.
We can all support a goal of ramping up testing capacity to the level necessary to meet the purpose, whatever that number is. In fact it would be a good idea. But the minute you set it as a numerical target, it is subject to Goodhart. Managing backwards from an outcome plucked from thin air is a feature of command-and-control management, the only kind of management that government knows. But it is back-to-front. Targets are a disease. They destroy purpose, distort priorities, and soak up energy in games-playing and bureaucracy. They are the problem, to be avoided like, well, the plague.

 

 


<< Back to Index

User comments

Douglas Board :: 14th May 20
Many of us will remember when, post GFC, the Queen visited (?) the Stock Exchange and asked why no-one had seen it coming. Provided that she had a Simon Caulkin script in her handbag, I would pay to watch her visit the COBRA room in the coming months to ask a rather opposite question.
Henning Sieverts :: 14th May 20
Simon, please send a copy of this to Sir Kier, or his health care advisor.
Andy Lippok :: 14th May 20
Simon, oh how many times have you written about this topic in the past 10, 15, 20 years?!! And they still don't learn or get it - I despair, truly despair of them all! I thought of tweeting Hancock about this, I desisted as it seems so pointless as he won't ever read it. Ah well, the bullshit and nonsense of targets will continue!
Andy Lippok :: 14th May 20
Good point Henning - also send it to the leader of the SNP in Westminster, Simon.
Ray Ison :: 16th May 20
Simon as ever right to the nub of the matter. But as Andy says - we/you have been relentless in your articulation of this systemic failing, yet little seems to happen. In our book launched recently we set out the case for the reinvention of our governance systems and their population by those with Systems Thinking in practice capabilities - see https://www.youtube.com/watch?v=kZYSeTmjjZo
Ray Ison :: 16th May 20
Simon as ever right to the nub of the matter. But as Andy says - we/you have been relentless in your articulation of this systemic failing, yet little seems to happen. In our book launched recently we set out the case for the reinvention of our governance systems and their population by those with Systems Thinking in practice capabilities - see https://www.youtube.com/watch?v=kZYSeTmjjZo
Paul Reading :: 16th May 20
Our problem is we have for a long time had leaders who aim not to achieve progress for society, but merely to appeal to the prejudices of the swing voters in order to preserve their own position and consequent options for personal advancement. It all looks remarkably like the fall of Athens, Rome, Luxor, every former civilisation.
Paul Reading :: 16th May 20
Our problem is we have for a long time had leaders who aim not to achieve progress for society, but merely to appeal to the prejudices of the swing voters in order to preserve their own position and consequent options for personal advancement. It all looks remarkably like the fall of Athens, Rome, Luxor, every former civilisation.
Simon Caulkin :: 25th May 20
Thank you all for your bracing comments and suggestions. I value them all, and will indeed send a copy to Starmer. Will let you know if I get a response!
Name:
Comment:
Check:5+4=