Faith vs Metrics
I was leaving the supermarket a while ago.
My hands were full of shopping and I was late.
As I hurried along I was accosted by someone on a promotional stand.
They were handing out free attachments for showers, which they said would reduce water consumption.
“Great, that sounds like a good idea”, I said. “Just pop one in my bag would you?”
They replied: “You’ll need to come over to the stand and fill in a form first. It’s a government-backed scheme you see.”
Standing there with all my shopping, my interest started to wane.
I asked them how it worked-starting to wonder whether this device was going to be worth the hassle.
“I haven’t got a clue, I’m not a plumber”, they said.
My interest level dropped to zero. I made my excuses and hurried away.
I’m all for measuring and evaluating. Large sums of money are invested in marketing and anything that can improve its effectiveness should be embraced-particularly when it's being funded from the public purse.
But all that survey would have ascertained was that I’d taken a shower attachment.
They’d have been able to produce a report showing how many shower attachments had been handed out. But if the promotional staff were trustworthy, they could have reported this themselves.
I’m sure they’d have collected lots of other data to show breakdowns by sex, age, socio-economic group, geography etc. All very interesting, but not critical.
What it wouldn’t have told them was whether I got around to installing it.
Or how much water it actually saved me.
So why did they let the survey get in the way of achieving their goal?
Evaluation must always measure the right things.
And measuring the effect mustn’t get in the way of achieving an effect.
Sometimes this requires a little more faith and a little less reliance on incidental metrics.