Does Your Business Suffer From ‘Survivorship Bias’?


Are you already getting bored [or should that be terrified?] of the phrase ‘big data?’ If so, I’m hardly surprised. The spectre of having lots of data to sift through (and establish casual links between), is a prospect few but the most highly analytical experts look forward to. HR professionals may well be ‘people-people’, but seldom are they mathematicians in their spare time too.

Given that understanding data – and employee data in particular – is only going to become more, not less important – the ‘accessibility-problem’ data presents is an issue. Perhaps that’s why we’re starting to hear organisations talk much more about ‘smart data’ instead – this, apparently, is the ‘good data’ – that which we can use to make real business change, as opposed to the majority of bad data (or ‘noise’), which has no real use at all.


However, even the concept of smart data requires HRDs to know what is and what isn’t useful, and it’s this differentiation that reminds me of Abraham Wald – a man most people won’t have heard of, but whose thinking I believe is as relevant today as it has ever been.

For those who don’t know, Wald was a Hungarian statistician who, during World War II, was employed by the US Centre for Naval Analyses to come up with solutions to ensure more fighter planes and bombers returned home safely. In a radical departure from what previous researchers had concluded – which was to give planes more armour in the areas that had sustained bullet-hole damage – he suggested the opposite – to give them more protection in the areas where returning planes hadn’t been hit. Crazy? Well, actually, no. The holes that researchers had looked at in returning aircraft were, Wald argued, precisely the areas where they could take damage but still make it back in one piece. What he proposed was that the Navy should actually reinforce the areas of the planes where damage hadn’t been observed, as these were (statistically speaking), most likely to be the areas that – if hit – would bring the plane down.

When looked at this way, Wald’s thinking actually makes perfect sense. And, it also explains why commanders back then tended to jump to other conclusions. Because no one ever saw the planes that had actually been lost, they placed all of their effort looking at the ‘survivors’ – those which came back, but which would then give a false reading about what their most useful course of action should be.

This inherent tendency to look first at what seems to be the most obvious conclusion has since been called the ‘Survivorship Bias’. By only looking at the data that’s presented to us, at best it means decision makers are only focusing on one side of the argument. At worst, it means they ignore absent information that may be extremely pertinent – if not essential.

It’s my belief that survivorship bias is a lot more prevalent in organisations today than many are willing to admit to.

This brings us back to big data, or even smart data. It’s clear to me that before we even deem what data is the most important to look at, we must actually go back to basics first, and determine what maybe ‘isn’t’ being shown, rather than what ‘is’. It’s well known, for instance, that the employees who fill in staff surveys are self-selecting and do so time and time again. It means that the answers they give (while useful) possibly don’t reflect the broader views of the silent majority – those who didn’t answer, because they couldn’t be engaged to take part in the survey in the first place. It’s my view that before you start analysing the data you get back from your employee surveys, you need to look at what you’re not getting back, and try and change this first.

Employee data is hugely valuable, hugely relevant, and can lead to insights that can make a real impact. But there’s a employeeslesson in here for all of us. Sometimes you shouldn’t just listen to what the survivors say – those who are visible to you. It’s your job to hunt those missing in action – those who never take part – because questions aren’t couched in a way that’s relevant to them. Unlike those who were piloting the shot-down planes, these people are alive and well in your organisation – they simply need to be rescued.

Maybe engaging these lost people, to contribute to your employee research, should be your New Year’s data resolution!

Silverman Research
Better Tools, Better Data, Better Science.
Silverman Research
Building Purpose Through Listening