This is the first part in 'The Dangers of Data' series. Over the three parts, I'll be discussing the dangers, traps and pitfalls involved with data.

Nappies and beer: urban legends in the world of data

For those who take an interest in the world of big data and analysis at some point there is a good chance you will encounter the story about Beer and Nappies.

An anecdotal case study from the early days of data analysis this story holds that a large grocery retailer noticed a strong correlation between beer and nappies appearing in the same baskets. The story goes the retailer moved its beer fridges next to the nappies aisle and sales doubled! The accompanying story was there were a lot of new dads that couldn't go to the pub so buy beer to drink at home and stock up with nappies at the same time.

Sounds intuitive and believable and the story has become something of an urban legend in world of data. But like most urban legends...it's not true! The hypothesis was actually tested on a range of retailers and it was found that baskets containing nappies were no more likely to also contain beer than a standard basket that didn’t contain nappies.

This example shows the danger of not fully scrutinising a set of results simply because it offers a convenient, believable narrative. The beauty of big data is it allows you to test these myths, they need not be accepted at face value. It's easy to imagine that there might be some retailers who hoped to replicate the success of the mythical grocer and moved all their beer fridges only to find that sales remained flat. This also raises the issue of what to do when the results of big data seem to contradict what we might intuitively believe to be true.

That's all for now. In tomorrow's article I'll be looking in greater detail at the mixed messages given by data as I explain Simpson's Paradox. 

 

By Peter Smith, Analyst at Webalytix.

PrivSec Conferences will bring together leading speakers and experts from privacy and security to deliver compelling content via solo presentations, panel discussions, debates, roundtables and workshops.
For more information on upcoming events, visit the website.

comments powered by Disqus