Data can tell us a lot, but not everything.

The numbers, so the saying goes, do not lie. The idea that cold, hard data tells us everything we need to know about everything is hugely popular at the moment. Algorithms and machine learning will tell us all we need to know, right?
Perhaps, but history says perhaps not. A very good example of how important field research is, is shown by Tom Kelley, General Manager at IDEO in this video. 
No matter how big the dataset, data mining and analysis have limits— or the “head” approach— and the same is true of “softer” method like customer

surveys, the “heart” approach. The truth is you need both.

The Head And The Heart
Don’t get us wrong: Data has its use. When working on innovation, it makes sense to sit down with the data you have and look at what it might be telling you. Who your customers are, what products have sold the most, which products tend to bring in new customers, and the effectiveness of promoting different features of a product are all worth considering.
That said, data has a host of problems. You don’t have a perfect sample of your industry, for example—just the customers who’ve come to your company. If you’re trying to break into a new market, historical data won’t tell you anything.
And it’s far too easy to mistake historical trends for the future. Energy is a perfect example. When energy prices were deregulated in the 1980s, it was assumed that coal would remain the cheapest form of energy for decades. But the invention of hydraulic fracturing, better known as fracking, drove the price of natural gas well below that of coal. To add insult to injury, government agencies using historical data severely underestimated both the infrastructure and per-kilowatt cost of renewable energy, in particular failing to anticipate that they were technologies, which always get cheaper over time.

Balancing Head And Heart
Conversely, anyone who’s dealt with self-reporting surveys can tell you that the painful truth is that customers don’t tell you everything. This can be for any number of reasons; for example, if a product is of a deeply personal or intimate nature, like certain forms of medicine, people are sometimes just too embarrassed to tell you the whole truth. Even when customers are generally honest, though, you’re still speaking only to customers who are highly motivated to share their ideas. How much of your customer base are you really getting data from?
The solution is to balance what customers tell you with what the data says, and not let one override the other. This is particularly true when customer data and customer surveys contradict each other. Let’s say for example that customers tell you they’d like a particular feature on a product, but your competitors introduced that feature and it flopped. There are a few answers here: Perhaps, it’s a feature that’s strongly desired only by a small subset of customers, and the majority are uninterested. Or there was a drawback to the feature that wasn’t obvious at first. It might even be a matter of poor marketing. But it’s a question you have to find an answer to.

If you want to learn more about research in projects, we highly recommend the online course „Basics of Design Research“ on OpenSAP, which is free for everyone. 
For working material and more information you can have a look at our download center on Mindshift.One.
You are interested in a training at your location in Germany or in our workshop space in Mannheim? Mindshift.One offers workshops from 1-3 days. Here you can find more information.