We are at the beginning of the Big Data era, and there is widespread anticipation that this will be a huge benefit to companies. I’ve been attending the World Economic Forum in Davos and in my `Data to Decisions’ panel we heard CEOs tell how Big Data can reinvent everything from CRM to internal processes to product design.
We also heard that there are significant challenges in data sourcing, permission agreements, data quality and of course privacy concerns, as most Big Data is personal data about customers. Fortunately these challenges can be addressed by conventional business practices.
There is, however, a more subtle but even bigger problem: The use of Big Data turns the process of decisionmaking inside-out. Normally, you approach a problem with some understanding and some guess — or hypothesis — about what to do. Then you construct a test of your hypothesis in order to validate it, and statistics tell you if it’s true or false.
The power and danger of Big Data is that you don’t need to start with a good guess. Instead, you can ‘data mine’ patterns from the data. But these patterns are often beyond human understanding and can depend on context in unexpected ways, as we saw during Wall Street’s `flash crash’ and during the 2008 derivatives meltdown.
Because normal statistical methods of decision making and validation fail us when we turn to Big Data, we have to seek validation through ‘Living Labs.’ These can be the sort of rapid, real-world consumer testing that companies like Amazon or Nike use, or the ‘special trade zones’ where governments test new rules of commerce.
The lesson is that Big Data can dramatically improve your company, but you have to match Big Data with real-world testing.
Prof. Alex ‘Sandy’ Pentland directs MIT’s Human Dynamics Laboratory and the MIT Media Lab Entrepreneurship Program, and advises the World Economic Forum. He also will be co-teaching in the upcoming MIT Sloan Executive Education program, “Big Data: Making Complex Things Simpler,” on March 27-28
Share your thoughts