This article was originally published at The Wall Street Journal at Online.WSJ.com
on November 29th, 2012.
The next Next Big Thing is Big Data. Evangelists claim it has the power to reveal hidden truths about our companies, about our lives, about society as a whole.
So important is it that last week's Silicon Valley Comes to Oxford annual event was built around the topic.
Inevitably the real world crashes into digital utopia. According to Peter Tufano, the dean of Oxford's Said Business School, which played host to the event, while awareness of the topic was high among enterprises, only about 6% of companies have got beyond a pilot stage, and 18% are still in one.
"That means three-quarters of industries are looking at this and saying 'what is this all about?'"
Why aren't they looking at Big Data? "The answer across all business," he said, "was 'we don't know what the business case is.'"
But according to speakers at the event, the business case has already been answered. Michael Chui has extensively researched the area for McKinsey Global Institute. His conclusion is emphatic: "The use of data and analytics in general is going to be a basis of competition going forward for individual firms, for sectors and even for countries. Those companies that are able to use data effectively are more likely to win in the marketplace."
MGI's research showed that in just one field—personal location data—some $100 billion of value can be created globally for service providers through use of data. He suggested at a talk last year that the benefits for consumers could be six times that. "We find that Big Data tends to accelerate the capture of surplus [value] by consumers." In other words, not only do companies do well, but customers do even better.
And if companies need even more persuading, what about the claim at the conference that Big Data played a part in re-electing Barack Obama to the White House? John Aristotle Phillips, Chief Executive of Aristotle International—a nonpartisan company that applies technology to politics and political communication—said the use of data analytics had a material effect on outcomes.
"This was the first presidential election campaign where all of the data that was coming into the campaign was successfully collected and centralized," he said. "The Obama campaign did a successful job with that; the Romney campaign did not."
Mr. Phillips was quick to add: "The election was not won because of Big Data, but it played a very important part."
According to Mr. Phillips by bringing together all of the information about individuals it was possible to build a much more accurate picture of a voter and so focus efforts on them in a much more targeted way than simple crude TV electioneering.
So if Big Data can get a man into the White House, how can companies use the same, or similar tools, to achieve those productivity gains that McKinsey was suggesting?
Stephen Sorkin, vice president of Engineering for the U.S.-based Splunk, a Big Data analytics company, suggested that often companies overlook one really important source of information that they are already sitting on—computer log files. Computers generate huge log files that record all manner of information. A Web server, for instance, keeps very detailed records of every thing it does. His company provides tools for customers to mine that data.
"That data is often used just to troubleshoot problems, but you can do a whole lot more with it—it is a categorical record of everything that has happened."
Mr. Sorkin said this gave companies "the opportunity to know, at a very fine grain what customers are doing, how they are doing it and perhaps why they are doing it."
One false promise that some proponents of Big Data hold out is that somehow vast oceans of digital data can be sifted for nuggets of pure enterprise gold.
Mr. Sorkin quickly shoots such hopes down. "It is not going to happen magically. The software only finds correlations, not causations. In order to find causal relationships you have to do work.
"If you take any sufficiently large data sets, you are going to find correlations," he said. "You need a human in the loop to work out which are important."
So given the evidence and the success stories, why are so few companies actually embracing the opportunities?
Andrew Grant, Chairman of Satalia, a U.K. university spinout that applies algorithms to optimize Big Data, suggests cultural obstacles are the biggest impediment.
"At this time you would think companies would be saying we need to innovate our way out of the financial crisis. What seems to be happening is the opposite—everyone is retrenching. But there are real opportunities here."
There are a raft of other obstacles, including a regulatory framework that was designed for a different data world and a lack of skills to actually do the work.
There is also the "creepy factor" says Mr. Sorkin. "The richest examples of Big Data are to understand consumer behavior and optimize your product for it. That is where the danger can lie."
He suggests thatunless companies are careful, optimizing a product can end up putting consumers off, like ads that follow you from site to site. "Companies will customize some aspect based on the consumer and the consumer can think it is a violation of their privacy, or it can just feel creepy. I can make something perfect, but perfect may not be what the consumer is looking for."
Write to Ben Rooney at ben.rooney@wsj.com
Corrections & Amplifications:
MGI's research showed that in just one field—personal location data—some $100 billion of value can be created globally for service providers through use of data. A previous version of this article said that $100 billion of value can be created in the U.S. alone.
.jpg)
.jpg)
