Disgruntled over big data? Maybe it's that visualization, magic box dependence
Big data, artificial intelligence and visualization may be nearing an emperor has no clothes intersection as we regular humans---business managers---start believing in beautiful but statistically insignificant graphics and magic boxes that are marketed as experts systems.
Simply put, the whole big data and analytics movement may be hitting Gartner's trough of disillusionment. You knew the trough was coming. Consider the following developments:
- Technologies like Hadoop emerged so we could store data thrown off from applications and the Internet of things.
- Business intelligence and the dashboards that went with it were lacking.
- The ability to store structured and unstructured data made us all information packrats.
- First movers---not surprisingly companies that were already leaders in their industries---swooped in with real returns and business use cases for big data projects.
- Second, third and fourth movers jumped on the big data bandwagon only to find returns weren't so easy. First, the leaders gobbled up the low hanging fruit. Second, there was a reason these enterprises weren't first.
- Artificial intelligence emerged in systems such as IBM's Watson.
- And visualization tools were either built into cloud applications such as Salesforce or Workday or brought to the enterprise masses via companies like Tableau.
Today, you can't escape the Internet of things and all that data that'll be thrown off from devices. Let's table the discussion about what we'll do with that information. We'll figure it out at some point, say business execs.
Sure they will.
The key words in that previous sentence are statistically relevant and narrative. Sengupta's beef is that today's artificial intelligence is taking human intuition out of the equation. That movement means two things: First, we'll need data scientists to tell us what to do. Second, we'll be marketed magic boxes that aim to lead us. Either way we're outsourcing way too much intelligence and instead of becoming data savvy we'll just become visualization slaves. After all, we all look smarter with a fancy graphic to back us up. The catch is visualization isn't analysis.
Also: Big data initiatives not quite delivering yet, survey shows | Samsung at CES 2015: How enterprise fits it in with Internet-of-Things | The Internet inside the enterprise: We don't have it, and we need it | Lowe's at CES 2015: Smart homes are about lifestyles, demystifying home automation | Welcome to the dystopian Internet of Things, powered by and starring you | Five years until the Internet of Things arrives? Why I hope it's a lot, lot longer | Cisco's next stop on Internet-of-Everything roadmap: Connected analytics
"If artificial intelligence (AI) systems can't let a business user understand it, the insight will remain the domain of data scientists or machines," said Sengupta, an Oracle and Microsoft alum. "We're just getting to the point where the market is beginning to understand how dangerous believing in dashboards and magical boxes."
In other words, we can't let the data scientists and engineers run away with big data or we'll have information haves and have nots.
BeyondCore's technology, which I played with, provides text summaries and animated briefings. Computers compute and humans overlay their knowledge. But the real win is that all the narratives are built on statistically sound data---there's even a regression proof available (view source for data scientists)---so the business user can keep the wonks at bay. "Humans look at graphs and get excited," quipped Sengupta. "The solution to the world's problems is not more data scientists. The communication chasm between the business user and analytics is a problem."
Specifically, BeyondCore's algorithms spin Hadoop's approach around. Hadoop is optimized for search and reducing data to something manageable via MapReduce. The emphasis is on the Map part of the equation. BeyondCore is optimized for the reduce part of the big data. "So far big data has been about bragging about how much stuff you had stored in the attic," said Sengupta. "The promise of big data is broken."
Sengupta has a lot of examples of how the new approach to artificial intelligence and data needs to work. For instance, BeyondCore highlighted how young women with diabetes had a 49 percent rehospitalization rate. Out of 30 million patients, the natural conclusion was that there was some gender issue or drug reaction. Turns out that many women weren't taking insulin because they gain weight. A box that requires you to ask a question probably couldn't deliver that insight. "How would a researcher know to look at that question?" asks Sengupta.
Due to the current state of big data and the intersection with artificial intelligence, Gartner created a category dubbed "smart pattern discovery." These vendors are likely to spring up to make sense of data lakes. These vendors are also likely to be acquired because there just aren't enough people to manage the data or interpret it. Five years from now we're going to need to bring statistically relevant data to business users.
"I don't believe in magic models. Prove it to me," said Sengupta.
ZDNet's Monday Morning Opener is our opening salvo for the week in tech. As a global site, this editorial publishes on Monday at 8am AEST in Sydney, Australia, which is 6pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and the US.
Previously on Monday Morning Opener: