Big Data Starting Path Traps

At a recent CIO Executive Summit with over 90 CIO’s in Chicago, I stated that I thought that we were entering Phase 3 of the Big Data journey, where:

• Phase 1 was the Big Data Educational phase, with folks reading magazines and blogs, and attending conferences to learn more about what is big data
• Phase 2 was Big Data Experimentation, where folks would install a “big data technology” like Hadoop and start playing with the technology to gain some hands-on experience
• Phase 3 is Big Data Business Transformation where IT and the business collaborate to identify opportunities to exploit big data to power the organization’s key business initiatives

When I asked where these CIO’s felt they were today in their big data journey, I was shocked when over 80% of them raised their hands to indicate that they were still in Phase 1.

I hear stories every day of IT organizations that are reluctant to engage the business in developing their big data strategy.  Many IT folks seem more comfortable going down the technology path treating big data as a technology innovation instead of a business transformation opportunity.

These organizations are missing the opportunity to leverage the big data discussion to move from a retrospective, “rearview mirror” view of their business using summarized data to monitor the business, to a forward-looking, real-time, predictive organization that is leveraging all available data in order to optimize the business (see Figure 1).

Figure 1:  Big Data Business Transformation

When IT organizations try to go down the big data strategy path by themselves, they end up focusing on only the technology aspects of big data.  They apply the lens of what they already know – in particular, data warehousing and business intelligence – as they try to map out their big data journey.  Unfortunately, this misses the larger data monetization and business transformation opportunities (see Big Data Business Maturity Chart).  You simply cannot replace engaging the business stakeholders early in the big data journey.  Doing so will establish an environment of collaboration whereby big data innovations can transform the organization’s value creation processes.

Technology-led Traps on the Big Data Journey

When IT organizations try to move down the big data technology path without engaging the business, they tend to fall into the following traps:

• Start with Hadoop and load up a bunch of data.  First, big data does not equal Hadoop.  Hadoop, along with its brethren like MapReduce, Pig, Python, Ruby on Rails, Mahout, and others provide a unique capability to manage, transform, and analyze massive amounts of semi-structured and unstructured data (it can also be used on structured data as well) that previously could not easily be mined for customer, product, and operational insights.  And while Hadoop is an important tool in the IT big data tool bag, it can not replace the need to understand the business questions and decisions that the business stakeholders are seeking to address in support of the organization’s key business initiatives.
• Acquire in-memory computing technology to accelerate the existing data warehouse environment.  Organizations have invested tens and hundreds of millions of dollars, and countless years of development and training, into their data warehouse and business intelligence environments.  And organizations continue to invest in these environments at a healthy clip today.  However, simply trying to accelerate an existing BI and data warehouse environment with an in-memory capability without considering how new big data innovations can do things better, or even do things that could not be done before, is the wrong first step.  It’s like saying that you’re going to the moon and then climbing to the top of a tree to get there.  While it’s true that you are closer to the moon from the top of the tree, you just can’t get to the moon from the top of a tree.
• Hire a big consulting firm to develop a 400-slide Big Data Strategy PowerPoint.  Consultants can play an integral major role in helping organizations build out their big data strategies.  Consultants can bring hands-on experience and perspectives from different industries that can be the catalyst for accelerating your big data journey.  However, 1) the consulting firm should complement, not replace, the skills and capabilities that your organization is ultimately going to need and 2) starting with the uber big data strategy just might not be the right place to start.  Start small and focus on leveraging big data to power one of your organization’s key business initiatives.  It’s the only way to build support and relevance with the business side of the organization.
• Wait for your business intelligence (BI) or data warehouse vendor to solve the technology problem for you.  While this approach seemed to work well in the past, where monolithic products evolved slowly to engulf incremental technology innovations, this approach is not going to work in a world where:
  1. Open source innovations are challenging every part of the traditional BI, data warehouse, and analytics stack.
  2. Universities (and not just the Stanford’s and MIT’s of the world) and venture capitalists are fueling entrepreneurial technologies and approaches with the goal of disrupting old school technology capabilities (and grabbing a piece of the $90B business analytics market[1]).
  3. Adoption of these new technologies can provide distinct and compelling competitive advantage to first movers.  Think Amazon versus Barnes and Noble, or Netflix versus Blockbuster, or Apple versus the entire music distribution industry.  There is only one chance to be the first mover, and waiting for your current software vendor might mean the difference between being Walmart and Kmart.