For CIOs, there is a lot more than Hadoop to managing large data sets
No matter the topic at hand, lately my conversations with CIOs turn to Big Data — or, to use my editor’s preferred term, large data sets. CIOs have Big Data on the brain — and for good reason. Business leaders are convinced that mining varied, complex and unstructured large data sets generated internally and from all corners of the world will give their companies a competitive edge.
As Yvonne Genovese remarked in my story this week on the impact of Big Data on CIO careers, business leaders are convinced that wisdom — or better yet, money — lurks in these vast amounts of distributed data, and they are counting on their CIOs to find it.
That is the good news, of course, for CIOs. In this era of commoditized IT, leveraging the value in Big Data gives CIOs a plum role, or as Genovese bluntly said, puts them “back in the boardroom.” They become the heroes of the enterprise again, she said. But what is becoming clearer with every conversation I have with CIOs, turning Big Data into information, and information into actionable knowledge, is a huge challenge.
Sure, there are technologies and applications that can help do that, as consultants and vendors are eager to inform you, from the Hadoop and MapReduce frameworks to SAP’s HANA (High-Performance Analytic Appliance) and Watson, IBM’s super-duper computer. Prospecting for nuggets in the goldmine of data out there, however, means getting through a minefield of organizational challenges. Those include finding the right IT people for the job, for example, and competing against other departments interested in Big Data that might not answer to IT, such as marketing and operations.
“The CIO’s role is very difficult,” Boris Evelson of Forrester Research told me. “CIOs know the reason for Big Data, they know the technology out there; but no one knows literally how to create an organizational structure and best practices around it.”
Evelson said he has talked to clients with multimillion-dollar budgets for Big Data, approved by the business to get started tomorrow! But “when they called IBM, Accenture, Deloitte and PwC, you name it, all [the vendors] were doing was pitching Hadoop,” he relayed.
As for what kind of organizational structure they should aim for, figuring out how business processes will change, and who owns this stuff, “CIOs have to figure it out on their own,” Evelson said. “We are on the cusp of an era where we will learn from our mistakes over the next few years.”
So, there you are, out there again — the enterprise’s guinea pigs (or is it monkeys?) in deep space — braving the unknown. Here’s hoping you return from your journey as conquering heroes. And please, if you are inclined to talk about how you are tackling the puzzle of leveraging Big Data, let me know. I’m even willing to negotiate terms of disclosure.