As the ongoing drive to merge big data software with traditional data warehouse technology generates new and interesting opportunities to provide products and services that integrate these two systems more closely, channel partners are taking advantage of this new hybrid data model which presents challenges and possibilities to improve their clients' data management architecture.
Undoubtedly, value added resellers (VARs) and systems integrators relish the demand for products and services that come with the implementation of Hadoop, NoSQL, in-memory analytics and other big data technologies. Combined with enterprise data warehouse (EDW) systems, big data software allows them to apply new tools to offload some aspects of staging, data hub creation and extract, transform and load from relational databases onto Hadoop and other big data tools.
As businesses continue to spend on IT and companies increasingly recognize the importance of big data investments, channel partners will want to jump into a growing market. According to research from Capgemini Consulting, global organizational spending on big data exceeded $31 billion in 2013, and is predicted to reach $114 billion in 2018.
Not only is there money to be made in big data engagements, but channel partners and their clients alike are finding that information stored in data warehouse systems, as well as new data often generated from mobile devices, social media, the Internet of Things and other sources, can provide actionable insights that materially improve the way companies deliver products and services, as well as enhance the way they engage with their customers.
Essentially, by augmenting traditional EDW with big data technology, companies are primed to take advantage of past investments while upgrading their data management systems with newer, cheaper big data technology.
"This approach preserves the investments made in existing EDW people, process and technology, while allowing companies to get benefits from faster, less expensive big data solutions," said Malcolm Smith, director of data and integration practice at Cloud Sherpas, a company that provides cloud advisory and technology services.
David Loshin, president of Knowledge Integrity Inc., a consulting, training and development services company, observed that while there is a continuing role for the data warehouse to support a multitude of reporting and analysis tasks to help run the business, technologies like Hadoop, as well as many of the NoSQL and graph data management environments, are a disruptive force because they enable developers to build creative solutions that address tough challenges that are difficult to deal with using a more traditional data warehouse model.
"An example is using Hadoop to create a data lake that can provide storage augmentation for 'cold' data that is accessed less frequently, which makes migrating a conventional system to an in-memory system a reasonable alternative," Loshin said.
He predicts that over time more organizations will increasingly adopt big data technologies, but there will be a long transition period where both traditional data warehouse systems will coexist with growing big data platforms, and many of these deployments will be in the cloud.
"Keeping this in mind, it is clear that there are many opportunities for system integrators and service providers to complement, and in some cases replace, the internal management of the business intelligence and analytics ecosystems," Loshin said.
Facing big data software challenges
As VARs and systems integrators strive to manage data stored in data warehouses while introducing new big data technology, several challenges exist that are familiar barriers and have often impacted other data projects of the past. These obstacles include:
The inability to share data between departments;
Difficulty establishing internal teams to effectively manage IT projects; and
The inability to tap the skills and expertise necessary to steer companies into a new data warehouse technology world.
"It's not easy," said Peter Guerra, vice president at Booz Allen Hamilton Inc., who co-leads the firm's data science practice. "Ultimately it comes down to what their data architecture and data strategy is within the organization."
The impact of big data on traditional data warehouse management is only just beginning.
Malcolm Smithdirector of data and integration, Cloud Sherpas
According to Guerra, one of the biggest barriers to using big data technologies on top of data warehouses is coping with clients that have five or six data warehouses. These separate data warehouses are structured to manage and store information from different segments of the organization including the financial department and sales and marketing, as well as information from enterprise resource planning systems, customer relationship management systems and supply chain networks. In this siloed structure, various teams operate different data, and data sharing is limited.
"It's harder for these companies to achieve the strategic objective of being able to leverage all that value they collect in their data," Guerra said.
Collating data across disparate systems is an area of data management that Cloud Sherpas' data and analytics practice often grapples with.
"When we bring together disparate customer relationship management systems or merge service center operations onto a single system (usually in Salesforce1 Cloud) we are mostly tackling new problems for client master data management and cloud integration," Smith said.
Smith observed that most projects tend to focus on new capabilities rather than on upgrading existing data warehouse infrastructure. He also noted that many of his company's projects provide or consume data from the existing warehouse, but don't fundamentally change it. He said companies are keeping their relational data warehouses which he describes as "alive and well" and noted that most companies are moving to hybrid big data/traditional models rather than ripping and replacing their core EDW infrastructure.
In this new environment, Smith said typical projects for Cloud Sherpas involve data cleanup, client master data management, migration from in-house infrastructure and deployment of new cloud-based capabilities (usually on Salesforce).
"We are in the new application space and we frequently have the flexibility to introduce new infrastructure to support them -- which include these big data solutions," Smith said.
Partners develop services around data warehouse technology
As channel partners look for ways to help their clients, many companies are tailoring their approach to customers, providing consulting services that advise on technology and guide clients through the process of developing a new architecture that works for them.
Carey Moretti, principal consultant in big data intelligence at Trace3 Inc., a data consulting firm, said her company offers a six-to-eight week consulting engagement program which helps companies develop a strategy that involves developing a business case for why they should adopt new technology, creates use cases that identify, clarify and organize system requirements, and helps customers choose the right technologies to address their data challenges.
"There are hundreds of technologies [for] data platforms, whether it be traditional data warehousing or big data," Moretti said. "When you look at the landscape quarter over quarter, companies are developing and changing their capabilities quickly to meet the demand of their clients and of the industry."
Guerra said his company's approach is multifaceted and involves assessing a company's business, evaluating a company's data management team and finding out who is in charge of the data. Additionally, questions to be answered include: What data is being captured in the current data warehouse system? Why? And what data isn't being captured and why?
Essentially, big data, Guerra said, is changing data architecture, as well as the approach systems integrators are taking as they tackle these challenges. Guerra said in the past if a company had a data warehouse system, the data architect would think about its model and the star schema versus other schemas which they would set and revise over time.
"Really good data architecture now looks at the variety of different data sources, how they are going to be analyzed and comes up with a model for what data should be stored within which platform and how you can configure it within that platform," Guerra said.
As EDW merges with big data software technologies, other significant signs of change can only benefit channel partners. Loshin observed what he calls an "interesting phenomenon" going on in the conventional data warehouse world.
"There is a raft of emerging businesses focused on data warehouse automation, either through the combination of automation tools and service providers providing consulting, or cloud-based data warehouses spun up by companies providing data warehousing as a service," Loshin said.
Scanning the landscape, Loshin also pointed out that the exploding interest in using big data technologies, the success that Amazon is experiencing in the growth of its Amazon Web Services business unit, and the complexity of the fast- evolving suites of big data software are signs of the "perfect storm" that has laid the groundwork for channel partners to increase their data engagements.
"Third-parties [can] help businesses with all aspects of the information production process, including data ingestion, preparation, integration, design of analytical platforms, design and implementation of analytical models, and integration of those models with the day-to-day operations of the business," Loshin said.
Cloud Sherpas' Smith said he sees similar trends and believes big data projects are moving from proof-of-concept engagements to core, must-have technology. He also said in this new data environment he envisions his company driving success for its broad base of customers by enabling big data for new scenarios, while also standing up hybrid environments that augment traditional EDW.
"Big data is in the broader adoption phase where more and more businesses can gain the capabilities it offers without having to run a 'science project' to get them implemented," Smith said. "The impact of big data on traditional data warehouse management is only just beginning."
About the author: Nicole Lewis a freelance business and technology writer based in Miami.
Next Steps
Read more about the big data explosion
Learn about the tricks data warehouse systems use to stay relevant
