Cloud-based integration seeks to tear down data silos

Application anarchy

Silos first became widespread in the era of mainframe computers, as companies installed separate business applications, both bought and homegrown, for individual departments. They proliferated further as mainframes gave way first to minicomputers and then to client-server architectures that put applications on dedicated Unix, Windows and Linux servers. The emergence of enterprise application integration, or EAI, tools made it easier for organizations to string loner systems together, helping to reduce the number of silos in companies that adopted EAI software—but then came the rise of cloud computing.

While many consider the cloud a silo buster, it can lead to more silos, not fewer, according to Jeff Kaplan, a consultant at THINKstrategies. "You have applications running on-premises, applications that you migrated to a cloud service provider or infrastructure as a service, and subscription-based SaaS solutions you've signed up for," he said. Each may also have different security and governance considerations, further complicating integration approaches to get the disparate apps working together.

Falling into a cloud silo trap can happen quickly, said Ross Mason, founder of cloud integration services provider MuleSoft. "The cloud's service-based delivery model makes it easy for enterprises to sign up for and deploy new applications. It's that very effortless sign-up and near-instantaneous deployment that can lead to silo proliferation."

Adding to the difficulty of integrating applications and data from multiple sources is the increased use of mobile computing. According to market researcher Gartner, mobile apps are having a profound impact on IT infrastructures. For example, Gartner predicts that by 2017, wearable devices will drive 50% of application interactions. Because they typically require data from more than one system, the importance of data integration is likely to be thrust even more to the forefront.

Integration, please

We did better communicating through APIs, letting the application do all the heavy lifting.

Mason's recommended strategy to clients embarking on integration projects is to develop two work streams. "We talk about a long-term view of connecting the pieces going forward, especially with mobile dominating the picture; otherwise, you have a spaghetti-like mess on your hands," he said. "But you also need a short-term strategy. What are the projects you need to get done this year and next?"

While migrating custom-built, on-premises applications to the cloud can work, Kaplan says, it's unlikely to provide the long-term flexibility needed for a business environment in which software updates are made frequently. But adopting a cloud-centric approach creates an opportunity to retire aging software and replace it with new applications that are inherently cloud-aware.

"On-premises systems, developed in a custom fashion, did their specific jobs, but creating each one separately reinforced a siloed approach. Often, this meant that IT failed to ensure that data was properly integrated and capitalized upon," Kaplan said. "Now that we're adding a complex layer of cloud-based services on top, there's the inclination among organizations to adopt wholly new SaaS and mobile applications, built specifically for the cloud."

The response from the software industry to that demand has been increased specialization -- for example, Aria Systems in transaction monetization, SafeLogic and CipherCloud in security, Asigra for backup and recovery, and SAP's Concur for travel and expense management all the way up to mammoth Salesforce.com for customer relationship management. But all the specialization adds to the potential for silos, creating the need for a modernized, more cloud-based integration.

Tear down these walls

Getting the walls to come tumbling down is essential to businesses of all sizes—and sometimes can be a trial-and-error process. At Formaspace, its infrastructure for acquiring and processing sales leads consisted of SugarCRM software running on an in-house server connected to "a glorified home router," according to Alspach, who today constitutes the company's entire IT department.

To improve access, the company hired an offshore IT consultancy that installed a business-grade Meraki router. The consulting company also determined that little maintenance had been performed on the server over its lifetime. Rather than take the system down to apply hundreds of updates and patches, its functions, including the SugarCRM implementation, were migrated to Amazon Web Services. A year later, Formaspace chose to move hosting of SugarCRM from AWS to Rackspace.  "As utilization increased, the need for a bigger, better, more powerful server became clear," Alspach said.

Kenan Rappuchi

To perform the migration, Formaspace brought in consulting company WSM. Its analysis painted an ugly picture, according to WSM partner Kenan Rappuchi. "We discovered that this offshore provider had built a highly customized integration to SugarCRM, manipulated for their specific business applications." 

Though moving SugarCRM into the cloud was a positive step, the website that received leads from SugarCRM to generate price quotes couldn't integrate into the public cloud. The configuration was far out of standard, essentially siloed. "[It] was so customized, if SugarCRM was upgraded to the latest modules, all of those custom modules would break," Rappuchi said.

WSM built a new Web application, also hosted by Rackspace, for generating price quotes that incorporated design specifications for custom furniture manufacturing. In a move to avoid paying for unneeded capacity, SugarCRM was switched to a pay-per-use model.

Visitors to the Formaspace website complete an online form to request a price quote, ask for product information or get a brochure, Alspach said. In each case, the completed form is routed through Salesforce's Pardot SaaS marketing platform. For pricing requests, Pardot hands the form off to SugarCRM. "Brochure and information requests continue all the way through Pardot and come back to us as a finished form," Alspach said.

All about that data

Ultimately, it isn't the applications that matter, said David Strom, a longtime networking consultant based in St. Louis. It's the data that's in them -- getting to it and using it. In addition, it's one thing to get the data conduits between cloud and on-premises applications working, but quite another to do so in a way that can handle both the volume of data and the speed at which it travels.

Taking control of its CRM and e-commerce data and maximizing information throughput in internal workflows were the primary drivers for V-Belt Global Supply LLC's integration of its SaaS and private cloud data, according to Gunnar Gustafson, e-commerce operations manager for the Sioux Falls, S.D., distributor of belts for automotive, agricultural and industrial applications.

"We were in a services situation where we leased software with the data living on the provider's servers," Gustafson said. "We felt like we didn't own the data."

The first step toward regaining control of the company's data was implementing the Magento e-commerce platform on a dedicated server running on-premises in a private cloud, followed by an integration phase in which the Magento system was given access to existing CRM data stored in a cloud version of Microsoft’s SQL Server database. "Eventually, we replicated the data and migrated it to a fresh Magento install," Gustafson said.

Gunnar Gustafson

Magento's use of APIs for data connectivity simplified the data-sharing and integration process, he added. The V-Belt team first tried SQL Server Integration Services software but found it to be "very fragile as a connection method," Gustafson said. "We did better communicating through APIs, letting the application do all the heavy lifting."

Unfortunately, system performance turned out to be underwhelming, Gustafson said. So his team spread the Magento platform over three private cloud servers and added application performance analytics tools from New Relic to help monitor how things were working. "If we found that performance was lagging, we could just spin up another cloud. That was a huge part of the appeal of our private-cloud architecture."

Underlying client-server architectures often aren't conducive to cloud migrations—as a result, companies can run into roadblocks when attempting to move back-end business applications to the cloud. "An application that is not, at a minimum, capable of running inside a Web browser almost certainly should not be on a cloud server," WSM's Rappuchi said. On-premises installations of software from established vendors such as SAP, Oracle, Microsoft and Kronos -- and millions of homegrown applications built by IT departments -- might not work well in the cloud, he cautioned.

If a company wants to move toward the cloud, those applications might need to be replaced—or conversely, they can continue to function where they are and co-exist with new apps designed for the cloud age through a well-planned integration strategy. That requires a detailed understanding of the capabilities and operating requirements "of every single application, new or old, custom-built or purchased," Rappuchi said.

A cloud-based integration initiative is also likely to take a lot of time and effort, not to mention money. But getting it right can literally bring an organization's business processes and operations closer together. "Ultimately," he said, "the goal is to break down the walls and knock down the silos."