Next Pathway update targets Hadoop migrations

Next Pathway has deciphered the codes for Hadoop migration.

The latest versions of Next Pathway's Crawler360 migration planner and Shift migration engine became generally available on Wednesday. Crawler360 now allows customers to scan through their Hadoop ecosystems and identify all the legacy applications within, along with their data dependencies. Crawler360 already covers legacy data warehouse systems such as Teradata and Netezza, as well as ETL frameworks such as Informatica and Talend.

Shift can then shift Hadoop workloads to cloud-based data warehousing platforms such as Snowflake, Amazon Redshift, Azure Synapse and Google BigQuery. The ability to translate Hadoop-based code to cloud targets has been in beta for the past two months and became generally available in Wednesday's update.

Hadoop is the core of many organizations' business intelligence (BI) strategies today, but according to Next Pathway CEO Chetan Mathur, customers have been looking for ways to move off it in favor of cloud-based data warehouses. However, due to the complexity of Hadoop environments, such a migration required developers to manually rewrite code in Hadoop-based data lakes to modern cloud architecture.

"Unlike an EDW [enterprise data warehouse], where you can lift and shift, you can't just do that to Hadoop. You need to first intelligently figure out the workloads," Mathur said.

With the new Hadoop support, Next Pathway can now provide a streamlined, automated alternative to this labor-intensive and expensive manual process, Mathur said.

Next Pathway stands out from other data management and migration vendors such as Komprise and WANdisco by doing more than moving data between environments. Its migration tools are designed to move on-premises data lakes and data warehouses to the cloud while keeping all data dependencies intact, ensuring legacy applications still work once they're in the cloud.

The challenge with translating Hadoop to cloud wasn't in deciphering the various codes and query engines Hadoop uses, but in developing a way to elegantly capture an open, non-standard environment, Mathur said. With this update, Crawler360 was upgraded to be able to search for duplicated data across multiple data silos.

"Netezza, Teradata, they're all pretty standard. But Hadoop was the Wild West -- there were no guardrails in place, and people were ingesting data willy-nilly," Mathur said. "We needed to understand how people stitched their environments together."

Next Pathway has no direct competitors, Mathur said, as customers looking to lift and shift data warehouses to the cloud do so through global systems integrators or not at all. Many Hadoop environments are stuck on premises, with companies continuing to sink money into storage for them because they don't know of an alternative. Those are the businesses Next Pathway wants to target, Mathur said.

Many of the most resource-intensive enterprise workloads have to do with BI, making them prime considerations for cloud migration. Moving BI workloads to the cloud has been a challenging prospect, however, because of their size and complexity, said Charles Araujo, principal analyst at Intellyx and founder of the Institute for Digital Transformation.

The task is daunting enough that organizations are hesitant to perform these large-scale migrations, especially in the case of Hadoop. The market demand is there, but the tools are not, Araujo said.

Next Pathway's big market differentiator is its focus on these BI workloads, Araujo said. Its tools address the full lifecycle of the migration process, from initial planning to cut-over, and allows customers to identify and fix redundancies and inefficiencies in their environments before moving them.

"There is plenty of pent-up desire to move these intensive and expensive workloads to the cloud, if enterprises can do so in a manageable and scalable way," Araujo said.