With data coming from on-premises and cloud sources and spanning legacy systems, databases and even flat files, City Furniture faced problems bringing all its data together.
Just before COVID-19 started to hit hard in early 2020, Ryan Fattini was facing a data challenge that he really needed to overcome.
The answer turned out to be data virtualization.
The challenge for Fattini, director of data engineering at City Furniture, an online retailer with a chain of physical stores in Florida, is that all the different teams in the company were asking for data they could use for business intelligence and analytics.
The teams included sales, marketing, supply chain and business operations. Each used various sources of data coming from a diverse set of sources -- from something as basic as flat file Excel spreadsheets, to cloud databases, to a legacy IBM iSeries server platform.
Meanwhile, Fattini's team was struggling to build out its own data pipelines to bring in all the data to a data warehouse where it could be analyzed.
"The demand was overwhelming to build out these pipelines," Fattini said. "We knew it wasn't the right approach to burden our software teams to build the software to build these pipelines."
The path to data virtualization
So City Furniture engaged with analyst firm Gartner, which brought up the concept of data virtualization.
"We didn't know what the heck they were talking about and what the relationship of data virtualization was with the data warehouse," Fattini said.
Ryan Fattini
Fattini and his team did end up learning what the relationship of data virtualization with data warehouse was all about -- a way to easily enable the connection of data sources. City Furniture ultimately did a proof-of-concept deployment with data virtualization vendor Denodo in early 2020 as COVID-19 began spread across the country.
The result of the proof of concept was positive. It helped remove the pressure for Fattini's team to build its own data pipelines. Fattini noted that with data virtualization from the Denodo platform, he and his team were able to connect data much faster.
"What Denodo solved for us was eliminating all the heavy software engineering efforts around processes to build all of these pipelines," Fattini said.
Why City Furniture chose data virtualization
Fattini explained that City Furniture has an old IBM iSeries system that has been in place since the 1980s. That legacy system is streaming data via a change data capture approach into an IBM DB2 warehouse running in the IBM cloud.
The retailer also maintained other systems including MySQL, MongoDB and flat file databases. On top of the Denodo layer are Cognos BI and Power BI for analytics.
The demand was overwhelming to build out these pipelines. We knew it wasn't the right approach to burden our software teams to build the software to build these pipelines.
Ryan FattiniDirector of data engineering, City Furniture
"Denodo sits on top of all the different data sources," Fattini said.
With Denodo atop of the profusion of data sources, Fattini said City Furniture is able to virtualize all the sources and create a semantic layer. That layer makes it easier for him to deliver data to the various departments in the company.
Performance is often a difficult challenge for organizations that decide to use data virtualization instead of loading data directly into a data warehouse. Data virtualization creates latency from the source to the destination, but the data doesn't move.
By not keeping the data in the exact location where it is being analyzed, there is potential for some data latency.
Fattini noted that in general the performance of Denodo's data virtualization works well for City Furniture, though latency can sometimes be a problem.
However, not all reports need real-time data, so low latency doesn't matter as much, Fattini explained. And with Denodo he can set up different time intervals for data caching that can reduce latency when needed.
City Furniture has been running data virtualization in production for a year now and it's an approach that Fattini suggested can work for other organizations that face the same data challenges that his does.
"If you're a company where you have a lot of software development work going into data pipelines, I would look into the data virtualization layer to try to take some of the burden off of your software teams and get some of those data sources combined quicker and get those insights out faster," he said.
Ryan Fattini
