Kyligence 4.5 adds Clickhouse to Intelligent Data Cloud
The open source Clickhouse OLAP database joins the Apache Kylin analytical data warehouse in a new update that provides a platform for different types of data queries.
Kyligence expanded the capabilities of its Intelligent Data Cloud platform with a new update revealed on Aug. 10 and set for general availability by the end of the month.
The Kyligence Cloud 4.5 update is the second major update of the Kyligence platform in 2021, following the Kyligence 4.0 release that came out in January.
At its core, Kyligence is based on the open source Apache Kylin project that provides users with the ability to enable OLAP cubes (online analytical processing), which create an aggregated view of data for query optimization. The challenge with the OLAP cubes approach is that it is not optimized for ad hoc queries, which is what the Kyligence 4.5 update aims to address.
In the update, the open source Clickhouse database is being integrated into the platform, providing a mechanism for faster responses for ad hoc queries.
Kyligence is now integrating both Clickhouse and Kylin in its Intelligent Data Cloud by way of what the vendor refers to as a smart tiered storage model that routes queries to the appropriate underlying technology.
The new Kyligence model will make the platform helpful for organizations that are querying different types of data, said Dave Menninger, an analyst at Ventana Research.
"Two of the big challenges of OLAP implementations has been the proprietary formats of the data structures and the scalability of those structures," Menninger said. "The tiered storage model of Kyligence 4.5 helps address both of those issues."
Bringing Clickhouse to Kyligence 4.5
Li Kang, head of North America operations at Kyligence, explained that Kyligence previously experienced a performance bottleneck for ad hoc queries that were not in the aggregate index that Apache Kylin enables.
Kang said ad hoc queries that might have taken 10 minutes with Kylin can be computed in under 20 seconds with Clickhouse.
Two of the big challenges of OLAP implementations has been the proprietary formats of the data structures and the scalability of those structures. The tiered storage model of Kyligence 4.5 helps address both of those issues.
Dave MenningerAnalyst, Ventana Research
Even at just 20 seconds, that query response time isn't fast enough for an interactive dashboard, from which a business analyst is looking for instant results. Kang noted that over time as ad hoc queries are generated and processed with Clickhouse, the results can be used to enable an aggregated query in Kylin, which provides a real- time response.
For users, the Kylin and Clickhouse technologies are abstracted away, with the Kyligence platform intelligently choosing the appropriate technology as queries come in, Kang added.
Support for real-time streaming from Kafka
Another new feature in the update is the ability to handle real-time data sources for event streaming.
The first data streaming that Kyligence supports is the open source Apache Kafka technology.
Kang said Kyligence will support other technologies, likely including Amazon Kinesis, in the future. Kang said Kyligence's goal is to enable users to query any type of data, including historical as well as real-time streaming data.
"With this hybrid model, we can build one data model, and behind the scenes, we can support both historical data as well as new streaming messaging or events coming from the streaming source," he said.
