Speed, ease of use guide Snowflake approach to analytics

Since its inception, Snowflake has aimed to make the analytics process faster and easier.

The vendor, founded in 2012 and now based in Bozeman, Mont., at its core is a cloud data warehouse, a repository where organizations could store their data before importing it into their business intelligence platforms for analysis.

Key to its mission, however, was that from the time it emerged from stealth in 2014, Snowflake has also sought to be automate certain data management tasks that reduce the time it takes to prepare and query data.

In addition, as it grew beyond its core capabilities, Snowflake started to work on enabling customers to not only manage their data in Snowflake but also, through integrations with various BI vendors, do their analytics work in-warehouse as well. No longer do they have to extract, transform and load data from their data warehouse to their BI platform for analysis.

Everything can be done in Snowflake, and the results are speed and simplicity, according to the vendor and some users.

Now, Snowflake is continuing to add features and integrations aimed at speeding up the analytics process and making it as frictionless as possible. Among the capabilities now in preview are Serverless Tasks and Query Optimization.

Serverless Tasks, in preview with a group of Snowflake customers, will enable customers to automate data pipeline tasks by scheduling SQL statements to be executed as a recurring event, while Query Acceleration, still in private preview, will speed up the exploration of large data sets.

"Performance shouldn't be hard," Bharath Sitaraman, a senior product manager at Snowflake, said on June 9 during a session at Snowflake Summit 2021, a virtual user conference. "Our goal is a near zero user maintenance so [customers] don't need to worry about knobs, indices, query hints, distribution columns, sorting keys and so on."

While Serverless Tasks and Query Optimization are in preview, other Snowflake capabilities that are designed to accelerate the analytics process are already generally available now as a service and address such needs as resource management, data engineering and SQL optimization.

Among them are Automatic Clustering, which automatically reorganizes table data to enable efficient manipulation of micro partitions; Search Optimization to speed up the location of subsets of rows from large tables; and Materialized Views to pre-compute what might otherwise be expensive calculations, such as aggregations.

Performance shouldn't be hard. Our goal is a near zero user maintenance so [customers] don't need to worry about knobs, indices, query hints, distribution columns, sorting keys and so on.

One Snowflake customer taking advantage of Snowflake's capabilities to make analytics faster and easier is HYAS, a cybersecurity vendor founded in 2015 and based in Victoria, Canada.

Its job is to develop software for clients that proactively detects and mitigates cyber risks before attacks happen. To do so, the tools collect massive amounts of data that gets sent to a data lake, and then the system sifts through that data to identify potentially harmful activity.

"Our customers are truly searching for a needle in a haystack," said Randy Fox, senior software engineer at HYAS.

That search, however, needs to be efficient. To build its tools, HYAS needed a platform that can search for that kind of granular data, find it, and do so quickly and easily without HYAS needing to allocate extra resources toward that search.

"Being a small company, HYAS needs to focus on our domain problem and not spend our time and money wresting with a barely manageable database," Fox said.

Specifically, HYAS needed to scale without being limited by the number of queries being run concurrently, improve its customer experience through better performance, make sure its ETL stopped affecting end users, and reduce technology and operational costs.

Using Snowflake's capabilities, HYAS has done that, according to Fox.

In particular, Search Optimization has benefited HYAS, speeding performance over its previous system, Fox said.

HYAS was a beta tester for Search Optimization. Before using the beta version of Search Optimization, searching a large table of location data took HYAS 37 minutes. With the beta version of Search Optimization, that was reduced to about 45 seconds. And with the generally available version of Search Optimization, which was released in March 2021, it took just 18.54 seconds, according to Fox.

"We were satisfied with the performance, so we took the leap to go all-in on Snowflake," Fox said.

Now, instead of spending time conducting lengthy searches of large amounts of data, built on Snowflake, HYAS' tools can search data in seconds to fuel the analytics process. HYAS' employees, meanwhile, are able to spend their time elsewhere in attempting to thwart cyber attacks.

"We want to free [customers] from the cumbersome burden of tuning exercises so they can focus on the actual task at hand," Sitaraman said.