olly - Fotolia
Big data platform broadens place in analytics architecture
Big data platforms stumbled a bit getting out of the prototyping stage. But a view from the Strata Data Conference in New York sees broader use in the offing.
Big data platform deployments hit some rough patches in recent years, as the bright ideas of application developers and newly minted data scientists struggled to emerge from proof-of-concept and prototype projects. But a bellwether data management and analytics conference this month showed new signs of progress.
As a whole, the big data platform now appears to be finding greater maturity in enterprise applications, even as it adapts to the cloud services architectures that came into use just after the first glimmers of Hadoop and MapReduce appeared on the horizon.
The recent Strata Data Conference in New York offered evidence that the big data platform is encompassing more structured data. That's noteworthy because big data systems first cut their teeth on unstructured forms of data, while enterprise data warehouses (EDWs) continued to feast on the structured kind.
This broadening of focus on the big data side will affect BI and analytics architecture design, according to Mike Matchett, principal analyst at IT industry consulting firm Small World Big Data.
"What we are seeing is a convergence. The big data platform is really winning because it can handle both structured and unstructured data," Matchett said in this edition of the Talking Data podcast. As he also noted in a post-Strata article for SearchDataManagement on the evolution of big data technologies, his bet is on big data platforms to outpace legacy EDWs.
The rising trajectory of big data environments is also coinciding with a general trend toward microservices-oriented architectures; add to that event processing, said Jay Kreps, CEO and co-founder of Kafka platform vendor Confluent, in an interview at the Strata event.
Kreps and colleagues at LinkedIn created Kafka before starting Confluent in 2014; a form of middleware, the open source distributed messaging technology has become a linchpin in streaming big data analytics applications. Kreps said more developers are now focusing on the notion of event processing, which involves taking streams of events and joining them together for analytics -- not at the end of the day, but in real time.
From Kreps' point of view, the rise of real-time event processing systems is another factor with the potential to drive big data platform deployments forward. Listen to the podcast to hear more from him and Matchett on what they see happening with big data technologies.