AWS Elasticsearch distro stokes open source competitive fires
The new AWS-backed Open Distro for Elasticsearch may constitute a fork of the popular distributed search and systems analytics engine. What it means for customers is yet to come.
AWS has teamed up with Expedia and Netflix on a new version of Elasticsearch that is fully open source. Moreover, it was necessary to create in light of actions by the distributed search and analytics engine's primary maintainer, Elastic.
Open Distro for Elasticsearch will deliver security, monitoring, SQL support and other features on top of the core code base, and it's licensed under Apache 2.0, AWS said.
Elastic's own actions forced this move, according to AWS, which has offered AWS Elasticsearch, a managed service based on the main codeline, since 2015.
"Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base," wrote Adrian Cockroft, vice president of cloud architecture strategy at AWS, in a blog post. "While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on."
AWS talked to Elastic about its worries and offered to put more people and effort behind a purely open source version of Elasticsearch, but the vendor was rebuffed, Cockcroft claimed.
Ultimately, "the majority of new Elasticsearch users are now, in fact, running proprietary software," Cockcroft claimed.
The Elasticsearch GitHub site includes a directory that groups together all the proprietary code Elastic offers under its Elastic License. But, in AWS' view, this doesn't go far enough.
Elastic founder and CEO Shay Banon characterized AWS' move as nothing new and something his company will survive.
"Our products [have been] forked, redistributed and rebundled so many times I lost count," Banon said in a blog post. "From various vendors, to large Chinese entities, to now, Amazon. ... None of these have lasted. They were built to serve their own needs, drive confusion and splinter the community."
Elastic has always sought to make its open source code easy to implement cleanly, and its commercial code has been copied by others, he added.
It has joined forces with commercial vendors such as the application performance monitoring provider Opbeat, but rejected others that asked for preferential treatment, according to Banon. AWS was among the latter group, he said.
AWS' gravitational pull poses change for open source
AWS compared its move with Elasticsearch to Corretto, a distribution of OpenJDK, the open source version of Java SE that it released last year after Oracle said it would stop commercial support for SE in January.
It also follows long-borne criticism from parties who say AWS takes too much from open source projects for its own commercial gain and gives back too little.
Today's open source software communities are as much about end users as they are about the providers.
Jay Lymananalyst, 451 Research
The waters got muddied further in January, with the release of Amazon DocumentDB. The MongoDB-compatible NoSQL data store implements an API that supports workloads from MongoDB version 3.6 and earlier. That API is licensed under Apache 2.0, in contrast to current versions of MongoDB, which its parent company placed under a more restrictive license in October.
As for Open Distro for Elasticsearch, AWS' vast resources and industry influence make it likely that the Elasticsearch community will experience at least some fragmentation, as developers decide whether to commit changes to the core project or the new one.
Still, Open Distro for Elasticsearch should not be viewed as a fork, and its backers will contribute new features back to the Elasticsearch upstream project, AWS said. What remains to be seen is how Open Distro changes or augments its work on the AWS Elasticsearch managed service.
Whether Open Distro is a good or bad thing depends on who you ask, said Stephen O'Grady, an analyst at RedMonk in Portland, Maine.
"Elastic would undoubtedly see a fork as a negative," he said. "Whether third parties do will depend largely on their views of whether the commingling of open and proprietary code is acceptable or not."
As for the purity of Open Distro going forward, AWS makes its money operating a service, not selling software, so it is probable that all of its contributions to Elasticsearch will be Apache 2.0-licensed, as promised, O'Grady added.
Still, Elastic has reason to be concerned.
"The hyperscale cloud providers are large, incredibly well-resourced entities with major advantages in economies of scale, customer acquisition efficiency and so on," he said. "If they decide a fork is their best competitive mechanism, you just inherited a very large, highly capitalized competitor."
Overall, the situation reflects a couple of things about modern, enterprise open source software projects and communities, said Jay Lyman, an analyst at 451 Research.
First, when these types of issues arise, they are aired out publicly. While this may seem like a negative development, it is the sign of vital software that matters to big companies such as AWS, Expedia and Netflix, he said.
"Second, today's open source software communities are as much about end users as they are about the providers, since both are consumers of open source software code," Lyman added. "This may put more pressure on how a commercial company operates or monetizes open source software, but it may ultimately boost the open source software project upon which it is based."
