Logistics firm taps Netflix Conductor service to manage AI

SPI weighs microservices orchestration vs. choreography

In 2022, Peralta's team evaluated alternatives to its internally built microservices workflow approach and came across microservices orchestration tools -- one of two major categories for microservices management, the other being microservices choreography.

Ezequiel Peralta

"We had some sequences of data synchronization services that were very compatible with orchestration, and we were doing data orchestration with Airflow, which was cool. But that's more for loading our data warehouse, which we still do," Peralta said. "But we were starting to get into something Airflow wasn't designed for: more transactional operations. And we decided to look into something that's more compatible with that."

Microservices orchestration tools use a centralized control plane to manage sequentially ordered workflow steps that are distributed among compute nodes called workers. Microservices choreography distributes workflows using a more loosely coupled event-driven system. Microservices orchestration allows more control over workflows, but microservices orchestrators can represent a single point of failure if they're not managed well.

AWS Step Functions can be used in either type of workflow. Microservices choreography can also be managed with event-driven architecture frameworks, such as Apache Kafka and Amazon EventBridge. Microservices orchestration tools include Temporal; Camunda; and open source Conductor, created at Netflix and released to open source in 2016.

SPI considered commercial microservices orchestration tools from Camunda, implementing microservices orchestration on its own using AWS Step Functions and tools other than Airflow for data management, and self-managing Netflix Conductor.

The AWS Step Functions pricing model charges per state transition or each time a step of a workflow runs and would quickly become expensive under SPI's transactional workflows, according to Peralta. In the meantime, Conductor stood out because of its approach to scheduling compute nodes or workers.

"Conductor is a queue system. We're setting up the task queue, and workers are calling it for work," he said. "It might seem like a small difference, but then you can come up with very generic workers, and the orchestrator handles rate limiting. … You're scaling your system without touching the orchestrator. I think that makes it more reliable."

In Peralta's view, supplying security credentials only for workers that need it is also more secure than using an orchestration system that pushes workloads to a waiting set of credentialed workers via an internet connection.

Conductor runs on a Kubernetes cluster, which can automatically restart containerized workloads within the orchestrator if necessary, to shore up the resilience of that potential single point of failure. But given the inherent complexity of managing Kubernetes on top of managing microservices, SPI would need help from a vendor to keep it running.

Eventually, Peralta encountered Orkes, a cloud managed service provider founded by the creators of Conductor to run the microservices orchestration framework for enterprises. A year ago, SPI put Orkes Cloud into production.

"Conductor has a lot of configuration of moving parts to optimize it for the reduced latency and output we need to have. [And] you need to have multiple databases running," Peralta said. "The value [of Orkes] for us is huge. We get a lot of the features that go into the open source version earlier because of the support we have."

At SPI, a newly efficient platform awaits AI apps

Orkes added a set of prepackaged workflows for AI microservices orchestration this month. Peralta said he's eager to try them as the company looks to add generative AI and machine learning based apps to replace rules-based automation in the company's fraud detection and freight spot market bidding processes.

Orkes' AI Orchestration feature, which supports large language models, including Microsoft's Copilot, OpenAI's GPT-4, Google's Vertex AI, and vector databases from Pinecone and Weaviate, also allow for human-driven tasks to be incorporated into these workflows.

Having microservices orchestration in place as the company adopts generative AI to analyze images and documents for classification or summarize available contracts in the bidding process will help SPI work more efficiently, Peralta said.

"[Orkes Cloud] allows us to parallelize our work more because you can have, for example, business specialists designing the workflows while the backend developers are working on all the deployment pieces and operations handles API access and database access," he said. "You can actually have the entire workflow designed with placeholder tasks [as it's implemented]."

We needed to make decisions about how to speed up development without compromising security and focus on building products.

Enterprises face dizzying array of AI workflow choices

As the initial generative AI hype dies down and enterprises search for ways to use it that align with existing workflows and governance practices, vendors from virtually every corner of the IT market are vying for their attention. These include microservices choreography and orchestration vendors as well as Kubernetes-based developer platforms, such as Red Hat OpenShift Data Science, and emerging MLSecOps pipelines from JFrog and others. Business process management vendors and integration-platform-as-a-service providers comprise an entirely separate landscape of tools for enterprises to consider.

"The lines are blurring among all of these, because modern apps are inherently distributed," said Matt Brasier, an analyst at Gartner. "People are also realizing that one great big [AI] model is not the best way to solve problems. Often, business problems are a mix of classification and content generation, so you want to chain some stuff together."

Deciding which IT automation tools to use will come down to the nature of the distributed applications businesses need. This and the skills available among IT teams to manage various automation tools are factors to consider when selecting the choreography or orchestration approach, Brasier said.

"I normally tell clients to begin with the skills of the person doing the orchestration, who is going to be creating that and managing that workflow," he said. "These things, like any integration, are going to break; a new version will come out that breaks all of your orchestration. So the skills of the person that is building and managing that orchestration will be key."

Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.