6 open source workflow engines and how to use them

Open source workflow engine features to consider

Several factors should be considered when selecting an open source workflow engine, including the application, environment, user interface, compliance, support and cost.

Application. Human-heavy processes, such as trouble ticket management, employee onboarding and password management, have different requirements than IT-heavy processes, such as spinning up data infrastructure, network configuration, disaster recovery or automating continuous integration and continuous delivery (CI/CD) for complex scenarios.

User interface. Identify the suitability of UI capabilities for different types of users. An IDE interface might work for technical processes primarily overseen by IT engineers, while GUI might be better at supporting different stakeholders such as security, business and finance teams. These GUI-based approaches can support low-code/no-code capabilities, making it easier for nontechnical users to support different use cases.

Artifacts. There are many ways to create, capture and organize information related to workflows. Business Process Modeling Notation (BPMN) is popular for business, security and compliance teams. YAML is popular with engineering teams. Developers could benefit from support for programming languages like Python, Java or C. Various workflow engines can map representations of the workflow for different users.

Compliance. Understand how a given tool will directly support compliance requirements or integrate with other compliance automation tools. At the very least, consider IT compliance requirements, such as ensuring services are configured correctly and recorded for auditors. Finance and healthcare enterprises might need additional oversight to enforce data privacy, residency and security compliance.

Microservices support. Some workflow engines run on top of a microservices architecture on top of Kubernetes to improve resilience and scalability. Many of these tools can automate provisioning, scaling and managing microservices architectures for other apps and use cases.

Hybrid/multi-cloud. Businesses that need to manage infrastructure across public clouds or on-premises private clouds should investigate tools that provide native integration into their existing or planned environments.

Cost controls. Workflow automation costs can sometimes spiral out of control due to configuration problems or unexpected events. Also, business and finance teams might require some oversight and the ability to assign costs to specific departments or projects. Look for support either directly within the workflow automation tool or by integrating with third-party cloud cost management tools.

GitHub likes. The number of "likes" and activity on the proprietary developer platform GitHub reflects the community support behind the open source tools. Many workflow engine projects are offered as open source to encourage experimentation and adoption as well as loyalty to the platform's vendor, while tools with strong community support can make it easier for users to adopt and scale workflow automation projects on their own.

6 open source workflow engine platforms to consider

Several open source workflow engine tools are available. The six detailed here were selected based on their popularity on GitHub and their support for novel IT operations automation capabilities. Some of these platforms received at least 10,000 stars on GitHub.

1. Apache Airflow

Apache Airflow enables teams to programmatically author, schedule and monitor workflows using Python scripts. The platform was developed by Airbnb in 2014 and contributed to the open source community in 2016. Airflow pioneered the use of directed acyclic graphs (DAGs) to define dependencies in complex automation, offering greater flexibility compared to traditional scripting approaches. IT operations teams use the platform to track the status of workflows and troubleshoot issues. Airflow can provision, back up and restore cloud infrastructure; automate system health checks; streamline patch management; conduct security audits; and collate IT compliance documentation.

2. Argo Workflows

Argo Workflows is a container-native workflow engine that supports complex provisioning and management processes on Kubernetes for CI/CD, data processing and machine learning (ML) model training processes. Like Airflow, it supports DAGs. IT engineers, developers and data scientists can define workflows using YAML files, making them easy to manage and version control. The platform is a good fit for deploying applications, setting up data processing pipelines, backing up and restoring processes, and monitoring complex infrastructure.

3. Camunda

Camunda is a legacy BPM vendor committed to building a thriving open source workflow automation community around its various core offerings. In 2019, the company rolled out Zeebe to orchestrate microservices at scale while also providing visibility for various stakeholders. The platform provides strong support for BPMN tooling, which can facilitate collaboration across business, finance, security and IT operations teams. It supports event streaming and a distributed architecture that improves scalability and resilience compared to traditional BPM workflow engines. Other complementary open source tools support modeling, implementing, connecting, monitoring, troubleshooting and optimizing workflows. Zeebe is a good candidate for workflows that span multiple human and IT services, such as trouble ticket response and employee onboarding.

4. Conductor

Conductor was open sourced by Netflix in 2016 and facilitated the streaming service's rapid growth. The core team behind the orchestration platform founded the software company Orkes to steward the project and provide support and customization for enterprise users. Conductor supports a variety of workflows for IT operations use cases, including cloud infrastructure provisioning, cloud security scanning, CI/CD pipeline orchestration, human-driven processes, BPMN workflows and data protection workflows. Orkes and the Conductor community are also working on agentic AI workflows that support autonomous decision-making, enforce security and apply AI governance.

5. Dagster

Dagster is emblematic of an open source workflow engine designed from the ground up to support complex data pipeline requirements for ETL -- extract, transform and load -- as well as data science, ML and AI workflows. Released in 2019, the platform defines data applications using a graph of functional computations for producing and consuming data assets. Different roles can use a variety of tools to create and manage pipelines, including Spark, SQL and Python, while collaborating on data infrastructure. These tools can help multiple roles -- such as infrastructure teams, developers, data scientists and data engineers -- visualize, configure, develop, test and monitor complex data pipelines in production. Dagster supports similar capabilities as Apache Airflow but also improves visibility into data lineage, understands dependencies and facilitates more granular data compliance-related workflows.

6. N8n

N8n is a workflow automation tool created by Jan Oberhauser in 2019. It models and automates business logic and IT processes, supports more than 400 integrations and over 900 templates, and has native AI platform capabilities. IT teams can create AI agent workflows using the LangChain software framework. N8n is a good fit for IT operations workflows such as automating patch updates, checking servers and resolving server failures. Common use cases include automating help desk ticketing, employee onboarding, alert aggregation, IT infrastructure provisioning and compliance processes.

George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.