Introducing Argo — A Container-Native Workflow Engine for Kubernetes

Pratik Wadher
Argo Project
Published in
4 min readAug 31, 2017

--

Today, we are excited to announce the launch of the Argo Project, an open source container-native workflow engine for Kubernetes conceived at Applatix.

The Applatix team is an experienced group of enterprise software engineers from companies like Data Domain (Data Protection), Nicira (SDN), Bebop (Enterprise Development Platform acquired by Google), Apigee (API Platform pioneer acquired by Google) and Tintri (Virtualization) to name a few. These companies provided key infrastructure building blocks to solve fundamental system-level problems for enterprises.

Having led R&D and Operations teams for close to two decades, one of the challenges I have had to address is to continue delivering quality products while continuously scaling the teams, infrastructure, and services. Virtualization and automation were transformative in achieving these goals. Containers and cloud have the potential to provide another quantum leap.

The emergence of containers and Kubernetes is driving a major shift in how applications and services will be developed, distributed, and deployed in the future. This will require new approaches to address the same basic system-level problems in monitoring, data protection, and workflow automation to name a few. We founded Applatix to focus on providing similar building blocks for this new ecosystem.

As we have worked with Kubernetes over the last 2 years, our enthusiasm for its potential as a core building block in enterprise software deployments has grown immensely. At the same time, we discovered that the ecosystem would benefit from the addition of essential elements that we are uniquely qualified to contribute. For example, an integrated workflow engine is fundamental in a distributed system for orchestrating jobs as well as distributing and deploying complex microservices-based applications.

We can tell you from experience that jobs don’t run themselves! Development teams typically build a framework to serve as a scalable “hopper” to manage the software development lifecycle workflows such as CI/CD, testing, experimentation, and deployments. This motivated us to create a generic container-native workflow engine for Kubernetes that makes it easier to use other services and enable the workflows to do useful work.

Introducing Argo

Our first contribution to the Kubernetes ecosystem is Argo, a container-native workflow engine for Kubernetes. You can think of Argo as an engine for feeding and tending a Kubernetes cluster. Argo makes it easy to specify, schedule and coordinate the running of complex workflows and applications on Kubernetes.

Argo accomplishes this by combining a workflow engine with native artifact management, admission control, “fixtures”, built-in support for DinD (Docker-in-Docker), and policies. There are several uses for Argo workflows including:

  • Traditional CI/CD pipelines
  • Complex jobs with both sequential and parallel steps and dependencies
  • Orchestrating deployments of complex, distributed applications
  • Policies to enable time/event-based execution of workflows

Why a container-native workflow engine?

Argo is a robust workflow engine for Kubernetes that enables the implementation of each step in a workflow as a container. It provides simple, flexible mechanisms for specifying constraints between the steps in a workflow and artifact management for linking the output of any step as an input to subsequent steps. Integrating artifact management with workflows is critical for achieving portability as well as greater efficiency and simplicity.

Unlike VMs, containers provide lightweight and reliable packaging of the execution environment and application together into a portable self-contained image. By building workflow steps entirely from containers, the entire workflow itself including all execution of steps as well as interactions between steps are specified and managed as source code.

While there are many use cases for workflows, one simple example would be a continuous integration (CI) environment to enable fast iteration and innovation. With Argo and Kubernetes, a user not only can create a “pipeline” for building an application, but the pipeline itself can be specified as code and built or upgraded using containers. In other words, you can use CI to manage your CI infrastructure :)

A sample Argo workflow

Another advantage of an integrated container-native workflow management system is that workflows can be defined and managed as code (YAML). With Argo, workflows are not only portable but are also version controlled. A workflow that runs on one Argo system will run exactly the same on another Argo system. Just point your new Argo system to the same source repo and container repository as used by your previous system.

Argo workflow illustrating passing artifacts between steps

Our goal at Applatix is enable developers and operators to adopt containers and Kubernetes for developing and deploying distributed applications. Argo is being used in the community today by a number of open source projects and enterprises including RocksDB Cloud, Heron, Apache BookKeeper, Apache Pulsar, and AppDynamics.

Argo is our first step toward advancing the Kubernetes ecosystem and making Kubernetes accessible and more useful to the broader community. We are grateful to the early users who provided invaluable feedback. We invite you to give Argo a try and welcome your input.

We hope that you find it an indispensable and delightful part of your containers and Kubernetes journey. To see our demo at the Kubernetes Community Meeting, please click here.

--

--