Argo Workflows v3.1 is coming…

Hot on the heels of v3.0, v3.1 introduces some features which we know we’re excited about, but first some numbers!

• 40 new features
• 125 fixes
• 243 pull requests
• 52 contributors

You can see previews of the features in our CNCF webinar at around 20m in:

Data Template

Users often source and transform data as part of their workflows. The data template provides first-class support for these common operations.

A data template is especially useful for iterating over artifacts in a bucket. This allows you to have one step in your workflow create the artifacts, and then fan out in parallels over all the artifacts

spec:
  entrypoint: data-transformations
  templates:
    - name: data-transformations
      steps:
        - - name: list-log-files
            template: list-log-files
        - - name: process-logs
            template: process-logs
            withParam: "{{steps.list-log-files.outputs.result}}"
            arguments:
              artifacts:
                - name: file
                  s3:
                    key: "{{item}}"
parameters:
                - name: file-name
                  value: "{{item}}"
- name: list-log-files
      data:
        source:
          artifactPaths:
            name: test-bucket
            s3:
              bucket: my-bucket
transformation:
          - expression: "filter(data, {# endsWith \"main.log\"})"
- name: process-logs
      inputs:
        parameters:
          - name: file-name
        artifacts:
          - name: file
            path: /file
      container:
        image:  argoproj/argosay:v2
        command: [ sh, -c ]
        args:
          - |
            echo {{inputs.parameters.file-name}}
            head /file

Learn more

Conditional Artifacts

The Conditional Artifacts and Parameters feature enables to assign the Step/DAG level artifacts or parameters based on an expression. This introduces a new field fromExpression: ... under Step/DAG level output artifact and expression: ... under step/DAG level output parameter. Both use the expr syntax.

- name: coinflip
  steps:
    - - name: flip-coin
        template: flip-coin
    - - name: heads
        template: heads
        when: "{{steps.flip-coin.outputs.result}} == heads"
      - name: tails
        template: tails
        when: "{{steps.flip-coin.outputs.result}} == tails"
  outputs:
    artifacts:
      - name: result
        fromExpression: "steps['flip-coin'].outputs.result == 'heads' ? steps.heads.outputs.artifacts.headsresult : steps.tails.outputs.artifacts.tailsresult"

Learn more

Emissary Executor + Container Set Template

v3.1 introduces a new executor called the emissary. The emissary works by replacing the container's command with its own command. This allows that command to capture stdout, the exit code, and easily terminate your process. The emissary can also delay the start of your process. The container set template allows you to run multiple containers within a workflow pod. This allows the containers to share a workspace.

Starting containers is cheaper and faster than starting pods, so a template that uses a container set can be faster and cheaper.

The Container set template is designed to work with the emissary to allow you to execute a DAG within a pod, for example:

spec:
  entrypoint: main
  templates:
    - name: main
      containerSet:
        containers:
          - name: a
            image: argoproj/argosay:v2
          - name: b
            image: argoproj/argosay:v2
            dependencies:
              - a
          - name: c
            image: argoproj/argosay:v2
            dependencies:
              - a
          - name: d
            image: argoproj/argosay:v2
            dependencies:
              - b
              - c

Learn more about the emissary + container set template

Expression Template Tags

In version v3.0 you can use simple template tags within your workflow to replace values, e.g.:

args: [ "{{ inputs.parameters.message }}" ]

This only allows search-and-replace of variables. Expression tag templates allow these to be expressions that are evaluated, which allows you to have logic in your templates rather than in a pod.

This allows you to write workflows that are simpler and faster to execute.

spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: task-0
            template: pod-0
            arguments:
              parameters:
                - name: foo
                  value: "{{=item}}"
            # withParam must be a JSON list encoded as a string, so you use `toJson` to perform the conversion
            withParam: "{{=toJson(filter([1, 3], {# > 1}))}}"
    - name: pod-0
      inputs:
        parameters:
          - name: foo
      container:
        image: argoproj/argosay:v2
        # in this example, we use `asInt` to cast a parameter (which are ALWAYS strings) to an int so we can
        # multiple by 10
        args:
          - echo
          - |
            hello {{=asInt(inputs.parameters.foo) * 10}} @ {{=sprig.date('2006', workflow.creationTimestamp)}}
          - /output
      outputs:
        parameters:
          - name: output
            valueFrom:
              path: /output

See how the expressions appear in the inputs and ouputs

Learn more

Notable Contributors

A big thank you to all our contributors!

• BOOK
• Iven
• Michael Crenshaw
• Peixuan Ding
• Riccardo Piccoli
• Shoubhik Bose
• Stephan van Maris
• Tim Collins
• Vladimir Ivanov
• Yuan Tang
• Zach Aller
• dinever
• kennytrytek
• tczhao

Photo by Marek Piwnicki on Unsplash