Argo Workflow — 云原生工作流

什么是 Argo Workflow

Argo Workflow 是 K8s 原生的工作流引擎，每个步骤都是一个 Pod，支持 DAG（有向无环图）和顺序执行。

安装

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.0/install.yaml

基础工作流

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: hello-world
  templates:
  - name: hello-world
    container:
      image: busybox
      command: [echo]
      args: ["Hello, Argo!"]

DAG 工作流

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ml-pipeline-
spec:
  entrypoint: ml-pipeline
  templates:
  - name: ml-pipeline
    dag:
      tasks:
      - name: data-preprocessing
        template: preprocess
      - name: feature-engineering
        template: features
        dependencies: [data-preprocessing]
      - name: train-model-a
        template: train
        dependencies: [feature-engineering]
        arguments:
          parameters:
          - name: model-type
            value: "random-forest"
      - name: train-model-b
        template: train
        dependencies: [feature-engineering]
        arguments:
          parameters:
          - name: model-type
            value: "xgboost"
      - name: evaluate
        template: evaluate
        dependencies: [train-model-a, train-model-b]

  - name: preprocess
    container:
      image: python:3.11
      command: [python, preprocess.py]

  - name: features
    container:
      image: python:3.11
      command: [python, features.py]

  - name: train
    inputs:
      parameters:
      - name: model-type
    container:
      image: python:3.11
      command: [python, train.py, --model, "{{inputs.parameters.model-type}}"]

  - name: evaluate
    container:
      image: python:3.11
      command: [python, evaluate.py]

WorkflowTemplate（可复用模板）

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ci-pipeline
  namespace: argo
spec:
  templates:
  - name: build
    inputs:
      parameters:
      - name: repo
      - name: tag
    container:
      image: docker:24
      command: [docker, build, -t, "{{inputs.parameters.repo}}:{{inputs.parameters.tag}}", .]
      volumeMounts:
      - name: docker-sock
        mountPath: /var/run/docker.sock
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock

常用操作

# 提交工作流
argo submit workflow.yaml -n argo

# 查看工作流
argo list -n argo
argo get my-workflow -n argo

# 查看日志
argo logs my-workflow -n argo

# 删除工作流
argo delete my-workflow -n argo