# Docker Automation Executor Design

> Canonical path: `specs/automation/docker-automation-executor.md`
> Status: living design
> Scope: MeshX server-managed automation job execution substrate.

## Summary

MeshX server-managed automation execution uses Docker as the job execution
substrate. MeshX remains the automation control plane: AGS owns automation definition source
files, while MeshX owns run and job ledger state, queue leases, waits, logs,
artifacts, authorization, scheduling, reconciliation, and final state
transitions. Docker owns only the process isolation boundary for shell-only
automation jobs.

Host-local shell execution is not a production fallback. When the
server-managed runner is enabled with the Docker executor, Docker executor
configuration errors are startup-fatal. If the Docker daemon is unavailable,
MeshX must not fall back to host shell execution or advertise a server-managed
runner; MeshX disables the server-managed runner for that process and leaves
the rest of MeshX online.

## Background

The server-managed runner lives in the automation runtime package under
`internal/service/automation/runtime`. It claims ready jobs from MeshX
storage, accepts leases through `runtime.Worker`, writes bounded log chunks
through the automation blob store, and records final job state through
lease-checked worker transitions.

Host shell execution of supported `run` steps inside the MeshX service process
is only defensible for trusted development workspaces because automation code can
observe host tooling and filesystem state that belong to the MeshX server
process.

## Goals

- Execute each server-managed automation job inside a Docker container.
- Resolve `runs-on` to an operator-controlled image whitelist.
- Preserve existing MeshX queue, lease, wait, retry, log, timeout, and
  cancellation semantics.
- Fail closed when Docker is unavailable, the image map is invalid, or the
  Docker executor receives an unsupported execution environment; the server
  runner must not execute unadvertised runner targets.
- Keep the automation MVP shell-only: Docker executes `run` steps, while the
  server runner may handle explicitly supported AgentTeam-native `uses` steps
  outside the container. This does not add arbitrary GitHub Actions features.
- Keep unit and state-machine tests independent of Docker by injecting fake
  executors.

## Non-goals

- No GitHub Actions `container:`, `services:`, matrix, expressions, checkout,
  or arbitrary `uses` support.
- No automation-authored Docker image selection.
- No Docker socket mount inside job containers.
- No host shell fallback for production server-managed execution.
- No Run9 backend in this design; Run9 belongs behind a separate executor
  implementation.

## Ownership Boundary

`runtime.ServerRunner` owns automation control-plane behavior:

- bounded ready-workspace recovery scan;
- queue claiming with runner labels;
- internal lease owner derivation;
- `Worker.AcceptLeasedJob`, completion, terminal failure, and retry behavior;
- shell-only job shape validation;
- isolated host workdir creation and removal;
- bounded stdout/stderr log chunk persistence;
- run-cancellation observation through the automation store.

The executor boundary owns only process execution:

```go
type JobExecutor interface {
    ExecuteJob(ctx context.Context, req ExecuteJobRequest) (ExecuteJobResult, error)
}
```

The Docker executor receives a validated shell-only job, a host workdir, the
allowlisted environment, stdout/stderr writers, and a handler for supported
AgentTeam-native `uses` steps. It returns step results and an execution error when a
step exits nonzero, times out, a supported uses handler fails, or the container
cannot be created or cleaned up.

## Runtime Flow

```text
ServerRunner
  |
  | claim ready queue row matching advertised Docker labels
  v
Worker.AcceptLeasedJob
  |
  | validate shell-only job shape
  | create isolated host workdir
  | prepare allowlisted env and bounded log writers
  v
DockerExecutor
  |
  | resolve runs-on label -> whitelisted image
  | create and start one container for the job
  | bind-mount workdir at /workspace
  | exec each run step sequentially with /bin/sh -e -c
  | stream stdout/stderr back through MeshX log writers
  | kill/remove container on timeout or cancellation
  v
ServerRunner
  |
  | Worker.CompleteLeasedJob or terminal failure
  v
MeshX automation ledger
```

## Image Resolution

Docker execution supports exactly one normalized `runs-on` label per job. The
label must resolve to an image from `MESHX_AUTOMATION_DOCKER_IMAGES`:

```text
MESHX_AUTOMATION_DOCKER_IMAGES=ubuntu-latest=ubuntu:24.04,node-22=ghcr.io/pingcap/agentteam/automation-runner-node:22,python-3.12=ghcr.io/pingcap/agentteam/automation-runner-python:3.12
```

The server-managed runner advertises only labels that have image mappings, and
queue claiming remains label/group selective so unrelated runner targets stay
outside the Docker runner hot path. If a claimed job still reaches the Docker
executor with no matching image, more than one `runs-on` label, or a runner
group, it is terminally failed before container creation.

## Docker Security Defaults

Container creation must use these defaults:

- no privileged mode;
- no Docker socket mount;
- `no-new-privileges`;
- Linux capabilities dropped by default;
- configured CPU, memory, and pids limits;
- only the per-job host workdir bind-mounted to `/workspace` by default;
- optional operator-configured bind mounts may expose local source directories
  outside `/workspace`; automation definition source cannot declare host paths or override
  these mounts;
- `/workspace` as the working directory;
- default container user set to the MeshX server process UID:GID so the
  bind-mounted workspace is writable and host cleanup can remove job output;
- MeshX allowlisted environment only;
- configured Docker network mode, defaulting to `none`;
- timeout- and cancellation-driven kill plus forced remove.

The default job environment is:

- `CI=true`
- `MESHX_AUTOMATION_RUN_ID`
- `MESHX_AUTOMATION_JOB_RUN_ID`
- `MESHX_AUTOMATION_JOB_ID`
- `TMPDIR=/tmp`

## Configuration

Docker executor configuration:

```text
MESHX_AUTOMATION_EXECUTOR=docker
MESHX_AUTOMATION_DOCKER_IMAGES=...
MESHX_AUTOMATION_DOCKER_NETWORK=none
MESHX_AUTOMATION_DOCKER_CPUS=1
MESHX_AUTOMATION_DOCKER_MEMORY_BYTES=1073741824
MESHX_AUTOMATION_DOCKER_PIDS_LIMIT=256
MESHX_AUTOMATION_DOCKER_USER=<meshx-uid>:<meshx-gid>
MESHX_AUTOMATION_DOCKER_MOUNTS=
MESHX_AUTOMATION_WIKI_PUBLISH_MAX_PAGES=5000
MESHX_AUTOMATION_WIKI_PUBLISH_MAX_ENTRIES=10000
MESHX_AUTOMATION_WIKI_PUBLISH_MAX_PAGE_BYTES=2097152
```

If `MESHX_AUTOMATION_DOCKER_USER` is empty, MeshX uses its own process
UID:GID. Operators that override this with an image user must ensure that user
can write to the server-runner workdir bind mount and that the MeshX process can
delete generated files during cleanup.

`MESHX_AUTOMATION_DOCKER_MOUNTS` is an optional comma-separated list of
`host_path:container_path[:ro|rw]` entries. Sources must be absolute host paths
or use `~/`, container targets must be absolute paths outside `/workspace`, and
entries default to read-only. This is intended for trusted local source inputs
such as demo documentation directories; it is not an automation-authored volume
mechanism.

`agentteam/publish-wiki` is bounded by operator-configured publish limits. The
automation-authored `max_pages` value can reduce the page limit for a job, but it
cannot exceed `MESHX_AUTOMATION_WIKI_PUBLISH_MAX_PAGES`; filesystem entry and
single-page byte limits are operator-only controls. The server-side publisher
maps Markdown paths to AGS-writable wiki slugs and keeps the published slug set
prefix-free: the root `README.md` becomes `overview`, nested `README.md` files
become `overview` pages under their directory, invalid slug characters are
normalized, and overlong path segments are shortened with stable hash suffixes.
When a publish step supplies labels, stale pages with those labels are pruned
before new pages are written.

`agentteam/publish-tree` is bounded by operator-configured publish limits. The
automation-authored `max_files` value can reduce the file limit for a job, but it
cannot exceed `MESHX_AUTOMATION_TREE_PUBLISH_MAX_FILES`; filesystem entry,
single-file byte, and total byte limits are operator-only controls. The
server-side publisher preserves relative file paths, skips `.git` control
directories and symlinks, creates a new AGS Git tree from the staged files,
commits it, and advances the target repository default branch. It does not use
wiki page, slug, or label APIs. The publisher currently accepts UTF-8 regular
files and fails closed for other binary content so inline Git tree writes do not
corrupt bytes.

Existing server-runner controls remain:

```text
MESHX_AUTOMATION_SERVER_RUNNER_ENABLED
MESHX_AUTOMATION_SERVER_RUNNER_ID
MESHX_AUTOMATION_SERVER_RUNNER_CONCURRENCY
MESHX_AUTOMATION_SERVER_RUNNER_WORKDIR
MESHX_AUTOMATION_SERVER_RUNNER_POLL_INTERVAL
MESHX_AUTOMATION_SERVER_RUNNER_JOB_TIMEOUT
```

`MESHX_AUTOMATION_SERVER_RUNNER_LABELS` is not the Docker executor advertisement
source. Docker runner labels come from the Docker image map keys.

When `MESHX_AUTOMATION_SERVER_RUNNER_ENABLED=true` and
`MESHX_AUTOMATION_EXECUTOR=docker`, startup validates the image map, verifies
Docker connectivity, and inspects each configured image before advertising the
corresponding runner label. Invalid executor configuration remains
startup-fatal. Docker daemon unavailability disables the server-managed runner
for that process; it must not create a host-shell fallback or advertise labels
until Docker connectivity is restored and the runner is constructed on a
subsequent startup. Operators that require automation dispatch availability should
pre-provision the configured images and Docker access correctly or disable
server-managed automation execution explicitly.

Docker executor errors carry stable low-cardinality failure kinds for
operations and alerts:

- `docker_unavailable` for Docker daemon/API operations that fail before or
  during execution;
- `image_missing` when startup image inspection or container creation fails
  because the configured image is not present or cannot be resolved by Docker;
- `container_cleanup` when the job's primary execution path finished but
  container kill/remove cleanup failed.

The original Docker error remains in the error chain for logs and debugging.
Server-runner terminal failures persist the stable kind in `error_json` as
`failure_kind` when a classified executor, timeout, or log-write failure is
available.

## Validation Strategy

Required automated coverage:

- fake executor verifies scheduler, queue, worker, logs, retry, and final run
  state without requiring Docker;
- config tests cover Docker image-map parsing, defaults, and invalid mappings;
- server construction tests cover fail-closed Docker startup and test executor
  injection;
- Docker executor unit tests use a fake Docker client for image resolution,
  container defaults, step success, step failure, timeout, cancellation, and
  cleanup behavior;
- optional integration coverage runs only when an explicit Docker test
  environment flag is set.

## Trade-offs

Docker adds an operational dependency and image management requirement, but it
removes the most unsafe behavior of the host shell runner: silently executing
automation code inside the MeshX server process environment. Keeping Docker behind
a narrow executor interface preserves the existing MeshX control plane and
leaves room for a Run9 or worker-node backend without changing automation ledger
semantics.