# MeshX Automation Control Plane MVP Design

## Summary

MeshX Automation Hub is a GitHub Actions-compatible automation control plane with
a Docker-backed default shell-job execution path for AgentTeam. Automation definitions
live in agent-git-service repositories as `.agentteam/**/*.yml` or `.agentteam/**/*.yaml`;
Git provides versions, branches, history, permissions, and protocol
compatibility. MeshX owns the run ledger, scheduling state, runner assignment
queue, authorization decisions, status/audit timeline, the server-managed
Docker runner, and runner-compatible APIs for debug and future self-hosted
execution paths.

## Product boundary

### Source of truth

- Automation collection: one agent-git-service repository.
- Automation definition: an automation definition file under `.agentteam/` in that repository.
- Automation version: Git ref plus resolved commit SHA.
- Automation authoring: Automation Hub edits files in the repository and commits through agent-git-service.
- Automation execution: the MeshX server-managed runner executes supported
  shell-only jobs through Docker by default when MeshX Automation is enabled.
  External runner-compatible clients remain a debug and future self-hosted
  runner path.
- MeshX control plane: tracks runs, GitHub Actions job state, leases, events, logs/artifact references, and authorization.

### Goals

- Reuse agent-git-service automation metadata and repository storage instead of duplicating automation definitions in MeshX.
- Keep authoring syntax compatible with GitHub Actions-compatible automation syntax.
- Provide a AgentTeam-styled first-level visual editor for supported MVP fields (`on`, `jobs`, `needs`, `steps`, `uses`, `run`, and `with`) while preserving YAML round-trip.
- Start automation runs from an AGS repo automation definition file at a selected ref and record immutable run provenance in MeshX.
- Reuse the same run ledger and trigger provenance for automatic starts created
  from AGS-local repository updates such as merged local PRs landing on a
  branch.
- Execute supported shell-only jobs automatically through the Docker-backed
  MeshX server-managed runner, while keeping lease-protected runner-compatible
  APIs available for explicit external runner integrations.
- Maintain durable run/job/event ledger in MeshX for state management, scheduling, authorization, audit, and UI timeline.
- Keep the automation API surface clean: no deprecated automation-definition APIs and no transitional MeshX-owned definition/version model.

### Non-goals

- MeshX does not store automation definition/version rows as source of truth.
- MeshX does not provide a general automation-authored container runner.
- MeshX does not execute unsupported automation features such as
  non-`agentteam/checkout` `uses`, expressions outside the supported `env` subset,
  container/service jobs, matrix jobs, `actions/checkout`, external actions,
  implicit checkout, or arbitrary checkout targets in the
  server-managed runner.
- The normal product path does not require a Desktop runner, manual
  `actions/runner` process, browser token, repository owner/name pair, or
  runner directory from the user.
- MeshX does not implement compatibility with old MeshX automation definition/version APIs.
- No Redis/Kafka/Temporal dependency for MVP queue state.

## Architecture

### Components

- `agent-git-service`: repository, Git ref, automation definition file, automation metadata, commit, branch, and permission source.
- `meshx-api`: Automation Hub API, run creation API, status API, runner auth API, and runner callback API.
- `meshx-source-adapter`: reads/writes `.agentteam/**/*.yml` or `.agentteam/**/*.yaml` through agent-git-service and resolves refs to commit SHAs.
- `meshx-scheduler`: expands an automation definition file at a commit into job-run ledger rows and enqueues runner-claimable jobs when dependencies are satisfied.
- `meshx-runner-dispatch`: exposes runner-compatible long-poll acquire and callback endpoints backed by TiDB leases.
- `meshx-server-runner`: internal background runner that claims ready jobs from
  MeshX storage, delegates supported shell steps to the configured automation
  executor, and writes job status plus logs through the same worker and
  blob-store transitions used by runner-compatible callbacks.
- `meshx-reconciler`: repairs expired leases, stuck runs, and inconsistent queue/ledger state through conditional updates.
- `automation hub web`: lists AGS-backed automation collections, edits automation definition files visually, and shows MeshX run timeline.

### High-level flow

1. User opens Automation Hub and selects an AGS repository as the automation collection.
2. MeshX lists `.agentteam/**/*.yml` and `.agentteam/**/*.yaml` through agent-git-service.
3. User creates or edits an automation definition file; MeshX validates GitHub Actions-compatible syntax and commits the file through AGS.
4. User starts an automation run from repo, automation path, and ref.
5. MeshX resolves the ref to commit SHA, reads automation YAML at that commit, validates it, and stores run provenance in `automation_runs`.
6. MeshX expands `jobs` and `needs` into `automation_job_runs` and `automation_jobs` rows.
7. The server-managed runner is signaled for the run workspace, claims ready
   shell jobs with DB leases, executes supported `run` steps through the Docker
   executor with isolated working directories bind-mounted into job containers,
   and writes stdout/stderr to automation log blobs.
8. External runner-compatible clients may also claim jobs through the
   runner-compatible API when explicitly configured for debug or future
   self-hosted runner paths.
9. MeshX records runner results in lease-checked transactions, appends
   `automation_events`, and schedules downstream jobs.
10. Run creation and terminal run transitions upsert a MeshX-owned desired
    repository commit-status row in the same MeshX database transaction as the
    run ledger change. A background worker leases those rows and writes AGS
    commit statuses asynchronously, so AGS latency or failures do not slow the
    runner hot path or corrupt the automation ledger.
11. Reconciler recovers expired leases and marks exhausted failures in the run ledger.

## GitHub Actions-compatible authoring

MeshX accepts GitHub Actions automation YAML shape as the canonical authoring format. The visual editor may normalize flexible YAML forms for UI editing, but saved files remain GitHub Actions-compatible.

Supported MVP authoring fields:

- top-level: `name`, `run-name`, `on`, `env`, `permissions`, `jobs`.
- job-level: `name`, `needs`, `runs-on`, `env`, `permissions`, `steps`.
- step-level: `id`, `name`, `uses`, `run`, `with`, `env`.
- expressions: automation/job/step `env` values support only a safe full-value
  subset: literal strings, `${{ secrets.NAME }}`, and `${{ inputs.NAME }}`.
  Concatenation, operators, functions, and all other contexts are rejected.
  Expressions remain unsupported outside `env`; `run` script text is stored and
  executed as written.

Validation rules:

- Automation definition file must parse as YAML and contain `jobs`.
- Job ids and explicit step ids must be stable identifiers.
- `needs` must reference existing jobs and the job DAG must be acyclic.
- Step must define `uses` or `run` according to GitHub Actions syntax.
- Job-level and step-level `if` conditions are not supported in the current
  execution MVP and must be rejected instead of silently ignored.
- Automation/job/step `env` names must be stable shell-style identifiers. Values
  must be literal strings or full `${{ secrets.NAME }}` / `${{ inputs.NAME }}`
  expressions; partial interpolation such as `Bearer ${{ secrets.TOKEN }}` is
  rejected.
- Automation/job `defaults` are not supported in the current execution MVP and
  must be rejected instead of silently ignored.
- Automation job `outputs` definitions are not supported in the current
  execution MVP because the official-runner job payload does not materialize
  job-output evaluation.
- Automation expressions outside the supported `env` subset must be rejected
  instead of dispatching a job that MeshX cannot evaluate or that the official
  runner would receive as unresolved shell script text or display metadata.
- Flexible GitHub Actions shapes such as `on` string/list/map and `needs` string/list are accepted and normalized for internal validation.

## Data model

MeshX stores runtime/control-plane state only.

### `automation_runs`

One row per run.

- `id`
- `workspace_id`
- `idempotency_key`
- `source_kind`: `agent_git_service_repo`
- `repository_owner`
- `repository_name`
- `repository_full_name`
- `definition_path`
- `git_ref`
- `commit_sha`
- `automation_id`
- `name`
- `run_name`
- `status`: `queued`, `running`, `waiting`, `failed`, `succeeded`, `canceled`, `blocked`
- `input_json`
- `output_json`
- `error_json`
- `trigger_kind`
- `trigger_ref`
- `target_kind`
- `target_ref`
- `owner_principal_id`
- `created_by_principal_id`
- `current_job_id`
- `summary`
- `metadata`
- `started_at`
- `completed_at`
- `created_at`
- `updated_at`

`metadata` may carry the short-term V1 `trigger_context` envelope until the run
ledger grows dedicated columns. The read contract still exposes first-class
structured `error` and `trigger_context` fields on `AutomationRun`, and runner
acquire payloads expose `trigger_context_json` alongside `input_json` so
trigger provenance does not get mixed into job step definitions.

### `automation_repo_status_outbox`

One row per `(workspace_id, repository_owner, repository_name, commit_sha,
context)` desired AGS commit status. This table is MeshX-owned and stores the
latest desired repository status for a commit/status context, not an append-only
history. The row still records the owning `run_id` and `run_created_at` for the
currently desired run. The commit status context is stable for the automation
path, such as `meshx/automation/.agentteam/build.yml`; it does not include
the run id, so branch protection can configure required checks once per
automation. If the context would exceed the storage/API limit, MeshX keeps the
stable prefix and appends a hash suffix.

Run creation writes desired state `pending`. Terminal run transitions map to
AGS status states as follows:

| MeshX run status | AGS commit status |
| --- | --- |
| `succeeded` | `success` |
| `failed` | `failure` |
| `canceled` | `error` |

Each accepted desired-state update increments `sequence`, resets delivery
attempts, and clears any expired lease. An update from the same run is always
accepted. An update from a different run is accepted only when that run was
created after the row's currently desired run, which prevents a delayed terminal
transition from an older run from replacing a newer retry on the same commit
status context. A worker that leased an older sequence may still write the older
status to AGS, but it cannot mark the row delivered; the newer sequence remains
pending and is delivered in a later worker pass. The worker uses bounded
retries with `next_attempt_at`, `attempts`, and `max_attempts`.

- `id`
- `workspace_id`
- `run_id`
- `run_created_at`
- `repository_owner`
- `repository_name`
- `commit_sha`
- `context`
- `state`: `pending`, `success`, `failure`, `error`
- `description`
- `sequence`
- `status`: `pending`, `delivered`, `failed`
- `attempts`
- `max_attempts`
- `next_attempt_at`
- `lease_owner`
- `lease_expires_at`
- `last_error`
- `delivered_at`
- `created_at`
- `updated_at`

### `automation_run_slots`

One row per active automation run capacity slot. This table is the DB-enforced
gate for `max_active_runs_per_workspace`: starting a run reserves one
workspace-local slot in the same transaction that creates the run ledger rows,
and terminal run transitions release the slot.

- `workspace_id`
- `slot_index`
- `run_id`
- `created_at`
- `updated_at`

### `automation_job_runs`

One row per GitHub Actions job execution unit tracked by MeshX. Steps remain runner payload/log detail, not MeshX scheduling rows.

- `id`
- `workspace_id`
- `run_id`
- `job_id`: GitHub Actions job id
- `job_type`: `github_actions_job`
- `status`: `pending`, `queued`, `leased`, `running`, `waiting`, `blocked`, `failed`, `succeeded`, `skipped`, `canceled`
- `attempt`
- `max_attempts`
- `runs_on_json`
- `input_json`
- `output_json`
- `error_json`
- `depends_on_json`
- `lease_owner`
- `lease_until`
- `scheduled_at`
- `started_at`
- `completed_at`
- `created_at`
- `updated_at`

### `automation_jobs`

TiDB-backed runner assignment queue.

- `id`
- `workspace_id`
- `run_id`
- `job_run_id`
- `queue_name`
- `status`: `ready`, `leased`, `done`, `failed`, `canceled`
- `priority`
- `scheduled_at`
- `not_before`
- `lease_owner`: opaque derived runner lease identity, generated from runner id and session id
- `lease_until`
- `attempt`
- `max_attempts`
- `idempotency_key`
- `runner_group`
- `runner_group_key`
- `labels_json`
- `label_key`
- `label_key_hash`
- `label_count`
- `shard_bucket`
- `created_at`
- `updated_at`

### `automation_events`

Append-only audit timeline.

- `id`
- `workspace_id`
- `run_id`
- `job_run_id`
- `event_type`
- `actor_principal_id`
- `resource_type`
- `resource_id`
- `summary`
- `metadata_json`
- `created_at`

### `automation_waits`

Durable AgentTeam-native wait subscriptions for jobs implemented by `agentteam/agent-task` or `agentteam/wait-event`. Active waits do not hold runner leases or `automation_jobs` rows.

- `id`
- `workspace_id`
- `run_id`
- `job_run_id`
- `wait_kind`: `agent_task`, `wait_event`
- `status`: `active`, `resolved`, `canceled`
- `active_key`: nullable unique key set only while active; resolved/canceled rows set it to NULL so TiDB/GORM can enforce active uniqueness without partial unique indexes
- `signal_key`: compact deterministic key for the workspace-scoped wait kind, resume event type, resource type, and resource id; used by service event delivery to avoid broad active-wait scans
- `resource_type`
- `resource_id`
- `resume_event_type`
- `input_json`
- `result_json`
- `created_by`
- `resolved_by`
- `created_at`
- `updated_at`
- `resolved_at`
- `canceled_at`

### `automation_log_chunks`

Metadata for automation log chunks stored outside the hot automation ledger. The
payload bytes are stored through the automation blob store; the database row is
only the bounded lookup and integrity record for one job-run stream sequence.

- `id`
- `workspace_id`
- `run_id`
- `job_run_id`
- `stream`
- `sequence`
- `object_key`
- `size_bytes`
- `sha256`
- `content_type`
- `created_at`

`(job_run_id, stream, sequence)` is unique. Retrying the same chunk with the
same object key, size, hash, and content type is idempotent. Reusing the same
sequence for different bytes is rejected.

### `automation_artifacts`

Metadata for automation artifacts stored outside the hot automation ledger. Artifact
payload bytes are written through the automation blob store; the database row does
not store artifact content. `object_url` is retained for schema compatibility;
blob-store-backed artifact callbacks use `object_key` as the active location.

- `id`
- `workspace_id`
- `run_id`
- `job_run_id`
- `name`
- `object_key`
- `object_url`
- `size_bytes`
- `sha256`
- `content_type`
- `expires_at`
- `created_at`
- `updated_at`

## APIs

All URLs in this section are canonical AgentTeam public/control-plane URLs. The
`:workspaceId` path segment is the routing authority and must agree with any
workspace headers. Embedded MeshX may receive the same suffix beginning at
`/api/v1/automations/...` only as a private in-process target after AgentTeam
authorizes the canonical request and strips the workspace prefix; it is not a
public URL.

### Automation definition source APIs

These APIs are AGS-backed and do not persist automation definitions in MeshX.

- `GET /api/v1/workspaces/:workspaceId/automations/repositories`
- `GET /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions?ref={ref}`
- `GET /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}?ref={ref}`
- `PUT /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}`
- `DELETE /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}`
- `POST /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}/validate`

`GET /api/v1/workspaces/:workspaceId/automations/repositories` returns the repository array shape used by
the Automation Hub UI. The request accepts normalized `page` and `per_page`
parameters with the automation API maximum of 100 items per page. The merged list
combines AGS repositories with active canonical project and knowledge-base
repositories, deduplicated by full repository name and ordered
deterministically. MeshX does not perform an unbounded workspace scan to prove an
exact global ordering across AGS, project, and knowledge-base sources; each
source read is capped to the requested page window plus one extra page, with a
fixed product maximum of 1000 candidates per source, to tolerate local filtering
and deduplication without allowing arbitrary deep scans.

`PUT /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}` saves an
automation definition source file through AGS. The request body contains raw source text, not
base64:

```json
{
  "content": "name: Build\non:\n  manual: {}\njobs:\n  build:\n    steps:\n      - run: echo hi\n",
  "message": "Save automation",
  "branch": "main",
  "ref": "main",
  "sha": "current-blob-sha"
}
```

`content` is required. `message`, `branch`, `ref`, and `sha` are optional.
`ref` is accepted only as a branch-name alias; tag refs and commit-SHA refs are
not valid write targets. If both `branch` and `ref` are supplied, they must
refer to the same branch. MeshX validates that `path` is under
`.agentteam/`, has a `.yml` or `.yaml` suffix, and does not contain path
traversal before calling AGS. MeshX enforces the configured automation definition source size
limit, decodes the automation with the standard automation parser, and runs the MVP
automation DSL validator before writing. Unsupported GitHub Actions fields remain
fail-closed at save time.

The MeshX source adapter base64-encodes `content` for AGS and preserves AGS
contents API update semantics: creating a new automation definition file does not require
`sha`; updating an existing automation definition file requires the current blob `sha`; a
wrong `sha` returns a conflict response and must not overwrite the file. A
successful response is `201 Created` for creates and `200 OK` for updates:

```json
{
  "content": {
    "repository_owner": "pingcap",
    "repository_name": "agentteam",
    "path": ".agentteam/build.yml",
    "name": "build.yml",
    "sha": "new-blob-sha",
    "size": 92,
    "ref": "main",
    "encoding": "utf-8"
  },
  "commit": {
    "sha": "commit-sha",
    "message": "Save automation"
  }
}
```

`DELETE /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/definitions/{path}` removes
an existing automation definition source file through AGS. The request body follows contents
API precondition semantics:

```json
{
  "message": "Delete automation",
  "branch": "main",
  "ref": "main",
  "sha": "current-blob-sha"
}
```

`sha` is required so deleting a stale editor snapshot cannot remove a file that
changed concurrently. `message`, `branch`, and `ref` are optional; `ref` is
accepted only as a branch-name alias with the same restrictions as automation
source writes. MeshX validates the automation path before calling AGS. A
successful delete returns `204 No Content`; wrong or stale `sha` values return a
conflict response.

### Automation secret APIs

Automation secrets are MeshX-owned, repo-scoped values. They are keyed by
`workspace_id + repository_owner + repository_name + name` and stored in
`automation_secrets` as encrypted ciphertext. They do not reuse AGS plaintext
secret storage.

Persistent secret sealing is owned by `internal/core/secrets`. The storage
contract is AES-256-GCM with no additional authenticated data, encoded as
unpadded standard base64 over `nonce || ciphertext`. Readers must remain
compatible with existing values; changing the package owner does not rewrite
stored ciphertext.

- `GET /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/secrets`
- `PUT /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/secrets/{name}`
- `DELETE /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/secrets/{name}`

`GET` accepts the standard `page` and `per_page` query parameters, caps
`per_page` at 100, and returns metadata only, including name, repository scope,
created/updated timestamps, and created/updated principal ids. The metadata
list query must not load plaintext or ciphertext. `PUT` accepts plaintext in
the request body as `value`, encrypts it with the MeshX automation credentials
master key, stores only ciphertext, and returns metadata only. `DELETE` removes
one secret row without reading plaintext. Secret names must match the safe
expression identifier subset `[A-Za-z_][A-Za-z0-9_]*`.

### Run APIs

- `POST /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/runs`
- `POST /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/events/push`
- `GET /api/v1/workspaces/:workspaceId/automations/runs`
- `GET /api/v1/workspaces/:workspaceId/automations/dashboard`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/overview`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/jobs`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/job-runs` (alias using explicit job-run terminology)
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/events`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/logs?job_run_id={job_run_id}&stream={stream}`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/jobs/{job_run_id}/logs/{stream}/{sequence}`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/artifacts?job_run_id={job_run_id}`
- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/artifacts/{artifact_id}`
- `POST /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/cancel`
- `POST /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/retry`

`GET /api/v1/workspaces/:workspaceId/automations/runs` accepts normalized `page`, `per_page`,
`status`, `repository`, and `q` parameters. `per_page` uses the automation API
maximum of 100 items per page. Text search with `q` is an interactive recency
search over the matching workspace/status/repository run set: MeshX first caps
the candidate window to the most recent 1000 runs by `updated_at, id`, then
applies case-insensitive matching across run IDs, names, summaries, repository
names, automation paths, automation IDs, and current job IDs. A dedicated search
index is required before `q` can promise arbitrary historical deep search. The
`q` value is treated as literal text; SQL wildcard characters such as `%` and
`_` do not broaden the match.

Run creation body includes `definition_path`, `ref`, `input_json` (or the `inputs`
alias), optional `idempotency_key`, and trigger/target metadata. MeshX resolves
`ref` to `commit_sha` and reads `definition_path` from AGS before creating ledger
rows. Manual starts require the resolved automation to declare a
`manual` trigger; retries that preserve manual-start semantics use
the same repository automation creation path and therefore require
`manual` as well. Schedule-triggered starts use the same endpoint
with `trigger_kind = schedule`, but their capability checks are distinct and do
not depend on `manual`. There is no generic public create/patch run
API that can bypass AGS provenance. When an
`idempotency_key` is provided, MeshX returns the already-created run for the
same workspace key instead of creating a duplicate run only when the replaying
caller and immutable start fields match the original request; mismatched
replays fail with conflict rather than exposing an unrelated run.

`POST /api/v1/workspaces/:workspaceId/automations/repositories/{owner}/{repo}/events/push` is an
internal normalized repository-update ingestion endpoint. The primary caller is
the AgentTeam-owned AGS local-repository bridge after a local PR merge or branch
update resolves to an immutable repository commit; this path is not a public
external webhook contract. Any upstream delivery verification, webhook receipt
dedupe, or external subscription routing stays with the bridge that produces
the normalized repository update. The request body carries a stable `event_id`
for that repository update event; `delivery_key` remains accepted as a legacy
alias. It also carries the updated `ref`, `before`, `after`, optional
`changed_paths`, optional `actor_ref` / `actor_display`, and optional
normalized `event_payload`. Empty or all-zero `after` values represent ref
deletion and return `runs: []` without reading automation definition files. For
non-deletion repository updates, MeshX lists `.agentteam/**/*.yml` and `.agentteam/**/*.yaml` at the
immutable `after` SHA, resolves each matching automation definition file at the same SHA,
and creates one run per automation whose `on` value matches the repository
update.

Repository-update matching reuses the MVP GitHub Actions `on.push` forms
`on: push`, `on: [push]`, and `on: { push: ... }` because AGS-local merges and
branch updates materialize as ref updates. Map-form `push` supports
`branches`, `branches-ignore`, `paths`, and `paths-ignore` with exact matches
and the simple `*`, `**`, and `?` glob operators. Branch filters evaluate
against branch names normalized from refs such as `refs/heads/main` to `main`.
Path filters evaluate against repository-relative changed paths. When
`changed_paths` is omitted, MeshX treats the path set as unknown and only
matches automations that have no `paths` or `paths-ignore` filter, avoiding
false-positive execution of path-filtered automations.

Each automatically triggered run created from this endpoint uses `GitRef` and
`CommitSHA` from the immutable `after` commit, `TriggerKind=push`,
`TriggerRef=event_id`, `TargetKind=repository_ref`, and
`TargetRef={owner}/{repo}@{ref}#{after}`. The run owner and creator principal
are `agentteam-automation-eventer`. The idempotency key is derived from the workspace,
repository, `event_id`, `ref`, `after`, and automation path, so AGS bridge
retries, worker retries, and concurrent repository-update deliveries return the
already-created run for the same immutable request and fail with conflict when
the same key is replayed with different immutable run fields. Repository-update
ingestion does not choose a Docker image or runner target; created jobs use the
same `runs-on` label matching and server-managed runner execution path as
manually started or retried runs.

`GET /api/v1/workspaces/:workspaceId/automations/dashboard` is workspace-scoped and returns the Automation
Dashboard first-paint aggregate. It includes the observed timestamp, health
status, DB-derived queue/run/wait/runner metrics, and bounded attention items
that link to the owning automation section. Dashboard metrics cover fixed counters
such as active runs, queue demand, pending approvals, active waits, registered
runners, active sessions, expired leases, expired sessions, active run slots,
leaked active run slots, log/artifact object counts, expired artifacts, and the
process-local reconciler heartbeat/error counters when the reconciler role is
enabled. The dashboard contract must not add high-cardinality dimensions such
as repository path, run id, job id, artifact id, object key, or wait id, and
must not run unbounded byte-total aggregates on the first-paint path; detailed
run records and per-job byte summaries remain under the run list and run-detail
APIs.

`GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/overview` is the run-detail first-paint
aggregate. It returns the authorized run, job runs, queue rows for that run,
runner sessions referenced by active lease owners, waits, the first bounded
event/log/artifact metadata pages, per-job log/artifact count and byte
summaries, and workspace-scoped dashboard metrics. It must not inline log or
artifact payload bytes; payload content remains behind explicit log chunk and
artifact content APIs.

### Runner-compatible APIs

- `POST /api/v1/workspaces/:workspaceId/automations/runners/register`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/sessions`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/sessions/{session_id}/heartbeat`
- `GET /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs:acquire`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/accepted`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/completed`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/failed`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/renew`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/logs`
- `POST /api/v1/workspaces/:workspaceId/automations/runners/{runner_id}/jobs/{job_id}/artifacts`

### Server-managed runner

When MeshX Automation is enabled, MeshX starts a server-managed automation runner
by default. The runner is an internal background component, not a user-visible
runner registration API. It uses the same TiDB queue leases, `automationruntime`
worker state transitions, automation log blob store, and reconciler recovery path
as the runner-compatible API.

The default runner supports only the shell-job subset:

- GitHub Actions jobs with `run` steps.
- Explicit `uses: agentteam/checkout` steps that materialize the current automation
  run's immutable AGS repository snapshot
  (`automation_runs.repository_full_name` at `automation_runs.commit_sha`) into the
  host job work directory. Docker-backed run steps then observe the checked-out
  files through the `/workspace` bind mount.
- Explicit `uses: agentteam/publish-wiki` steps that publish Markdown files staged
  under the job work directory into an AGS wiki through MeshX server-side
  handlers. The automation definition source supplies only a relative `source_path`; it does
  not get to declare host paths or container networking. Page, filesystem
  entry, and single-page byte limits are bounded by operator configuration;
  automation-authored `max_pages` can only reduce the page limit for that job.
  Markdown paths are normalized to AGS-writable, prefix-free wiki slugs before
  writing. The root `README.md` becomes `overview`; nested `README.md` files
  become directory `overview` pages instead of occupying the directory slug. A
  labeled publish prunes stale pages carrying the same labels before writing the
  current page set.
- Explicit `uses: agentteam/publish-tree` steps that publish files staged under the
  job work directory into an AGS repository default branch through MeshX
  server-side handlers. The publisher preserves relative file paths, skips
  `.git` control directories and symlinks, writes a new AGS Git tree and commit,
  accepts UTF-8 regular files, and does not use wiki page, slug, or label APIs.
- Docker-backed step execution through `/bin/sh -e -c` inside the image selected
  by the job's normalized `runs-on` label.
- Runner label matching against `MESHX_AUTOMATION_DOCKER_IMAGES` keys. The
  production Docker executor does not advertise
  `MESHX_AUTOMATION_SERVER_RUNNER_LABELS`.
- Jobs containing `agentteam/checkout`, `agentteam/publish-wiki`, or
  `agentteam/publish-tree` are assigned to an internal server-managed runner group
  so external runner-compatible clients cannot claim them before the
  server-side step runs.
- One isolated working directory per job:
  `<runner-workdir>/<workspace-id>/<run-id>/<job-run-id>/`, bind-mounted as
  `/workspace` inside the container.
- A default container user equal to the MeshX server process UID:GID, so
  `/workspace` is writable and generated files can be removed by host cleanup.
  Operators that override `MESHX_AUTOMATION_DOCKER_USER` must ensure the explicit
  image user has compatible bind-mount write and cleanup permissions.
- A job-level timeout, defaulting to 30 minutes.
- Stdout and stderr stored as bounded automation log chunks.
- The safe automation/job/step `env` subset. Literal values and `${{ inputs.NAME
  }}` values are injected directly; `${{ secrets.NAME }}` values are resolved
  from repo-scoped automation secrets at job execution time and are masked in
  stdout/stderr before log chunks are stored.

The server-managed runner fails closed for unsupported execution features:
`uses` steps other than the supported AgentTeam-native steps, expressions outside
the supported `env` subset, job `outputs`, job `defaults`, container jobs,
service containers, strategy/matrix jobs, job concurrency, and empty `run`
steps. AgentTeam-native waits such as
`agentteam/human-approval` has been removed; `agentteam/agent-task` and `agentteam/wait-event` do not enter the
runner queue; they stay in the wait runtime and schedule downstream jobs only
after the wait resolves.

Secret resolution is repeated when the server-managed runner accepts a job. If a
referenced secret has been deleted or cannot be decrypted, the job fails closed
without executing shell steps. Resolved secret values are held only in runner
process memory for the job environment and log masker; they are not written to
run, job, event, queue, log metadata, or artifact metadata.

No user-facing API is added to configure the server-managed runner. Operators
configure it through MeshX automation configuration:

- `MESHX_AUTOMATION_EXECUTOR`
- `MESHX_AUTOMATION_DOCKER_IMAGES`
- `MESHX_AUTOMATION_DOCKER_NETWORK`
- `MESHX_AUTOMATION_DOCKER_CPUS`
- `MESHX_AUTOMATION_DOCKER_MEMORY_BYTES`
- `MESHX_AUTOMATION_DOCKER_PIDS_LIMIT`
- `MESHX_AUTOMATION_DOCKER_USER`
- `MESHX_AUTOMATION_DOCKER_MOUNTS`
- `MESHX_AUTOMATION_WIKI_PUBLISH_MAX_PAGES`
- `MESHX_AUTOMATION_WIKI_PUBLISH_MAX_ENTRIES`
- `MESHX_AUTOMATION_WIKI_PUBLISH_MAX_PAGE_BYTES`
- `MESHX_AUTOMATION_CHECKOUT_MAX_BYTES`
- `MESHX_AUTOMATION_CHECKOUT_MAX_FILES`
- `MESHX_AUTOMATION_CHECKOUT_TIMEOUT`
- `MESHX_AUTOMATION_SERVER_RUNNER_ENABLED`
- `MESHX_AUTOMATION_SERVER_RUNNER_ID`
- `MESHX_AUTOMATION_SERVER_RUNNER_CONCURRENCY`
- `MESHX_AUTOMATION_SERVER_RUNNER_WORKDIR`
- `MESHX_AUTOMATION_SERVER_RUNNER_POLL_INTERVAL`
- `MESHX_AUTOMATION_SERVER_RUNNER_JOB_TIMEOUT`
- `MESHX_AUTOMATION_CREDENTIALS_MASTER_KEY` for automation secret writes and
  server-managed runner secret resolution. It must be a base64-encoded 32-byte
  AES-GCM key.

When the server-managed runner is enabled with the Docker executor, startup
fails fast if the Docker image map is empty or invalid, any configured image
cannot be inspected by Docker, or the server-runner working directory cannot be
created. If the Docker daemon is unreachable, MeshX must not fall back to host
shell execution or advertise server-runner labels; MeshX leaves the rest of the
server online with the server-managed runner disabled for that process.
`AGENTTEAM_DEV_FEATURES=MESHX ./dev --server` must be capable of starting and completing a
shell-only automation without a separate Desktop runner or `actions/runner`
process when the local Docker daemon is available and the configured runner
images are present.

### Wait APIs

- `GET /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/waits`
- `POST /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/waits/{wait_id}/resume` (generic resume endpoint)
- `POST /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/waits/{wait_id}/complete-task` for `agentteam/agent-task` waits
- `POST /api/v1/workspaces/:workspaceId/automations/runs/{run_id}/waits/{wait_id}/signal-event` for `agentteam/wait-event` waits
- `POST /api/v1/workspaces/:workspaceId/automations/waits:deliver-event` for service actors delivering `agentteam/agent-task` and `agentteam/wait-event` signals by workspace-scoped resource identity

## Authorization boundaries

MeshX automation APIs must authorize against the concrete workspace-scoped
resource before reading or mutating automation control-plane state. The
authorization boundary is an automation-domain port, not automation table fields or
API DTO state. Checks must stay bounded by one of these request scopes:

- workspace plus repository owner/name for automation definition source reads, source writes,
  validation, automation secret metadata reads, automation secret management, and
  run start;
- workspace plus run id for run detail, job list, event list, wait list, log
  and artifact metadata lists, retry, cancel, and dashboard metrics;
- workspace plus run id plus job run/stream/sequence or artifact id for log
  and artifact content reads;
- workspace plus run id and wait id for wait resume actions;
- workspace plus wait kind, resume event type, resource type, and resource id
  for service event delivery that resolves active waits by signal identity;
- workspace plus runner id for runner registration and management.

Automation definition source permissions continue to be enforced by agent-git-service for
repository contents. MeshX must still verify that the caller is authorized for
the selected workspace before proxying to AGS, and source writes and run starts
must require explicit automation write/run capabilities. When MeshX calls AGS
through the in-process source adapter, it forwards the configured trusted
identity header instead of assuming a hard-coded `Authorization` header, and
automation definition source responses are capped by the configured source-size limit before
base64 decoding.

External Git/AGS data-plane credentials for repository access follow
[Repository Access Tokens](./repository-access-tokens.md).

Claim-backed automation definition source reads require `automation:repo:read`. Automation definition source
writes require `automation:repo:write`. Starting runs requires
`automation:run:start`. `automation:*` permits all automation-domain actions, but read
and run-start permissions do not imply source write permission.

Normalized repository-update ingestion requires the MeshX
`AutomationRoleEventer` server role. In claim-backed authorization mode, it also
requires `automation:eventer` and a service actor capability; `automation:run:start`
does not authorize event ingestion.

Automation secret permissions are separate from run permissions:

- `automation:secret:read` lists metadata only.
- `automation:secret:manage` creates, updates, and deletes secret values.
- `automation:secret:use` is required when an automation start references secrets.

`automation:run:start` must not imply `automation:secret:use`; otherwise any actor
who can start automations could exfiltrate repo secrets through a shell step.
Run start validates secret references after resolving and validating the
automation definition file: if any automation/job/step `env` value references
`${{ secrets.NAME }}`, MeshX first requires `automation:secret:use` for that
repository, then checks the explicit deduplicated name set against
`automation_secrets`. Missing secrets fail the start before `automation_runs`,
`automation_job_runs`, queue rows, waits, or events are created. This validation
does not decrypt secrets and does not write plaintext or ciphertext into run,
job, event, log, artifact, or queue metadata. `${{ inputs.NAME }}` env
references do not require secret-use permission.

Human approval waits have been removed from the runtime; `agentteam/agent-task` and
`agentteam/wait-event` use the generic wait-resume permission boundary, not the
run-read boundary. Resuming a wait requires `automation:wait:resume` and a
concrete wait assignment to the authenticated principal. `automation:*` acts as
the workspace-wide automation administrator permission and may resume assigned
or unassigned waits.

Claim-backed automation authorization may be enabled only when MeshX receives JWT
claims from an upstream component that already verifies signature, issuer, and
audience and passes them through an explicitly configured non-`Authorization`
trusted identity header. If no explicit automation authorizer is wired and trusted
upstream claims are not enabled, automation control-plane actions fail closed.
Trusted claims mode must not fall back to mock or client-supplied bearer tokens
when the configured trusted identity header is absent.

Runner execution APIs use runner registration/session tokens and active session
leases. Ordinary user JWTs must not acquire jobs, renew leases, or submit
runner callbacks. Human or admin registration of a runner is a management action
on the workspace and must not grant execution authority until the runner
exchanges its registration token for a session token.

The server-managed runner does not use public runner registration/session
tokens. It derives an internal lease owner from its configured runner id and
uses that lease owner only inside MeshX background execution. This internal
identity must not be exposed as a public runner credential.

Server-managed automation execution runs supported shell jobs in Docker
containers selected from an operator-controlled image map; it must not fall back
to executing automation code on the MeshX server host. MVP deployments must still
treat Automation as a controlled capability because the Docker executor is a local
execution substrate with operator-managed images, mounts, resource limits, and
network policy. The server-managed runner must not inject AgentTeam server secrets
into job environments. Its job environment is allowlisted, each job uses an
isolated work directory mounted at `/workspace`, log and artifact limits remain
enforced, timeouts are mandatory, checkout is limited to explicit
`agentteam/checkout` of the current automation run's immutable AGS snapshot,
`actions/checkout` and external actions are unsupported, and unsupported
automation features fail closed before container creation.

AgentTeam service-event delivery that resumes automation waits acts as a service
actor. Eventer-style wait resume APIs must require service actor capability
rather than accepting an ordinary user JWT as proof of authority. `agentteam/agent-task`
waits must carry a concrete task identity (`task_id` or `resource_id`) that
AgentTeam task completion events can deliver. Task completion delivery into MeshX must
use a bounded retryable handoff rather than relying on a single best-effort HTTP
request. While AgentTeam uses the in-process task service, the handoff row is written
in the same database transaction that marks the task done; a remote task service
must provide an equivalent transactional outbox before it can drive MeshX wait
delivery. Reconciler and other internal automation workers do not use public user
JWTs for their runtime authority.

## TiDB queue semantics

### Enqueue

Scheduler inserts queue rows with idempotency key `run_id:job_id`. Queue rows contain references and runner labels only, not full automation payloads.

### Acquire

Runner-compatible clients claim ready jobs with conditional updates by
workspace, `queue_name`, shard bucket, normalized runner group, normalized label
subset key, `status='ready'`, `scheduled_at <= now`, and lease owner. The
workspace-leading claim index orders ready work by `scheduled_at`, then
`priority`, then row id; runner label subset enumeration is capped by the
configured runner-label limit. Runner group and label matching use the active
session's registered runner capabilities, not acquire-query overrides. The
normalized runner group and label key are rejected before persistence when they
exceed their queue-column budgets, so invalid matching inputs fail at validation
rather than during queue insert or runner registration. The
stored `lease_owner` is an opaque derived identity, not the raw runner
`session_id`; callbacks must prove the active session token plus runner/session
path/body ids before using that lease owner for conditional updates. The claim
response includes enough source provenance for the runner to fetch or receive
the automation job payload.

The server-managed runner uses the same claim semantics but does not discover
work by scanning all automation jobs on every request. Run creation and scheduled
automation starts signal the in-memory runner queue for the affected workspace.
Runner-compatible long-poll acquire first attempts a claim, then waits on a
workspace-scoped wake channel and a capped exponential backoff before retrying
the database claim. This keeps idle external runners from polling TiDB at a
fixed subsecond interval while preserving prompt wake on enqueue/resume signals.
After server restart, and periodically as a recovery path, the runner performs a
bounded ready-workspace lookup:

```sql
SELECT workspace_id
FROM automation_jobs
WHERE status = 'ready'
  AND queue_name = ?
  AND scheduled_at <= ?
GROUP BY workspace_id
ORDER BY MIN(scheduled_at) ASC, workspace_id ASC
LIMIT ? -- capped workspace rows
```

This lookup is capped by configuration and is workspace-fair: one workspace
with many old ready rows cannot consume the whole recovery budget before other
ready workspaces are discovered. It is a background recovery scan, not a user
first-paint or per-request read path.

### Callback

Runner success/failure callbacks update job status, run aggregate status, and event rows in a single lease-checked transaction. If the lease is expired or owned by another runner, MeshX rejects the callback and leaves ledger state unchanged.
Server-managed runner terminal failures preserve the human-readable failure
message and, when available, a stable low-cardinality `failure_kind` in
`error_json` for workspace-level failure aggregation and alert routing.

Log callbacks write the chunk bytes to the automation blob store, then record only
metadata in `automation_log_chunks` and `automation_events`. The request is bounded
by `max_log_chunk_bytes`, and an optional SHA-256 supplied by the runner must
match the received bytes.

Artifact callbacks use a small inline server-side blob upload in the MVP runner
API: the runner submits artifact bytes as `content_base64` together with `size`,
required `sha256`, and optional `content_type` / `expires_at`. MeshX decodes and
validates the bounded request, writes bytes to the automation blob store, and
records only metadata in `automation_artifacts` and `automation_events`. Artifact
names are bounded before blob writes so both the metadata row and generated
object key stay within their storage budgets. The inline JSON endpoint is
bounded by `max_artifact_bytes`, which defaults to 16 MiB; larger artifact
transport requires a separate streaming or init/complete upload protocol before
increasing that bound. Reads are scoped by `workspace_id` plus `run_id`;
metadata for another workspace or run must not be returned, and missing blob
objects return an explicit missing-blob error instead of silently returning
incomplete artifact/log content.

Blob lifecycle maintenance is bounded and explicit. Artifact metadata with
`expires_at <= now` is eligible for deletion together with its blob object.
Object stores that support bounded listing can run an orphan sweep under the
automation object prefix: objects without `automation_log_chunks` or
`automation_artifacts` metadata are deleted, while in-flight temporary objects are
ignored by store-specific implementations. Health checks can sample bounded
metadata references and report metadata rows whose blob object is missing.

### Recovery

Reconciler scans expired leases and requeues jobs when attempts remain. If
attempts are exhausted, it marks the job failed, appends failure events, and
marks the run failed or blocked according to policy. It also reconciles active
runs whose job ledger is already terminal and repairs leaked active-run slots
whose run is missing or already terminal. The repair path releases only leaked
slots; it does not synthesize slots for active runs.

Each process-local reconciler exposes an in-memory heartbeat snapshot with last
start time, last completion time, last success time, last failure time, last
error, total error count, consecutive failure count, and the last successful
reconciliation counts. Operators should alert when the reconciler has no recent
success, has consecutive failures, or when dashboard metrics show expired
leases, terminal-inconsistent runs, or leaked active-run slots.

### AgentTeam-native waits

Jobs whose payload uses `agentteam/agent-task` or `agentteam/wait-event` are persisted as `automation_waits` and marked `waiting` in `automation_job_runs`. They are resumed by AgentTeam service events, complete the job run idempotently, append `wait_resolved`, and then let the scheduler enqueue downstream GitHub Actions jobs. Canceling a run clears active wait keys and marks active waits canceled.

## Automation Hub UI

- Sidebar lists AGS repositories available as automation collections.
- Browser routes select an automation collection through path segments such as
  `/automations/repositories/{owner}/{repo}` and
  `/automations/repositories/{owner}/{repo}/runs`. Query parameters are reserved
  for list filters and editor-local state, not repository-selection aliases
  such as `repo` or `automation`.
- Repository page lists automation definition files from `.agentteam`.
- Editor opens an automation definition file, shows a AgentTeam-styled visual editor, and keeps YAML/source mode available.
- Saving commits to AGS repo; branch/commit UX follows AGS capabilities.
- Run page starts an automation from selected ref and inputs.
- Run detail page reads MeshX ledger and shows run status, progress, current blockers, jobs, waits, queue/runner lease state, events, logs, artifacts, retries, and cancellation state. The first viewport should prioritize actionable status, while logs, artifacts, event history, and raw JSON remain available as drill-down debug surfaces.

## Review gates

- No MeshX-owned automation definition/version source-of-truth tables or APIs.
- MeshX-enabled development and product paths must not require a Desktop
  runner or manually launched `actions/runner` process for shell-only automation
  execution.
- Server-managed runner execution must remain shell-only, fail closed for
  unsupported GitHub Actions features, and enforce timeout, log/blob limits, and
  isolated job work directories.
- Automation definition source must resolve to AGS repo, automation path, ref, and commit SHA.
- Runner callbacks must be lease-safe and atomic.
- Specs and implementation must not include deprecated transitional automation definition/version APIs.