# Unified AgentTeam Runtime Architecture

> Canonical path: `specs/architecture/unified-agentteam-runtime.md`
> Status: living-spec
> Scope: build artifacts, runtime composition, deployment topologies, and safe
> rollout of the AgentTeam gateway and unified business runtime.
>
> Related specs:
> - [`specs/api/server-profile-routing.md`](../api/server-profile-routing.md)
> - [`specs/deployment/agentteam-server-workspace-isolation.md`](../deployment/agentteam-server-workspace-isolation.md)
> - [`specs/deployment/workspace-gateway.md`](../deployment/workspace-gateway.md)
> - [`specs/architecture/meshx-service-boundaries.md`](./meshx-service-boundaries.md)

## 1. Decision

AgentTeam converges on two resident core server entrypoints:

```text
agentteam-gateway-server
agentteam-server --profile central|workspace
```

`agentteam-server --profile workspace` embeds the MeshX runtime. The server profile,
not a runtime feature flag or service-discovery URL, determines process
topology: `workspace` and `all` compose MeshX in process, while `central` must
not initialize MeshX, AGS, workspace files, MeshX secrets, or MeshX workers.
The converged release has no selectable standalone MeshX mode.

The names above are executable names. Deployments may use logical service names
such as `gateway-server`, `central`, and `workspace-<group>`, but must not create
a second binary contract for them. The unified business executable remains
`agentteam-server`, rather than being named `meshx-server`, because it owns both the
AgentTeam central control plane and the workspace data plane; MeshX is only one
workspace-profile component.

`profile=all` remains the compatibility default for local and single-process
private deployments. It is a convenience composition of central and workspace
behavior, not a third production ownership model.

Operational rollback restores the complete previous release: its AgentTeam and
MeshX binaries/images, deployment manifests, routing configuration, and worker
ownership configuration. The new release does not carry a disabled standalone
server or embedding toggle as an in-place rollback mechanism.
This keeps one runtime topology per release and makes rollback provenance
explicit. Schema changes and repository/filesystem formats must remain readable
by the retained previous release for the declared rollback window; otherwise
rollback requires restoring the coordinated data recovery point.

MaaS and Ops remain independent runtimes. Consolidating them into the AgentTeam
business process would merge distinct control and fault domains and is not part
of this architecture.

## 2. Artifact model

### 2.1 Before convergence

The current build and deployment model has three resident core server
executables and colocates runtime and administrative tools in some images:

```text
agentteam-gateway-server ── routes central and workspace traffic
agentteam-server         ── serves AgentTeam central/workspace/all profiles
meshx-server        ── serves MeshX APIs and workers

agentteam-schema                ── AgentTeam core schema tool, not a resident service
agentteam-tidb-admin            ── AgentTeam core administration tool, not a resident service
meshx-server migrate       ── current MeshX/AGS schema command
meshx-server agent-cloud-memory-backfill
                           ── bounded compatibility backfill, not a runtime capability
```

A typical composed environment therefore runs separate AgentTeam and MeshX service
containers, in addition to gateway/ingress, Web, MaaS, Ops, and databases.
Container count is deployment-specific and is not an architectural invariant.

```text
                         ┌─ agentteam-server
client ─ ingress/gateway ┤
                         └─ meshx-server

independent: agentteam-maas, agentteam-ops, Web, TiDB, external model providers
```

### 2.2 Target state

The target has two resident core server executables:

```mermaid
flowchart LR
    Client["Browser / Desktop / Daemon / Git client"]
    Web["AgentTeam Web\nstatic artifact"]
    Ingress["Public ingress\nTLS + coarse routing"]
    Gateway["agentteam-gateway-server\nplacement-aware routing"]
    Central["agentteam-server --profile central\nidentity + org + workspace + placement"]
    MaaS["agentteam-maas\nindependent service"]
    Ops["agentteam-ops\nindependent admin service"]
    CoreDB[("agentteam_core\nTiDB schema")]
    MeshXDB[("agentteam_meshx\nTiDB schema")]
    AGSDB[("agentteam_ags\nTiDB schema")]
    RepoVolume[("repository / workspace volumes")]
    Tools["agentteam-schema / agentteam-tidb-admin\nunified explicit jobs"]

    subgraph Workspace["agentteam-server --profile workspace --workspace-group-id G"]
        AgentTeamAPI["AgentTeam workspace API\nrealtime + runtime owner"]
        MeshX["embedded MeshX runtime\nhandler + domain services"]
        AGS["embedded AGS\nGit + repository APIs"]
        Workers["workspace-owned workers"]

        AgentTeamAPI -->|"in-process http.Handler adapter"| MeshX
        AgentTeamAPI -.->|"narrow typed ports where suitable"| MeshX
        MeshX -->|"in-process Handler / typed adapters"| AGS
        Workers -->|"typed service calls"| MeshX
    end

    Client -->|"HTTPS / WSS / Git smart-HTTP"| Ingress
    Ingress -->|"static asset request"| Web
    Ingress -->|"API and runtime traffic"| Gateway
    Gateway -->|"HTTP: control-plane routes"| Central
    Gateway -->|"HTTP: workspace-owned routes"| AgentTeamAPI
    Central -->|"SQL: control-plane data"| CoreDB
    AgentTeamAPI -->|"SQL: workspace-owned AgentTeam data"| CoreDB
    MeshX -->|"SQL: MeshX-owned data"| MeshXDB
    AGS -->|"SQL: AGS metadata"| AGSDB
    AGS -->|"filesystem / Git object access"| RepoVolume
    AgentTeamAPI -->|"filesystem: workspace runtime data"| RepoVolume
    Central -->|"authenticated internal HTTP where enabled"| MaaS
    AgentTeamAPI -->|"authenticated internal HTTP where enabled"| MaaS
    Ops -->|"authenticated internal admin HTTP"| Central
    Ops -->|"authenticated internal admin HTTP"| AgentTeamAPI
    Ops -->|"authenticated internal admin HTTP"| MaaS
    Tools -->|"explicit migration / admin job"| CoreDB
    Tools -->|"explicit migration / admin job"| MeshXDB
    Tools -->|"explicit migration / admin job"| AGSDB
```

The diagram shows the production-shaped split topology. `profile=all` collapses
Central and Workspace into one process for local and single-process private
deployments; it does not change the logical call or data-ownership boundaries.
Multiple SaaS workspace groups repeat the Workspace subgraph, and Gateway routes
each request to the group selected by placement.

Component placement alone does not select a call mechanism. The following table
is the call contract represented by the labeled edges:

| Caller → callee | Call mechanism | Boundary requirement |
| --- | --- | --- |
| Client → ingress/gateway | HTTPS, WSS, Git smart-HTTP | Public authentication, rate limits, and streaming semantics remain intact |
| Gateway → central/workspace | Service HTTP reverse proxy | Gateway resolves placement; the target revalidates identity and ownership |
| AgentTeam workspace API → embedded MeshX | In-process `http.Handler` adapter first | Preserve middleware and wire-contract parity; do not open loopback TCP |
| AgentTeam workspace domain → MeshX domain | Narrow typed port, only where stable | Pass trusted typed identity/workspace context; do not expose store or HTTP DTO types |
| MeshX → embedded AGS | In-process Handler or typed adapter by protocol | Preserve AGS authorization and repository ownership checks |
| Workspace workers → MeshX services | Typed service calls | Start once and fence work to the owning workspace group |
| Central/workspace/Ops → MaaS | Authenticated internal HTTP | MaaS remains an independent release, scaling, and fault domain |
| Ops → central/workspace admin surfaces | Authenticated internal HTTP | Never expose internal admin paths through public ingress |
| Runtime → owned data/volume | SQL, filesystem, or Git object access | Preserve separate schema, credential, transaction, and volume ownership |
| Tool job → schemas | Explicit migration/admin command | Never execute implicit DDL during normal replica startup |

There is no default Central-to-Workspace business RPC edge. Public requests are
classified at Gateway and sent directly to the owning service; cross-profile
work requires a separately specified durable handoff rather than an accidental
in-process dependency.

Build entrypoints must distinguish resident servers from one-shot tools even
when a release packages them in the same image:

| Artifact class | Contents | Lifecycle |
| --- | --- | --- |
| Gateway runtime | `agentteam-gateway-server` | Resident, horizontally scalable subject to gateway state contracts |
| Business runtime | `agentteam-server` | Resident, selected by `all`, `central`, or `workspace` profile |
| Migration/tools | `agentteam-schema`, `agentteam-tidb-admin` extended across owned schemas | Explicit job/operator artifacts; never normal replica startup |
| Independent products | MaaS server, Ops server, Web artifacts | Separate release, fault, and scaling domains |

A release may package gateway, business, migration, and administration
executables in one image or in separate minimal images. Image layout is not the
contract; executable entrypoints and lifecycle boundaries are. Resident pods
must start only a server entrypoint. Migration and administration commands run
only as explicit jobs/operator actions and must never be part of normal replica
startup.

## 3. Profile responsibilities

| Responsibility | `central` | `workspace` | `all` |
| --- | --- | --- | --- |
| Identity, organizations, workspace metadata and placement | Yes | No | Yes |
| Workspace-owned AgentTeam APIs and realtime/runtime paths | No | Yes | Yes |
| MeshX HTTP surface | No | Embedded | Embedded |
| MeshX database/services and AGS/filesystem dependencies | No | Yes | Yes |
| Workspace background loops and MeshX workers | No | Yes, group-fenced | Yes, local/single-owner semantics |
| `ManagedWorkspaceSet` | Empty | Placement-owned set | All workspaces |
| `--workspace-group-id` | Invalid/not used | Required | Optional local convention |

Route ownership, placement, and owner checks remain normative in the related
profile and deployment specs. Embedding MeshX does not permit central to mount
workspace routes or resources, and it does not permit a workspace process to
handle a workspace it does not own.

## 4. Embedded MeshX contract

### 4.1 Composition surface

MeshX exposes a small composition package to the AgentTeam server owner at the
canonical root-module path `internal/meshxruntime`. The composition root may
provide:

```go
type Runtime interface {
    Handler() http.Handler
    Ready(context.Context) error
    StartWorkers(context.Context, WorkerOwnership) error
    QuiesceWorkers(context.Context) error
    Shutdown(context.Context) error
}
```

The exact Go shape may differ, but it must preserve these capabilities and
lifecycle rules:

1. construction validates configuration without starting background work;
2. the handler can be mounted below the existing AgentTeam routing surface;
3. worker startup requires explicit ownership scope and never defaults to
   unbounded/global scans;
4. readiness includes required database, AGS, filesystem, and worker state;
5. quiesce stops new claims while allowing in-flight work to drain;
6. shutdown drains requests and workers before dependencies close;
7. errors identify the failed component without exposing secrets.

The root AgentTeam module uses canonical internal owners directly. Process assembly
depends on the narrow `internal/meshxruntime` lifecycle surface; only that runtime
owner imports `internal/resource/server`. API, service, store, database, worker,
and typed-port callers import their semantic owners rather than the resource
server or a compatibility facade. The transport-neutral KB Approval bridge
contract is owned by `internal/kbgovernance`. AgentTeam Approval composition and
MeshX KB behavior consume its canonical payload, publication,
terminal-notification, and authority values directly; the package contains no
domain implementation and must not depend on runtime composition packages.
Resource domain services retain the ownership rules in
`meshx-service-boundaries.md`; embedding and physical relocation do not move
business logic into the AgentTeam server composition root.

### 4.2 Handler and typed-port boundary

Embedding makes an in-process transport possible, but does not mean every API
call should become a direct handler invocation.

- Public REST, GraphQL, Git smart-HTTP, upload/download, streaming, and webhook
  surfaces continue through `http.Handler`. This preserves middleware,
  authentication, status codes, and streaming behavior without opening a
  loopback TCP connection.
- Existing internal AgentTeam-to-MeshX HTTP calls may first use an in-process
  `RoundTripper` or handler adapter. This is a low-risk transport change and
  keeps the HTTP contract while removing network discovery and serialization
  overhead where practical.
- Stable internal business operations may later move to narrow typed ports.
  Ports accept authenticated typed identity/workspace context and return domain
  results; they must not expose HTTP DTOs, database models, or MeshX internal
  service types.
- Handlers must not be called as ordinary functions with forged request headers.
  The embedding adapter strips untrusted internal headers and derives identity,
  workspace ID, and workspace group from trusted AgentTeam context.

This sequence avoids coupling AgentTeam to handler implementation details while
allowing measured, incremental removal of internal HTTP hops.

### 4.3 MeshX and AGS route ownership migration

The predecessor release's central process carries MeshX/AGS forwarder routes.
The unified release moves them to the workspace route set and gives central no
MeshX dependency.
The migration inventory includes every registered method for these families:

| Route family | Trusted workspace identity source |
| --- | --- |
| `/api/v1/workspaces/:workspaceId/{kb,memory,skills,roles,automations}*` | Authenticated principal plus path-owned workspace ID; target revalidates context and ownership |
| `/api/v1/workspaces/:workspaceId/{repository-owners,repository-names,repository-access-tokens,personal-access-tokens}*` | Authenticated principal plus path-owned workspace ID; target revalidates token/repository ownership |
| `/api/v1/workspaces/:workspaceId/{mcp,repositories}*` | Path-owned workspace ID; missing or conflicting identity fails closed |
| `/api/v1/workspaces/:workspaceId/agent-thread/session-channel` | Authenticated current-workspace context plus explicit route workspace ID |
| Native AGS REST, raw, and Git protocol routes | Preserve the native URL only for separately activated AgentTeam product operations; authority and placement follow the canonical [request workspace routing matrix](../api/request-workspace-routing-matrix.md). Non-product discovery/bootstrap compatibility remains 404 at the split edge. |
| Legacy `/daemon/*`, `/internal/app/runtime/*` | Authenticated runtime credential association; Gateway normalizes these compatibility families to canonical workspace routes |
| Retired `/internal/agent/*` | Unsupported at the Gateway and rejected with 404; daemon/Desktop/CLI callers use `/api/v1/workspaces/:workspaceId/agents/:agentId/runtime/*` with the canonical wire contract |
| Repository/account/organization management APIs, anonymous `POST /api/ext/v1/agents`, and `/api/graphql` | Unsupported at the split Gateway; reject fail-closed until a storage-aware ownership contract exists |

The route inventory is generated from the central and workspace routers and
compared by `(method, path)`. A new or changed MeshX/AGS route must update this
classification and its accepted identity source in the same change. A route
without a trustworthy workspace identity fails closed; the presence of an
embedded handler is never sufficient to activate the unified release.

Route cutover order is:

1. add workspace identity to Browser/Desktop/daemon call sites that already
   know the workspace;
2. add the bounded identity resolvers and route activated AGS operations according
   to the canonical request workspace routing matrix; unresolved workspace-local
   operations never fall back to the designated default owner;
3. validate the complete route inventory, owner checks, and headerless protocols
   against the final central/Gateway/multi-group workspace topology before the
   maintenance window;
4. quiesce the previous release's standalone workers and wait for in-flight
   claims and writes to drain;
5. start the unified workspace owners, enable final Gateway routing, and prove
   central constructs no MeshX dependency before restoring traffic;
6. remove the previous standalone owner only after the final topology remains
   healthy for the rollback observation window.

Supported public paths, request/response semantics, and credentials remain
unchanged. The split Gateway explicitly rejects deferred GraphQL, AGS
management, and anonymous public-agent registration surfaces; the all profile
retains its existing AGS surface.
Old clients that omit the route header remain compatible only when their
runtime credential itself has an authoritative workspace association. An
unrelated user session is never used as a default-workspace fallback. Rollback
restores the retained previous release in reverse
dependency order: restore its MeshX service and compatible central version,
wait for both to become ready, then restore its Gateway decisions. Gateway must
never send a route to central before that route and its MeshX upstream are
ready. This rollback path belongs to the previous release bundle; the new
`agentteam-server` does not switch its own topology at runtime.

Contract tests must prove the supported public route inventory is explicit,
every workspace family has an identity source, central/workspace route
ownership has no unintended conflict, wrong-owner requests fail closed, legacy
clients follow only the machine/Agent/App resolver, deferred split routes return
404, and the final central profile has neither MeshX routes nor a MeshX runtime
dependency.

### 4.4 Worker ownership and fencing

Every MeshX background worker is registered in a worker inventory before it can
run embedded. The inventory names its durable work records and classifies it as
one of:

- **group-fenced:** every discovery query, claim, retry, cleanup, and write is
  restricted to the supplied `workspace_group_id` and `ManagedWorkspaceSet`;
- **singleton:** exactly one dedicated process owns the worker globally until a
  group-fenced data/query contract exists.

`WorkerOwnership` (or an equivalent public contract) carries the non-empty
workspace group ID, `ManagedWorkspaceSet`, worker mode, durable owner ID, and
fencing epoch. Group-fenced workers must claim work with a lease or claim key
scoped by `(worker_class, workspace_group_id)`. A stale owner cannot commit a
write after its fence epoch changes. Merely filtering the first list query is
insufficient: retries, cleanup scans, queued jobs, and repository/file work
must repeat the ownership check at claim and commit boundaries.

Automation runtime/reconciliation, KB code-wiki work, Knowledge Q&A cleanup,
and every future MeshX worker must be present in this inventory. A worker that
cannot yet derive workspace ownership blocks activation of the unified release.
During rollback, a previous-release standalone owner may start only after all
new-release owners have quiesced, drained, and relinquished their fences. An
unscoped standalone worker cannot run beside any unified-runtime instance of
the same worker class.

Singleton is a predecessor-release state, not a target architecture. This
architecture does not define a permanent independent worker artifact. Before
the unified release can activate, every still-required singleton worker must
implement the group-fenced query/claim/commit contract, pass the isolation and
lease-loss suite, and transfer to workspace runtimes through the no-overlap
state machine below. A worker that is no longer required must be explicitly
retired with its queues, leases, alerts, and readiness checks. The final
inventory must report `remaining_singleton_compatibility_workers=0`.

Isolation tests use at least two workspace groups and assert that discovery,
claim, retry, cleanup, writes, AGS/Git operations, and filesystem access never
cross groups. Lease-loss tests prove a stale process cannot claim or commit new
work.

Ownership transfer and rollback use the following no-overlap state machine:

```text
active owner
  -> quiesce (stop new claims)
  -> drain in-flight work
  -> release lease / advance fence epoch
  -> start replacement owner
  -> acquire lease and pass readiness
  -> switch request traffic
```

Failure before replacement readiness restores the previous owner only after it
reacquires the current fence. At no point may both processes report worker-ready
for the same `(worker_class, workspace_group_id)`.

### 4.5 Agent turn-state snapshot and Workflow callback recovery

Workflow callback recovery judges whether an awaiting `send_message.callback`
responder is actually working from the **real Agent turn execution state**
(`active` / `queued` / `idle`) reported by the daemon and mirrored in the server
Hub overlay. It never reads the Workflow capacity lease or callback FIFO, never
synthesizes a TTL/heartbeat liveness, and never infers working from
process/status/activity/session presence.

The Hub keeps the turn-state snapshot connection-scoped and **connection-epoch
fenced** with a **same-connection monotonic ordering** contract:

- The daemon serializes fresh turn-state capture, revision allocation, and
  enqueue under one publication boundary; every modern ready publication carries
  an explicit **watermark** (even for an empty snapshot or zero running agents)
  and every incremental `agent:turn_state` carries a **strictly newer revision**.
- Server decodes watermark presence separately from its numeric value: a present
  watermark (including zero) is a modern snapshot claiming ordering authority; a
  ready that omitted the field is legacy and applied with clear-and-replace
  semantics, never claiming modern ordering authority. Server drops a stale
  `sessionId` (replaced session), a non-matching `epoch` (old connection), and an
  increment whose `revision` is older than or equal to the applied high-water. A
  snapshot is applied **by per-agent merge**: an older snapshot never overwrites
  a newer per-agent increment, its still-current entries for other agents remain
  materializable, and an agent absent from a snapshot whose stored revision is
  older/equal is cleared (an explicit empty current snapshot retires stale
  working state). Only an accepted (non-regressing) snapshot updates the
  high-water and snapshot diagnostics. Ordering resets only on a genuinely new
  authenticated connection.
- `Hub.MachineConnectedAt` records when the current connection established; it is
  anchored once and never moved by ready re-application, duplicate ready,
  polling, restart, or races.

The recovery working reader (`agentruntime.HubWorkingReader`) exposes the state
matrix used by the RecoveryRunner:

| Hub facts | Recovery judgment |
| --- | --- |
| machine offline (not connected) | **defer** — no send, no timing; resumes once reconnected and a snapshot is established |
| online + snapshot + `active` / `queued` | **working** — never wake, no timing |
| online + snapshot + `idle` | **recoverable** — existing five-attempt 1/2/4/8/16-minute schedule |
| online + no current snapshot (or responder absent) within grace | **defer** — bounded grace, anchored to connection time |
| online + no current snapshot past grace | **recoverable** — treated as idle so recovery cannot stall on an empty Hub |

The bounded grace (default 60s) is deliberately shorter than the first recovery
interval (1 minute). Because it is anchored to the connection timestamp,
polling, duplicate ready, restart, and old-epoch races cannot extend it forever.

## 5. Data, worker, and isolation boundaries

Binary consolidation does not imply storage consolidation:

- AgentTeam core, MeshX, and AGS retain separate schema and domain ownership.
- A workspace process may share connection-pool infrastructure only when
  credentials, pool budgets, transactions, and ownership remain explicit.
- Cross-domain transactions are not introduced by embedding. Workflows that
  span domains use explicit service contracts and idempotent compensation.
- Schema migration is an explicit release job. Normal central/workspace/all
  replicas open schemas without migration, verify compatible versions, and fail
  fast when schema is incompatible; they must not apply implicit DDL.
- MeshX APIs, repositories, filesystem paths, secrets, asynchronous jobs, and
  worker leases are fenced by `workspace_group_id` and the managed workspace
  set. A placement mismatch fails closed.
- A workspace group remains single-owner where realtime, filesystem, or worker
  semantics require it, until the related ownership/lease contract permits
  safe horizontal scale.

## 6. Deployment topologies

### 6.1 Local development and test

The default local path optimizes startup and debugging:

```text
browser ─ local ingress ─ agentteam-server --profile all
                          └─ embedded MeshX

optional: agentteam-gateway-server for routing integration tests
separate: MaaS, Ops, Web, TiDB as required by the test profile
```

Local compose may run `central` and `workspace` separately when validating
production routing. Standalone MeshX testing belongs to the retained previous
release and is not a selectable topology in the unified release.

### 6.2 Private delivery

Small installations may use the combined profile:

```text
ingress ─ agentteam-server --profile all
          └─ embedded MeshX
```

Larger or isolation-sensitive installations use the production-shaped split:

```text
                       ┌─ agentteam-server --profile central
ingress ─ gateway ─────┤
                       └─ agentteam-server --profile workspace --workspace-group-id private-default
```

Both shapes use the same business executable and configuration contract. The
split must not require a different product build.

### 6.3 SaaS

SaaS always keeps gateway, central, and workspace groups independently
deployable:

```text
public ingress
  └─ agentteam-gateway-server
       ├─ central service: agentteam-server --profile central
       ├─ workspace group A: agentteam-server --profile workspace --workspace-group-id A
       └─ workspace group B: agentteam-server --profile workspace --workspace-group-id B
```

Central and each workspace group have independent scaling, readiness, rollout,
and rollback. Gateway resolves placement and workspace servers re-check
ownership. MaaS, Ops, Web delivery, databases, and external providers remain
separate services.

## 7. Rollout and removal gates

Stages are correctness gates, not separately selectable production modes. One
unified release contains the final resident and tool entrypoints and satisfies
Stages 0–6 before production activation. It does not ship a standalone MeshX
binary/image, standalone service manifest, discovery adapter, or runtime
topology switch.

Maintenance mode removes the need for production shadow/canary waves and an
online worker-owner handoff, but not their correctness evidence. Route identity,
wrong-owner, two-group isolation, fencing, schema, backfill, and rollback tests
must pass in staging before the window. During the window, request intake is
closed before old workers quiesce and drain; new Central, Workspace, and Gateway
instances must all be ready and pass offline route/API/Git/worker smoke before
public entry reopens.

Rollback uses the immutable previous release bundle, not a mode in the unified
release. Delivery retains the previous images, manifests, configuration, and
recovery instructions for the declared rollback window. The old topology may
remain scaled to zero in the deployment system, but none of its resident
artifacts or compatibility code is copied into the new release. A GitOps revert
restores that whole bundle only while current schemas and repository/filesystem
formats remain backward-readable. Otherwise rollback restores the coordinated
data recovery point captured with all writers stopped.

Before data migration starts, maintenance mode must close every external and
internal write path, quiesce all request/background writers of AgentTeam core, AGS,
MeshX, and repository/workspace storage, and establish the final recovery point.
If databases and filesystem storage cannot be captured atomically, the delivery
runbook must define a rehearsed capture order and restore consistency invariant
while all writers remain stopped.

Moving from a standalone MeshX volume to the unified runtime volume is an
explicit data migration, even when both releases expose the same container
path. The old MeshX data root and new unified data root must be mounted at
different paths and processed by
[`scripts/migrate-unified-runtime-volume.sh`](../../scripts/migrate-unified-runtime-volume.sh)
according to the
[Unified Runtime Data Migration](../deployment/unified-runtime-data-migration.md)
runbook. Source preflight, expected AGS repository inventory, filesystem and
Git verification, and a coordinated backup of all three databases plus the old
data root are production activation gates. An empty or mismatched source fails
closed, and the old databases and volume remain available through the rollback
window.

### Stage 0: separate resident and tool entrypoints and lifecycles

Prepare data and release artifacts without changing routing or ownership:

- The shared release image may contain resident and tool entrypoints. Resident
  replicas start only their server entrypoint; schema/admin commands run only
  through explicit Jobs/operator actions and never during normal startup.
- AgentTeam core retains its schema package; MeshX and AGS expose narrow public
  `Init/Verify` surfaces. Each surface accepts only its domain configuration,
  owns its migration definitions, returns domain-qualified errors, and exposes
  no store, persistence model, or server construction.
- With `AGENTTEAM_MESHX_ENABLED=true`, `agentteam-schema init|verify` orchestrates three
  explicit DSNs in AgentTeam core → AGS → MeshX order, preserving the current
  AGS-before-MeshX dependency. With MeshX disabled it manages only AgentTeam core,
  so supported AgentTeam-only/private deployments do not need placeholder MeshX or
  AGS DSNs. It must not
  import `internal/database`. Completed steps are idempotent; verification
  checks all domains, performs no repair/DDL, and exits non-zero on
  incompatibility. Reordering requires dependency analysis plus clean-database
  and all supported upgrade-path tests.
- This replaces `meshx-server migrate`, not the migration capability. Normal
  replicas never migrate; they verify compatibility and fail fast.
- One application-level `AgentCreator` owns `create → provision owner-controlled
  Cloud Memory → finalize` for regular, duplicate, role-backed, and every
  Ops/import/automation entry. Any failure rolls back the agent and partial
  role/resource state. An AST/import guardrail restricts raw store creation to
  this adapter; tests prove each successful path has exactly one active binding
  and provisioning failure leaves no active agent.
- After that invariant is deployed, each environment runs the paginated
  backfill with zero failures, then a from-start dry-run with `wouldEnable=0`.
  Record the result and invariant version before deleting the bounded command.
- Update local, private, and SaaS manifests plus migration, upgrade,
  backup/restore, and operator runbooks to use the new tools.

`agent-cloud-memory-backfill` is temporary, not a permanent admin capability.
If an environment has not converged by the resident-server removal window, move
only this bounded command, including cursor/dry-run behavior, temporarily to the
`agentteam-tidb-admin` tool entrypoint. Remove old resident subcommands only after
every environment has stopped invoking them.

**Gate:** artifact inventories and smoke tests pass in every topology; schema
parity, independent verification, partial-success retry, no-auto-migrate, and
fail-fast tests pass; manifests/runbooks use the unified tool; `AgentCreator`
guards all production creation; every environment has convergence evidence or
the temporary bounded backfill remains packaged.

**Rollback:** point jobs to the retained compatible command before restoring
previous images. Migrations stay forward-compatible; traffic is unchanged.

### Stage 1: extract the MeshX runtime surface

Move MeshX server composition behind the narrow Runtime surface from
[§4.1](#41-composition-surface). During development, predecessor standalone
tests may exercise the same Runtime to prove behavioral parity, but the unified
release does not retain that executable or expose a runtime selection flag.
Construction accepts explicit ownership and never starts workers implicitly.

**Gate:** the existing MeshX API/worker suite runs against the extracted
runtime; construction, readiness, quiesce, shutdown, and double-start behavior
have contract tests; schema and Cloud Memory gates from Stage 0 are closed or
the bounded backfill is explicitly retained.

**Rollback:** restore the previous release bundle; do not add a compatibility
branch to the new Runtime.

### Stage 2: migrate MeshX/AGS route ownership

Execute [§4.3](#43-meshx-and-ags-route-ownership-migration): add trusted
workspace identity, update clients and the bounded legacy runtime resolver,
validate the final route inventory before activation, deploy workspace owner
checks, route workspace families by placement, and route supported AGS
REST/raw/Git to the designated AGS owner during the maintenance cutover.
Central forwarders remain only in the retained previous release rollback
artifact.

**Gate:** generated `(method, path)` inventory has no unclassified route;
contract tests preserve public behavior and prove owner mismatch fails closed;
the final-topology harness proves central has no MeshX/AGS route and owner loss
does not fall back. Central startup constructs no MeshX client, handler,
database, AGS, filesystem, secret, or worker dependency.

**Rollback:** restore and ready the previous release's central and standalone
services, then restore its Gateway revision.

### Stage 3: compose MeshX by server profile

Mount MeshX in `agentteam-server --profile workspace`, using the in-process
HTTP-compatible adapter where applicable. `workspace` and `all` select this
composition structurally; `central` excludes it. There is no
embedding toggle, resident MeshX base URL, or standalone
fallback in the unified server. A worker does not enter the release until its
fencing contract and isolation tests pass.

**Gate:** contract tests preserve status, headers, body, streaming,
authorization, and error semantics across public and in-process transports.
Central-profile tests continue to prove no MeshX route or dependency is
constructed. Configuration tests prove retired topology variables are ignored
and cannot restore standalone routing. Worker readiness reports every required class as
group-fenced; no disabled singleton backlog may be hidden by readiness.

**Rollback:** restore the previous release bundle. No schema rollback is
required while the previous release remains compatible with current data.

### Stage 4: validate and activate workspace groups

Use staging groups to validate mixed ownership and fence requests to placement.
A worker joins production only after satisfying
[§4.4](#44-worker-ownership-and-fencing); otherwise the maintenance cutover
cannot proceed. Production switches every ready group in the same maintenance
window rather than running an online canary.

**Gate:** no ownership mismatch or duplicate worker/lease activity; error rate,
p99 latency, RSS, database pool use, worker lag, Git/AGS operations, filesystem
failures, and shutdown drain remain within agreed baselines. Two-group isolation,
lease-loss, quiesce/drain, and stale-fence tests pass for every migrated worker.

**Rollback:** while public entry remains closed, quiesce, drain, advance the
fence, restore and ready the previous release owner, restore the previous
Gateway/Ingress mapping, then reopen traffic. Two owners must never be ready
simultaneously.

### Stage 5: validate the unified resident topology

Local/private uses `all`; SaaS uses independent `central` and `workspace`
profiles. Central stays MeshX-free. Ops uses the protected admin surface on the
owning unified AgentTeam server; the unified release does not carry two admin
endpoints for topology fallback.

**Gate:** every supported topology has upgrade, downgrade, backup/restore,
readiness, and failure-injection evidence. The previous release bundle is tested
as the rollback path before SaaS traffic reopens. The final worker inventory
reports `remaining_singleton_compatibility_workers=0`: every required worker is
group-fenced and owned by the unified Runtime, and every unneeded worker is
retired with its queue, lease, alert, and readiness dependency. The rollback
window retains the immutable old release and, when desired, its scaled-to-zero
resources; it does not add an active or disabled singleton migration backlog to
the unified release.

**Rollback:** close public entry, restore the retained GitOps revision and old
Central/standalone owners, verify readiness, restore Gateway/Ingress routing,
then reopen entry. Do not reverse schema migrations.

### Stage 6: remove standalone contracts from the unified release

Before publishing the unified release, remove the resident binary/image,
discovery, adapters, and topology switch when:

1. every supported environment has a tested whole-release rollback bundle;
   compatibility evidence covers the full supported public route inventory,
   Git, worker, admin, and lifecycle surfaces, and proves deferred split routes
   remain fail-closed;
2. Ops and all deployment manifests use combined admin/readiness. The Runtime
   does not read `AGENTTEAM_MESHX_BASE_URL` or equivalent discovery; stale values may
   remain temporarily in deployment-owned environment files and are ignored;
3. the unified tool, with explicit AgentTeam core/AGS/MeshX DSNs, owns schema
   migration, repair, verification, backup/restore, and operator workflows;
   clean-install, upgrade, independent-verify, and partial-retry tests pass;
4. every production agent creation path uses the guarded Cloud Memory owner;
   each environment has full-apply plus from-start `wouldEnable=0` evidence,
   and the backfill is deleted or remains a temporary tool with an owner and
   removal gate;
5. `remaining_singleton_compatibility_workers=0`; required workers are fenced
   and pass two-group, lease-loss, quiesce/drain, stale-fence, readiness, and
   rollback tests; retired workers leave no queue, lease, alert, or readiness
   dependency.

Removal includes the standalone executable and image build, deployment/service
manifests, local compose overlay and port, resident release packaging, dev
dual-process orchestration, Ops alternate endpoint, discovery environment
variables, and obsolete tests and docs. The Runtime composition surface,
unified schema tool, and embedded AGS transport remain because they are owned
by the target runtime; §7.1 later narrows their visibility inside the root
module.

### 7.1 Post-convergence MeshX module consolidation

MeshX module consolidation is a structural follow-up, not part of the runtime
topology cutover. The unified runtime and standalone-retirement gates in Stages
0–6 must complete first. A release must demonstrate the unified bundle,
profile-owned composition, schema tools, worker lifecycle, embedded AGS mirror,
activation, and whole-release rollback contract before module or package paths
move. This ordering keeps topology failures distinguishable from build-graph
and import-path failures and prevents a large file move from obscuring runtime
review.

After that gate, MeshX is an implementation domain of the root AgentTeam Go
module. Its composition API is rooted at `internal/meshxruntime`; supporting
schema/tool capabilities also remain a narrow internal surface. Generic
in-process HTTP adaptation is owned by `internal/inprochttp` rather than
the MeshX domain. Unified-process environment parsing, defaults, and validation
are owned by `internal/config`; resource-runtime configuration remains a named
surface within that owner rather than a second configuration package.
Protected-listener JWT identity extraction and request context
propagation are owned by `internal/identity/trusted`. Shared embedded-AGS trust,
identity mapping from that common identity value, and in-process request
construction are owned by `internal/agsbridge`; the process-local `meshxags`
go-git protocol and handler registry are owned by its independent
`internal/agsbridge/gittransport` child package so ordinary bridge callers do
not trigger go-git's process-global protocol registration. Transport-neutral KB
Approval contracts are owned by `internal/kbgovernance`. None of these root
owners or the child package depends on runtime composition packages. MeshX API,
service, store, database, worker, and AGS implementation lives in its canonical
root-module domain owners under `internal/api`, `internal/service`,
`internal/store`, `internal/database`, `internal/resource/server`, and
`internal/agsbridge`. This ownership boundary does not merge schemas, stores, transactions,
DTOs, worker ownership, or fault-handling semantics across AgentTeam and MeshX.

The canonical repository state has one Go dependency graph, no nested MeshX
module or root self-dependency, and no retired MeshX source namespace. Runtime,
schema tooling, API, services, stores, database, workers, and embedded-AGS
implementation live in their canonical root-module owners. Resource server
composition lives at `internal/resource/server`; shared configuration, trusted
identity, embedded-AGS bridging, its independent Git transport, governance
contracts, and generic handler transport live at `internal/config`,
`internal/identity/trusted`, `internal/agsbridge`,
`internal/agsbridge/gittransport`, `internal/kbgovernance`, and
`internal/inprochttp`. Build contexts, dev watchers, test inventories, and
canonical documentation use those paths. No alias module, transitional source
tree, or old-import compatibility shim may recreate the retired layout.

Executable guardrails preserve these dependency rules:

- process composition imports the narrow runtime lifecycle surface, and only
  `internal/meshxruntime` imports `internal/resource/server`;
- foundation, API, service, and store packages do not import resource-server
  composition for domain behavior, DTO translation, or persistence access;
- the runtime-topology guard rejects recreation of retired source namespaces,
  standalone listeners, commands, images, and process residues, while
  domain-specific service and store dependency rules remain independently
  enforced; and
- the relocation must not recreate a standalone command, listener, image,
  discovery URL, worker owner, or separately versioned MeshX module.

Repository-wide build and tests, MeshX domain tests, profile and lifecycle
tests, the production Runtime source-mirror test, schema init/verify, release
build/activate smoke, import-boundary guardrails, and path residue scans enforce
this state. No build, test, release, watcher, guardrail, or canonical-doc
reference may rely on the retired physical root.

**Rollback:** revert the structural step as a source change. Do not use runtime
topology flags, restore a standalone resident, or change data formats to roll
back an import-path or directory-layout migration.

## 8. Observability and rollback contract

All rollout stages preserve correlation across gateway, central, AgentTeam
workspace, and embedded MeshX spans. Structured logs include the server profile,
workspace group, route decision, and safe owner-mismatch reason where known.
Metrics follow the low-cardinality rules in the deployment specs and never use
workspace IDs, repository IDs, paths, tokens, or secrets as labels.

Readiness must distinguish at least:

- AgentTeam API/router readiness;
- MeshX handler dependencies;
- AGS and filesystem access;
- worker ownership and lease health.

A workspace instance does not become ready until the embedded runtime is ready.
Central readiness must remain independent of all MeshX components. Rollback
must restore a versioned deployment bundle, not flip a topology flag in the
current binary. No stage may rely on reversing a destructive schema migration.

## 9. Non-goals

- merging AgentTeam, MeshX, and AGS schemas or domain models;
- merging MaaS or Ops into the business runtime;
- bypassing authentication or middleware through direct handler calls;
- changing public API paths or response contracts for the supported route
  inventory; split GraphQL, AGS management, and anonymous public-agent
  registration surfaces are the explicit deferred exceptions defined in
  [§4.3](#43-meshx-and-ags-route-ownership-migration);
- enabling multi-writer workspace groups without ownership, routing, and worker
  lease support;
- renaming every deployment resource to match executable names; or
- retaining standalone MeshX as a selectable topology in the unified release.

## 10. Acceptance criteria

The architecture is complete when:

- releases have `agentteam-gateway-server` and one `agentteam-server` business executable
  as the resident core server contract;
- the same business executable runs `central`, `workspace`, and compatible
  `all` profiles with fail-fast dependency validation;
- workspace embeds MeshX through a narrow composition surface and central has no
  MeshX route or runtime dependency after the identity-aware route cutover;
- external protocols retain handler semantics, while suitable internal calls
  can evolve from in-process HTTP compatibility to narrow typed ports;
- every required worker is fenced to its owning workspace group before the
  unified release activates, and the remaining singleton count is zero;
- local, private, and SaaS deployment shapes pass their topology-specific
  readiness, upgrade, and rollback tests;
- each domain owns a public schema init/verify surface, while `agentteam-schema`
  orchestrates explicit AgentTeam core → AGS → MeshX jobs with independent DSNs,
  verification, and idempotent partial-success retry; normal replicas never
  auto-migrate and fail fast on incompatible schemas;
- every production agent-creation entry uses one mandatory Cloud Memory
  provision/rollback owner before the bounded backfill is removed;
- the unified release contains no standalone MeshX resident artifact,
  discovery path, or runtime topology switch; and
- rollback restores a tested previous release bundle whose data compatibility
  or coordinated recovery-point requirement is explicit.