# AgentTeam Workspace Multi-Group Phase 2 Spec

> Canonical path: `specs/deployment/workspace-multigroup-phase2.md`
> Status: draft
> Scope: Phase 2 multi workspace group architecture, workspace migration, workspace router/gateway, and cross-repository responsibilities for `agentteam`, `agentteam-ops`, and `agentteam-saas-delivery`.
> Builds on: [AgentTeam Server Workspace Isolation And Deployment Profiles](./agentteam-server-workspace-isolation.md), [AgentTeam Server API Profile Routing](../api/server-profile-routing.md)

## 1. Decision summary

Phase 2 extends the Phase 1 production split from a single workspace group to multiple workspace groups.

The accepted Phase 2 model is:

```text
central: horizontally scalable
workspace groups: many groups, exactly one owner pod per group
workspace migration: move a workspace from one group to another
```

The first target use case is moving a large workspace from the shared default group to a dedicated workspace group:

```text
workspace-default  -> shared workspace group, replicas=1
workspace-large-a  -> dedicated large-workspace group, replicas=1
```

Phase 2 is **not** same-workspace horizontal scaling. A single workspace still has exactly one execution owner at a time.

## 2. Goals

1. Support more than one workspace group in SaaS deployments.
2. Keep each workspace group single-owner (`replicas=1`) so current runtime-local contracts remain valid.
3. Allow operators to migrate a workspace between groups without copying business rows.
4. Route workspace REST, realtime, daemon, internal agent/app, and compatible runtime traffic to the group that currently owns the workspace.
5. Move workspace placement management into `agentteam-ops` while keeping `agentteam` as the authoritative owner of placement state and invariants.
6. Keep `agentteam-saas-delivery` as the GitOps and rollout owner for Kubernetes Deployments, Services, routing, sync waves, cutover gates, and rollback.
7. Retire Phase 1-only `agentteam-tidb-admin workspace-placement-verify` after `agentteam-ops` owns the multi-group verify/preflight path.

## 3. Non-goals

Phase 2 must not include:

- same-workspace multi-pod active-active execution;
- workspace-level HPA or more than one owner pod per workspace group;
- owner-hop, leader election, or lease-based multi-writer runtime ownership;
- durable fanout, cross-pod browser realtime, or socket.io-cluster;
- daemon owner registry or daemon owner rebalancing;
- trigger/worker outbox or cross-owner trigger execution;
- business table sharding or copying workspace business data between databases;
- a complete agentteam-ops UI if API/CLI is enough to safely operate the first migration flow.

Those are separate projects after multi-group migration is stable.

## 4. Phase 1 handoff

Phase 1 is considered ready for production cutover when the production split gate in [agentteam-server-workspace-isolation.md §11](./agentteam-server-workspace-isolation.md#11-staging--production-parity-and-production-split-gate) passes.

The production shape remains:

```text
central: replicas >= 2
workspace-default: replicas = 1
```

Phase 1 workspace identity propagation supports the single default group cutover:

- browser workspace REST sends route workspace headers;
- new machine connect / deep link writes per-channel `config.workspace_id`;
- daemon and internal clients send `X-AgentTeam-Route-Workspace-Id` when the channel has `config.workspace_id`;
- legacy/manual daemon configs without `workspace_id` keep routing to the default group;
- `agentteam` treats route headers as hints and enforces owner mismatch checks after authentication.

Phase 1 propagation does not by itself choose a non-default group. That requires the Phase 2 placement resolver and workspace router/gateway described below.

## 5. Repository ownership

### 5.1 `agentteam`

`agentteam` owns runtime invariants and the authoritative placement state:

- `workspaces`, `workspace_placements`, workspace group metadata, and migration metadata schemas;
- store/service methods that enforce placement and migration state transitions;
- admin HTTP/gRPC endpoints used by `agentteam-ops` and automation;
- workspace router/gateway implementation or resolver library;
- owner mismatch validation and `ManagedWorkspaceSet` refresh behavior;
- route contract and entry ownership tests.

`agentteam` must not delegate direct writes of `workspace_placements` or migration state to another repository.

### 5.2 `agentteam-ops`

`agentteam-ops` owns operator-facing placement management:

- list workspace groups;
- create/disable workspace groups;
- list workspace placements;
- verify placement and group readiness;
- start workspace migration;
- inspect migration progress;
- rollback failed migration;
- surface audit logs and operator intent.

`agentteam-ops` should be a thin frontend/API/CLI over `agentteam` admin endpoints for placement and migration. It must not directly mutate `agentteam` core placement tables.

Existing `agentteam-ops` direct reads or writes for other operational data do not become a precedent for placement management. Placement and migration need `agentteam`-owned APIs because `agentteam` owns the schema, state machine, fencing, and runtime owner checks.

### 5.3 `agentteam-saas-delivery`

`agentteam-saas-delivery` owns deployment and rollout mechanics:

- central Deployment/Service;
- workspace group Deployment/Service templates;
- workspace router/gateway Deployment/Service;
- ArgoCD sync waves;
- schema/backfill/verify/migration gates;
- service discovery names and upstream mapping;
- environment-specific resource requests/limits;
- rollout windows, dry-run/render validation, readiness checks, and rollback runbooks.

Delivery must call a reviewable automation entrypoint owned by `agentteam-ops` for placement verify and migration gates, and `agentteam-ops` calls the authoritative `agentteam` admin API.

The intended call chain is:

```text
agentteam-saas-delivery gate -> agentteam-ops API/CLI -> agentteam admin API -> agentteam placement store
```

## 6. Cross-repository access mode

Phase 2 must choose API-mediated access before implementation tasks are split.

Recommended mode:

1. `agentteam` exposes authenticated admin HTTP endpoints first. gRPC can be added later if needed.
2. `agentteam-ops` authenticates to `agentteam` with a service credential and forwards the human operator identity for audit. The `agentteam` admin API accepts either `X-AgentTeam-Internal-Admin-Token` or `Authorization: Bearer <token>` so operators and service clients have one explicit internal credential contract.
3. Mutating calls require an idempotency key and produce an audit event.
4. `agentteam-saas-delivery` calls `agentteam-ops` for preflight/verify/migration steps instead of connecting to `agentteam` core DB.
5. `agentteam-ops` may cache read results for UI responsiveness, but every mutation goes through `agentteam`.
6. `agentteam` admin endpoints must be reachable only through an internal network path, for example a Kubernetes ClusterIP Service or equivalent private service mesh route. They must not be published through the public Edge Gateway.
7. `agentteam-saas-delivery` must fail render or review when `/internal/admin/*` or the chosen admin API prefix is exposed through public ALB/Ingress. Only `agentteam-ops` and delivery automation with service credentials may call it.

Minimum admin API surface in `agentteam`:

| Operation | Method shape | Notes |
| --- | --- | --- |
| List groups | `GET /internal/admin/workspace-groups` | Returns group state and readiness summary. |
| Create group | `POST /internal/admin/workspace-groups` | Registers stable group metadata before delivery sends traffic. Requires idempotency key. |
| Disable group | `POST /internal/admin/workspace-groups/:groupId/disable` | Blocks new placements after safety checks. Requires idempotency key. |
| List placements | `GET /internal/admin/workspace-placements` | Supports filtering by workspace, group, and state. |
| Verify placements | `POST /internal/admin/workspace-placements/verify` | Multi-group replacement for Phase 1 CLI verify. |
| Start migration | `POST /internal/admin/workspace-migrations` | Creates the fenced migration intent and marks source placement draining. Requires source, target, operator intent, and idempotency key. |
| Get migration | `GET /internal/admin/workspace-migrations/:id` | Read state, timestamps, and last error. |
| Move migration | `POST /internal/admin/workspace-migrations/:id/move` | Switches new traffic to the target group by updating placement to `moving`. Requires idempotency key. |
| Complete migration | `POST /internal/admin/workspace-migrations/:id/complete` | Marks target placement `active` after verify/smoke succeeds. Requires idempotency key. |
| Rollback migration | `POST /internal/admin/workspace-migrations/:id/rollback` | Fenced rollback to the previous group when safe. Requires idempotency key. |

The exact path prefix can change during implementation, but the ownership rule cannot: `agentteam` is the placement authority, `agentteam-ops` is the operator frontend, and delivery uses `agentteam-ops` automation.

## 7. Data model

### 7.1 `workspace_groups`

Phase 2 introduces explicit group metadata. Recommended fields:

| Field | Meaning |
| --- | --- |
| `workspace_group_id` | Stable id such as `workspace-default` or `workspace-large-a`. |
| `display_name` | Operator-friendly label. |
| `state` | `active`, `draining`, or `disabled`. |
| `resource_profile` | Optional tier or sizing hint used by delivery/runbooks. |
| `service_ref` | Optional logical service target used by the router/gateway. |
| `created_at` / `updated_at` | Audit timestamps. |

Kubernetes pod membership remains in `agentteam-saas-delivery`; `workspace_groups` stores stable routing and operator intent, not pod IPs.

### 7.2 `workspace_placements`

The existing table remains the one-to-one workspace-to-group map.

Phase 2 keeps these invariants:

1. Every active workspace has exactly one placement row.
2. `workspace_group_id` points to a known `workspace_groups` row.
3. `placement_epoch` increments on every group change or migration state transition that can affect routing.
4. `state=active` means the router sends new traffic to `workspace_group_id`.
5. `state=draining` means the current group is still authoritative but a migration is preparing or draining old traffic.
6. `state=moving` means new traffic is routed to the target group while old long-lived connections are draining.
7. `state=disabled` means the workspace must fail closed until an operator repairs placement.

### 7.3 `workspace_placement_migrations`

Recommended migration metadata:

| Field | Meaning |
| --- | --- |
| `migration_id` | Stable id for operator/API tracking. |
| `workspace_id` | Workspace being moved. |
| `source_group_id` | Group before migration. |
| `target_group_id` | Intended new group. |
| `state` | `requested`, `preflight`, `draining`, `moving`, `verifying`, `completed`, `failed`, or `rolled_back`. |
| `placement_epoch_start` / `placement_epoch_current` | Fencing values. |
| `requested_by` | Operator identity forwarded by `agentteam-ops`. |
| `reason` | Human-provided intent. |
| `last_error` | Context-rich failure reason. |
| `created_at` / `updated_at` / `completed_at` | Audit timestamps. |

A migration record is the durable audit and idempotency anchor. The placement row is the routing source of truth.

## 8. Workspace router/gateway

### 8.1 Responsibility

The workspace router/gateway is the dynamic resolver that static ALB/Ingress rules cannot be.

It receives workspace-scoped traffic, resolves the workspace's current group, and forwards to the corresponding Workspace Service.

```text
Edge Gateway -> workspace router/gateway -> agentteam-server-workspace-<group>
```

It reads placement through a `agentteam` service/resolver API or a controlled cache. It must not mutate placement state.

### 8.2 Routing inputs

| Traffic | Workspace identity source |
| --- | --- |
| Browser/SDK workspace REST | `X-AgentTeam-Route-Workspace-Id`, verified against business/auth context by the target workspace server. |
| Browser realtime | Workspace-specific path/query/session context; ambiguous legacy `/socket.io` without workspace identity remains default-group only until client support is explicit. |
| Daemon websocket | Per-channel `config.workspace_id` header when present; legacy/manual configs without workspace id route to default group. |
| Internal agent/app | Authenticated token row plus optional route header. |
| LLM/runtime compatibility | Authenticated workspace context or explicit route header. |
| Provider/webhook callbacks | Central handler recovers workspace from signed state/subscription row before creating workspace side effects; not every callback goes through the router. |

A router may reject ambiguous workspace traffic instead of guessing, except for legacy/manual daemon fallback explicitly assigned to `workspace-default`.

### 8.3 Safety checks

The router/gateway is not an authorization layer. It only chooses a target group.

The target workspace server still must:

1. authenticate the request;
2. derive the authoritative workspace from session, machine key, agent token, app token, or stored provider state;
3. compare route hints with the authenticated workspace when present;
4. run `ManagedWorkspaceSet.Has(workspaceID)`;
5. fail closed with owner mismatch when the route target is wrong.

Router cache staleness is allowed only if owner mismatch remains fail-closed and retries can converge after placement refresh.

## 9. Migration workflow

The first migration implementation is operator-triggered, one workspace at a time.

### 9.1 Preflight

Before migration starts:

1. Target group metadata exists in `agentteam`.
2. Target group Deployment/Service exists in delivery.
3. Target group `/readyz` reports the expected profile and group id.
4. Current placement is `active` and source group matches operator intent.
5. No other migration is active for the workspace.
6. Entry audit checklist is complete for the production split path relevant to the workspace.
7. `agentteam-ops` verify reports no missing placements, malformed states, or unknown groups.

### 9.2 State sequence

Nominal sequence:

```text
active(source)
  -> draining(source)
  -> moving(target)
  -> active(target)
```

Step details:

1. `agentteam-ops` creates a migration request through `agentteam` admin API.
2. `agentteam` records migration metadata and increments the placement epoch.
3. Delivery confirms target group readiness.
4. Placement becomes `draining` on the source group to block concurrent migrations and signal old-connection cleanup.
5. Placement becomes `moving` with the target group. Router sends new traffic to target. Old long-lived connections on the source group are allowed to drain or are forced to reconnect by the client/runtime policy selected for that traffic class.
6. Verify confirms the target group owns the workspace and is serving ready traffic.
7. Placement becomes `active` on the target group.
8. `agentteam-ops` marks the migration `completed` and records audit metadata.

### 9.3 Rollback

Rollback is allowed while the migration is not `completed`, or after completion only if no incompatible follow-up migration has advanced the epoch.

Rollback sequence:

1. Mark migration as rollback requested.
2. Restore placement to `source_group_id` with a new `placement_epoch`.
3. Router sends new traffic back to the source group.
4. Drain target connections.
5. Run verify against the restored source group.
6. Mark migration `rolled_back` or `failed` with contextual error details.

Rollback must not silently rewrite business rows or delete group metadata.

## 10. Verify and gate strategy

### 10.1 Phase 1 verify

Phase 1 production split uses:

```bash
agentteam-tidb-admin workspace-placement-verify --workspace-group-id workspace-default
```

This command remains valid for the first production split because it is already merged and because reworking it before GA would delay the cutover.

### 10.2 Phase 2 verify

Phase 2 replaces the Phase 1 CLI with `agentteam-ops` verify/preflight backed by `agentteam` admin API.

Minimum checks:

1. Every active workspace has exactly one placement.
2. Every placement references an existing group.
3. Every group referenced by an active placement is `active` and ready.
4. No workspace is stuck in `draining` or `moving` beyond the configured timeout.
5. Router/gateway resolves sampled workspaces to the same group as `workspace_placements`.
6. Target group `/readyz` returns expected `server_profile=workspace` and `workspace_group_id`.
7. Legacy/manual daemon fallback remains intentionally mapped to `workspace-default` until a daemon upgrade/backfill gate is approved.

### 10.3 Phase 1 CLI retirement

The Phase 2 spec must track retirement of:

```text
agentteam-tidb-admin workspace-placement-verify
```

Retirement criteria:

1. `agentteam-ops` verify/preflight covers Phase 1 single-group and Phase 2 multi-group checks.
2. `agentteam-saas-delivery` calls `agentteam-ops` verify in staging and production gates.
3. Runbooks no longer reference the Phase 1 CLI.
4. A cleanup PR removes the CLI command or marks it deprecated with a removal date.

Backfill/repair commands may stay in lower-level tooling until `agentteam-ops` owns their safe replacement.

## 11. Entry audit checklist

Before production cutover and before multi-group implementation, `agentteam` maintainers must publish a route/entry audit checklist for human review.

The checklist should list every externally reachable entry family and classify:

| Entry family | Workspace identity source | Phase 1 fallback | Phase 1 action |
| --- | --- | --- | --- |
| Browser workspace REST | `X-AgentTeam-Route-Workspace-Id` from workspace-aware API helpers | none for workspace APIs | verify every workspace REST call uses workspace-aware helper. |
| Central REST | no route workspace header | central fallback | verify `workspace:false` is intentional. |
| Browser realtime | socket path/query/session workspace context | default group for legacy paths if applicable | verify routing path and owner check. |
| Daemon websocket | channel `config.workspace_id` route header | manual/legacy daemon -> default group | verify new connect and old manual connect. |
| Internal agent/app | token row plus optional route header | explicit path fallback | verify owner mismatch tests. |
| Provider callbacks/webhooks | signed state/subscription lookup | central recovery path | verify workspace side effects happen on owner group. |
| LLM/runtime routes | authenticated workspace context | workspace explicit path | verify owner group. |
| Background loops | `ManagedWorkspaceSet` filter | none | verify central does not run workspace loops. |

This checklist is a review artifact, not a substitute for route tests or delivery render validation.

## 12. Observability and readiness

Phase 2 needs enough observability to diagnose group routing and migration.

Required structured log fields where available:

- `server_profile`;
- `workspace_group_id`;
- `route_workspace_id`;
- `resolved_workspace_id`;
- `resolved_workspace_group_id`;
- `placement_epoch`;
- `route_decision`;
- `fallback_reason`;
- `owner_mismatch`;
- `migration_id`.

Metric labels must remain low-cardinality. Do not use `workspace_id` as a metric label.

Readiness checks:

1. Central readiness reports `server_profile=central`.
2. Workspace group readiness reports `server_profile=workspace` and its `workspace_group_id`.
3. Router/gateway readiness reports placement resolver connectivity and configured upstream groups.
4. Delivery gates validate readiness before and after migration.

## 13. Implementation sequencing

Recommended sequencing:

1. Finish Phase 1 production split readiness: schema migration, backfill, Phase 1 verify, delivery render/dry-run, entry audit, cutover window, and rollback plan.
2. Add Phase 2 data model and `agentteam` admin API behind feature-gated routes.
3. Add `agentteam-ops` API/CLI views over `agentteam` admin API for list groups, list placements, verify, start migration, status, and rollback.
4. Add `agentteam-saas-delivery` workspace group templates and router/gateway service skeleton.
5. Implement router/gateway for workspace REST and explicit runtime-compatible paths that already carry workspace identity.
6. Add migration state transitions and one-workspace-at-a-time migration runbook.
7. Validate a non-production migration from `workspace-default` to `workspace-large-a` and rollback.
8. Only after multi-group is stable, evaluate same-workspace multi-pod work as a separate project.

## 14. Acceptance criteria

A Phase 2 implementation satisfies this spec when:

- `central` can run with more than one replica;
- at least two workspace groups can be declared, each with `replicas=1`;
- `agentteam` is the authoritative writer for placement/group/migration state;
- `agentteam-ops` manages placement through `agentteam` admin API, not direct placement table writes;
- `agentteam` admin placement APIs are private internal-service endpoints and are not exposed through public Edge Gateway / ALB / Ingress;
- delivery can run verify/preflight through `agentteam-ops`;
- workspace router/gateway routes new workspace traffic to the group in `workspace_placements`;
- target workspace servers still enforce owner mismatch fail-closed;
- a workspace can be migrated from `workspace-default` to a second group and rolled back in staging;
- Phase 1 `agentteam-tidb-admin workspace-placement-verify` has a documented deprecation/removal path;
- same-workspace multi-pod execution remains explicitly out of scope.