# AgentTeam Gateway Server Spec

> Canonical path: `specs/deployment/workspace-gateway.md`
> Status: living-spec
> Scope: production-ready `agentteam-gateway-server` architecture for SaaS central/workspace server split and future multi-workspace-group routing.
> Related specs: [`specs/architecture/unified-agentteam-runtime.md`](../architecture/unified-agentteam-runtime.md), [`specs/api/server-profile-routing.md`](../api/server-profile-routing.md), [`agentteam-server-workspace-isolation.md`](./agentteam-server-workspace-isolation.md), [`workspace-multigroup-phase2.md`](./workspace-multigroup-phase2.md)

## 1. Decision

AgentTeam SaaS must stop using public ALB / Kubernetes Ingress as the workspace-group router. ALB remains the public host/TLS/path entry layer; workspace group selection moves into an in-cluster `agentteam-gateway-server` Deployment and Service.

Canonical component names:

| Role | Component name |
| --- | --- |
| central server | `agentteam-central-server` |
| workspace server | `agentteam-workspace-server` |
| gateway server | `agentteam-gateway-server` |

Rejected alternatives:

1. Raising the ALB quota as the primary fix. This hides the current rule explosion and keeps every new workspace group coupled to public-edge rule growth.
2. Continuing to duplicate ALB path/header rules per workspace group. This is fragile, hard to review, and cannot scale with `workspace-large-a`, `workspace-large-b`, or future dedicated groups.

Target shape for SaaS split deployment:

```text
Public ALB / Ingress
  agentteam.pingcap.com /api
    -> agentteam-gateway-server
  agentteam.pingcap.com /
    -> agentteam-web
  auth.agentteam.pingcap.com /
    -> agentteam-auth

agentteam-gateway-server
  central route family
    -> agentteam-central-server
  workspace route family + workspace id + placement
    -> agentteam-workspace-server-<group>
```

The gateway is a routing and reverse-proxy boundary only. It is not an authorization layer and must not mutate placement state.

## 2. Goals and non-goals

Goals:

- Keep ALB rule count roughly constant when new workspace groups are added.
- Make route classification reviewable in one AgentTeam-owned spec/code contract instead of spreading it across many hand-written Ingress objects.
- Route workspace traffic by `workspace_placements` while preserving AgentTeam Server owner mismatch checks.
- Support HTTP, WebSocket, daemon, and internal agent/app compatibility entries needed for the central/workspace-default production split.
- Fail closed for missing placement or unknown routes before production cutover.
- Keep single-pod and private single-binary deployments unchanged.

Non-goals:

- Same-workspace horizontal scaling. Each workspace group still has exactly one workspace owner pod until owner-hop / durable fanout exists.
- Moving browser realtime, daemon owner, agent wake, trigger fanout, or background loops out of the workspace profile.
- Replacing backend authorization. The target AgentTeam Server still authenticates and authorizes every request.
- Implementing the full Phase 2 migration state machine in the first gateway PR.
- Exposing `agentteam` internal admin APIs through the public edge.
- Making `agentteam-gateway-server` mandatory for private or single-pod deployments.

## 3. Deployment topologies

The gateway is optional and only needed when one public SaaS entry must route to more than one workspace-server group.

| Topology | Runtime shape | Gateway required? | Notes |
| --- | --- | --- | --- |
| A. Private / single-binary | One `agentteam-server` runs the existing all-in-one profile, often `profile=all`. | No | Existing private deployment and local single binary keep direct Ingress / reverse-proxy to `agentteam-server`. |
| B. SaaS single workspace server | `agentteam-central-server` + one `agentteam-workspace-server` group, or the current single `agentteam-server` before split. | Optional | Gateway may be shadowed for validation, but ALB can still route directly while there is only one workspace owner target. |
| C. SaaS multi workspace group | `agentteam-central-server` + `agentteam-workspace-server-default` + one or more named workspace groups. | Yes | Gateway prevents ALB rule count from growing linearly with workspace groups. |

Private and single-pod deployments must not start `agentteam-gateway-server` by default and must not require gateway-specific config to boot.

## 4. Code placement and entrypoint

`agentteam-gateway-server` should live in the AgentTeam repo because it must share route contracts, placement resolution code, logging, metrics, and error conventions with AgentTeam Server.

Implementation boundary:

- Add an independent entrypoint such as `cmd/agentteam-gateway-server`.
- Do not fold gateway behavior into the central or workspace server profiles.
- Do not change the default private deployment startup path.
- Reuse the route contract and placement resolver packages instead of duplicating ad hoc route tables in manifests.
- Keep gateway tests close to the route contract / placement resolver so new APIs cannot bypass classification review.

## 5. SaaS deployment model

Minimum runtime shape for SaaS split:

| Component | Profile / role | Replica rule |
| --- | --- | --- |
| `agentteam-central-server` | central server responsibilities | Start with `replicas=1` for the current rollout. |
| `agentteam-workspace-server-default` | workspace server for `workspace-default` | Must stay `replicas=1` for the group. |
| `agentteam-gateway-server` | stateless reverse proxy and placement resolver | May start with `replicas=1`; can scale after cache behavior and metrics are safe. |

The gateway must not reuse the central profile to avoid accidentally running central process dependencies in the public routing path or coupling central scaling to gateway traffic.

Recommended service names:

| Logical target | Kubernetes Service |
| --- | --- |
| central | `agentteam-central-server:3001` |
| `workspace-default` | `agentteam-workspace-server:3001` |
| future group `workspace-large-a` | `agentteam-workspace-server-large-a:3001` |
| gateway | `agentteam-gateway-server:3001` |

The group-to-service mapping should be explicit configuration at first, for example a ConfigMap consumed by the gateway:

```yaml
workspaceGroups:
  workspace-default: http://agentteam-workspace-server:3001
  workspace-large-a: http://agentteam-workspace-server-large-a:3001
central: http://agentteam-central-server:3001
```

A later implementation may read `service_ref` from group metadata, but the first production split should prefer explicit config that is easy to diff and review.

Initial `cmd/agentteam-gateway-server` runtime config:

| Environment variable | Purpose |
| --- | --- |
| `AGENTTEAM_GATEWAY_CENTRAL_UPSTREAM_URL` | Required central upstream base URL, for example `http://agentteam-central-server:3001`. |
| `AGENTTEAM_GATEWAY_WORKSPACE_UPSTREAMS` | Required comma-separated `group=url` list, for example `workspace-default=http://agentteam-workspace-server:3001,workspace-large-a=http://agentteam-workspace-server-large-a:3001`. |
| `AGENTTEAM_DEFAULT_WORKSPACE_GROUP_ID` | Default workspace group used for workspace bootstrap/configuration and, when no explicit resolver URL is set, to locate the private legacy-runtime identity resolver. It is not a public AGS compatibility owner or a fallback for unresolved workspace traffic. |
| `AGENTTEAM_GATEWAY_WORKSPACE_IDENTITY_RESOLVER_URL` | Optional explicit override for the private legacy-runtime identity resolver. When unset, the Gateway appends `/internal/meshxruntime/workspace-route-identity` to the upstream selected by `AGENTTEAM_DEFAULT_WORKSPACE_GROUP_ID`. An invalid explicit URL or a designated group without a configured upstream fails startup. |
| `AGENTTEAM_INTERNAL_ADMIN_TOKEN` | Required shared credential used only for Gateway calls to the designated owner's private identity resolver. Gateway and workspace owner must receive the same deployment-managed Secret; there is no fixed or empty production default. |
| `AGENTTEAM_DATABASE_DSN` | Read-only placement resolver source for the initial implementation. |

The resolver URL, internal credential, designated group, and designated group
upstream form one deployment unit. Rollout tooling must validate that the
derived or explicit resolver is reachable and authenticated before directing
public traffic to a new Gateway. Resolver failure never falls back to central,
the first configured workspace group, or a client-supplied route header.

AGS authority and placement are normative only in the [Request Workspace Routing
Matrix](../api/request-workspace-routing-matrix.md). Non-product AGS compatibility
routes, including discovery/bootstrap, are not exposed by the split Gateway and
do not use the configured default group. Session and repository operations use
authenticated workspace placement; public and opaque-credential discovery remain
blocked until they have a trustworthy locator. No unresolved operation defaults
or fans out.

`agentteam-gateway-server` must not require `AGENTTEAM_ADMIN_BASE_URL` or
`AGENTTEAM_ADMIN_SERVICE_TOKEN`; those belong to agentteam-ops/admin automation calling
central internals. Its `AGENTTEAM_INTERNAL_ADMIN_TOKEN` must not be shared with
MeshX admin, AGS admin, or credential-encryption keys.

## 6. Public edge contract

ALB / Ingress should route coarse public entry families only:

| Host | Path family | Target |
| --- | --- | --- |
| `agentteam.pingcap.com` | `/api` | `agentteam-gateway-server` |
| `agentteam.pingcap.com` | `/healthz`, `/readyz` | Gateway readiness or central readiness, as chosen by the cutover runbook. |
| `agentteam.pingcap.com` | `/` | `agentteam-web` |
| `auth.agentteam.pingcap.com` | `/` | `agentteam-auth` |

ALB must not contain workspace-group-specific rules such as `workspace-large-a` header conditions or per-workspace `/api/v1/workspaces/{workspaceId}/...` overrides. Adding a workspace group must be a gateway/config change, not an ALB rule-count change. Browser realtime rides `/api/v1/workspaces/{workspaceId}/events/stream` and is classified as an ordinary workspace V1 IM route, so it does not need its own ALB entry.

## 7. Gateway route classification

The gateway route table must be explicit and fail closed for unknown public entries.

The target canonical names, old-family rewrites, protocol exceptions, and
maintenance-window activation contract are defined by
[Workspace-Owned URL Canonicalization](./workspace-url-canonicalization.md).
The caller, Desktop, Gateway, and final-upstream header behavior is defined by
the [Request Workspace Routing Matrix](../api/request-workspace-routing-matrix.md).
The route table and that inventory must change together.

| Route family | Gateway decision |
| --- | --- |
| `/api/auth/*` | central |
| `/api/public/*` | central |
| `/api/v1/users/*/avatar`, `/api/v1/me/avatar`, `/api/v1/me/model-profiles*`, `/api/v1/me/behavior-settings` | central user-owned routes. |
| `/api/avatars/*` | Deprecated compatibility for historical avatar object URLs before 2026-06-12; canonical avatar reads use user/avatar resource URLs. |
| `/api/github/callback`, `/api/github/setup` | central |
| `/api/v1/workspaces`, `/api/v1/workspaces/:workspaceId/*` entity-management routes | central unless an explicit workspace data-plane subresource rule below matches first |
| `/api/orgs/*`, `/api/admin/*`, `/api/developer/apps/*` | central |
| `/api/apps/*` workspace app routes | workspace |
| `/api/llm/*` | Legacy URL compatibility rewrite to `/api/v1/workspaces/:workspaceId/llm/*`; Gateway validates workspace identity before routing. |
| `/api/v1/workspaces/:workspaceId/{attachments,stickers}/*` workspace media mutations | workspace |
| `/api/v1/workspaces/*/snapshot`, `/api/v1/workspaces/*/scopes/*/snapshot` | workspace |
| `/api/v1/workspaces/:workspaceId/{channels,threads,messages,tasks,task-board,schedules,triggers,attachments,stickers,agents,llm}/*` | workspace |
| `/api/v1/workspaces/:workspaceId/events/stream` | workspace browser realtime WebSocket. Classified as an ordinary workspace V1 IM route; no path rewrite at the gateway. |
| `/healthz`, `/readyz` | gateway or central health; must not imply workspace ownership. |
| AGS native routes | Apply the canonical routing matrix. Non-product discovery/bootstrap compatibility returns 404 at the split edge. Separately activated workspace-local, public, credential, and unsupported policies must not be inferred from a URL prefix. |
| Repository management routes such as collaborators, transfer, rename, hooks, keys, rulesets, actions secrets, and numeric environment APIs | unsupported in split mode; reject fail-closed. |
| `/api/graphql` | unsupported in split mode; reject fail-closed until a storage-aware multi-owner contract is defined. |
| `/internal/admin/*` | never public; reject if it reaches the gateway. |
| unknown public route | reject fail-closed. |

The route table should be generated from or checked against `go run ./cmd/agentteam-route-contract` once route-contract CI becomes mandatory. Until then, the spec table above is the review source of truth for gateway implementation.

## 8. Workspace identity contract

The gateway uses workspace identity only to choose an upstream. It is a route hint, not authorization.

Hard boundary:

- `agentteam-gateway-server` must use `X-AgentTeam-Route-Workspace-Id` as the canonical HTTP route header.
- `agentteam-gateway-server` must not read `X-Workspace-Id` as a long-term compatibility routing input.
- `X-Workspace-Id` remains the backend business-context header used by existing handlers; it is not the edge owner-selection contract.
- Browser API clients must set `X-AgentTeam-Route-Workspace-Id` for workspace-owned REST calls during the gateway migration.
- If both headers are present, target AgentTeam Server must still reject authenticated workspace mismatches; gateway should reject conflicting route identity once it has enough information to compare.

Workspace identity sources by traffic class:

| Traffic | Accepted identity source at gateway | Missing identity behavior |
| --- | --- | --- |
| Browser / SDK workspace REST | `X-AgentTeam-Route-Workspace-Id`. | Reject 400/421; do not default. |
| Browser realtime `/api/v1/workspaces/:workspaceId/events/stream` | workspace id in path. | Reject if path id is empty or malformed. |
| Daemon websocket | workspace id from connect token or per-channel `config.workspace_id` propagated as `X-AgentTeam-Route-Workspace-Id`; legacy `/daemon/*` may resolve the machine key. | Reject when neither an explicit workspace nor a valid machine credential association exists. |
| Internal agent/app | `X-AgentTeam-Route-Workspace-Id` or the authenticated Agent/App token association used by the legacy runtime resolver. | Reject invalid credentials as 401 and unresolved identities fail closed; never default. |
| AGS REST/raw/Git | See the canonical routing matrix: authenticated session operations require an authorized claim and placement; public and opaque-credential discovery do not default. | Preserve native protocol behavior or reject before upstream when authority is unresolved. |
| Provider callbacks / webhooks | central recovers workspace from signed state/subscription row; not every callback goes through gateway workspace routing. | Central handles or rejects. |

The target AgentTeam Server must compare route hints with the authenticated workspace when present and still run `ManagedWorkspaceSet.Has(workspaceID)`.

## 9. New API and workspace lifecycle guardrails

The main correctness risk is not the extra pod. The risk is an API or client path that forgets to carry workspace identity or is not classified by the route contract.

Required guardrails:

1. New public API route families must be classified as central, workspace, gateway health, web, auth, or internal-only before merge.
2. Unknown gateway routes must fail closed instead of defaulting to central or `workspace-default`.
3. Workspace creation must synchronously create a default `workspace_placements` row for the new workspace.
4. Workspace switching in the web app must update a single API client source of truth that sends `X-AgentTeam-Route-Workspace-Id` and uses `/api/v1/workspaces/{workspaceId}/events/stream` for browser realtime.
5. Staging shadow validation must compare gateway decisions with live routing for sampled REST, events/stream realtime, daemon, internal agent/app, and central routes.
6. Production cutover must be blocked if placement backfill or verify reports any missing, non-default, inactive, or not-routable workspace.
7. AGS operation capabilities must derive from registration-owned method-plus-pattern metadata. Gateway composition separately owns the operation-and-authority-mode activation overlay; an empty overlay creates no compatibility handler or route table, while coverage proves every future activated key maps to one real registration.
8. `agentteam-ops` calls AgentTeam placement admin through the private central Service and calls MeshX admin through its private Workspace Service. Neither admin client may use the public Gateway.

## 10. Placement resolution

The gateway resolves workspace id to workspace group through one of these approved mechanisms:

1. A private AgentTeam central/internal API that reads `workspace_placements`; preferred once available because it centralizes schema and future group metadata.
2. A read-only DB resolver limited to `workspace_placements` and optional group metadata; acceptable for the first gateway implementation if the API is not ready.

Rules:

- The gateway must not insert, update, or repair placement rows.
- Cache entries may use a short TTL, for example 5-30 seconds.
- Cache staleness is safe only because the target workspace server still fails closed on owner mismatch.
- Negative cache TTL must be short to avoid prolonging recovery after backfill.
- `state=active` routes to the placement group.
- `state=draining` or `state=moving` must follow the migration spec before production migration is enabled; until then, reject as not routable.
- `state=disabled` rejects.
- Placement group without a configured upstream rejects and logs a configuration error.

## 11. Missing placement policy

Production cutover policy: missing placement must fail closed 4xx. The gateway must not silently fallback missing placements to `workspace-default`.

Rationale:

- `workspace_placements` is the source of truth for owner routing.
- Silent defaulting hides data repair gaps and can route a workspace to the wrong owner once more groups exist.
- The production gate already requires `workspace-placement-backfill` followed by `workspace-placement-verify --workspace-group-id workspace-default`; traffic should not cut over until missing count is zero.

Staging shadow policy:

- Shadow mode may observe and log missing placement decisions without receiving public traffic.
- If a request is mirrored or replayed and placement is missing, the shadow result records `missing_placement` but must not affect the live response.
- A temporary fallback may exist only in shadow-only tooling and must not be enabled on the public gateway path.

HTTP status guidance:

| Condition | Suggested status | Error code |
| --- | --- | --- |
| Missing workspace identity on workspace route | `400 Bad Request` | `missing_route_workspace_id` |
| Malformed workspace identity | `400 Bad Request` | `invalid_route_workspace_id` |
| Missing placement | `404 Not Found` or another 4xx chosen by implementation | `workspace_placement_missing` |
| Placement state not routable | `409 Conflict` or `421 Misdirected Request` | `workspace_placement_not_routable` |
| Unknown route classification | `404 Not Found` | `unknown_gateway_route` |
| Upstream group not configured | `503 Service Unavailable` | `workspace_group_unavailable` |

Error bodies must not expose credentials, internal DSNs, or raw upstream URLs.

## 12. Browser realtime streaming path

The browser realtime entry is the streaming WebSocket:

```text
/api/v1/workspaces/{workspaceId}/events/stream
```

Why this shape:

- Browser WebSocket upgrades cannot reliably carry custom route headers through every client/proxy path.
- The gateway needs workspace identity before it can choose the owning workspace group.
- The streaming route lives under the standard `/api/v1/workspaces/{workspaceId}/...` workspace V1 IM family, so the gateway classifies it via the same code path as the rest of the workspace REST surface — no realtime-specific ALB rule or gateway special case.

Gateway behavior:

1. Extract `{workspaceId}` from the path.
2. Resolve placement.
3. Forward to the owning workspace service with the path preserved.
4. Preserve WebSocket upgrade headers and query parameters.

There is no legacy `/socket.io` browser entry. Earlier compatibility windows for a Socket.IO edge have been removed; the streaming WebSocket is the only browser realtime contract and is treated as an ordinary workspace V1 IM route by both ALB and gateway.

## 13. Daemon workspace-id contract

New daemon connect flows must carry workspace identity.

Required behavior:

1. Channel configs that include `config.workspace_id` must send `X-AgentTeam-Route-Workspace-Id` during daemon websocket and runtime-compatible HTTP calls.
2. Connect-token exchange should include or derive `workspaceId` so daemon config can persist it.
3. Manual or legacy daemon configs without workspace id remain compatible only
   when their machine key resolves to exactly one authoritative workspace.
4. Missing or invalid machine credentials return 401; an unresolved identity
   never maps to `workspace-default`.
5. Before cutover, smoke tests must cover both explicit-workspace and
   credential-only legacy daemon requests.

The target workspace server remains responsible for authenticating machine keys / agent tokens and verifying owner membership.

## 14. Resource requests and ResourceQuota baseline

For the initial SaaS split, resource requests for `agentteam-central-server`, `agentteam-workspace-server`, and `agentteam-gateway-server` should be based on current production observed peak usage multiplied by 1.5, then adjusted after 1-2 weeks of staging/production observation.

Current decision:

- Do not copy staging's large test reservations into production.
- Start `agentteam-central-server` as `1/1` for this rollout.
- Increase requests/limits later if observed load requires it.
- Keep production unchanged until staging gateway shadow and cutover validation are stable.

This section is a sizing policy for deployment manifests; it does not itself
set staging or production resources.

## 15. Observability and readiness

Gateway structured logs should include low-risk routing fields:

- `route_family`;
- `route_decision`;
- `route_workspace_id`;
- `resolved_workspace_group_id`;
- `placement_epoch` when available;
- `fallback_reason`;
- `upstream_service` logical name;
- `owner_mismatch` when propagated from upstream;
- `request_id`.

Metrics must remain low-cardinality. Do not use `workspace_id`, API keys, tokens, or raw path parameters as metric labels. Allowed labels include `route_family`, `route_decision`, `fallback_reason`, `status_class`, and `upstream_group` only if group cardinality is bounded by configured workspace groups.

Readiness:

- `/healthz` checks process liveness.
- `/readyz` checks central upstream config, placement resolver connectivity, and every configured workspace group upstream readiness.
- Gateway readiness must fail if a configured group lacks an upstream service.

## 16. Staging and production sequencing

Required sequence:

1. Gateway spec PR reviewed and merged, including naming, deployment topologies, optional gateway scope, route classification, missing-placement behavior, daemon workspace-id contract, and streaming browser-realtime classification.
2. Staging deploys `agentteam-gateway-server` in shadow/internal mode. Public ALB rules stay unchanged during shadow.
3. Staging validates gateway decisions against live ALB decisions for sampled REST, events/stream realtime, daemon, internal agent/app, and central routes.
4. Staging ALB changes to coarse gateway entry and removes per-group `workspace-large-a` duplicated rules only after shadow validation passes.
5. Staging multi-group shadow uses `workspace-placement-topology-verify`, not the Phase 1 default-only `workspace-placement-verify --workspace-group-id workspace-default`; the topology gate must fail on missing placements, unknown groups, inactive groups, inactive placements, malformed placements, or active migrations.
6. Production completes `workspace-placement-backfill` and automated `workspace-placement-verify --workspace-group-id workspace-default` with zero missing, non-default, inactive, and not-routable counts while production remains single-default before cutover.
7. Production agentteam-ops admin config lands: `AGENTTEAM_ADMIN_BASE_URL` points at internal central service, and `AGENTTEAM_ADMIN_SERVICE_TOKEN` is available through secret-managed config.
8. Production single PR atomically deploys central, workspace-default, gateway, coarse Ingress, backfill/verify gate, ResourceQuota baselines, and agentteam-ops admin config.

During this sequence, staging and production live manifests must not be modified outside reviewed GitOps changes. Production must not change before staging shadow and staging coarse-ingress cutover are stable.

## 17. Guardrails

Before a production gateway cutover PR can be reviewed:

1. Render validation must fail if any public Ingress contains workspace-group-specific ALB conditions or paths for a named non-default workspace group.
2. Render validation must fail if `/internal/admin/*` is exposed publicly.
3. Route contract validation must ensure every public path family is classified as central, workspace, gateway health, web, auth, or internal-only.
4. Gateway tests must cover:
   - central exception outranks workspace route hints;
   - workspace REST missing `X-AgentTeam-Route-Workspace-Id` fails closed;
   - gateway does not route by `X-Workspace-Id`;
   - missing placement fails closed 4xx;
   - configured `workspace-default` routes to the default workspace service;
   - `/api/v1/workspaces/:workspaceId/events/stream` is classified as a workspace route without a realtime-specific special case;
   - legacy daemon requests resolve only through explicit workspace or machine credentials;
   - unknown public route fails closed.
5. Delivery budget validation must prove adding a workspace group changes only gateway config / workspace Service resources, not ALB rule count.
6. Private / single-pod deployment validation must prove `agentteam-gateway-server` is not required for `profile=all` or private single-binary startup.

## 18. Acceptance criteria

This spec is satisfied when:

- ALB routes coarse entry families to `agentteam-gateway-server` instead of duplicating rules per workspace group.
- Gateway routes central and workspace families according to the route classification table.
- Missing workspace identity, missing placement, unknown route, and unavailable group all fail closed with contextual but safe errors.
- New workspace groups can be added without adding public ALB rules.
- Target workspace servers still enforce `ManagedWorkspaceSet` owner checks.
- Staging shadow validation proves gateway decisions match intended routing before public traffic is cut over.
- Staging multi-group shadow uses topology placement verify, and production cutover remains blocked until production default placement verify, ResourceQuota baseline, and agentteam-ops admin config are green.
- Single-pod and private single-binary deployments remain gateway-free and unchanged by default.