# Production Deployment

This guide describes a production-oriented `agent-git-service` deployment. It
assumes one persistent TiDB database and one durable Git root per tenant,
explicit production configuration, and operational probes.

For local evaluation, use [Quick Start](quickstart.md). TiDB Zero is useful for
trying the service quickly and can be claimed from its claim URL for longer
experiments, but production deployments should use a persistent TiDB Cloud
Starter instance.

## Deployment Shape

The production baseline is:

- one or more `gh-server` processes or containers
- TLS terminated by a load balancer or reverse proxy
- `LISTEN_MODE=production`, which exposes one HTTP listener on `PORT`
- TiDB Cloud Starter databases for relational metadata
- a durable POSIX filesystem with a non-overlapping root for every tenant
- `/readyz` for readiness and `/metrics` for Prometheus scraping

Single-node deployments may use a local persistent disk. Horizontally scaled
deployments need shared POSIX-compatible storage so every instance can read and
write each configured tenant root.

### Tenant Domains

One AGS process can serve multiple fully isolated business tenants through one
shared REST, GraphQL, Git HTTP, OAuth, and service graph. Do not model the
domains as aliases for one user and data space. For example:

- `git.clawmem.ai` belongs to the Clawmem application tenant.
- `git.ags.example` belongs to an unrelated engineering-code tenant whose
  stable tenant ID is `ags`.

Each domain resolves to an immutable `TenantID` before authentication. Unknown
hosts fail with `421 Misdirected Request`; there is no default-domain fallback.
Users, databases, Git/wiki roots, encryption keys, browser policy, and login
clients are selected independently for each tenant. Database credentials and
embedding API keys may be reused across those tenant configurations. See
[First-Class Business Multitenancy](design/business-multitenancy.md).

## Prerequisites

- Go 1.25+ if building from source
- Git available in the runtime image or host
- A TiDB Cloud Starter database for each tenant
- A persistent, non-overlapping filesystem root for each tenant
- A reverse proxy or load balancer for TLS
- A secrets manager that can materialize the protected YAML file

## Prepare TiDB Cloud Starter

Create or open a TiDB Cloud Starter instance, then create one dedicated
metadata database per tenant:

```sql
CREATE DATABASE IF NOT EXISTS `ags_clawmem`;
CREATE DATABASE IF NOT EXISTS `ags_ags`;
```

Open the TiDB Cloud **Connect** dialog, choose the public MySQL connection type,
and build one Go MySQL DSN per tenant. The DSNs may reuse one database user and
password, but each must select its tenant's database. Keep `tls=true`:

```text
<shared-user>:<password>@tcp(<tidb-cloud-host>:4000)/ags_clawmem?parseTime=true&timeout=10s&tls=true
<shared-user>:<password>@tcp(<tidb-cloud-host>:4000)/ags_ags?parseTime=true&timeout=10s&tls=true
```

AGS writes an immutable tenant ownership record into each database before
running business migrations. Startup fails if a database is already owned by a
different tenant, even when two DSNs use different aliases for the same
database. Shared credentials remove database-level credential isolation: one
leaked DSN may expose every database granted to that user. Use separate
least-privilege users when that boundary is required.

## Configure Runtime

For multitenant deployments, store one complete YAML document in a secrets
manager and mount it read-only into the container. The file contains secret
values directly; it must not be stored in a ConfigMap, Git repository, container
image, or log output.

```yaml
environment: production
listen_mode: production
port: 8080

tenants:
  - id: clawmem
    base_url: https://git.clawmem.ai
    db_dsn: "<shared-user>:<password>@tcp(<tidb-cloud-host>:4000)/ags_clawmem?parseTime=true&timeout=10s&tls=true"
    git_repo_dir: /data/tenants/clawmem/repos
    secret_encryption_key: "<base64-encoded-32-byte-key>"
    browser:
      console_base_url: https://console.clawmem.ai
      oauth_device_verification_url: https://console.clawmem.ai/device-login
      cors_allowed_origins:
        - https://console.clawmem.ai
    embedding:
      api_key: "<shared-embedding-api-key>"
      base_url: https://api.openai.com
      model: text-embedding-3-small
    repository_sync:
      enabled: true
      github_api_url: https://api.github.com
      github_app_id: 123456
      github_private_key: "<secret-manager-materialized-github-app-private-key-pem>"
      poll_interval: 5s
      lease_ttl: 2m
      full_audit_interval: 168h
      max_concurrency: 8
      max_asset_bytes: 2147483648

  - id: ags
    base_url: https://git.ags.example
    db_dsn: "<shared-user>:<password>@tcp(<tidb-cloud-host>:4000)/ags_ags?parseTime=true&timeout=10s&tls=true"
    git_repo_dir: /data/tenants/ags/repos
    secret_encryption_key: "<different-base64-encoded-32-byte-key>"
    browser:
      console_base_url: https://console.ags.example
      oauth_device_verification_url: https://console.ags.example/device-login
      cors_allowed_origins:
        - https://console.ags.example
    embedding:
      api_key: "<shared-embedding-api-key>"
      base_url: https://api.openai.com
      model: text-embedding-3-small
```

Point the process at the mounted file. Logging and subsystem-specific settings
that are not part of the application schema remain independent environment
variables:

```env
AGS_CONFIG_FILE=/etc/agent-git-service/config.yaml
LOG_FORMAT=json
LOG_LEVEL=info
```

Important production notes:

- `AGS_CONFIG_FILE` is strict: unknown fields, duplicate fields, multiple YAML
  documents, and environment-reference fields such as `db_dsn_env` fail
  startup. Its top-level fields are `environment`, `listen_mode`, `port`,
  optional `trust_forwarded_host`, and `tenants`; it has no schema-version
  field.
- `AGS_CONFIG_FILE` cannot be combined with process or tenant configuration
  environment variables such as `ENVIRONMENT`, `PORT`, `TENANT_CONFIG`,
  `TENANT_ID`, `DB_DSN`, or `EMBEDDING_API_KEY`.
- Every `id` is a stable business identifier persisted into the database and
  Git-root markers. Do not derive it from a hostname that may change.
- Set a structured tenant's optional `enabled` field to `false` to take it out
  of routing without allowing another tenant to claim its configured domains.
  At least one tenant must remain enabled.
- Every tenant needs a distinct DSN selecting its dedicated database, canonical
  domain, non-overlapping Git root, persistent 32-byte encryption key, and login
  client identity. Login client secrets must also be distinct when present.
  Database credentials and embedding API keys may be shared.
- Canonical domains and aliases for different tenants must not share a
  hostname, even on different ports. Browser cookies are scoped by hostname,
  not port, so port-only separation is not a credential boundary.
- `domain_aliases` is optional and adds alternate hosts for the same tenant; it
  does not create another tenant. `trust_forwarded_host` defaults to `false` and
  should be enabled only when the ingress removes client-supplied forwarding
  headers before setting its own value.
- Tenant-owned OIDC, connected-login, embedding, workflow, repository sync,
  CORS, and console
  settings belong inside the corresponding tenant object.
- `environment: production` skips local seed data. Tenant `admin_login` and
  `admin_token` settings do not bootstrap production users.

For an explicitly single-tenant deployment, the flat format remains supported,
but `TENANT_ID` is required:

```env
TENANT_ID=ags
BASE_URL=https://git.ags.example
TENANT_DOMAIN_ALIASES=https://code.ags.example
DB_DSN=<ags-dsn>
GIT_REPO_DIR=/data/tenants/ags/repos
SECRET_ENCRYPTION_KEY=<base64-encoded-32-byte-key>
```

Flat configuration is normalized into one explicit tenant and does not create a
default tenant. `TENANT_DOMAIN_ALIASES` is an optional comma-separated list of
alternate public origins for that tenant. Every compatibility hostname must be
listed explicitly; unknown hosts still fail with `421 Misdirected Request`.
The JSON `TENANT_CONFIG` environment variable also remains available as a
transition format for existing deployments, but new multitenant deployments
should use `AGS_CONFIG_FILE`.

### Existing Clawmem State

Existing pre-multitenancy databases and non-empty Git roots require no separate
adoption step. On first startup, AGS records ownership from the tenant that
selects each resource through `db_dsn` or `git_repo_dir`. A resource with an
existing marker for another tenant always fails startup.

## Bootstrap Access

Production does not seed `octocat` or `local-dev-token`. Use one of these
supported access paths. The uppercase names below are the flat single-tenant
equivalents; a multitenant deployment configures the corresponding
`oidc` or `connected_login` object independently in each tenant entry.

- Configure OIDC for human login with `OIDC_PROVIDER`, `OIDC_ISSUER`,
  `OIDC_CLIENT_ID`, and optional `OIDC_AUDIENCE`.
  Auth0 migrations should keep `OIDC_PROVIDER=auth0` (or rely on the default
  inferred from an `*.auth0.com` issuer) so existing `UserIdentity` rows keep
  matching the same human accounts.
- Configure connected login with `CONNECTED_LOGIN_ORIGIN`,
  `CONNECTED_LOGIN_API_ORIGIN`, `CONNECTED_LOGIN_CLIENT_ID`, and
  `CONNECTED_LOGIN_CLIENT_SECRET` when a non-OIDC provider should mint local AGS
  sessions. Use `CONNECTED_LOGIN_PROVIDER`, `CONNECTED_LOGIN_LOGIN_PATH`, and the
  `CONNECTED_LOGIN_*_CLAIM` settings to describe provider-specific paths and
  claims in configuration. The callback URL is derived from the tenant
  `base_url` as
  `/auth/connected/callback`; do not configure a separate `APP_ORIGIN`.
- Register agent accounts through `POST /api/ext/v1/agents`, which returns an agent
  login, token, and default repository.

Connected login is for providers that do not expose standard OIDC
discovery and signed ID tokens, but do support browser login, authorization-code
exchange, and bearer-token userinfo. The request flow is:

1. `GET /auth/connected/login` redirects the browser to
   `{CONNECTED_LOGIN_ORIGIN}{CONNECTED_LOGIN_LOGIN_PATH}` with `client_id`,
   callback URL, and CSRF `state`. Requests made through a tenant domain alias
   first redirect to the tenant canonical domain so the host-only state cookie
   is created on the callback host.
2. The provider returns to `{tenant.base_url}/auth/connected/callback` with
   `code` and `state`. Callback requests through an alias fail with
   `421 Misdirected Request`.
3. AGS exchanges the code at
   `{CONNECTED_LOGIN_API_ORIGIN}{CONNECTED_LOGIN_TOKEN_PATH}` using
   `CONNECTED_LOGIN_CLIENT_ID` and `CONNECTED_LOGIN_CLIENT_SECRET`.
4. AGS calls `{CONNECTED_LOGIN_API_ORIGIN}{CONNECTED_LOGIN_USERINFO_PATH}` with
   the returned bearer token.
5. AGS maps configured userinfo claims into the same local identity/session
   path as OIDC.

Slock-compatible example:

```bash
CONNECTED_LOGIN_PROVIDER=slock
CONNECTED_LOGIN_ORIGIN=https://app.slock.ai
CONNECTED_LOGIN_API_ORIGIN=https://api.slock.ai
CONNECTED_LOGIN_CLIENT_ID=...
CONNECTED_LOGIN_CLIENT_SECRET=...
CONNECTED_LOGIN_LOGIN_PATH=/login-with-slock/setup
CONNECTED_LOGIN_SUBJECT_NAMESPACE_CLAIM=server_id
CONNECTED_LOGIN_SUBJECT_NAMESPACE_SLUG_CLAIM=server_slug
```

For Slock, `CONNECTED_LOGIN_PROVIDER=slock` preserves the local
`UserIdentity.provider` key. `CONNECTED_LOGIN_SUBJECT_NAMESPACE_CLAIM=server_id`
preserves the subject shape `<server_id>:<sub>` and avoids cross-server subject
collisions; when this claim is configured, Slock must return `server_id`.
Callback JSON and console redirect metadata include both `subject_namespace`
and the configured alias `server_id`, preserving the old Slock response shape
without Slock-specific code.
`CONNECTED_LOGIN_SUBJECT_NAMESPACE_SLUG_CLAIM=server_slug` only improves
generated login candidates.

These values keep their defaults unless the provider differs:

```bash
CONNECTED_LOGIN_TOKEN_PATH=/api/oauth/token
CONNECTED_LOGIN_USERINFO_PATH=/api/oauth/userinfo
CONNECTED_LOGIN_RETURN_TO_PARAM=return_to
CONNECTED_LOGIN_SUBJECT_CLAIM=sub
CONNECTED_LOGIN_ACTOR_TYPE_CLAIM=type
CONNECTED_LOGIN_HUMAN_TYPE_VALUE=human
CONNECTED_LOGIN_AGENT_TYPE_VALUE=agent
CONNECTED_LOGIN_CLIENT_ID_CLAIM=client_id
CONNECTED_LOGIN_PREFERRED_USERNAME_CLAIM=preferred_username
CONNECTED_LOGIN_NAME_CLAIM=name
CONNECTED_LOGIN_PICTURE_CLAIM=picture
CONNECTED_LOGIN_AVATAR_URL_CLAIM=avatar_url
CONNECTED_LOGIN_DESCRIPTION_CLAIM=description
CONNECTED_LOGIN_SCOPE_CLAIM=scope
```

The Slock sample is based on the historical adapter contract. Confirm the
provider's userinfo response still returns the configured claims, especially the
actor type claim, before enabling it in production.

Open agent self-registration is intentional in the current product model. If
that is not appropriate for your environment, restrict access at the reverse
proxy or network layer until an admission flow is in place.

## Optional Capabilities

### Browser Console And CORS

For a single tenant using flat configuration, set the console URL and allowed
browser origins:

```env
CONSOLE_BASE_URL=https://console.example.com
OAUTH_DEVICE_VERIFICATION_URL=https://console.example.com/device-login
CORS_ALLOWED_ORIGINS=https://console.example.com
```

When `CORS_ALLOWED_ORIGINS` is empty, the server derives local development
origins from `CONSOLE_BASE_URL`.
When `OAUTH_DEVICE_VERIFICATION_URL` is set, device-flow clients show that
external console URL instead of AGS's built-in `/login/device` fallback page.

Allowed browser origins can also read the `X-Request-Id` response header via
`Access-Control-Expose-Headers` so frontend and API clients can correlate
server responses with request-scoped logs.

### Workflow Execution

Workflow execution is fail-closed by default. These flat settings apply to a
single tenant; multitenant deployments put the equivalent values in each
tenant's `workflow` object. Enable it only when the container runtime and
sandbox limits are ready:

```env
ENABLE_WORKFLOW_EXEC=true
WORKFLOW_EXEC_IMAGE=bash:5.2
WORKFLOW_EXEC_TIMEOUT=2m
WORKFLOW_EXEC_CPUS=1.0
WORKFLOW_EXEC_MEMORY=256m
WORKFLOW_EXEC_PIDS_LIMIT=128
WORKFLOW_EXEC_NOFILE=1024
WORKFLOW_EXEC_TMPFS_SIZE=64m
```

### Semantic Search

Lexical search works without an embedding provider. These flat settings apply
to a single tenant; multitenant deployments put provider credentials in each
tenant's `embedding` object. Multiple tenant entries may reference the same API
key; each tenant still owns a separate provider client and search state. Set
them only when semantic/vector-backed issue and pull request search should be
enabled:

```env
EMBEDDING_API_KEY=<provider-api-key>
EMBEDDING_BASE_URL=https://api.openai.com
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_DIMENSIONS=
EMBEDDING_CONCURRENCY=50
```

### Git HTTP Push Limits

The normal API body limit is not used for Git pushes. Tune Git push spooling
when large repositories or memory-constrained hosts need it:

```env
GITHTTP_MAX_PUSH_BYTES=
GITHTTP_SPOOL_DIR=/data/spool
```

An empty `GITHTTP_MAX_PUSH_BYTES` uses the default 2 GiB push limit. An empty
`GITHTTP_SPOOL_DIR` uses the system temp directory.

## Run From Source

Build and run the binary with the production environment loaded by your process
manager:

```bash
make build
./gh-server
```

For a simple systemd-style deployment, mount the protected YAML file with mode
`0400` or `0440`, set `AGS_CONFIG_FILE`, and run the binary as an unprivileged
user with write access to every configured tenant Git root.

## Run As A Container

The Dockerfile builds a static `gh-server` binary and includes Git in the
runtime image:

```bash
docker build -t gh-server:local .
docker run --rm \
  -p 8080:8080 \
  -e AGS_CONFIG_FILE=/etc/agent-git-service/config.yaml \
  -v /etc/agent-git-service/config.yaml:/etc/agent-git-service/config.yaml:ro \
  -v /srv/agent-git-service/tenants:/data/tenants \
  gh-server:local
```

In Kubernetes, put `config.yaml` in a Secret and mount that single key as a
read-only file. Set only `AGS_CONFIG_FILE` on the Deployment for application
configuration; do not also inject the old application ConfigMap or Secret keys
through `envFrom`.

```yaml
spec:
  template:
    spec:
      containers:
        - name: gh-server
          env:
            - name: AGS_CONFIG_FILE
              value: /etc/agent-git-service/config.yaml
          volumeMounts:
            - name: ags-config
              mountPath: /etc/agent-git-service
              readOnly: true
      volumes:
        - name: ags-config
          secret:
            secretName: gh-server-config
            defaultMode: 0400
            items:
              - key: config.yaml
                path: config.yaml
```

Terminate TLS at the load balancer or reverse proxy and forward HTTP to the
container port.

## Verify The Deployment

Wait for readiness. The first start can take a few minutes while schema
migrations and TiDB full-text indexes are created:

```bash
curl -fsS https://git.clawmem.ai/readyz | jq .
```

Expected ready response:

```json
{
  "status": "ready",
  "version": "unknown",
  "checks": {
    "tenant_clawmem_db": {
      "status": "ok"
    },
    "tenant_ags_db": {
      "status": "ok"
    },
    "tenant_clawmem_storage": {
      "status": "ok"
    },
    "tenant_ags_storage": {
      "status": "ok"
    }
  }
}
```

Probe the GitHub-compatible discovery endpoints:

```bash
curl -fsS https://git.clawmem.ai/api/v3/ | jq .
curl -fsS https://git.ags.example/api/v3/ | jq .
curl -i -H 'Host: unknown.example.com' http://<load-balancer-address>/api/v3/
```

The first two requests must return tenant-specific canonical URLs. The unknown
host request must return `421`.

Scrape Prometheus metrics from:

```text
https://git.clawmem.ai/metrics
```

Application request and business metrics carry the validated `tenant_id` label;
raw request hosts are never used as metric labels.

The Grafana dashboard JSON lives in
[monitoring/grafana/gh-server-operations-dashboard.json](monitoring/grafana/gh-server-operations-dashboard.json).

## Backups And Upgrades

Back up both persistence layers as a tenant-scoped bundle:

- the tenant's TiDB database
- the matching tenant Git/wiki root
- the tenant ID and ownership-marker metadata needed to verify the pair

Never restore one tenant's database or Git root under another tenant ID. Keep
each pair consistent enough that repository rows and bare repositories can be
restored together. Before upgrading:

1. Back up TiDB metadata and Git repository storage.
2. Deploy the new binary or image to one instance first.
3. Wait for `/readyz` to return `ready`.
4. Run API and Git smoke tests.
5. Roll the remaining instances.

Schema migrations run at startup. During the first deployment of a new version,
allow enough startup time for migrations and full-text index changes. Startup
also backfills empty tenant provenance on persisted jobs, audit records, and
webhook deliveries from the database ownership marker; a conflicting tenant ID
is treated as corruption and prevents listeners from opening. An unmarked
database or Git root is claimed for the tenant that selects it in configuration;
an existing ownership mismatch is never overwritten.

## Production Checklist

- the protected YAML sets `environment: production` and
  `listen_mode: production`
- `AGS_CONFIG_FILE` points to a read-only Secret mount that defines every
  production tenant; no application configuration environment variables are
  mixed into the process
- each tenant has a stable ID, distinct HTTPS canonical domain, dedicated
  database selected by its DSN, non-overlapping durable Git root, and unique
  encryption key; database credentials may be shared
- TLS terminated before traffic reaches `gh-server`
- `/readyz` wired as the load balancer readiness check
- `/metrics` scraped by Prometheus
- tenant-scoped database and Git storage backups configured
- OIDC, connected login, or agent registration controls chosen independently
  for each tenant