# First-Class Business Multitenancy Status: Accepted target architecture; implementation in progress `agent-git-service` is one multitenant product. A tenant is a complete business and credential boundary inside one AGS deployment, not a domain alias and not a second single-tenant runtime hidden behind a dispatcher. This decision replaces the isolated-runtime design that constructed a complete service and handler graph for every configured consumer. ## Context `git.clawmem.ai` is the AGS backend endpoint for the Clawmem application. Unrelated engineering code is not part of the Clawmem product scope. Both workloads should use one deployed AGS product while retaining independent users, credentials, repositories, policy, and storage. The previous design selected a domain and then dispatched to one fully duplicated runtime: ```text host -> runtime A -> handlers A -> service A -> database A -> runtime B -> handlers B -> service B -> database B ``` That shape isolates state, but it does not make the AGS business layer multitenant. It multiplies handler graphs, service singletons, caches, workers, and lifecycle state. It also allows business methods to run without a tenant because the selected runtime acts as an ambient default. The product goal requires the tenant to be explicit throughout request, authentication, business, persistence, Git, and asynchronous execution paths. ## Goals The implementation is complete only when all of the following are true: 1. One process owns one shared REST, GraphQL, Git HTTP, OAuth, and business service graph. 2. Every tenant-owned business operation requires a validated `TenantContext`. 3. Domain resolution occurs before authentication and produces an immutable internal `TenantID`. 4. Users, credentials, relational data, Git/wiki data, keys, login providers, caches, locks, and background work cannot cross tenant boundaries. 5. Tenant-specific resources are selected by trusted `TenantID`, never by a raw host, request header, user-supplied identifier, or default runtime. 6. Missing, unknown, disabled, or mismatched tenant state fails closed. 7. Isolation is proven with cross-tenant negative tests over every public transport and detached-work path. ## Terminology - **TenantID**: immutable, normalized internal identifier. It is the business partition key and must not be derived again below the ingress boundary. - **TenantInfo**: immutable public policy for one tenant, including canonical URL, allowed browser origins, console URL, and enabled login flows. - **TenantContext**: a context carrying validated `TenantID` and `TenantInfo`. - **TenantResources**: tenant-owned database, Git/wiki storage, cryptographic keys, login providers, embedding provider, and worker state. - **TenantRegistry**: process-owned mapping from `TenantID` to normalized `TenantInfo` and initialized `TenantResources`. - **Domain binding**: configured mapping from an exact public host to a `TenantID`. Domains are mutable ingress aliases, not business identifiers. The term `consumer` may describe an external application in product conversation, but it is not used as the internal ownership primitive. ## Decision The request and business flow is: ```text request host -> strict TenantResolver -> immutable TenantContext -> tenant-aware authentication -> shared router and handlers -> shared business Service -> TenantResourcesProvider.ForTenant(TenantID) +-- dedicated database +-- dedicated Git/wiki root +-- dedicated secret keypair +-- tenant-configured login and embedding providers +-- tenant-scoped caches, locks, jobs, and policy ``` Only stateless implementation code and explicit process concerns are shared. Tenant-owned state is never stored in a process-global singleton. ## Tenant Context Contract `internal/tenant` owns `TenantID`, `TenantInfo`, and context helpers. It has no dependency on HTTP headers, configuration parsing, databases, or Git storage. The following rules are mandatory: - `server` is the only package that resolves a request host into a tenant. - A `TenantContext` is injected before any authentication middleware runs. - `tenant.Require(ctx)` returns the validated tenant or `ErrTenantRequired`; it has no default or fallback behavior. - Business methods do not accept a raw tenant string from request payloads. - `context.Background()` is not a valid context for tenant-owned business work. Maintenance commands and tests must bind a tenant explicitly. - Transactions may replace the database session on a context only when the session belongs to the same tenant. - Detached work copies the validated tenant onto the server lifecycle context before launching a goroutine or enqueueing a job. - Platform-only operations, such as aggregate readiness, enumerate the registry explicitly and do not impersonate a tenant. No helper may interpret absence of a tenant as the default tenant. A single-tenant deployment still receives an explicit tenant at ingress. ## Routing Contract The process has one application router. Process endpoints are registered outside tenant routing: - `/readyz` - `/metrics` Every other request passes through strict tenant resolution before request logging, authentication, CORS, REST, GraphQL, OAuth, or Git HTTP logic. - Known configured hosts resolve to exactly one `TenantID`. - Canonical domains and compatibility aliases are explicit bindings. Structured configuration uses `domain_aliases`; flat single-tenant configuration uses `TENANT_DOMAIN_ALIASES`. - Different tenants cannot bind the same browser cookie hostname, including on different ports. Domain aliases used for connected login redirect to the canonical login endpoint, and callbacks are accepted only on the canonical hostname. - Unknown or malformed hosts return `421 Misdirected Request`, including when only one tenant is configured. - When trusted forwarded-host support is enabled, the proxy-provided host is authoritative. Invalid forwarded input never falls back to `Host`. - The resolver accepts forwarded-host input only from a deployment explicitly configured to trust and sanitize its ingress proxy. - Raw hosts never become metric labels. Logs and metrics use validated `TenantID`. - Canonical URLs come from `TenantInfo`, not from request headers. There is no default tenant for HTTP routing and no `BASE_URL` data fallback. ## Business Service Contract There is one shared `service.Service`. Tenant-owned dependencies are removed from direct fields on that service and resolved through a provider: ```go type TenantResourcesProvider interface { ForTenant(tenant.ID) (*TenantResources, error) } ``` At the beginning of each public business operation, service code resolves a tenant scope from the context. That scope owns access to: - the tenant database or current transaction - Git and wiki storage - canonical URL and browser policy - encryption keys - OIDC and connected-login clients - embedding client - tenant-scoped caches, locks, and job submission Helpers such as `DBForCtx`, `GitForCtx`, and `BaseURLFor` must fail closed when the context lacks a tenant. They must never fall back to fields belonging to a default runtime. Handlers, GraphQL resolvers, OAuth endpoints, and Git HTTP code share the same service and resolve resources from the same request context. They do not retain per-tenant service pointers or handler trees. ## Persistence And Placement Tenantization of the business layer and physical database isolation are orthogonal. This design uses both. The supported placement contract is one tenant per database: - every tenant has a distinct DSN selecting its dedicated database - database credentials may be reused across tenant DSNs - users, organizations, repositories, tokens, and all other business rows are physically absent from every other tenant database - identical usernames, repository names, token shapes, and numeric IDs may exist independently Every tenant database contains a singleton metadata record: ```text ags_tenant_metadata tenant_id initialized_at ``` Startup atomically creates or verifies this record before seed data, workers, or listeners start. An unmarked existing database is claimed for the tenant that selects it in configuration; a database already claimed by a different `TenantID` causes startup to fail. This check uses database state rather than comparing DSN strings, so DNS aliases and equivalent connection strings cannot bypass the boundary. Reusing a database principal does not merge tenant state, but it does mean that database credential compromise reaches every database granted to that principal; deployments that need a database-level credential boundary should still provision separate least-privilege principals. Business identities, audit envelopes, background jobs, logs, and external events carry `TenantID`. Persisted audit records, queued work, workflow runs, and webhook deliveries store that ID as immutable provenance. Startup backfills legacy empty provenance from the database ownership marker and fails if any record names a different tenant. Creation of a provenance-bearing record additionally requires the exact registry-owned `TenantContext`; a missing tenant, different tenant, or same-ID context with different policy is rejected instead of being silently stamped from the database owner. Ordinary tenant data-plane tables inherit ownership from their exclusive database. The provenance fields above validate detached work and audit artifacts; they are not a second row-filtering mechanism. This design intentionally does not create two competing data partition mechanisms inside the same database. Shared-schema row tenancy is not supported by this decision. Adding it later requires a separate design covering composite uniqueness, foreign keys, every raw query, migrations, and database-enforced row access. ## Git And Wiki Storage Every tenant owns a non-overlapping Git/wiki root and a persistent root marker containing its `TenantID`. Startup verifies: - configured paths are not equal or nested - canonical paths and filesystem identities are distinct - the root marker is absent and can be atomically claimed, or already matches the configured `TenantID` - no root already belongs to another tenant Git repository names are unique only inside a tenant. Every Git Smart HTTP, post-push, workflow, wiki, release, and maintenance operation resolves the tenant store from `TenantContext`. ## Identity And Credential Isolation Users are tenant-owned records. There is no process-global user identity and no implicit account linking between tenants. - the same email or login may identify different users in different tenants - PATs, agent tokens, sessions, device codes, authorization codes, SSH keys, GPG keys, and deploy keys are looked up only in the selected tenant database - a credential presented on another tenant domain returns an authentication failure without searching other databases - cookies are host-only and derive secure behavior from the tenant canonical URL - different tenants cannot share a cookie hostname, even when their listeners use different ports - encryption and signing keys are distinct per tenant - development seed admin tokens are distinct per tenant - OIDC and connected-login clients are tenant-owned, including provider policy, client identity, callback URL, and claim mapping - embedded authenticators are registered per tenant; a process-global authenticator is permitted only for an explicitly single-tenant server Existing single-tenant credentials remain in the database explicitly adopted for that tenant. Credential lookup never adds an empty or global tenant scope: the selected tenant database is the credential boundary. Cross-tenant identity linking, if added later, must be an explicit platform feature with user consent and audit records. It must not merge tenant users. ## Asynchronous Work Every job and detached operation has an immutable tenant envelope. - in-process goroutines copy `TenantContext` onto the server lifecycle context - in-memory queue envelopes and persisted job records store `TenantID` - persisted jobs live in the database claimed by the same `TenantID` - workers resolve resources from the registry and compare the record envelope before reading job payload data - unknown, disabled, or mismatched tenants fail the job without using another tenant - cache keys, mutex keys, idempotency keys, and distributed locks include `TenantID` - external worker and sandbox identifiers, including workflow container names, include `TenantID` - webhooks and notifications derive canonical URLs from the job tenant - embedding calls use the tenant provider and credentials No background path may use a first, default, or previously active tenant. ## Configuration The strict YAML document selected by `AGS_CONFIG_FILE` defines process settings and all tenants. Each tenant entry has an explicit: - immutable `id` - optional `enabled: false` administrative disable state - canonical `base_url` and optional domain aliases - database DSN - Git/wiki root - encryption key - admin seed policy for development - console, device verification, and CORS policy - complete connected-login and OIDC policy - embedding policy and credentials - workflow execution policy Tenant-owned fields do not inherit from another tenant or from a process-global default in multitenant mode. Encryption keys, development admin tokens, and login client identities and secrets cannot be reused across tenants. Database credentials and embedding API keys may be reused while each tenant retains its own database, resource bundle, and provider client. The YAML is fully materialized because the file itself is mounted from a secret store. Unknown keys, duplicate keys, multiple YAML documents, and indirect secret-reference fields fail startup. Application configuration environment variables cannot be combined with `AGS_CONFIG_FILE`; subsystem-only operational variables such as logging overrides remain independent. The legacy flat environment variables remain a single-tenant input format. They require an explicit stable `TENANT_ID`, and normalization materializes exactly one tenant from them. Flat values cannot silently add a default tenant beside structured configuration. The JSON `TENANT_CONFIG` environment variable remains a transition format for existing multitenant deployments. Startup initializes all tenant resources, verifies database and filesystem ownership markers, and only then opens listeners. Disabled tenants retain their reserved IDs and domain ownership in configuration but do not initialize resources or enter the resolver; requests for their hosts fail with `421 Misdirected Request`. ## Operations Readiness reports every configured tenant resource. A deployment policy may choose aggregate readiness or per-tenant routing readiness, but a failed tenant must never route to another tenant. Maintenance commands require a `TenantID`, even in a single-tenant deployment. Backups and restores are tenant-scoped bundles containing: - the tenant database snapshot - the matching Git/wiki root - the tenant resource metadata needed to verify the pair Restoring a database or Git root under a different tenant requires an explicit import procedure that rewrites and verifies ownership markers. ## Security And Failure Model This architecture provides tenant data, identity, credential, configuration, and storage isolation inside one AGS process. It does not provide operating-system or availability isolation. Tenants still share process memory, CPU, deployment rollouts, listener availability, and the host identity. A process compromise can affect every tenant. Separate deployments remain necessary for independent compromise containment, network policy, release cadence, or failure domains. ## Verification Isolation tests create two tenants containing deliberately identical logins, repository names, numeric IDs, and credential-shaped values. They must prove: - missing and unknown tenants fail before authentication - a token, OAuth code, SSH key, or embedded identity from tenant A cannot authenticate in tenant B - REST, GraphQL, Git HTTP, OAuth, and browser flows see only their tenant data - Git and wiki writes touch only the selected tenant root - canonical URLs, redirects, CORS, cookies, notifications, and webhooks use the selected tenant policy - background jobs retain the issuing tenant after request cancellation - caches, locks, and rate-limit actors cannot collide across tenants - database and storage ownership-marker mismatches prevent startup - one shared handler and service graph serves both tenants Tests that only swap a base URL or context value on one default service do not prove multitenancy. ## Migration And Implementation Order The redesign is implemented in dependency order: 1. Introduce `internal/tenant`, strict host resolution, and missing-tenant errors. 2. Build one shared router and handler graph. 3. Introduce the tenant resource registry and make database/Git/policy access context-bound. 4. Move authentication, crypto, login, embedding, transforms, and browser policy onto tenant resources. 5. Propagate tenant envelopes through every background and maintenance path. 6. Add database and filesystem ownership markers. 7. Migrate the existing Clawmem state into its explicit tenant identity. 8. Delete duplicated runtime graphs, `internal/consumer`, ambient URL state, default-tenant fallbacks, and legacy empty credential scope. 9. Run the full cross-tenant matrix and existing compatibility gates. The final implementation must not retain both runtime dispatch and tenant-aware business routing. A mixed architecture would preserve two sources of ownership truth and is not an acceptable end state. ## Non-Goals - Tenant selection does not replace repository authorization within a tenant. - This decision does not change the `/api/v3`, `/api/graphql`, and `/api/ext/v1` namespace boundary. - This decision does not provide cross-tenant search, repository sharing, or implicit identity linking. - This decision does not claim process-level fault or exploit isolation.