# drive9 API Key Management Design: A Single /v1/tokens Resource Surface (Data Plane + Control Plane, Two Credential Types)

> **Scope**: server side only (this repository). Client changes (CLI/SDK) are out of scope.
> **Audience**: implementers and maintainers. Every design point is presented as "Decision / Rationale / Alternatives" so each one can be reviewed individually.

---

## 0. Requirements and Summary of Conclusions

Requirement A: use TiDB Cloud control-plane API keys (public key + private key, resolved via IAM into an `org:owner` / `project:owner` identity) to manage a tenant's data-plane drive9 API keys: generate, delete, activate, deactivate; list is included for operability.

Requirement B: a data-plane drive9 API key refreshes itself using itself as the credential (rotation / lease renewal — extending its expiry).

Summary of conclusions (details in the referenced sections):

- **Single resource surface**: no new route family is introduced. `/v1/tokens` carries every operation — data-plane scoped issuance (status quo), control-plane generate/list/activate/deactivate/delete, and data-plane self-refresh (§3, §4, §5).
- **Two-credential dispatch**: the `/v1/tokens` mux registration becomes a credential dispatcher — a Bearer `drive9_*` token follows the existing `tenantAuthMiddleware` semantics; control-plane credentials (body `public_key`/`private_key` first, falling back to the `X-TiDBCloud-Public-Key`/`X-TiDBCloud-Private-Key` headers) go through IAM identity resolution + provider-fork organization authorization. Neither credential present → 401; **both present → 400** (ambiguous requests are never silently interpreted) (§3.2, §3.3).
- The only entry point for scoped tokens is `POST /v1/tokens/refresh` (the single authoritative gate lives inside `handleTokens`); every other `/v1/tokens*` path remains 403 for scoped tokens. The control-plane path does not pass through the Bearer middleware, so it is structurally unreachable by scoped tokens (§3.6, §4.1).
- Response fields are unified across the whole family as `token` / `token_id` (§4.1).
- Carried over unchanged: the `active/disabled/revoked` state machine (`expired` derived), the `expires_at` ↔ JWT `exp` write-together invariant with lazy backfill for legacy rows, the `SetAPIKeyStatus` / `RotateAPIKey` contracts, the capacity cap (generate + scoped issue + slock rotation), list defaulting to non-expired rows + `include_expired`, schema evolution and the security design (§2.2, §5, §6, §7).

---

## 1. Current State and Gaps

### 1.1 What "generate" means today

There are currently two independent "generation" paths in the code:

1. **Provision generates the owner key**. `handleProvision` (`pkg/server/server.go:5052`) calls `issueOwnerAPIKey` (`pkg/server/server.go:6476`) → `buildOwnerAPIKey` (`pkg/server/server.go:6441`) during tenant provisioning: it issues a JWT **with no expiry** (`token.IssueTokenWithJournalPermissions(..., time.Time{}, ownerJournalPermissionList())` hard-codes a zero `expiresAt`), encrypts it via `s.pool.Encrypt`, and writes it into `tenant_api_keys` with `scope_kind = owner`, `status = active`. The issuance source is recorded in the `issued_by_provider` / `issued_by_subject_key` fields of `apiKeyIssueSource` (`pkg/server/server.go:5444`). The plaintext token is returned only once, in the provision response. The same function is also reused by admin pool backfill (`pkg/server/admin_tenant_pool.go:2547`) and slock rotation (`pkg/server/slock.go:664`); `buildOwnerAPIKey` has two additional direct call sites (`server.go:5851` shared provision, `admin_tenant_pool.go:2606` shared pool claim), all producing owner keys with no expiry.
2. **`/v1/tokens` issues scoped tokens**. `handleTokens` (`pkg/server/tokens.go:46`) → `handleScopedTokenIssue` (`pkg/server/tokens.go:101`): callable only by owner (`ownerScopeFromRequest`, `pkg/server/tokens.go:85`, invoked at `tokens.go:51` **before any path dispatch**). It builds a JWT with an `exp` claim from the requested `ttl_seconds` (`tokens.go:135-136`), but **no expiry column is persisted in the DB** (`tenant_api_keys` currently has no `expires_at` column; see the column list of `InsertAPIKey` at `pkg/meta/meta.go:1885-1892`) — the expiry exists only in the JWT `exp` claim. The TTL ceiling is `maxScopedTokenTTLSeconds` (`pkg/server/tokens.go:44`, ≈ 292 years — effectively "no ceiling"). Scopes are written to `tenant_api_key_fs_scopes` (`InsertAPIKeyFSScope`, `pkg/meta/meta.go:3317`).

Revocation paths: `POST /v1/tokens/revoke` revokes by plaintext api_key (`handleScopedTokenRevokeByAPIKey`, `pkg/server/tokens.go:220`); `DELETE /v1/tokens/{id}` revokes by id (`handleScopedTokenRevoke`, `pkg/server/tokens.go:207`). Both bottom out in `RevokeAPIKey` (`pkg/meta/meta.go:4002`): a conditional UPDATE + `insertTenantNotifyTx(TenantNotifyWorkAPIKeyCacheCleanup)` for cross-pod cache cleanup + in-process `evictTenant` (`pkg/meta/api_key_cache.go:136`). The bulk convergence paths `RevokeTenantAPIKeys` (`meta.go:3974`) and `RevokeAPIKeysByIssuer` (`meta.go:4035`) are likewise conditioned on `status='active'` and are called widely from tenant deletion, fork cleanup, admin tenant operations, pool cleanup, and failed-tenant cleanup (`tenant_delete.go`, `fork.go:974,994`, `admin_tenants.go:405,451`, `admin_tenant_pool.go:1753,1851,1898,1928`, `tenant_failed_cleanup.go:356`).

Routing and authentication structure (the target of this redesign):

- `/v1/tokens` and `/v1/tokens/` are registered on the `business` mux (`pkg/server/server.go:688-689`); the **entire family** passes through `tenantAuthMiddleware` (`pkg/server/auth.go:212`): Bearer hash lookup (`ResolveByAPIKeyHash`, 10s TTL in-process cache) → constant-time hash comparison → reject when `status != active` (`auth.go:258`) → decrypt-and-compare → JWT signature verification plus `tenant_id`/`token_version` checks → build the `TenantScope` (`auth.go:407`).
- Scoped tokens are subject to two gates: the `isScopedBusinessRequestAllowed` check at the entry of `handleBusiness` (invoked at `server.go:1790`, defined at `server.go:1951`) ends in default-deny (`server.go:2004-2005`; today the entire `/v1/tokens*` family is 403 for scoped tokens); and inside `handleTokens`, `ownerScopeFromRequest` runs before dispatch (`tokens.go:51-54`).
- Existing uses of control-plane credentials: the `X-TiDBCloud-Public-Key` / `X-TiDBCloud-Private-Key` headers (constants at `pkg/server/quota.go:74-75`) and body `public_key`/`private_key` (`adminCredentials`, `pkg/server/admin_tenants.go:586-599`, **body first, header fallback**), used by quota (`quota.go:96-149`), admin tenants (`admin_tenants.go:455-523`), and provision (`server.go:5350`); identity resolution via `resolveTiDBCloudIdentity` (`pkg/server/tidbcloud_iam.go:14`), with organization-belonging authorization forked by provider (native org binding / shared `GetSharedDBForTenant`, `admin_tenants.go:470-502`).

### 1.2 What is missing

| Capability | Current state | Gap |
|---|---|---|
| Control-plane management of data-plane keys | None. The whole `/v1/tokens` family accepts only data-plane Bearer tokens; control-plane credentials are used only by provision / quota / admin tenants | Introduce credential dispatch and a control-plane branch on `/v1/tokens` (§3) |
| Provider fork in control-plane authorization | quota/admin already fork (native binding / shared `GetSharedDBForTenant` / free pool exclusion); shared tenants **do not write** `tenant_tidbcloud_org_bindings` (`TestUpsertTenantTiDBCloudOrgBindingRejectsSharedTenant`, `pkg/meta/meta_test.go:1902`) | The control-plane branch must reuse the same fork (§3.4) |
| list | None. The meta layer only has `GetActiveAPIKeyByIssuer` (`pkg/meta/meta.go:4064`), a single-row lookup by issuer | Add `ListAPIKeys` and `GET /v1/tokens` |
| activate / deactivate | None. `APIKeyStatus` has only `active` / `revoked` (`pkg/meta/meta.go:80-84`) | Extend the state machine (add `disabled`) and widen the `Revoke*` convergence conditions accordingly (§2.2) |
| refresh / lease renewal | None. JWT supports `exp` (`token.go:121-127`), but owner keys have no expiry and scoped expiry is not persisted | Add `POST /v1/tokens/refresh`, the `expires_at` column, and the precise scoped-gate allowance (§5) |
| `expires_at` ↔ JWT `exp` consistency | Scoped issuance writes only the JWT, never the column; no invariant exists | Define the invariant and cover issuance / refresh / legacy rows (§5.3, §6.4) |

---

## 2. Terminology and Model

### 2.1 Terminology

- **owner key**: a row in `tenant_api_keys` with `scope_kind = 'owner'`. Created at provision time (§1.1-1); holds full data-plane privileges for the tenant plus data-plane management rights on `/v1/tokens`. Currently has no expiry.
- **scoped token**: a row with `scope_kind = 'fs_scoped'`, issued by an owner via `POST /v1/tokens`, carrying a `prefix + ops` allowlist (`tenant_api_key_fs_scopes`) and a mandatory TTL (`tokens.go:112-115` requires `ttl_seconds > 0`). **Expiry is an intrinsic security property of scoped tokens; no path may remove it.**
- **control-plane apikey (TiDB Cloud API key)**: a TiDB Cloud IAM public key + private key pair, **never persisted by drive9**; used only as a request credential and resolved via `resolveTiDBCloudIdentity` (`tidbcloud_iam.go:14`) into a `TiDBCloudAPIKeyIdentity` (`pkg/tenant/provisioner.go:53`, which carries only `OrganizationID` and `Role`); the role must be `org:owner` or `project:owner` (`tidbcloud_iam.go:35`).
- **tenant / org belonging**: for dedicated (`tidb_cloud_native`) tenants the org belongs in `tenant_tidbcloud_org_bindings` (DDL at `pkg/meta/meta.go:932`); for shared (`tidb_cloud_native_shared`) tenants it lives in `db_pool.org_id`, resolved via `GetSharedDBForTenant` (`db_pool.go:1016`). Free pool members (`tenant_pool_memberships.pool_status='free'` / binding `pool_status='free'`) are pooled tenants not yet claimed; they belong to no caller's org.
- **The three caller principals**: `owner` (data-plane owner key), `scoped` (scoped token), `control_plane` (control-plane identity). The single `/v1/tokens` resource surface determines the available operation set per principal (identity matrix in §4.1).

### 2.2 API key lifecycle state machine

**Decision**: `status` is extended to three stored states — `active` / `disabled` / `revoked`; `expired` is not stored, it is a derived state. `revoked` is terminal.

```
                generate / provision / issue
                      │
                      ▼
        ┌─────────► active ◄─────────┐
        │             │              │
        │      deactivate      activate
        │             │              │
        │             ▼              │
        │          disabled ─────────┘
        │             │
   delete/revoke      │ delete/revoke
        │             │
        ▼             ▼
        revoked (terminal; cannot be revived)

  Derived state (never written to the status column):
  expired := status='active' AND an effective expiry exists AND now >= effective expiry
  See §5.3 for the resolution order of "effective expiry" (expires_at → JWT exp → none).
  JWT-level exp expiry is rejected by token.ParseAndVerifyToken at auth.go:287 with 401.
```

**Convergence of state-transition primitives (decision)**:

- activate / deactivate: go through `SetAPIKeyStatus` (§8.2), allowing only `active ↔ disabled` (both `from` and `to` are forbidden from being `revoked`; contract in §5.5).
- delete / tenant teardown / issuer bulk revocation: unified under the revoke primitives. The conditions of `RevokeAPIKey`, `RevokeTenantAPIKeys`, and `RevokeAPIKeysByIssuer` are widened from `status = 'active'` to **`status IN ('active','disabled')`** (three sites: `meta.go:3981`, `4010`, `4042`) — a disabled key must be revocable via delete and must be converged by tenant-teardown paths, otherwise revivable ciphertext rows would be left behind.
- Idempotency semantics per primitive (unchanged from status quo; bulk semantics untouched): the single-entity primitive `RevokeAPIKey` returns `ErrNotFound` for already-`revoked` or missing rows (→ delete maps to 404); the bulk primitives `RevokeTenantAPIKeys` (`meta.go:3990-3992`) and `RevokeAPIKeysByIssuer` (`meta.go:4052-4054`) return `nil` when zero rows match (tenant-teardown / issuer-cleanup paths are naturally idempotent and re-entrant).

**Rationale**:

- The existing check at `auth.go:258`, `resolved.APIKey.Status != meta.APIKeyActive → 401`, naturally rejects any new non-`active` status, so `disabled` requires no middleware change, and old binaries during a rolling upgrade also correctly reject disabled keys (gradual-rollout compatibility, see §9).
- `disabled` vs `revoked` have distinct semantics: the former is reversible (temporary operational freeze, suspected-leak observation window), the latter is not (confirmed leak / deliberate destruction / tenant teardown). The `revoked_at COALESCE` pattern in `RevokeAPIKey` (`meta.go:4009`) already treats revoked as terminal.
- Deriving `expired` avoids introducing a background sweeper; the JWT layer already rejects expired tokens.

**Alternatives considered**:

| Option | Trade-off | Verdict |
|---|---|---|
| Separate `deactivated` and `disabled` states | Duplicated semantics, one more verb on the API surface | Rejected; merged into `disabled` |
| Store `expired` as a stored state | Requires a sweeper worker; dual writes with JWT `exp` could diverge | Rejected; derived state |
| `revoked` revivable (via activate) | Revocation must mean "irreversible" | Rejected; revoked is terminal |
| Keep revoke condition at `status='active'`, implement delete via `SetAPIKeyStatus(to=revoked)` | Disabled keys could not be closed out by delete / tenant teardown, and the two paths would diverge semantically | Rejected; widen the `Revoke*` conditions and forbid `SetAPIKeyStatus` from touching revoked |

---

## 3. Routing Model: Single Resource Surface with Two-Credential Dispatch

### 3.1 The single-resource-surface decision

**Decision**: no new route family is introduced. All data-plane and control-plane operations on `tenant_api_keys` are unified under the `/v1/tokens` family.

**Rationale**: both caller types manage the same tokens in the same table — the resource is one and the same; only the caller's credential differs. Splitting two route families by credential produces duplicated endpoint tables, duplicated documentation, and the lasting confusion of "one resource, two URIs". In REST terms the resource surface should stay singular; the difference belongs in the authentication and authorization layers.

**Alternatives**: a separate `/v1/api-keys` family — rejected, same reason (resource-surface split). "Mount control-plane operations under admin routes (`/v1/admin/...`)" — rejected: the admin surface is pooling/quota management; tokens are a tenant resource and would split the resource surface there too.

### 3.2 Mux registration and the credential dispatcher

**Decision**: `/v1/tokens` is **moved out of** the `business` mux (delete the existing registration on `business` at `server.go:688-689`, and delete the tokens dispatch case inside `handleBusiness` (`server.go:1811-1812` region) — avoiding double-registration dead code), and re-registered with an independent credential-dispatch assembly:

```go
var tokens http.Handler = http.HandlerFunc(s.handleTokens)
if cfg.Meta != nil && cfg.Pool != nil && len(cfg.TokenSecret) > 0 {
    tokens = s.tokensCredentialDispatcher(cfg, tokens)
}
mux.Handle("/v1/tokens", tokens)
mux.Handle("/v1/tokens/", tokens)
```

**The single assembly (decision)**:

```text
mux → tokensCredentialDispatcher
  ├─ Bearer branch        → tenantAuthMiddleware (logic unchanged) → handleTokens
  └─ Control-plane branch → controlPlaneTokenAuth (IAM identity)   → handleTokens (ControlPlaneScope)
```

`handleBusiness` no longer appears on the `/v1/tokens` path — the `isScopedBusinessRequestAllowed` check at its entry (`server.go:1790`) is therefore **no longer** the scoped gate for tokens; the scoped gate moves into `handleTokens` (§3.6). `isScopedBusinessRequestAllowed` is **not modified at all** (tokens are no longer on the business path; adding an allow-line there would only create dead code).

**Key points**:

- The Bearer path is **behaviorally equivalent** to today — `tenantAuthMiddleware` is reused as-is; only the wrapping position moves from mux-global to inside the dispatcher's Bearer branch.
- The control-plane path **does not pass through** `tenantAuthMiddleware`, structurally guaranteeing that scoped tokens cannot reach the control-plane branch (§3.6).
- GET requests carry no body, so control-plane list credentials can only travel via the `X-TiDBCloud-*` headers (§3.3); this matches quota GET's production usage (`decodeQuotaGetRequest`, `quota.go:209-215`).

**Alternatives**: route both credential types through `tenantAuthMiddleware` and recognize control-plane credentials inside the middleware — rejected. Every step of the middleware (hash lookup, decrypt-and-compare, JWT verification, `pool.Acquire`) presumes a drive9 token; a dual-mode retrofit carries far more regression risk than a dispatcher.

### 3.3 Credential recognition: peek-and-restore contract and conflict handling

**Body-sniffing contract (decision)**: dispatch-time body sniffing is performed by a dedicated function `peekTokenCredentials(r)`. Reusing `quotaCredentials` / `decodeQuotaRequest` is **forbidden** (they require `tenant_id` and apply `DisallowUnknownFields`, `quota.go:345-370` — they would kill legitimate Bearer issue/revoke bodies). Follow the repository's existing "bounded read + restore body" pattern in `rejectCredentialProvisionBody` (`server.go:5420-5439`):

```text
peekTokenCredentials(r) (called exactly once, at dispatch time):
  1. hasBearer := bearerToken(r) != ""        // Authorization: Bearer or X-API-Key (auth.go:559)
  2. headerControl := either X-TiDBCloud-Public-Key or X-TiDBCloud-Private-Key is non-empty
  3. bodyControl := false
     // Skip body sniffing when headers already matched (header-first, to shrink the
     // unauthenticated parsing surface)
     if !headerControl && method ∈ {POST, PUT, PATCH, DELETE}:
         buf := read the entire body through http.MaxBytesReader(w, r.Body, maxCredentialProvisionBodyBytes)
         // over the limit → 400 "request body too large" (bounded, safe even unauthenticated, no OOM)
         r.Body = io.NopCloser(bytes.NewReader(buf))   // restore is mandatory so downstream
                                                       // handlers can decode normally
         lenient decode (plain json.Decoder; no DisallowUnknownFields, no tenant_id requirement,
                         unknown fields and format errors tolerated) extracting only
                         public_key / private_key
         either field non-empty after TrimSpace ⇒ bodyControl = true
         // malformed JSON ⇒ bodyControl = false (not an error — the body may simply be
         // unexpected content of a Bearer request, handled by the downstream handler's own
         // decoding rules; strict validation happens only if the control-plane branch
         // is actually selected)
  4. hasControlCred := headerControl || bodyControl
  5. Dispatch:
     hasBearer && hasControlCred → 400 "ambiguous credentials"
     hasBearer                   → Bearer branch
     hasControlCred              → control-plane branch: only now validate pairing/format
                                   via adminCredentials (body first, header fallback,
                                   admin_tenants.go:586-599)
     neither                     → 401, reusing the current message
                                   "missing or malformed Authorization header" (auth.go:227)
```

**`controlPlaneTokenAuth` scope of responsibility (decision)**: it performs only IAM identity resolution and produces `ControlPlaneScope{Identity, OrganizationID}` (`Tenant = nil`). **Explicit tenant authorization** (`resolveTiDBCloudAuthorizedTenant`, §3.4) runs inside each handler, after the business decode has produced `tenant_id`. The dispatcher neither parses `tenant_id` nor consumes the body a second time (peek operates on the restored buffer; handlers read the complete original body). The `needTenant` parameter is deleted.
Alternative: pre-resolve the tenant in the auth layer — rejected; it forces the dispatcher to know where each endpoint keeps `tenant_id` (body/query) and duplicates the handler's decoding.

**Conflicting credentials (decision, locked)**: Bearer and control-plane credentials **present together → 400 `ambiguous credentials`** (including "Bearer + public_key/private_key inside the body" — body sniffing exists precisely to cover this shape; checking only headers would miss it and contradict the goal of never silently interpreting an ambiguous request).
Rationale: the two credential types point at different authorization models (token bound to a tenant vs IAM org identity); co-occurrence means the client was constructed incorrectly. Silently preferring one (e.g. Bearer-first — quota's production behavior is a silent Bearer-first branch, `quota.go:105-118`) would mask "wrong credential used" as a permissions debugging nightmare. The tokens surface returns an explicit 400; the difference from quota is documented in the API docs.

**Neither credential → 401**, reusing the current message `missing or malformed Authorization header`.

**Security boundary of pre-authentication body parsing (decision)**: body sniffing does happen before authentication — that is the inherent cost of supporting body credentials. Mitigations: ① a hard `MaxBytesReader` ceiling (≤ `maxCredentialProvisionBodyBytes`) — over the limit returns 400 immediately without allocating beyond the bound; ② body sniffing is skipped when headers matched; ③ the lenient decode scans only two fields with no schema validation; ④ brute-force traffic on this path is bounded by gateway rate limiting (§9.4).

### 3.4 Control-plane identity resolution and organization authorization (provider fork)

Every request in the control-plane branch executes the following steps in order:

1. `resolveTiDBCloudIdentity(ctx, cred, "tokens")` (`tidbcloud_iam.go:14`): served through the `tidbCloudRBACCache` (existing hit/miss metrics); the identity must carry an `OrganizationID` and a role in {`org:owner`, `project:owner`} (`tidbcloud_iam.go:28-37`). "IAM not enabled" is wrapped into the sentinel `errTiDBCloudIAMNotEnabled` → 404.
2. **Explicit tenant_id** (generate / activate / deactivate / delete): resolve org belonging forked by provider and compare. **Decision: extract a shared pure function** used both by this surface and by `authorizedAdminTenant`:

```go
// pkg/server/tidbcloud_iam.go (new; authorizedAdminTenant becomes a thin wrapper)
func (s *Server) resolveTiDBCloudAuthorizedTenant(ctx context.Context, tenantID, organizationID string) (*meta.Tenant, string, error)
```

Fork logic (aligned line-for-line with the current `authorizedAdminTenant`, `admin_tenants.go:455-523`):

| provider | org-belonging source | free pool | org mismatch |
|---|---|---|---|
| `tidb_cloud_native` | `GetTenantTiDBCloudOrgBinding` (`meta.go:2105`); binding `pool_status='free'` → 404 | 404 (`tenant.ErrQuotaBackendNotFound`) | 403 (`tenant.ErrQuotaPermissionDenied`) |
| `tidb_cloud_native_shared` | `GetTenantPoolMembership` excludes `pool_status='free'` (404) → `GetSharedDBForTenant` (`db_pool.go:1016`) for `TiDBCloudOrganizationID` | 404 | 403 |
| any other provider | unsupported: **409 Conflict**, sentinel `errTokensProviderUnsupported`, message states clearly "only tidb_cloud_native / tidb_cloud_native_shared are supported" | — | — |

`authorizedAdminTenant` (`admin_tenants.go:455`) is refactored into a thin wrapper over this pure function (behavior unchanged); control-plane branch handlers perform their own error mapping (§4.8). Tenant missing, `deleted`, binding/shared DB missing, and free pool are uniformly folded into `tenant.ErrQuotaBackendNotFound` → 404.

3. **Org-wide list** (`GET /v1/tokens` without `tenant_id`): `ListTenantsByTiDBCloudOrganization(ctx, identity.OrganizationID, offset, limit)` (`pkg/meta/meta.go:2832`, which already covers native + shared and excludes `deleted` and free pool); filtering and pagination semantics in §4.4.

### 3.5 Context propagation and the handleTokens control flow

**Decision: introduce `ControlPlaneScope`; do not reuse `TenantScope`.**

```go
// pkg/server/auth.go (or a new file)
type ControlPlaneScope struct {
    Identity       tenant.TiDBCloudAPIKeyIdentity // IAM resolution result (OrganizationID + Role)
    Tenant         *meta.Tenant                   // non-nil for explicit tenant_id; nil for org-wide list
    OrganizationID string                         // effective org (= tenant's org for explicit tenant;
                                                  // otherwise Identity.OrganizationID)
}
```

It is propagated through its own context key (`controlPlaneScopeKey`, parallel to `tenantScopeKey`); handlers read it via `ControlPlaneScopeFromContext`. Rationale: every field of `TenantScope` (`Backend`, `FSScopes`, `JournalPermissions`, `TokenVersion`) presumes a drive9 token; none of them can be populated for a control-plane request. Reusing it would create a "half-empty struct misread downstream" hazard.
Alternative: add a `Principal` field to `TenantScope` — rejected, same reason.

**The handleTokens control flow (decision: the scoped gate moves into the handler — the single authoritative gate)**: after the dispatcher, both identities converge into the same `handleTokens`; every scoped determination happens inside this function and does not depend on `handleBusiness`:

```text
handleTokens:
  if meta/pool/tokenSecret not enabled → 404                     // status quo tokens.go:47-50
  cps   := ControlPlaneScopeFromContext(ctx)                     // control-plane identity (dispatcher-injected)
  scope := ScopeFromContext(ctx)                                 // Bearer identity (middleware-injected)

  // Branch 0: refresh (the only entry open to scoped; Bearer identities only)
  if path == "/v1/tokens/refresh":
      if cps != nil → 403                                        // control plane may not refresh (§4.7)
      if method != POST → 405                                    // new endpoint: known path, wrong method → 405
      if len(r.URL.Query()) != 0 → 400                           // owner and scoped alike (§4.9)
      if scope == nil → 401
      handleTokenRefresh(w, r, scope)                            // both owner and scoped allowed (§5)
      return

  // Branch 1: control-plane identity (structurally unreachable by scoped —
  // scoped only ever appears in the Bearer branch)
  if cps != nil:
      switch method+path:
        POST   /v1/tokens/generate          → handleTokenGenerate (§4.3)
        GET    /v1/tokens                   → handleTokenList (§4.4)
        POST   /v1/tokens/{id}/activate     → handleTokenStatusTransition(cps, activate) (§4.5)
        POST   /v1/tokens/{id}/deactivate   → handleTokenStatusTransition(cps, deactivate)
        DELETE /v1/tokens/{id}              → handleTokenDelete(cps) (§4.6)
        POST   /v1/tokens/revoke            → handleTokenRevokeByAPIKey(cps) (§4.6)
        default (incl. POST /v1/tokens)     → 403 "control-plane credentials cannot issue scoped tokens"
      return

  // Branch 2: data-plane identity
  if scope == nil → 401                                          // defensive; dispatcher already blocked this
  if scope.IsScoped || scope.ScopeKind == meta.APIKeyScopeKindFS:
      → 403 "scoped token cannot manage tokens"                  // all non-refresh paths (the single scoped
                                                                 // gate; message identical to tokens.go:95
                                                                 // today, no behavioral drift)
  // Owner below. The legacy three endpoints' method semantics are preserved (§4.2/§4.6/§9.1):
  switch:
    path == "/v1/tokens":
        POST → handleScopedTokenIssue (status quo); GET → handleTokenList (owner branch, §4.4); other methods → 405 (status quo)
    path == "/v1/tokens/revoke":
        POST → handleScopedTokenRevokeByAPIKey (status quo); other methods → 405 (status quo)
    path == "/v1/tokens/generate" → 403 "owner key cannot generate owner keys" (§4.3)
    path == "/v1/tokens/{id}/activate" | "/v1/tokens/{id}/deactivate":
        POST → handleTokenStatusTransition(scope, fs_scoped targets only) (§4.5); other methods → 405
    path == "/v1/tokens/{id}" (well-formed id):
        DELETE → handleScopedTokenRevoke (status quo); other methods → 405 (status quo)
    default (unknown/malformed path) → 404 (status quo tokens.go:75)
```

Key points: the refresh branch stays "dispatched first"; the legacy three endpoints' 405/404 semantics are preserved item by item; illegitimate "identity × endpoint" combinations (control-plane issue, owner generate, etc.) get an explicit 403.

### 3.6 Scoped protection

**Decision**: the scoped gate **moves into `handleTokens`** (the explicit 403 in branch 2 of §3.5); `isScopedBusinessRequestAllowed` is **not modified** (it governs only the fs/uploads/layers paths on the business mux; once `/v1/tokens` leaves business it has nothing to do with tokens). Three layers of protection:

- **Gate 1 (the single authoritative gate)**: inside `handleTokens` — scoped may only enter the refresh branch (POST, no query); everything else is 403. A scoped refresh carrying query parameters gets **400** (not 403), same as owner — one unified contract.
- **Gate 2**: the refresh branch is evaluated before the owner branch (branch 0 in §3.5), so scoped is never hit by the owner branch's 403 message.
- **Structural protection**: the control-plane branch does not pass through `tenantAuthMiddleware`; a scoped token is classified into the Bearer branch at the dispatcher and can never appear as a "control-plane identity".

**Implementation note**: once `/v1/tokens` leaves the business mux, `isScopedBusinessRequestAllowed` has nothing to do with tokens — implementers **must not** add a refresh allow-line to it; that would create "gate not on the path" dead code.

---

## 4. Endpoint Surface Design

### 4.1 Endpoint overview and identity matrix

**Response field naming (decision, unified across the family)**: always `token` / `token_id` (aligned with today's issue response `scopedTokenResponse` of `POST /v1/tokens`, `tokens.go:31-38`). On a single resource surface, one concept gets one field name; it does not fork by caller identity.

| Method + path | Operation | owner (Bearer) | scoped (Bearer) | control plane |
|---|---|---|---|---|
| `POST /v1/tokens` | Issue scoped token | ✅ (status quo) | 403 (gate 1) | 403 (scoped issuance is data-plane semantics) |
| `POST /v1/tokens/generate` | Control-plane generate owner key | 403 (see §4.3) | 403 | ✅ `tenant_id` required |
| `GET /v1/tokens` | list | ✅ own tenant | 403 | ✅ `tenant_id` single tenant / omitted = org-wide |
| `POST /v1/tokens/{id}/activate` | `disabled → active` | ✅ `fs_scoped` targets only | 403 | ✅ |
| `POST /v1/tokens/{id}/deactivate` | `active → disabled` | ✅ `fs_scoped` targets only | 403 | ✅ |
| `DELETE /v1/tokens/{id}` | delete → `revoked` | ✅ (status quo) | 403 | ✅ |
| `POST /v1/tokens/revoke` | Revoke by plaintext api_key | ✅ (status quo) | 403 | ✅ (tenant resolved from the key row, §4.6) |
| `POST /v1/tokens/refresh` | Self-refresh rotation | ✅ | ✅ (only entry) | 403 |

**Owner activate/deactivate is limited to `fs_scoped` targets (decision)**: an owner may deactivate/activate scoped tokens of its own tenant (consistent with today's owner right to revoke scoped tokens, and deactivate is the lighter action). An owner **cannot** activate/deactivate owner keys (including its own) — rationale: ① freezing/unfreezing an owner key is a control-plane governance action (deciding "who may enter through the main gate") and should not be performed by an identity already inside; ② an owner deactivating itself immediately loses every management entry point (including the ability to re-activate), a self-lockout. Owner calling with an owner-key target → 403.
Alternative: owner may deactivate *other* owner keys (not itself) — rejected: the concept of "other owner key" does not exist in the current model (before control-plane generate, a tenant typically has exactly one owner), and the lockout risk remains with fuzzier semantics.

**Owner DELETE by id keeps the status quo: no `scope_kind` restriction (decision)**: today's `handleScopedTokenRevoke` (`tokens.go:207-218`) does not check `scope_kind`; an owner may DELETE any key of its tenant (including its own owner key). This design **deliberately preserves** that behavior — the asymmetry with "activate/deactivate limited to `fs_scoped`" is **not an oversight**: DELETE is a legacy endpoint and tightening it (to `fs_scoped` only) would be an observable behavior change, whereas activate/deactivate are new endpoints that can be defined strictly from day one. The lockout risk of an owner deleting its own owner key already exists today; the recovery path is control-plane generate (one of the very purposes of requirement A). §8.5 adds a self-delete case to lock in this semantics.
Alternative: tighten owner DELETE to `fs_scoped` as well — rejected; it would break the zero-change promise for legacy endpoints (§9.1).

**Tenant semantics (decision)**:

- Data-plane identity: the tenant always comes from the token (`TenantScope`); a `tenant_id` in the request body or query that disagrees with the token's tenant → **403**; agreement → accepted (redundant but harmless).
- Control-plane identity: generate / activate / deactivate / delete **require an explicit `tenant_id`** (body or query; both present with different values → 400, §4.9); list without `tenant_id` paginates org-wide; for revoke-by-key the tenant is resolved from the revoked key's DB row — if a `tenant_id` is also present and disagrees with the resolution → 404 (consistent with cross-tenant anti-enumeration).

**Tenant-state constraints (control-plane operations, decision)**:

| Tenant state | generate | activate / deactivate | delete / revoke | list |
|---|---|---|---|---|
| `active` | ✅ | ✅ | ✅ | ✅ |
| `suspended` | 403 (a frozen tenant should not receive new entry credentials) | ✅ (operational action) | ✅ (stop the bleeding) | ✅ |
| `failed` | 409 | 409 | ✅ (stop the bleeding; complements the automatic convergence of `tenant_failed_cleanup.go:356`) | ✅ |
| `pending` / `provisioning` | 409 (schema not ready) | 409 | 409 | 409 |
| `deleting` / `deleted` | 404 | 404 | 404 | 404 |

### 4.2 `POST /v1/tokens` (issue scoped, unchanged from status quo)

Data-plane owner only. Request / response / validation logic are all preserved (`handleScopedTokenIssue`, `tokens.go:101-205`): `subject` ≤ 64 bytes, `ttl_seconds > 0`, scopes validated by `validateScopedTokenScopes` (`tokens.go:256`); **JSON decoding stays lenient as today** (plain `json.Decoder` + `io.LimitReader`, `tokens.go:103` — unknown fields and trailing data tolerated; `DisallowUnknownFields` is **not** enabled, §4.9). **Only two functional changes**: ① at `InsertAPIKey` time the computed `expiresAt` (truncated to the second first, §5.3) is written into `meta.APIKey.ExpiresAt`, so newly issued scoped rows satisfy the invariant 100% (the `expires_at` in the response accordingly becomes second-precision — an intentional, compatible change, see "second-level alignment" in §5.3); ② issuance now goes through `InsertAPIKeyWithLimit` (count check + insert inside the tenant lock, §7.4; over the limit → 409). Control-plane identity → 403; scoped → 403 (gate 1, §3.6).

### 4.3 `POST /v1/tokens/generate` (control-plane generate owner key)

Control-plane identity only. Request:

```json
{
  "tenant_id": "t-abc",
  "key_name": "ci-deploy",
  "ttl_seconds": 7776000,
  "public_key": "(optional; header fallback)",
  "private_key": "(optional; header fallback)"
}
```

- `key_name`: optional, defaults to `"default"` (DDL default, `meta.go:905`); max 64 bytes.
- `ttl_seconds`: optional. When omitted, issues an owner key **with no expiry** (same as provision); when provided, issues a JWT with `exp` and writes `expires_at` in the same write. Ceiling: `maxAPIKeyTTL = 365 * 24 * time.Hour` / `maxAPIKeyTTLSeconds` (does not reuse the ≈292-year `maxScopedTokenTTLSeconds`).
- Cap check: issuance goes through `InsertAPIKeyWithLimit` (count + insert inside the tenant lock, §7.4); over the limit → 409.

Response `201 Created` (plaintext appears only this once, §7.1):

```json
{
  "token": "drive9_....",
  "token_id": "9f1c...",
  "tenant_id": "t-abc",
  "key_name": "ci-deploy",
  "scope_kind": "owner",
  "status": "active",
  "issued_at": "2026-08-06T16:00:00.000Z",
  "expires_at": null
}
```

**Issuer metadata values (decision, fixed)**: `IssuedByProvider = "tidbcloud"`, `IssuedBySubjectKey = "org:" + identity.OrganizationID`, `issued_by_metadata_json = {"role": identity.Role}` (the actual role, `org:owner` or `project:owner`). Rationale: `TiDBCloudAPIKeyIdentity` carries only OrgID and Role (`provisioner.go:53-56`); `org:<id>` is deterministic, contains no secret material, and is compatible with the `RevokeAPIKeysByIssuer` dimension.
Alternative: a digest of the public key — rejected; no security gain and loses readability.

**Owner identity calling generate → 403 (decision)**: an owner already holds the highest-privilege key; self-issuing additional owner keys is a governance action belonging to the control plane. This prevents an owner from minting long-lived credentials unknown to the org, bypassing control-plane audit.

### 4.4 `GET /v1/tokens` (list)

Three call shapes: ① data-plane owner — own tenant (tenant from the token); ② control plane + `tenant_id` — single tenant; ③ control plane without `tenant_id` — org-wide. scoped → 403 (gate 1).

Response `200 OK`:

```json
{
  "tokens": [
    {
      "token_id": "9f1c...",
      "tenant_id": "t-abc",
      "key_name": "ci-deploy",
      "scope_kind": "owner",
      "status": "active",
      "expired": false,
      "issued_by_provider": "tidbcloud",
      "issued_by_subject_key": "org:1234567890123456789",
      "issued_by_metadata_json": {"role": "org:owner"},
      "issued_at": "...",
      "expires_at": null,
      "revoked_at": null,
      "created_at": "...",
      "updated_at": "..."
    }
  ],
  "next_offset": 10
}
```

- **Never returns** `jwt_ciphertext` / `jwt_hash` / plaintext tokens; **no** JWT plaintext reverse-parsing.
- **`ListAPIKeys` listing scope (decision)**: `revoked` rows are not listed (they are audit data; `revoked_at` in the response sample is therefore always `null` — the field is kept only for schema stability). By default the listing is further restricted to **non-expired rows** — `status IN ('active','disabled') AND (expires_at IS NULL OR expires_at > UTC_TIMESTAMP(3))`. This is intentionally **broader than** the cap-count predicate (`CountActiveAPIKeys` / `GetActiveAPIKeyByIssuer`): list keeps legacy `fs_scoped` NULL rows visible for operators, while count excludes them from the soft cap (§7.4).
- **Visibility of expired rows (decision)**: single-tenant list (both the owner-own-tenant and control-plane `tenant_id` shapes) supports `?include_expired=1` to include expired rows (same key-level pagination, bounded); the `expired` field can only be `true` in that mode. **Org-wide list does not support `include_expired` (passing it → 400)**. Legacy scoped rows (`expires_at IS NULL` but JWT has `exp`) display `expires_at: null, expired: false` under the list predicate — a declared transitional deviation (§5.3, §6.4); they are **not** counted against the soft cap (§7.4).
- **Owner own-tenant list (decision)**: same function and same listing scope as the control-plane single-tenant list; an owner only ever sees its own tenant (tenant from the token). This is a new capability — previously an owner could not enumerate its tenant's tokens; this grants reasonable visibility (no secrets included).
- **Ordering**: `ListAPIKeys` is fixed at `ORDER BY created_at, id` (ascending).

**Pagination semantics (decision)**:

- **Single tenant**: key-level pagination, `offset` / `limit` (default 50, max 200); `next_offset` is a key offset; omitted when the returned row count `< limit`.
- **Org-wide**: **tenant-level pagination** — `offset` / `limit` applies to the tenant sequence from `ListTenantsByTiDBCloudOrganization` (default 10, max 50, aligned with that function's default, `meta.go:2842-2844`; its existing ordering is `ORDER BY t.created_at DESC, t.id DESC`, `meta.go:2872`, which the contract references directly). Keys of each returned tenant are enumerated per the listing scope (non-expired by default), **capped at 500 rows per tenant; beyond that the enumeration is truncated and the response carries a `"truncated_tenants": ["t-x", ...]` signal** (decision) — truncated tenants can be fully enumerated via the single-tenant key-level paginated list. Boundedness is thereby closed: one page ≤ limit × 500 + a little tenant metadata; 500 is far above the normal inventory under the cap (100), so truncation only fires on historically large tenants.
- **`next_offset` formula (based on the raw tenant page)**: if this page of `ListTenantsByTiDBCloudOrganization(offset, limit)` returned `n` tenants (**pre-filter** raw page size) — when `n == limit`, always return `next_offset = offset + limit`; when `n < limit`, omit it. Deciding from the post-filter content is forbidden: when a whole raw page consists of skipped tenants, `tokens` is empty but `next_offset` must still be returned, otherwise clients silently truncate mid-sequence.
- **Org-wide tenant filtering**: silently skips `pending` / `provisioning` / `deleting`; includes `failed` / `suspended` (disposing of their keys is precisely a control-plane duty).

### 4.5 `POST /v1/tokens/{id}/activate` · `/deactivate`

Control plane (any `scope_kind` target) and owner (`fs_scoped` targets only, §4.1). Request body:

```json
{ "tenant_id": "t-abc", "public_key": "...", "private_key": "..." }
```

`tenant_id` is required for the control plane; the owner branch ignores/validates `tenant_id` (tenant semantics in §4.1). Response `200 OK`:

```json
{ "token_id": "9f1c...", "tenant_id": "t-abc", "status": "active" }
```

Idempotency semantics (decision): already in the target state → `200` with the current `status` (`SetAPIKeyStatus` returns `changed=false`; no mutation, no "change" audit record); `revoked` → `409`; missing / cross-tenant / owner operating on an owner-key target → uniformly `404` / `403` (owner-key target for owner → 403 per §4.1; cross-tenant and missing are uniformly 404 for anti-enumeration).

**Pre-check for activating an expired key (decision)**: activating a disabled key whose `expires_at` is in the past returns **409** (the handler reads the row via `GetAPIKey`), with a message of "key already expired; cannot be revived; please generate a new key". Rationale: a successful activate whose JWT has already expired still gets 401 at authentication, which would mislead the caller into believing access was restored. An expired key has no "activate first, then refresh" self-rescue path (refresh requires the key to pass authentication, and disabled/expired is rejected at `auth.go:258` / JWT `exp`), so the only way out is issuing a new key. Rows with `expires_at IS NULL` (including legacy scoped rows) skip this check (undecidable; conservatively allowed through). deactivate has no such restriction.

The race contract is defined in `SetAPIKeyStatus` in §5.5.

### 4.6 `DELETE /v1/tokens/{id}` and `POST /v1/tokens/revoke`

- **`DELETE /v1/tokens/{id}`**:
  - **Data-plane owner (legacy, zero change)**: semantics, status codes, and **the response body `{"status": "ok"}` are all preserved** (`handleScopedTokenRevoke`, `tokens.go:207-218`); no `scope_kind` check (deliberately preserved, §4.1).
  - **Control plane**: `tenant_id` required; the underlying call is the same widened `RevokeAPIKey` (`status IN ('active','disabled')` convergence, §2.2). The response may use the extended body `{"token_id": "...", "tenant_id": "...", "status": "revoked"}` — **control-plane branch only**, a new contract for a new caller; legacy responses are unaffected.
  - Already `revoked` / missing / cross-tenant → 404.
- **`POST /v1/tokens/revoke` (by plaintext api_key)**:
  - **Data-plane owner (legacy, zero change)**: today's `handleScopedTokenRevokeByAPIKey` (`tokens.go:220-254`), restricted to `fs_scoped` and the same tenant; **the response body `{"status": "ok"}` and lenient JSON decoding are both preserved**.
  - **Control plane is also allowed (decision)**: request `{ "api_key": "drive9_...", "public_key": "...", "private_key": "..." }` — the tenant is resolved from the row found by `ResolveByAPIKeyHash` (hash miss → 404), then §3.4 organization authorization runs against that tenant; authorization failure → 404 (uniform anti-enumeration). The control plane is **not** restricted to `fs_scoped` — it must be able to revoke a leaked owner key by its plaintext (a core scenario of requirement A); the owner branch keeps today's scoped-only restriction (preventing an owner from revoking itself by mistake). Response: same control-plane extended body.
- Both delete and revoke share one semantics: terminal `revoked`, and the same cache-invalidation chain (§5.4).

### 4.7 `POST /v1/tokens/refresh` (self lease renewal)

Bearer identities only (owner and scoped alike); **control-plane identity → 403 (decision)**: refresh means "the credential holder rotates after proving possession" — the control plane holds no data-plane token, so renewing another key is not self-proof; to replace a key the control plane should use delete + generate. Full design in §5.

### 4.8 Error code table and error-mapping reuse

**Decision: error mapping reuses quota's existing machinery; nothing new is invented.** `writeQuotaCredentialError` (`pkg/server/quota.go:625-633`: `tenant.ErrQuotaBackendNotFound` → 404, everything else goes to `clientFacingErrorResponse`) + `clientFacingErrorResponse` (`pkg/server/server.go:5204`, called with a default status of `502`; it already maps: IAM 401 → 401, `ErrQuotaPermissionDenied` / `ErrTiDBCloudRoleInsufficient` / IAM 403 → 403, `ErrCredentialsRequired` → 400). New sentinels — **package ownership (decision)**: store-layer sentinels live in `package meta` (`meta.ErrAPIKeyTerminal` for terminal rows, `meta.ErrAPIKeyLimitExceeded` for over-cap), returned by store primitives and mapped to 409 by handlers — the meta package stays free of HTTP semantics; server-side sentinels: `errTiDBCloudIAMNotEnabled` → 404, `errTokensProviderUnsupported` → 409. **`backendErrorStatus` is not used** (defaults to 500 and does not recognize IAM errors, `auth.go:99`).

| HTTP | Scenarios |
|---|---|
| 400 | New-endpoint invalid JSON / unknown fields (new endpoints and control-plane branch only, §4.9) / explicit `ttl_seconds: 0` / ttl over the ceiling / `tenant_id` body-query conflict / refresh carrying query / **Bearer and control-plane credentials together (ambiguous credentials)** / org list passed `include_expired` or illegal pagination values / attempting to shorten expiry / dispatch-time body over the limit |
| 401 | No credentials at all (message reused: `missing or malformed Authorization header`) / invalid Bearer (existing middleware behavior) / control-plane credential failing IAM verification |
| 403 | Identity × endpoint not allowed (§4.1 matrix: all management paths for scoped, control-plane issue/refresh, owner generate and owner-target activate, etc.) / insufficient role / org mismatch / generate on suspended / data-plane `tenant_id` disagreeing with the token tenant |
| 404 | Tenant missing or deleted / free pool / binding or shared DB missing / key missing or cross-tenant / IAM not enabled / feature not enabled / unknown path (legacy semantics) |
| 405 | Wrong method on the legacy three endpoints (non-POST `/v1/tokens`, non-POST `/v1/tokens/revoke`, non-DELETE `/v1/tokens/{id}`, status quo preserved) / wrong method on known new-endpoint paths |
| 409 | Other providers / revoked terminal (`meta.ErrAPIKeyTerminal`) / activating an expired key / tenant pending/provisioning/failed (mutations only) / over the per-tenant cap (`meta.ErrAPIKeyLimitExceeded`) / concurrent refresh conflict |
| 502 | IAM or metadb unavailable |

Outbound messages stay generic (`invalid credentials` / `tenant not found` / `token not found`); 401/404 must not leak dispatch logic or tenant existence; error details are sanitized via `clientFacingErrorDetail` (`server.go:5242`).

### 4.9 Request decoding boundaries (decision)

**Two rule sets, split by endpoint generation**:

- **Legacy three endpoints** (`POST /v1/tokens` issue, `POST /v1/tokens/revoke`, `DELETE /v1/tokens/{id}`), **data-plane branch**: decoding stays lenient as today — plain `json.Decoder` + `io.LimitReader` (`tokens.go:103,222`), unknown fields and trailing data tolerated; `DisallowUnknownFields` is **not** enabled. Legacy clients sending extra fields must not be killed by new rules.
- **New endpoints** (generate / refresh / activate / deactivate / `GET /v1/tokens`) and **the entire control-plane branch**: JSON decoding uses `DisallowUnknownFields` (consistent with quota's `decodeQuotaRequest`, `quota.go:346`) — unknown fields → 400.

Unified rules (shared by both rule sets):

- Explicit `"ttl_seconds": 0` → **400** (omission means the field is absent; 0 is not an implicit "no expiry" switch).
- `tenant_id` present in both body and query with **different values** → 400; same value → accepted. GET only accepts query.
- **Pagination parameters (decision)**: negative or non-numeric `offset` → clamped to 0; non-numeric or ≤ 0 `limit` → default; `limit` over the maximum → **clamped to the maximum** (consistent with the clamp precedent of `adminPagination`, `admin_tenants.go:609-611`; not a 400). `include_expired` accepts only `1` / `true` (case-insensitive) as true; any other non-empty value → 400.
- The `X-Drive9-Tenant-ID` header is **not supported** (`quotaTenantIDHeader`, `quota.go:73`) — it is a quota-surface convention; this surface does not introduce a second way to pass the tenant.
- refresh rejects any query parameter for owner and scoped alike: `len(r.URL.Query()) != 0 → 400` (§5.1).

---

## 5. Data-Plane Refresh Design (Self Lease Renewal)

### 5.1 Endpoint, gates, and credentials

`POST /v1/tokens/refresh`: a Bearer identity passes through `tenantAuthMiddleware` and is dispatched by branch 0 of `handleTokens` (control flow in §3.5; gates in §3.6). The credential is `Authorization: Bearer <current drive9 API key>` — "renewing a lease using itself as the credential". Every identity field the handler uses (`tenant_id`, key id, `token_version`, `scope_kind`) comes only from the `TenantScope` (the middleware has already cross-checked it, `auth.go:295`); the request body **has no, and does not trust,** `tenant_id` / `token_id` fields; any query parameter → 400 (§4.9).

**Who may refresh (decision)**: both owner and scoped may refresh themselves. Refresh produces no privilege amplification — the new token inherits the original row's `scope_kind`, `tenant_api_key_fs_scopes`, `issued_by_*`, and the original JWT's journal permissions (carried verbatim from the current request's claims at re-issuance, `auth.go:155`). Scoped is excluded from the `/v1/tokens` management surface, and refresh is the only exception: long-running agents hold exactly scoped tokens, and forcing a return to owner re-issuance would enlarge the owner key's exposure surface.
Alternative: owner-only refresh — rejected, same reason.

**Claims retrieval (implementation convention)**: `TenantScope` does not carry JWT claims. The refresh handler takes the raw token via `bearerToken(r)` and calls `token.ParseAndVerifyToken(s.tokenSecret, tok)` again (`pkg/tenant/token/token.go:73`, pure HMAC + JSON) to obtain `exp` / `iat` / `journal_permissions`. The middleware has already verified the signature; this re-parse only reads claims.
Alternative: stuff claims into `TenantScope` — rejected; it would touch every construction site of the middleware struct (`auth.go:28-39`) for the benefit of saving one HMAC.

**DB row read path (implementation convention, decision)**: the refresh handler must read the current DB row directly through the new **`GetAPIKey(ctx, tenantID, apiKeyID)`** (§8.2), keyed by `scope.TenantID + scope.APIKeyID`, then compute the period per the three-level resolution of §5.3. **Forbidden**: reusing the cached record from `ResolveByAPIKeyHash` (its 10s TTL may return values from before a concurrent refresh); **forbidden**: computing the period from JWT claims alone (that would skip the lazy backfill and fail to write `expires_at`). An uncached direct `GetAPIKey` read plus `RotateAPIKey`'s `token_version` optimistic lock forms the closed loop of "read a stale value → UPDATE does not match → 409".

### 5.2 Request / response

Request:
```json
{ "ttl_seconds": 7776000 }
```

Response `200 OK`:

```json
{
  "token": "drive9_....",
  "token_id": "9f1c...",
  "tenant_id": "t-abc",
  "scope_kind": "fs_scoped",
  "expires_at": "2026-11-04T16:00:00.000Z"
}
```
The new plaintext is returned only once. Error codes in §4.8 (400 for illegal ttl / shortening / query; 401 when the current key is already invalid; 403 for scoped hitting non-refresh paths, the existing behavior; 409 for concurrent conflicts).

### 5.3 Renewal rules and the `expires_at` ↔ JWT `exp` invariant

**Core invariant (decision, repository-wide)**: any write path that issues or rotates a key with a TTL must, **in the same write**, set both the JWT `exp` and `tenant_api_keys.expires_at`, equal to each other (after truncation to the second); a key with no expiry omits both. The legal meaning of `expires_at IS NULL` is fixed per `scope_kind`:

| scope_kind | Meaning of `expires_at IS NULL` | May a new JWT have no `exp`? |
|---|---|---|
| `owner` | No expiry (status-quo semantics, e.g. provision issuance) | Allowed (only when the original JWT also has no `exp`) |
| `fs_scoped` | **Legacy row** (issued before the column existed; expiry lives only in the JWT `exp`) — never means "no expiry" | **Forbidden** |

Companion issuance-side change: `handleScopedTokenIssue` writes `ExpiresAt` (§4.2), so newly issued scoped rows satisfy the invariant 100%.

**Resolution order of the "effective expiry"** (shared by refresh and derived-state judgments):

1. `tenant_api_keys.expires_at` non-NULL → authoritative (`issued_at` of the same row is the effective issuance time);
2. Otherwise parse the current request's JWT claims: non-zero `exp` → claims are authoritative (`iat` is the effective issuance time) — the only path for legacy scoped rows;
3. Otherwise (owner with no expiry) → no effective expiry.

**Renewal rules (decision)**:

- When `ttl_seconds` is provided: `period = ttl_seconds`, ceiling `maxAPIKeyTTLSeconds` (365 days). Compare the ceiling in **seconds** (not `time.Duration` nanoseconds) to avoid int64 overflow for huge inputs.
- When `ttl_seconds` is omitted: `period = effective expiry − effective issuance time` ("renew one more period") **without clamping to 365d** — scoped issue still permits longer TTLs (Q7), and clamping then enforcing no-shortening would lock those keys out of refresh; an owner key with no effective expiry → rotation only, keeping no-expiry.
- Optimistic lock uses the **caller's proven** `token_version` (JWT claims / `TenantScope`), not the current DB row version alone: within the auth-cache residual window a stale Bearer may still authenticate while `GetAPIKey` already sees a newer generation; locking on the DB version would let that stale caller clobber the successor. Version mismatch → 409 before period math.
- New values are uniform: `issued_at = now`, `expires_at = now + period`, new JWT `iat = now`, `exp = now + period`. **`issued_at` must be updated on refresh** — otherwise the second default refresh's period grows monotonically with the key's age (counter-example: `issued_at=T0, expires_at=T0+30d`; a default refresh at T0+20d yields a new expiry of T0+50d while the DB `issued_at` is still T0, so the next computation produces a 50-day period). `RotateAPIKey`'s conditional UPDATE therefore includes `issued_at = ?` (§5.5), and the list's `issued_at` also agrees with the new JWT `iat`.
- **Second-level alignment (implementation convention)**: JWT `exp`/`iat` are Unix seconds; `expires_at`/`issued_at` are `DATETIME(3)`. refresh / generate / scoped issue all start with `nowSec := time.Now().UTC().Truncate(time.Second)`; `expires_at = nowSec.Add(period)` (exactly the same second as the JWT `exp`); `issued_at = nowSec`; `iat` is taken internally by the issuing function as `time.Now().Unix()` (`token.go:46`) and may differ from `nowSec` by sub-second at most — `iat` takes part in no judgment, and tests compare after truncation. **Impact on the existing issue response (decision)**: today's issue response carries the un-truncated `time.Time` as `expires_at` (`tokens.go:135-137,199`); the new behavior makes it second-precision — **an intentional, compatible change**: the precision loss affects no client comparing at second granularity (JWT `exp` was always second-level), and it is listed in the perceivable-change list of §9.2. Alternatives: add an explicit `iat` parameter to the token package / keep millisecond precision in the response while storing the truncated value in the DB — both rejected: the former changes a signature for test convenience, the latter reintroduces a two-source divergence.
- **Scoped anti-amplification**: when `scope_kind = fs_scoped` and no effective expiry can be resolved (no `expires_at` and no JWT `exp`) → 400, requiring owner re-issuance; under all circumstances the scoped JWT produced by refresh must carry an `exp`. For legacy rows (case 2), refresh persists the derived expiry semantics into the column, completing the **lazy backfill**.
- **Owner anti-accident**: when an owner key has an effective expiry, refresh must preserve the expiry semantics and must not regress to no-expiry.
- **No shortening**: `now + period` must be later than the original effective expiry (owners with no expiry are exempt). Rationale: prevent "shortening" from being used as "revocation" — immediate invalidation should go through revoke; also avoids concurrent refreshes shortening each other's windows.

**Legacy-row backfill strategy (decision)**: **lazy backfill** (derived at case 2 on first refresh and persisted), **no** one-shot full backfill. Rationale: a full backfill would require `pool.Decrypt` + JWT parsing row by row — a one-time operational action touching all key material; lazy backfill covers "keys still in use", while unused legacy scoped rows die naturally at their JWT `exp`. Cost (recorded and accepted): before their first refresh, legacy scoped rows display `expires_at: null, expired: false` in list (§4.4) and are **excluded from the soft-cap count** (§7.4).

### 5.4 Rotation strategy

**Decision**: rotate on commit, **no grace period**. Refresh updates the original row in a single transaction with new `jwt_hash` / `jwt_ciphertext` / `token_version` / `issued_at` / `expires_at`; after commit:

- The old token's DB resolution fails immediately (unique index `idx_api_keys_hash` on `jwt_hash`, `meta.go:918` → `ErrNotFound` → 401).
- The in-process resolve cache is invalidated via `evictTenant`; cross-pod invalidation goes through `insertTenantNotifyTx(TenantNotifyWorkAPIKeyCacheCleanup)` — the same chain as `RevokeAPIKey` (`meta.go:4007-4028`).
- **Known residual window**: the cache's fill-after-evict race (`api_key_cache.go:42-47`) means the old token can still be served from cache for at most one TTL in the worst case (default 10s, `DRIVE9_META_AUTH_CACHE_TTL_MS`). This matches today's revoke behavior and is accepted; the same applies to deactivate/delete (operational semantics: "not instantly effective globally", §9.4).

Alternative: an N-second dual-active grace period — rejected; it requires storing the old hash additionally and greatly increases authentication-path complexity.

### 5.5 Concurrency / replay protection and store-layer contracts

**Refresh idempotency (decision)**: refresh is **not idempotent**; each successful call rotates once. When a response is lost, retrying with the old key is not allowed (the old key may already be 401, or a concurrent conflict yields 409). Recovery paths: control-plane generate / owner re-issuance. **Client rule**: after a successful refresh, persist the new token before switching to it; if persisting fails, keep using the old token until its expiry rather than blindly retrying refresh.
Alternative: `Idempotency-Key` — rejected for v1 (§10 Q4).

**`RotateAPIKey` contract**:

```sql
UPDATE tenant_api_keys
SET jwt_hash = ?, jwt_ciphertext = ?, token_version = ?, issued_at = ?, expires_at = ?, updated_at = ?
WHERE tenant_id = ? AND id = ? AND token_version = ? AND status = 'active'
```

- `RowsAffected == 0` → 409 (only one of two concurrent refreshes can match; or the key has been concurrently revoked/disabled).
- The transaction template reuses `RevokeAPIKey`: conditional UPDATE → `RowsAffected` judgment → `insertTenantNotifyTx` → `evictTenant` after commit.

**`SetAPIKeyStatus` contract (race-safe idempotent state transition, decision)**:

Parameter validation (before any DB operation):

- `from` and `to` may only be `active` / `disabled`; **`from == revoked` and `to == revoked` are both rejected** (the former would revive a terminal row; the latter would create a second path rivaling delete).
- **`from == to` is rejected**: a same-state UPDATE would match, refresh `updated_at`, and report `changed=true`; moreover it depends on driver `RowsAffected` semantics (`CLIENT_FOUND_ROWS`-style drivers report 1 even when nothing changed), so an idempotent replay would be misreported as a real transition. The correct path for a same-state request is `from` set to the opposite state + read-back hitting `current == to`.

SQL (the condition matches only `from`; `status IN (from, to)` is not used — otherwise same-state rows would also match and `changed` would be uncomputable):

```sql
UPDATE tenant_api_keys SET status = ?, updated_at = ?
WHERE tenant_id = ? AND id = ? AND status = ?   -- status = from
```

Then a **read-back in the same transaction**, returning `(currentStatus, changed, err)`:

| UPDATE result | read-back | Return |
|---|---|---|
| `RowsAffected == 1` | (must be `to`) | `current = to, changed = true` |
| 0 rows matched | row missing | `ErrNotFound` |
| 0 rows matched | `current == revoked` | `meta.ErrAPIKeyTerminal` (sentinel → handler 409) |
| 0 rows matched | `current == to` | `current = to, changed = false` (idempotent replay, 200) |
| 0 rows matched | anything else | error (unexpected state, 500) |

The store layer returns only sentinels (`ErrNotFound` / `meta.ErrAPIKeyTerminal`); HTTP mapping is done by the handler (the meta package stays free of HTTP semantics, §4.8). Concurrency semantics: an activate racing a revoke is serialized by row locks — if the revoke commits first, this UPDATE matches 0 rows and the read-back shows `revoked` → 409; if this UPDATE commits first, the revoke converges normally. Cache invalidation and notify fire only when `changed = true`.

**token_version and the cache**: after refresh, the old hash's cache entry agrees with the old token's version (within the residual window the old token still works, same as revoke); the new hash misses on first use → DB hit. No new cache mechanism is needed.

### 5.6 Audit and metrics

Refresh success/failure writes a `server_event` log (only `tenant_id`, `token_id`, `scope_kind`, `result`; no token material of any kind); `metricEvent(ctx, "token_refresh", "scope_kind", ..., "result", ...)`. Refresh travels the Bearer branch, where the middleware has already set the `TenantScope` via `setRequestMetricScope` (`auth.go:419`), so the handler needs no extra metric scope.

---

## 6. Data Model and Schema

### 6.1 `tenant_api_keys` changes

**Decision: exactly one new column.**

```sql
ALTER TABLE tenant_api_keys ADD COLUMN expires_at DATETIME(3) NULL AFTER revoked_at;
```

The DDL in `metaInitSchemaStatements()` is updated accordingly (`pkg/meta/meta.go:902-921`):

```go
expires_at     DATETIME(3) NULL,
```

| Candidate column | Decision | Rationale |
|---|---|---|
| `expires_at` | **Add** | Persistence basis for refresh / TTL-carrying owner keys; source for list display and `expired` derivation; written together with the JWT `exp` (§5.3). NULL meaning per scope_kind (§5.3 table) |
| `disabled_at` | Not added | `status` + `updated_at` suffice |
| `refreshed_at` | Not added | `issued_at` updating on refresh already expresses "last rotation time" |
| `last_used_at` | Not added | Write amplification on the authentication hot path, breaking the design premise of `apiKeyResolveCache` (§10 Q2) |

**Indexes / constraints**: none added. list is covered by `idx_api_keys_tenant (tenant_id, status)` (`meta.go:919`).
Alternative: `UNIQUE (tenant_id, key_name)` — rejected; duplicate names are allowed today (`TestScopedTokenIssueAllowsRepeatedSubject`, `tokens_test.go:108`).

### 6.2 Go model changes

- `meta.APIKey` (`meta.go:250-266`) gains `ExpiresAt *time.Time`.
- New constant `APIKeyDisabled APIKeyStatus = "disabled"` (next to `meta.go:83-84`).
- `InsertAPIKey` (`meta.go:1878`): the INSERT column list gains `expires_at`; **and a new storage-layer guard (decision)**: `ScopeKind == APIKeyScopeKindFS && ExpiresAt == nil` → return an error outright (fail-closed). Rationale: "scoped must expire" is a repository-wide invariant (§5.3); honoring it only at the handler layer would let future callers / test helpers / backfill scripts silently bypass it and mint eternal scoped keys that escape the counting predicate. The storage-layer guard is the only backstop. Legacy rows are unaffected (the guard blocks only new writes).
- `RotateAPIKey` gets the same guard: `next.ExpiresAt == nil` for `fs_scoped` → error (refresh already forbids this at the rule level in §5.3; this is a storage-layer second line of defense).
- `resolveByAPIKeyHashDB` (`meta.go:3236`): SELECT/Scan gains `k.expires_at`.
- `cloneTenantWithAPIKey` (`api_key_cache.go:180-189`) gains an `ExpiresAt` clone — **mandatory**, otherwise the cache path would share the pointer.
- `GetActiveAPIKeyByIssuer` (`meta.go:4064`): column list gains `expires_at`, plus the filter `AND ((expires_at IS NULL AND scope_kind = 'owner') OR expires_at > UTC_TIMESTAMP(3))` — same non-expired predicate as `CountActiveAPIKeys`. Its caller `slockIssueForBinding` (`pkg/server/slock.go:590`) previously relied on implicit JWT verification to catch expired keys; with the column introduced, expired and legacy-NULL scoped rows are excluded explicitly.
- `RevokeAPIKey` / `RevokeTenantAPIKeys` / `RevokeAPIKeysByIssuer`: condition changed to `status IN ('active','disabled')` (§2.2).

### 6.3 How the meta schema evolves

`tenant_api_keys` is a **metadb** table, defined by `metaInitSchemaStatements()` (`meta.go:751`) and repaired automatically through `diffMetaSchema` / `plannedMetaSchemaRepairs` / `applyMetaSchemaRepairs` (`meta.go:1274/1401/1427`). A nullable `ADD COLUMN` is a safe repair (`isSafeMetaRepairDiff` → `isSafeAddColumnRepairSQL`, `meta.go:1639-1663`) and is applied automatically at startup.

**Relationship to the AGENTS.md "Schema synchronization" clause (conclusion)**: that clause governs the **tenant-database init schema** (`pkg/tenant/schema/tidb_auto.go`, `tidb_app.go`, `pkg/tenant/db9/schema.go`) and its external synchronization via `drive9-server schema dump-init-sql`. `tenant_api_keys` is a metadb table, **outside** the dump's coverage (`cmd/drive9-server/schema.go:17`); this change **requires no edit** to those three schema files and no dump-init-sql re-run. Adding the `disabled` value to `status` is a pure code change within `VARCHAR(20)`.

**Migration / rollback**: forward — add the column (auto-repair) + new binary; old binaries are oblivious to the new column (all column lists are explicit; there is no `SELECT *`). Rollback: leftover `disabled` rows are rejected by the old code's `status != active` check (`auth.go:258`) — safe; `DROP COLUMN` is not performed.

### 6.4 Legacy data strategy

- **Legacy scoped rows** (`expires_at IS NULL`, JWT has `exp`): authentication behavior unchanged; refresh lazily backfills (§5.3); list shows `expires_at: null, expired: false` before backfill (declared deviation, §4.4); counting treats them as non-expired (§7.4).
- **Legacy owner rows**: `expires_at NULL` = no expiry; semantics unchanged.
- **Existing raw INSERT paths**: the pool-backfill INSERTs at `pkg/meta/db_pool.go:1487` and `pkg/meta/tenant_pool_membership.go:407` do not write `expires_at` (intentional — no-expiry owners). **Contract**: both sites gain an explicit guard `if k.ExpiresAt != nil { return error }`, preventing a future caller from passing a TTL-carrying key into these paths and silently dropping the column; unit tests included. Alternative: add the column to those two INSERTs — rejected; an explicit guard exposes misuse better than widening the column lists.

---

## 7. Security Design

### 7.1 Plaintext appears only once

generate (§4.3) and refresh (§5.2) are the only two responses that return plaintext; scoped issuance (status quo, §4.2) likewise returns plaintext only in its issuance response. The storage-side separation of duties is preserved:

- `jwt_hash` (SHA-256, `token.HashToken`, `pkg/tenant/token/token.go:31`): the **lookup key**, used by `ResolveByAPIKeyHash` and the constant-time comparison (`auth.go:252`).
- `jwt_ciphertext` (`pool.Encrypt`): the **verification material**, decrypted at authentication and compared constant-time against the request's plaintext (`auth.go:266`, `auth.go:279`).

Neither can be omitted; refresh must update both in the same transaction (§5.5).

### 7.2 Audit

- **server_event logs**: generate / list / activate / deactivate / delete / refresh each record `tenant_id`, `token_id`, `organization_id`, `principal` (`owner`/`scoped`/`control_plane`), `action`, `result`, `duration_ms`, following the `eventFields` pattern. activate/deactivate distinguish `changed=true/false`; idempotent replays record only `changed=false`.
- **journal**: **not written to**. The journal is the tenant's data-plane audit (`pkg/journal/`); key management is a control-plane activity, and writing it there would blur the subjects of the two audit surfaces.
- **metrics**: see §8.4.

### 7.3 Credentials never enter logs

TiDB Cloud public/private keys and drive9 token plaintext **never** appear in logs, metric labels, or error bodies:

- Log fields are limited to `tenant_id` / `token_id` / `organization_id` / `role` / `principal`.
- IAM resolution failure uses a uniform outward message, `invalid credentials`; internal logs record only the error category.
- Dispatcher logs and 400 (ambiguous credentials) logs contain no credential material.
- Existing code already follows this pattern (`tidbcloud_iam.go:14-40`, `auth.go:253-262`); new code is reviewed against this checklist.
- Response error text is sanitized via `clientFacingErrorDetail` (`server.go:5242`).

### 7.4 Rate limiting, capacity cap, and brute-force enumeration protection

- **Enumeration protection**: key ids are UUID v4 — not enumerable; cross-tenant and missing are uniformly 404; list requires passing authorization first (Bearer → the token's tenant; control plane → IAM + org belonging).
- **Per-tenant key-count cap (decision)**: a **soft cap of 100** per tenant, counted by the predicate **`status IN ('active','disabled') AND ((expires_at IS NULL AND scope_kind = 'owner') OR expires_at > UTC_TIMESTAMP(3))`** (excluding revoked, expired, and legacy NULL `fs_scoped` rows; same predicate as `GetActiveAPIKeyByIssuer`). **Concurrency safety (decision)**: count and insert must be serialized by a **tenant-level row lock** — a new single-transaction primitive `InsertAPIKeyWithLimit(ctx, key, max int)`: inside the transaction, `SELECT id FROM tenants WHERE id = ? FOR UPDATE` (locking the tenant row and serializing all key writes of that tenant) → `CountActiveAPIKeys` → over the limit returns `meta.ErrAPIKeyLimitExceeded` → otherwise insert. A bare "count then insert" is unsafe even inside one transaction (two concurrent transactions can both count 99 and each insert a row, ending at 101). **The cap applies to every issuance path**:
  - Control-plane generate (`POST /v1/tokens/generate`) and data-plane scoped issuance (`POST /v1/tokens`): always via `InsertAPIKeyWithLimit`; over the limit → 409.
  - **slock rotation (`rotateIssuedOwnerAPIKey`, `pkg/server/slock.go:663-673`)**: today it "inserts the new key via `issueOwnerAPIKey` first, then revokes old issuer keys via `RevokeAPIKeysByIssuer`, with a compensating revoke of the new key on failure" — placing the cap check before the insert would make rotation permanently impossible for a tenant at the 100 boundary. **Decision: replace it with a single-transaction `RotateIssuerAPIKey`** (§8.2): inside the same `FOR UPDATE` tenant lock, first revoke the old issuer keys → then the count check → then insert the new key; any step failing rolls the whole transaction back with the old keys intact; still over the limit (keys other than the issuer's occupy all 100) → `meta.ErrAPIKeyLimitExceeded`, safe to retry. **Wiring the 409 (decision)**: today the slock callback wraps every rotation failure as 500 (`slock.go:618-621`, `newProvisionTenantError(http.StatusInternalServerError, ...)`) — it must be changed to recognize `meta.ErrAPIKeyLimitExceeded` and map it to 409 (all other errors stay 500), otherwise the "slock over-limit → 409" case in §8.5 cannot be met.
  - provision / pool backfill / fork (`server.go:6307`, `5851`, `admin_tenant_pool.go:2547/2606`, `fork.go:398`): the first key of a brand-new tenant — the count starts at 0 and cannot touch the cap, so no check is inserted (avoiding a pointless COUNT and lock on the provisioning hot path); this trade-off is recorded explicitly.
  - refresh creates no new rows and is not subject to the cap.
- **Counting semantics for legacy NULL rows (decision)**: count predicate is **`status IN ('active','disabled') AND ((expires_at IS NULL AND scope_kind = 'owner') OR expires_at > UTC_TIMESTAMP(3))`**. Owner NULL means no-expiry (always counted). Legacy `fs_scoped` NULL rows are **excluded** from the cap count — `scope_kind` distinguishes them from no-expiry owners, and the §6.2 storage guard prevents any new no-expiry scoped write, so excluding them does not create a lasting bypass. Residual risk is only a transient undercount of still-valid pre-migration scoped keys until they refresh (lazy backfill) or expire.
- **Credential brute force**: control-plane credential strength is guaranteed by TiDB Cloud; `resolveTiDBCloudIdentity` is cached via `tidbCloudRBACCache`, and failures are not cached — v1 relies on gateway rate limiting (the repository has no built-in limiter), **in an operable configuration form (decision)**: with two credential types on the same path, a gateway can only recognize control-plane requests by their headers — apply IP-level rate limiting to `/v1/tokens*` requests carrying the `X-TiDBCloud-Public-Key` header; body-only credential shapes (whose body features an L7 gateway usually cannot read) rely on the platform WAF / IAM-side quotas. Listed in the release checklist (§9.4).
- **Audit alerts**: org-level generate/delete frequency is included in metric alerting (the `api_key_manage` event carries an org label, §8.4).

### 7.5 Cross-tenant isolation

- **IAM role granularity (decision record)**: `TiDBCloudAPIKeyIdentity` carries only `OrganizationID` and `Role` (`provisioner.go:53-56`), so `project:owner` is authorized at **org scope** — this is drive9's existing policy (the comment at `tidbcloud_iam.go:31-36` states explicitly that "both owner roles are treated as organization-scoped operators"); extending it from quota/admin to token management is **an accepted term of the existing policy** and introduces no new authorization semantics. Project-level isolation would be a separate topic.
- Data-plane path: the tenant always comes from the `TenantScope` (the middleware has cross-checked claims against the DB row); handlers trust no identity field from the request; a co-present `tenant_id` that disagrees → 403 (§4.1).
- Control-plane path: `resolveTiDBCloudAuthorizedTenant` enforces that `identity.OrganizationID` equals the tenant's effective org belonging (§3.4); activate/deactivate/delete store UPDATEs carry `WHERE tenant_id = ? AND id = ?` as a second line of defense; revoke-by-key resolves the tenant from the hash row and then re-authorizes.
- Org-wide list outputs only tenants within the set returned by `ListTenantsByTiDBCloudOrganization(identity.OrganizationID)`.
- The dispatcher itself performs no authorization — it only classifies the credential shape; every authorization decision happens inside a branch handler (Bearer → the `TenantScope` tenant; control plane → org belonging). There is no "credential mixing amplifies privileges" path.

---

## 8. Implementation Landing Points

### 8.1 Files added / modified

| File | Changes |
|---|---|
| `pkg/server/server.go` | ① `/v1/tokens` is **moved out of** the `business` mux (delete the registration on business at `server.go:688-689`) and re-attached to `tokensCredentialDispatcher` (the single assembly, §3.2); ② delete the tokens dispatch case inside `handleBusiness` (`server.go:1811-1812` region — prevents double-registration dead code); ③ **`isScopedBusinessRequestAllowed` is not modified** (§3.6); ④ `buildOwnerAPIKey` (defined at `server.go:6441`) gains an `expiresAt time.Time` parameter (call matrix in §8.2) |
| `pkg/server/tokens.go` | ① `handleTokens` control flow rearranged into "refresh branch → control-plane branch → data-plane branch (scoped 403 → owner legacy dispatch)" (§3.5); ② new `handleTokenRefresh`; ③ `handleScopedTokenIssue` writes `ExpiresAt` (second-truncated) + switches to `InsertAPIKeyWithLimit` (§4.2, §7.4); ④ the legacy three endpoints keep their response bodies / method semantics / lenient decoding (§4.2/§4.6/§4.9) |
| `pkg/server/tokens_control_plane.go` (**new**) | `tokensCredentialDispatcher`, `peekTokenCredentials` (§3.3 contract), `ControlPlaneScope` and its context key, `controlPlaneTokenAuth` (IAM identity only, §3.3 scope of responsibility), `handleTokenGenerate`, `handleTokenList`, `handleTokenStatusTransition`, `handleTokenDelete`, `handleTokenRevokeByAPIKey` (control-plane branch), the `errTokensProviderUnsupported` / `errTiDBCloudIAMNotEnabled` sentinels and error mapping (reusing `writeQuotaCredentialError` / `clientFacingErrorResponse`, §4.8) |
| `pkg/server/tidbcloud_iam.go` | New pure function `resolveTiDBCloudAuthorizedTenant` (§3.4); "IAM not enabled" wrapped into a sentinel |
| `pkg/server/admin_tenants.go` | `authorizedAdminTenant` refactored into a thin wrapper over `resolveTiDBCloudAuthorizedTenant` (behavior unchanged; regression case in §8.5) |
| `pkg/server/slock.go` | `rotateIssuedOwnerAPIKey` (`slock.go:663`) switches to calling the transactional `RotateIssuerAPIKey` (§7.4); callback error handling recognizes `meta.ErrAPIKeyLimitExceeded` → 409 (status quo: everything is 500, `slock.go:618-621`) |
| `pkg/meta/meta.go` | `APIKey` gains `ExpiresAt`; `APIKeyDisabled` constant; DDL gains the column; `InsertAPIKey` / `resolveByAPIKeyHashDB` / `GetActiveAPIKeyByIssuer` column lists (the latter also gains the non-expired filter, §6.2); `InsertAPIKey` gains the fs_scoped no-expiry guard (§6.2); the three `Revoke*` conditions are widened (§2.2); sentinels `ErrAPIKeyTerminal` / `ErrAPIKeyLimitExceeded` (§4.8); new `GetAPIKey`, `ListAPIKeys`, `CountActiveAPIKeys`, `SetAPIKeyStatus`, `RotateAPIKey`, `RotateIssuerAPIKey`, `InsertAPIKeyWithLimit` (contracts in §5.5, §7.4) |
| `pkg/meta/db_pool.go` / `pkg/meta/tenant_pool_membership.go` | The two raw INSERTs gain `ExpiresAt == nil` guards (§6.4) |
| `pkg/meta/api_key_cache.go` | `cloneTenantWithAPIKey` gains an `ExpiresAt` clone (§6.2) |
| `pkg/server/instrumentation.go` | `classifyTokenAction` (`instrumentation.go:687`) extended: generate / list / activate / deactivate / refresh (§8.4); `requestRoute` **needs no change** (`/v1/tokens/*` already exists, `instrumentation.go:480-481`) |
| `pkg/metrics/operations.go` | `tenantAttributedBusinessEvent` (`operations.go:551-559`) gains `"api_key_manage"` and `"token_refresh"` (always true, §8.4) |
| `pkg/server/tokens_test.go` / `pkg/server/tokens_control_plane_test.go` (**new**) | Test matrix in §8.5 |
| `pkg/meta/meta_test.go` | New meta functions, revoke convergence, schema spec assertions |

### 8.2 Key function signatures and call matrix

```go
// pkg/meta/meta.go
func (s *Store) GetAPIKey(ctx context.Context, tenantID, apiKeyID string) (out *APIKey, err error)                                     // uncached direct read; the read path for refresh / activate expiry pre-check (§5.1, §4.5)
func (s *Store) ListAPIKeys(ctx context.Context, tenantID string, includeExpired bool, offset, limit int) (out []APIKey, err error)  // status IN ('active','disabled') only; non-expired by default; ORDER BY created_at, id (§4.4)
func (s *Store) CountActiveAPIKeys(ctx context.Context, tenantID string) (n int, err error)                                         // status IN ('active','disabled') AND ((expires_at IS NULL AND scope_kind='owner') OR expires_at > UTC_TIMESTAMP(3)) (§7.4)
func (s *Store) SetAPIKeyStatus(ctx context.Context, tenantID, apiKeyID string, from, to APIKeyStatus) (current APIKeyStatus, changed bool, err error) // §5.5 contract; from/to only active/disabled and from != to
func (s *Store) RotateAPIKey(ctx context.Context, tenantID, apiKeyID string, oldVersion int, next *APIKey) (err error)               // §5.5 contract, includes issued_at
func (s *Store) RotateIssuerAPIKey(ctx context.Context, tenantID, provider, subjectKey string, next *APIKey) (err error)             // single transaction + tenant-row FOR UPDATE: revoke old issuer keys → count check → insert (§7.4); over-limit meta.ErrAPIKeyLimitExceeded → 409
func (s *Store) InsertAPIKeyWithLimit(ctx context.Context, key *APIKey, maxKeys int) (err error)                                     // single transaction + tenant-row FOR UPDATE: count check → insert (§7.4 concurrency safety); over-limit meta.ErrAPIKeyLimitExceeded; includes the fs_scoped no-expiry guard

// pkg/server/server.go (signature change)
func (s *Server) buildOwnerAPIKey(ctx context.Context, tenantID, keyName string, tokenVersion int, expiresAt time.Time, source apiKeyIssueSource) (rawToken, apiKeyID string, rec *meta.APIKey, err error)
// zero expiresAt = no expiry (status-quo semantics); non-zero is truncated to the second, with JWT exp and rec.ExpiresAt written together (§5.3)

// pkg/server/tidbcloud_iam.go (new, §3.4)
func (s *Server) resolveTiDBCloudAuthorizedTenant(ctx context.Context, tenantID, organizationID string) (*meta.Tenant, string, error)

// pkg/server/tokens_control_plane.go (new, §3.2/§3.3/§3.5)
func (s *Server) tokensCredentialDispatcher(cfg Config, next http.Handler) http.Handler
func peekTokenCredentials(w http.ResponseWriter, r *http.Request) (hasControlCred bool, err error) // §3.3: bounded read + lenient decode + body restore; header-first
func (s *Server) controlPlaneTokenAuth(ctx context.Context, w http.ResponseWriter, r *http.Request) (*ControlPlaneScope, bool) // IAM identity only; tenant authorization inside handlers (§3.3)
func (s *Server) handleTokenGenerate(w http.ResponseWriter, r *http.Request, cps *ControlPlaneScope)
func (s *Server) handleTokenList(w http.ResponseWriter, r *http.Request)   // shared by both identities: ControlPlaneScope (tenant_id / org) or TenantScope (owner's own tenant)
func (s *Server) handleTokenStatusTransition(w http.ResponseWriter, r *http.Request, tokenID string, activate bool)
func (s *Server) handleTokenDelete(w http.ResponseWriter, r *http.Request, cps *ControlPlaneScope, tokenID string)
func (s *Server) handleTokenRevokeByAPIKey(w http.ResponseWriter, r *http.Request, cps *ControlPlaneScope)

// pkg/server/tokens.go (new / rearranged, §3.5/§5)
func (s *Server) handleTokenRefresh(w http.ResponseWriter, r *http.Request, scope *TenantScope)
```

**All `buildOwnerAPIKey` call sites and their `expiresAt` expectations** (the signature change forces a full audit; includes direct calls that bypass `issueOwnerAPIKey`; line numbers reflect the current tree — use symbol names as authoritative during implementation):

| Call site | Scenario | Call style | `expiresAt` |
|---|---|---|---|
| `server.go:6307` | provision (dedicated) | via `issueOwnerAPIKey` | zero (status quo) |
| `admin_tenant_pool.go:2547` | admin pool backfill | via `issueOwnerAPIKey` | zero (status quo) |
| `slock.go:664` | slock rotation (now via `RotateIssuerAPIKey`, §7.4) | via `issueOwnerAPIKey` | zero (status quo) |
| `server.go:5851` | shared provision atomic write | **direct** | zero (status quo) |
| `admin_tenant_pool.go:2606` | shared pool claim | **direct** | zero (status quo) |
| `handleTokenGenerate` (new) | control-plane generate | direct | request `ttl_seconds` → `nowSec+ttl`; omitted → zero |
| `fork.go:398` | fork pre-created owner key | bypasses `buildOwnerAPIKey`; independent `IssueToken` + `InsertAPIKey` | unaffected; `ExpiresAt` nil |

`issueOwnerAPIKey` (defined at `server.go:6476`) correspondingly gains an `expiresAt` parameter for pass-through.

### 8.3 Auth middleware changes

**Decision: `tenantAuthMiddleware` logic changes zero** — only its wrapping position moves from "mux-global" to "inside the dispatcher's Bearer branch" (§3.2). `disabled` is covered by the existing `status != active` check (`auth.go:258`); JWT expiry is covered by `token.ParseAndVerifyToken` (`token.go:121-127`), the single authority for authentication; the `expires_at` column serves display / issuance / renewal purposes, and the §5.3 invariant keeps the two written together.

### 8.4 instrumentation / metrics

- `classifyTokenAction` (`instrumentation.go:687`) extended: `POST /v1/tokens/generate` → `generate`; `GET /v1/tokens` → `list`; `POST /v1/tokens/{id}/activate|deactivate` → `activate`/`deactivate`; `POST /v1/tokens/refresh` → `refresh`; the existing issue / revoke / revoke_by_key branches are unchanged.
- `requestRoute` (`instrumentation.go:466`) **needs no change**: the existing `/v1/tokens` prefix → `"/v1/tokens/*"` already covers every new sub-path.
- **Metric scope**: the Bearer branch is set by the middleware via `setRequestMetricScope` (`auth.go:419`); handlers need nothing. The control-plane branch bypasses the middleware, so handlers must call `setRequestMetricTenant(r.Context(), t.ID, tokenIDorEmpty, t.Provider, orgID, classifyTenantRequest(r))` themselves before logging / metrics (for explicit tenants, aligned with `quota.go:130,148`); the **org-wide list** has no tenant context (`setRequestMetricTenant` returns early on an empty tenantID, `instrumentation.go:206-210`), so it instead passes `"tidbcloud_org_id", identity.OrganizationID` explicitly as a kv in `metricEvent`/`eventFields` (org-level aggregation; no high-cardinality tenant label introduced).
- **Attribution (decision)**: `metricEvent` attaches `tenant_id`/`tidbcloud_org_id` labels only for events whitelisted by `tenantAttributedBusinessEvent` (`operations.go:551-559`) — the whitelist is extended with `"api_key_manage"` and `"token_refresh"` (always true, same level as `"auth"`); org list keeps the explicit kv.
- Event and log naming: `metricEvent(ctx, "api_key_manage", "action", "generate|list|activate|deactivate|delete", "principal", "owner|control_plane", "result", ...)`; `metricEvent(ctx, "token_refresh", "scope_kind", ..., "result", ...)`; log event names `token_generate` / `token_list` / `token_activate` / `token_deactivate` / `token_delete` / `token_refresh`.

### 8.5 Test plan

Repository conventions (AGENTS.md): same-package white-box tests, no assertion library, MySQL-backed tests via `pkg/testmysql` + `testmysql.ResetDB(t, db)`, server tests via the raw-HTTP helpers in `pkg/server/apitest_test.go`.

**1. Meta unit tests** (`pkg/meta/meta_test.go`):
- `ListAPIKeys`: mixed statuses, returns only `active`/`disabled`, non-expired by default / included with `includeExpired=true`, pagination boundaries, `ORDER BY created_at, id` stability.
- `CountActiveAPIKeys`: counts only `active`/`disabled`; expired rows not counted; **owner `expires_at IS NULL` counted; legacy `fs_scoped` with `expires_at IS NULL` excluded**.
- `SetAPIKeyStatus`: round trips report `changed` correctly; same-state replay yields `changed=false` without refreshing `updated_at`; `from == to` errors; `from=revoked` / `to=revoked` error; missing → `ErrNotFound`; `revoked` read-back → `meta.ErrAPIKeyTerminal`; activate racing revoke produces no stale success.
- `RotateAPIKey`: old hash `ResolveByAPIKeyHash` → `ErrNotFound`; `issued_at` updated; wrong `oldVersion` / non-active → no match; `fs_scoped` with `next.ExpiresAt == nil` → error (§6.2 guard).
- `InsertAPIKey` storage guard: `fs_scoped` with `ExpiresAt == nil` → error; a raw-insert attempt (bypassing the server, calling the Store directly) is likewise rejected.
- `InsertAPIKeyWithLimit`: N goroutines concurrently generate/issue against one tenant; final row count is exactly 100 and exactly N−100 `meta.ErrAPIKeyLimitExceeded` errors (deterministic verification of the FOR UPDATE serialization).
- `RotateIssuerAPIKey`: single-transaction atomicity; at the 100 boundary — rotation succeeds after the issuer's own slot is freed; issuer-others filling all 100 → `meta.ErrAPIKeyLimitExceeded` with old keys preserved by rollback; interleaved with concurrent generate never exceeds the cap (same tenant lock).
- **Cross-pod notify assertions**: `RotateAPIKey` and `SetAPIKeyStatus` (when `changed=true`) write `TenantNotifyWorkAPIKeyCacheCleanup` in their transactions; idempotent replays do not.
- Revoke convergence: all three `Revoke*` affect `disabled` rows; no `active`/`disabled` rows remain after `RevokeTenantAPIKeys`; bulk primitives return `nil` on zero rows while the single-entity one returns `ErrNotFound` (§2.2).
- `GetActiveAPIKeyByIssuer`: expired rows no longer returned.
- Raw INSERT guards: non-nil `ExpiresAt` → error (§6.4).
- Schema spec: the `expires_at` column parses and its repair diff passes `IsSafeAddColumnRepairSQL` (following the `meta_test.go:1515` pattern).

**2. Cache invalidation tests**: after rotation / status change, the same Store's `ResolveByAPIKeyHash` on the old hash misses immediately; the new hash hits with `ExpiresAt` correctly cloned.

**3. Server handler tests**:

| Case | Expectation |
|---|---|
| **Dispatcher**: Bearer only → data-plane branch; control-plane credentials only (body / header fallback) → control-plane branch | Each endpoint responds per the §4.1 matrix |
| **Dispatcher**: Bearer and control-plane credentials together (including "Bearer + public_key inside body" — body conflict detection) | 400 ambiguous credentials |
| **Dispatcher**: neither credential | 401 with the message `missing or malformed Authorization header` (status quo) |
| **Dispatcher peek**: Bearer POST issue whose body has no control fields | 201, and the handler decodes the body normally (proves the body was restored after peek) |
| **Dispatcher peek**: header-only control credentials + GET list | 200 (no body sniffing) |
| **Dispatcher peek**: body-only control credentials + POST generate | 201 |
| **Dispatcher peek**: mixed body and header (body public_key + header private_key) | Authenticates normally per the `adminCredentials` pairing rule |
| **Dispatcher peek**: malformed JSON body (no control fields) | Not misclassified as control plane; processed normally on the Bearer/no-credential path |
| **Dispatcher peek**: body over the limit (> `maxCredentialProvisionBodyBytes`) | 400, bounded read without OOM; an oversized unauthenticated body is likewise rejected quickly with 400 |
| scoped token calls `POST /v1/tokens/refresh` (through full dispatch + middleware + handleTokens) | 200, new token works |
| scoped refresh with query parameters | 400 (not 403 — unified contract, §3.6/§4.9) |
| scoped token calls every other `/v1/tokens` path (issue / generate / list / activate / deactivate / delete / revoke) | 403 (the single gate inside `handleTokens`, message identical to today) |
| **Legacy method semantics**: non-POST `/v1/tokens` (PUT/PATCH/DELETE), non-POST `/v1/tokens/revoke`, non-DELETE `/v1/tokens/{id}` | All 405 (status quo preserved) |
| **Legacy path semantics**: unknown/malformed `/v1/tokens/xxx` | 404 (status quo preserved) |
| **Legacy response bodies**: owner `DELETE /v1/tokens/{id}` and `POST /v1/tokens/revoke` | Assert exactly `{"status":"ok"}` (not the revoked extended body) |
| **Legacy lenient decoding**: issue / revoke-by-key with unknown fields / trailing data | Still 201/200 (no DisallowUnknownFields) |
| Control plane calls `POST /v1/tokens` (issue scoped) / `POST /v1/tokens/refresh` | 403 |
| Owner calls `POST /v1/tokens/generate` / activate/deactivate with an owner-key target | 403 |
| Owner DELETEs its own owner key (self-delete, intentional compatibility) | 200 (§4.1 decision locked in); subsequent requests with that owner → 401 |
| Owner calls `GET /v1/tokens` | 200, sees only its own tenant, same scope as the control-plane single-tenant list |
| Data-plane request carrying a `tenant_id` that disagrees with the token tenant | 403 |
| owner key refresh | 200; TTL-carrying owner's expiry advances; no-TTL owner rotates only |
| Newly issued scoped row in the DB | `expires_at` non-NULL and same second as the JWT `exp` |
| Legacy scoped (`expires_at=NULL` + JWT with `exp`) refresh | New JWT still carries `exp`, lazy backfill applied, must not become non-expiring |
| scoped refresh with no resolvable effective expiry | 400 |
| Two consecutive default refreshes | period does not grow; `issued_at` equals the new JWT `iat` after truncation |
| Shortening ttl / explicit `ttl_seconds: 0` / unknown fields / `tenant_id` body-query conflict / refresh with query (owner and scoped) | All 400 |
| Two concurrent refreshes of the same key | Exactly one 200, one 409 |
| Control-plane generate: body credentials / header fallback | Both 201 |
| Control-plane generate: dedicated hit / cross-org / suspended / free pool binding | 201 / 403 / 403 / 404 |
| Control-plane generate: shared hit / cross-org / free pool membership | 201 / 403 / 404 |
| Non-TiDB Cloud provider tenant | 409 |
| IAM role insufficient / IAM not enabled | 403 / 404 |
| Over the per-tenant 100 cap (three paths: generate, `POST /v1/tokens`, slock rotation) | All 409; a tenant with 100 expired keys is not limited; slock rotation at the boundary succeeds after freeing the issuer's slot; the slock callback maps `meta.ErrAPIKeyLimitExceeded` to 409 (other errors stay 500) |
| Pagination parameter boundaries: negative/non-numeric `offset`, non-numeric/≤0/over-max `limit`, `include_expired=foo` | Clamped to 0 / default / max; illegal `include_expired` → 400 (§4.9) |
| Cap boundary: `expires_at IS NULL` legacy scoped rows | Excluded from cap count; expired keys also excluded; only non-expired active/disabled owner+TTL keys count toward 100 |
| activate/deactivate idempotency | Same-state 200 with `changed=false`; racing a concurrent revoke yields no stale 200 |
| activate an expired disabled key | 409; NULL `expires_at` → 200 |
| disabled key → delete (owner and control plane) | 200 (revoke convergence) |
| Control plane `POST /v1/tokens/revoke` revoking an owner key by plaintext | 200; cross-org → 404 |
| list: contains no `jwt_hash`/`jwt_ciphertext`/plaintext; no revoked rows; `issued_by_*` is `tidbcloud` / `org:<id>` / the actual role | Assert the response body field-by-field against the §4.4 sample |
| list boundedness: 100 non-expired + many expired keys | Default list contains only non-expired rows; `include_expired=1` paginates at key level; org list passed `include_expired` → 400 |
| Org-wide list: skips pending/provisioning/deleting, includes failed/suspended | Asserted |
| Org-wide list pagination: a full page of pending tenants followed by an active tenant | Empty `tokens` but `next_offset = offset + limit` still returned (no silent truncation) |
| Org-wide list large tenant: >500 non-expired keys in one tenant | That tenant's enumeration truncated + its id in `truncated_tenants`; single-tenant list can complete it (§4.4) |
| Metric assertion: org list event | Prometheus output has exactly one `tidbcloud_org_id` label and no `tenant_id` label |
| Metric assertion: `api_key_manage` / `token_refresh` whitelist events | Carry `tenant_id` + `tidbcloud_org_id` (the `tenantAttributedBusinessEvent` extension takes effect) |
| Auth middleware rejects `disabled` / JWT-expired keys | 401 (middleware-level, zero-change regression) |
| `authorizedAdminTenant` thin-wrapper regression | Admin tenant API behavior unchanged on every path (refactor regression) |
| **Legacy behavior regression**: all existing `tokens_test.go` cases (issue / revoke / scoped cannot manage / cross-tenant / invalid policy, etc.) | All green with unchanged semantics |

**4. Concurrency / races**: N goroutines concurrently refresh the same key; exactly one succeeds (deterministic reproduction via the DB conditional UPDATE; no failpoint needed). The activate-vs-revoke interleaving tests the read-back contract.

**5. Compatibility / mixed deployment**: old-binary behavior is simulated with unit-level assertions (`disabled` rows rejected by `status != active`); the rollback scenario is verified as documented (§9.3).

**6. e2e**: this change makes zero semantic change to existing endpoints; the current `e2e/api-smoke-test.sh` and `e2e/journal-smoke-test.sh` are unaffected; refresh / control-plane smoke cases can be added later (separate PR, per `e2e/AGENTS.md`).

---

## 9. Compatibility and Rollout

### 9.1 Relationship to the existing `/v1/tokens`

**Decision: zero semantic change to existing endpoints.** The requests / responses / status codes of `POST /v1/tokens` (issue), `POST /v1/tokens/revoke`, and `DELETE /v1/tokens/{id}` are all unchanged (§4.2, §4.6); `GET /v1/tokens` (previously no GET semantics — 405 today) and all new sub-paths (generate / activate / deactivate / refresh) are purely additive. Response fields keep the existing `token`/`token_id` naming, so old clients need no mapping.

### 9.2 Impact on old clients and the middleware refactor's blast radius

- Old clients do not use the new endpoints; their behavior is entirely unchanged.
- **Bearer-path behavioral equivalence**: after the dispatcher refactor, Bearer traffic to `POST /v1/tokens`, `POST /v1/tokens/revoke`, and `DELETE /v1/tokens/{id}` still passes through the same `tenantAuthMiddleware` (logic unchanged, §8.3) and the same handlers — the only structural change is the wrapping position moving from mux-global to inside the dispatcher. Blast-radius control: all existing `tokens_test.go` cases staying green serves as the regression baseline (last row of §8.5-3).
- Old binary + new schema: compatible (§6.3, explicit column lists; `disabled` rows rejected by the old check).
- Mixed deployment (during rollout): both versions agree on `active/revoked` semantics; `disabled` rows are rejected by both; leftover `disabled`/`expires_at` data is safe after rollback (§6.3).
- `POST /v1/tokens` gains the per-tenant 100 cap (§7.4): a **perceivable** behavior change for old clients — extreme tenants will receive 409 on issuance (previously uncapped); the remediation is clear (delete old keys). The transactionalized slock rotation adds a new failure mode, "issuer-others filling all 100 → 409", likewise affecting only extreme tenants and safe to retry.
- The issue response's `expires_at` becomes second-precision (second-level alignment, §5.3): an intentional, compatible change — JWT `exp` was always second-level, so clients comparing at second granularity are unaffected.
- Beyond these, nothing perceivable changes: the legacy three endpoints' method semantics (405/404), response bodies (`{"status":"ok"}`), lenient JSON decoding, and the no-credential 401 message are all preserved (§4.2/§4.6/§4.9).

### 9.3 Gradual rollout / rollback

1. Deploy the DDL-carrying version first (the column is added by meta schema auto-repair, idempotently).
2. Endpoint-level gray release: the new sub-paths (generate/list/activate/deactivate/refresh) are zero-risk while unused; the gateway can first allow control-plane-shaped requests only for internal orgs.
3. The dispatcher sits on the full `/v1/tokens` traffic path — after launch, watch this family's 401/403 rates and auth latency closely (the `tenant_auth_timing` log already has per-phase timing, `auth.go:394`).
4. Rollback: revert the binary directly; do not DROP COLUMN.

### 9.4 Operational notes (including the required release checklist)

- **Release checklist (mandatory)**: ① configure IP-level rate limiting at the gateway for `/v1/tokens*` requests carrying the `X-TiDBCloud-Public-Key` header (body-only credentials rely on the WAF / IAM quotas, §7.4); ② alert thresholds for org-level `api_key_manage` frequency; ③ verify the metadb auto-repair has applied `expires_at` (startup logs / spot-check `information_schema`).
- The resolve cache TTL (`DRIVE9_META_AUTH_CACHE_TTL_MS`, default 10s) determines the **worst-case effective delay** of refresh/revoke/disable: the operational semantics are "not instantly effective globally".
- Control-plane endpoints depend on TiDB Cloud IAM availability; when IAM fails, the control-plane branch returns 502 — this does **not** affect the data-plane Bearer branch (the dispatch branches are independent).
- The transitional list deviation for legacy scoped rows (`expires_at`/`expired`, §6.4) and the scope-aware count predicate (§7.4) are written into the operations FAQ.

---

## 10. Open Questions and Recommended Decisions

| # | Question | Recommended decision | Alternative |
|---|---|---|---|
| Q1 | Does refresh need an old-token grace period? | No (§5.4). Minimizing the leak window comes first | 5-30s dual-active grace |
| Q2 | Record `last_used_at`? | Not in v1 (§6.1; write amplification breaks the resolve cache's premise) | Async batched updates |
| Q3 | Should the control plane be able to issue scoped tokens directly? | No (§4.1; scoped is data-plane semantics) | Copy `validateScopedTokenScopes` into the control-plane branch |
| Q4 | Do refresh / generate need an `Idempotency-Key`? | Not in v1 (§5.5); failed recovery goes through re-generate | Idempotency record table + 24h window |
| Q5 | May refresh also modify a scoped token's prefix/ops? | No; it only touches time and secret material; policy changes require re-issuance | Combined rotate-and-amend endpoint |
| Q6 | Should owner keys have a mandatory maximum lifetime? | Not enforced in v1; the `ttl_seconds` ceiling of 365 days (§4.3) already constrains the TTL-choosing case | Mandatory expiry for all keys + alerting |
| Q7 | Should the `/v1/tokens` scoped-issuance TTL ceiling be aligned to 365 days? | Unchanged in v1 (keeps `maxScopedTokenTTLSeconds` ≈ 292 years — status quo, not a regression); **default refresh must not clamp the derived period to 365d**, otherwise no-shortening + long issue TTL permanently blocks refresh. Converge issue ceiling in a separate PR if desired | Change the issue ceiling to `maxAPIKeyTTLSeconds` |

**Decided items that are no longer open questions** (recorded for reference): TTL ceiling (365 days, §4.3); per-tenant cap (100 + 409, all issuance paths including slock, §7.4); credential fallback order (body first, header fallback, §3.3); conflicting-credentials handling (400, §3.3); issuer metadata values (§4.3); org list filtering and pagination semantics (§4.4); single resource surface (§3.1).