# Tenant App-Managed Embedding Configuration **Status:** Proposed, approved for implementation **Date:** 2026-08-27 ## Summary Drive9 will add tenant-scoped app-managed embedding configuration for shared tenants and native tenants whose effective embedding mode is `fts_only`. Native tenants in TiDB database-managed `auto` mode remain unchanged. Document text, file descriptions, and search queries use the same effective OpenAI-compatible provider. The vector dimension remains fixed at 1024 because the tenant schemas use `VECTOR(1024)`; dimensions are not configurable through the environment or tenant API. The implementation reuses the existing durable `embed` task, semantic worker, revision-gated writeback, vector search, encryption, tenant pool, admin tenant authorization, and tenant notification outbox. It does not add a periodic tenant scan or wake a cold native tenant database. ## Goals 1. Allow shared and native `fts_only` tenants to configure one embedding provider for document, description, and query vectors. 2. Keep native TiDB `auto` tenants on database-managed embedding. 3. Prevent model changes from mixing old document vectors with new query vectors. 4. Backfill eligible existing files asynchronously without scanning unrelated tenants or opening a cold tenant database from the config API. 5. Avoid a MetaDB lookup on every file write or query. 6. Validate a provider before storing its endpoint, key, or model. ## Non-Goals - Migrating native `auto` tenants to app-managed embedding. - Configurable vector dimensions. - Separate providers for document and query embedding. - Independent `DRIVE9_QUERY_EMBED_*` configuration. - A fleet-wide tenant scanner or periodic cold-database discovery. - A new embedding queue or a second worker implementation. - Embedding configuration for layered workspace entries. ## Mode Boundary | Tenant mode | Embedding owner | Tenant `embedding-config` PUT | |---|---|---| | Native `auto` | TiDB generated columns and DB-side query embedding | Rejected as not applicable | | Native effective `fts_only` | App-managed provider and durable `embed` tasks | Supported | | Shared | App-managed provider and durable `embed` tasks | Supported | `DRIVE9_DISABLE_AUTO_EMBEDDING` is only a default-selection input for a native tenant whose persisted `embedding_mode` is still NULL: `true` resolves that tenant to `fts_only`, while `false` resolves it to `auto`. It is not an app-managed embedding kill switch. It has no disabling effect on shared tenants or native tenants resolved to `fts_only`, including a NULL mode resolved by this default. ## Process Default ```bash DRIVE9_EMBED_ENABLED=false DRIVE9_EMBED_API_BASE="" DRIVE9_EMBED_API_KEY="" DRIVE9_EMBED_MODEL="" DRIVE9_EMBED_TIMEOUT_SECONDS=20 ``` Rules: - `DRIVE9_EMBED_ENABLED` defaults to `false`. - When it is `true`, `API_BASE`, `API_KEY`, and `MODEL` are all required and startup fails when the provider probe does not return exactly 1024 values. - When it is `false`, the process has no default app-managed provider. A complete enabled tenant configuration can still activate embedding. - `DRIVE9_EMBED_DIMENSIONS` is removed. Requests use 1024 and responses with a different length fail validation and worker execution. - All `DRIVE9_QUERY_EMBED_*` variables are removed. Queries use the same effective provider as document and description embedding. - The process default is a fallback, not a cluster kill switch. ## MetaDB Schema Add one table independent from `tenant_auto_embedding_profiles`: ```sql CREATE TABLE IF NOT EXISTS tenant_embedding_configs ( tenant_id VARCHAR(64) NOT NULL PRIMARY KEY, enabled TINYINT(1) NOT NULL DEFAULT 0, api_base VARCHAR(512) NULL, api_key_cipher VARBINARY(2048) NULL, model VARCHAR(255) NULL, generation BIGINT UNSIGNED NOT NULL DEFAULT 1, config_revision BIGINT UNSIGNED NOT NULL DEFAULT 1, created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3) ) ``` `tenant_auto_embedding_profiles` remains the source of native schema mode and TiDB auto-embedding provider configuration. Reusing it would couple shared runtime credentials to native schema provisioning and is therefore rejected. An off-marker has `enabled=FALSE` and all three provider fields NULL. It keeps the counters so a later re-enable cannot accidentally make vectors produced by an older contract current. `generation` changes when the vector contract changes: `api_base`, `model`, or disable followed by re-enable. An API-key-only replacement does not change the generation because existing vectors remain compatible. `config_revision` changes for every meaningful config replacement, including an API-key-only replacement. It identifies a backfill attempt and permits a credential repair to retry files that previously failed without claiming that their vector contract changed. An exact full-replacement PUT is a semantic no-op: it preserves both counters and `updated_at`. Tenant terminal deletion removes this row in the same transactions that remove `tenant_extract_configs`. Encrypted credentials never outlive the tenant. ## Admin API ```http GET /v1/admin/tenants/{tenant_id}/embedding-config PUT /v1/admin/tenants/{tenant_id}/embedding-config ``` Both methods reuse the existing two-stage tenant admin authorization: 1. resolve and authorize the TiDB Cloud organization; 2. call `authorizedAdminTenant` to prove the tenant belongs to that organization. IAM resources are reused rather than added: - GET: `/drive9/orgs/{org_id}/admin/tenants/view` - PUT: `/drive9/orgs/{org_id}/admin/tenants/modify` ### GET ```json { "enabled": true, "api_base": "https://api.openai.com/v1", "api_key": "sk-a********", "model": "text-embedding-3-small", "source": "custom", "generation": 3, "updated_at": "2026-08-27T10:00:00Z" } ``` `source` is `custom`, `default`, `none`, or `database_auto`. API keys are always masked using the existing fixed-length masking helper. A native `auto` tenant returns `enabled=false, source=database_auto` and exposes no provider credential. ### PUT Enable or replace the custom provider: ```json { "enabled": true, "api_base": "https://api.openai.com/v1", "api_key": "sk-...", "model": "text-embedding-3-small" } ``` Disable app-managed embedding: ```json { "enabled": false } ``` PUT is intentionally a full replacement rather than a partial patch: - `enabled` is required. - `enabled=true` requires all three provider fields and rejects empty strings, explicit JSON null, unknown fields, and an `api_key` containing `*`. - `enabled=false` rejects provider fields and writes the off-marker. - No default API key is copied into a tenant row. - Native `auto` tenants receive HTTP 409 because app-managed configuration is not applicable. Before persistence, an enabled PUT uses the tenant-safe HTTP transport to call the proposed provider with a small fixed probe. The request must succeed and return exactly 1024 finite vector values. Validation follows the existing tenant endpoint security boundary: HTTPS only, no environment proxy, all DNS answers must be public, the connection is pinned to a validated address, TLS and SNI use the original hostname, and redirects are rejected. Provider validation runs before the config transaction. A failure returns a sanitized response in this form and does not modify the row: ```text embedding provider validation failed: ``` The transaction then locks the tenant row, rejects deleting/deleted tenants, and writes the config. This preserves the existing tenant-delete lock order. ## Effective Configuration Resolver The resolver has four results: `ready`, `disabled`, `not_configured`, and `database_auto`. - An enabled tenant row wins over the process default. - An off-marker returns `disabled`; it never falls back. - A confirmed `meta.ErrNotFound` falls back to the process default. - Any other MetaDB, decrypt, or client-construction error fails closed. - Native `auto` returns `database_auto` without building a client. The resolver reuses the extraction resolver's bounded pattern: singleflight, a short success TTL, a one-second failure backoff, a process-wide generation fence around eviction, and a janitor that physically releases cached plaintext keys. Config changes and tenant deletion evict both the resolver and tenant pool entry on every pod. The existing `tenant_notify_outbox` gains one broadcast cache-invalidation bit. The config transaction emits it. Every pod already polls this table and only invalidates local memory and pool state; it does not acquire a tenant backend or touch a tenant database. This provides cross-pod propagation without a new poller or a new table. On a tenant-pool cache miss caused by real request or worker activity, backend construction resolves effective app-managed state once: - `ready`: enable embed admission and wire a tenant-resolving query client; - `disabled` or `not_configured`: do not enqueue app `embed` work; - `database_auto`: retain the existing native auto path; - resolver error: fail the acquire rather than silently use stale/default credentials. Native `auto` construction explicitly clears any process-default app runtime, generation, config revision, query client, and app semantic-task admission. This prevents a process default from leaking across the database-managed mode boundary. Hot file writes read only the backend's in-memory admission state. They do not query MetaDB. An invalidation removes the cached backend but never opens a cold tenant database. The next real request rebuilds it with the new config. ## Embedding Trigger Contract In app-managed mode, create embedding work when the current file revision has non-empty `content_text` or non-empty `description` and its vector is missing or stale for either file revision or effective config generation. | Source | Trigger point | |---|---| | Small text create/overwrite | Same transaction that writes `content_text` | | Create/overwrite with description | Same transaction that writes the description | | Image extraction | Successful, non-empty, revision-matched writeback transaction | | Audio extraction | Successful, non-empty, revision-matched writeback transaction | | Video extraction | Successful, non-empty, revision-matched writeback transaction | | Future text extraction | Successful, non-empty, revision-matched writeback transaction | One `embed` task produces both `embedding` and `description_embedding` when both source fields are present. Empty output, a revision miss, a deleted file, or disabled/not-configured app embedding does not create follow-up work. The current image-only pre-enqueue rule is removed. Merely recognizing an image is not a reason to create an `embed` task before extraction has produced text. This removes an obsolete ACK and a second queue write for each image. Native `auto` mode never creates app `embed` work. ## Generation Correctness App-managed semantic rows add nullable columns: ```sql embedding_generation BIGINT UNSIGNED NULL, description_embedding_generation BIGINT UNSIGNED NULL ``` These columns exist only in app-managed schemas (`db9`, shared, and local). Native and zero providers in database `auto` mode retain their generated `EMBED_TEXT(..., 1024)` columns and do not gain app generation columns. For an existing native tenant that changes to `fts_only`, the normal tenant schema-version diff/repair path adds both generation columns; no separate repair mechanism or fleet scan is introduced. Embed task identity already includes the expected file revision. Its payload also records the effective generation, config revision, and whether the runtime came from a tenant-managed row or the process default. Worker writeback stores the matching generation alongside each vector. A stale worker may finish after a config change, but its vector remains invisible because app-managed vector queries require both: ```text embedding_revision = inode.revision embedding_generation = effective_generation ``` Description vector search applies the equivalent predicates. Query embedding and document embedding resolve the same effective client and generation. Tenant-managed config revisions are monotonic. Enqueue and execution compare the complete payload identity: newer tenant revisions supersede older work, while an old backend cannot replace or rewrite a task admitted by a newer revision; tenant-managed work also cannot be replaced by a stale process-default backend. An old backend defers newer work without calling the provider. If a claimed `embed` task carries an older configuration, or has the empty legacy payload, the current backend rewrites it to the effective identity and defers it. This prevents a late media writeback from superseding the current task and leaving the file without a current vector. When a new config revision revives an existing terminal `embed` row, its `attempt_count` is reset so an API-key repair receives a full retry budget. A requeue under the same config identity preserves the counter and cannot create unbounded retries. Dead-lettered embed tasks retain only their bounded, non-secret config identity; media task payloads continue to be cleared because they may contain extracted text. The process default uses deterministic, non-secret fingerprints: generation hashes `api_base + model + dimensions`, while config revision also includes a one-way hash of the API key. Restarting with the same configuration does not schedule another backfill. Changing only the key schedules recovery without invalidating compatible vectors. ## Backfill Add `embed_backfill` to the existing `semantic_tasks` table. It is not a new worker or persistence system. When a real request or existing semantic kick opens an app-managed backend, it inserts one backfill task identified by the effective `config_revision`. Insertion is deduplicated by the existing task uniqueness contract. The config API itself never opens the tenant database. The backfill task keyset-paginates confirmed semantic rows and selects only files for which at least one non-empty source lacks a current-generation vector. For each page it ensures normal `embed` tasks, checkpoints the inode cursor in `payload_json`, and defers itself without consuming an attempt. The task ACKs at the end of the tenant. Shared queries and writes always carry the existing `fs_id` scope. This gives the changed tenant a durable, bounded backfill while preserving two invariants: - no scan of unrelated tenants; - no cold tenant database wakeup caused by configuration or periodic work. ## Provider and Worker Failures - Resolver failures are retryable and never call a fallback provider unless the config row was confirmed absent. - Authentication, model-not-found, rate-limit, timeout, invalid JSON, empty vectors, non-finite values, and dimension mismatch are classified into bounded failure categories. - Metrics, structured logs, and durable task `last_error` retain the bounded category and provider HTTP status where applicable. They do not retain the provider response body/message, model, endpoint, or API-key material. - Provider errors remain visible through the existing embedding operation and semantic worker metrics; the implementation adds tenant/org attribution at the resolver/worker boundary without high-cardinality error labels. - Query embedding failure degrades to lexical search according to the existing search contract and is logged; it never switches to an unrelated provider. ## Deletion and Cache Lifecycle `MarkTenantDeleted`, `FinalizeTenantDelete`, shared deletion finalization, and `DiscardUnclaimedSharedTenant` delete `tenant_embedding_configs` in their existing terminal transactions. The outbox cleanup signal evicts resolver and pool state on every pod. In-flight provider calls are not cancelled across pods; generation and revision predicates prevent their results from becoming current. ## Validation Tests must cover: - environment disabled/default/invalid configurations and removal of all query-specific/dimension variables; - shared and native `fts_only` enablement while `DRIVE9_DISABLE_AUTO_EMBEDDING=true`; - native `auto` GET and PUT behavior; - cross-organization GET/PUT denial; - API masking, JSON null/unknown-field rejection, complete replacement, off marker, invalid provider, dimension mismatch, and transaction/delete races; - tenant-safe HTTP behavior including proxy, redirect, private DNS, and mixed DNS answers; - resolver fallback only on confirmed not-found, cache eviction, concurrent resolve/update fencing, failure backoff, and plaintext-key janitor cleanup; - native auto runtime isolation, stale/legacy task generation refresh, and provider-error sanitization in HTTP responses, logs, metrics, and durable task errors; - provider schema exports and native `fts_only` schema repair boundaries; - no MetaDB lookup on a hot backend write; - config invalidation on every pod without opening a cold tenant database; - small text and description trigger paths; - image/audio/video writeback bridges and removal of image pre-enqueue; - one task producing both vectors; - stale file revision and stale config generation writeback; - app vector search filtering by file revision and config generation; - scoped shared backfill, checkpoint/retry, deduplication, and no unrelated tenant scan; - all tenant deletion paths removing the encrypted config row.