# Tenant-Level AI Extract Config and Media Extraction Date: 2026-08-26 Status: Implemented, pending review ## Background Image, audio, and video AI extraction originally used process-level singleton configuration from env vars (`DRIVE9_IMAGE_EXTRACT_*`, `DRIVE9_VIDEO_EXTRACT_*`, etc.), shared by all tenants. This design makes extraction configurable per tenant, extends image extraction to emit structured key=value attribute tags that are directly queryable via `find&tag=k=v`, and defines bounded media processing for images, standalone audio, and video. Existing mechanisms that stay untouched: - `tenant_auto_embedding_profiles` + `ApplyTiDBAutoEmbeddingProviderConfig` (`SET @@GLOBAL.TIDB_EXP_EMBED_*`): still required by native/starter-mode tenants whose `semantic` table carries `EMBED_TEXT` generated columns (schema init/diff + runtime). Shared-schema tenants never used it. - Shared-schema tenants use plain app-managed VECTOR columns (no EMBED_TEXT), so nothing in this area applies to them. ## Goals 1. Per-tenant extract config, admin API, resolver integration, and worker execution for `image`, `audio`, and `video`; `text` remains reserved in the storage model. 2. Admin API to get/set the config, with api_key masking. 3. Image, audio, and video extraction emit customer-defined KV attribute tags (top-level keys in `file_tags`) guided by server-controlled prompt suffixes. 4. No billing/isFree gating — free tenants can use custom configs. Existing media-LLM file quota and monthly LLM cost budget remain enforced for tenant-configured extraction (usage is recorded to the tenant's ledger). 5. Large media sources use a bounded, disk-backed per-task workspace rather than retaining complete source payloads in the Go heap. Non-goals this phase: - Text extract-config API and text extraction worker (`text` rows remain storable, but the API rejects them). - Re-extract/backfill of historical revisions (explicitly out of scope; see Activation contract). - Cross-pod cache invalidation (see Tenant deletion). - Embedding configuration or embedding policy changes. ## Storage: `tenant_extract_configs` (meta DB) Executable DDL, following existing meta schema conventions (`tenant_auto_embedding_profiles`): ```sql CREATE TABLE IF NOT EXISTS tenant_extract_configs ( tenant_id VARCHAR(64) NOT NULL, media_type VARCHAR(16) NOT NULL, enabled BOOL NOT NULL DEFAULT FALSE, api_base VARCHAR(512) NULL, api_key_cipher VARBINARY(2048) NULL, model VARCHAR(255) NULL, prompt MEDIUMTEXT NULL, protocol VARCHAR(32) NULL, created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3), PRIMARY KEY (tenant_id, media_type) ) ``` - Provider fields (`api_base`/`api_key_cipher`/`model`/`prompt`/`protocol`) are nullable; an off-marker row is `enabled=FALSE` with all five set to NULL. - `media_type` allowed values (`image`/`audio`/`video`/`text`) are validated in the meta API layer, not by a DB constraint (consistent with existing meta tables). - Encryption reuses `pool.Encrypt/Decrypt`; `VARBINARY(2048)` matches the existing cipher column capacity. Three-state semantics per (tenant, type): | State | Behavior | |---|---| | Row absent | Fall back to server default (env) config | | Row, `enabled=true` | Tenant custom api_base/api_key/model/prompt/protocol in effect | | Row, `enabled=false` (off-marker; provider fields NULL) | Extraction of this type explicitly disabled for the tenant | ## Activation contract Process-level enablement flags such as `DRIVE9_IMAGE_EXTRACT_ENABLED` indicate whether a **process default** exists; they do not decide whether tenant-configured extraction is available. The three concepts are kept separate: - **task capability** — the server has a meta/pool + resolver runtime and can execute image, audio, and video tasks; - **task delivery** — durable media tasks are created for confirmed candidate revisions, subject to the existing quota-admission rules, and can be claimed; - **effective config** — resolved at task execution time: custom enabled → tenant provider; custom disabled → no extraction; row absent → process default (or none). Rules: 1. A multi-tenant server registers `img_extract_text`, `audio_extract_text`, and `video_extract_visual` capability whenever meta/pool + the resolver are configured, regardless of whether a process-default provider exists. 2. The file-write path determines candidate task types without reading tenant config. It never couples upload/finalize to a MetaDB config lookup and never wakes a cold tenant database only to resolve extract config. 3. After claim, the worker resolves the task's media type before reading file bytes or making any external request. 4. `enabled=false` and row absent with no process default are terminal `disabled`/`not_configured` outcomes. The claimed image, audio, or fresh video task uses the existing lease-fenced discard path. The worker also deletes at most 1000 due queued peers of the same type when their `attempt_count=0`, collapsing unconfigured upload bursts without querying MetaDB from the write path. Retried tasks are excluded because an earlier attempt may already have performed chargeable provider work; a video task with checkpoint progress is ACKed so its quota record remains durable. 5. Custom enabled → execute with the tenant extractor. 6. Meta lookup / decryption / extractor-build failures → retryable outcome; no extractor (default or custom) is called. 7. Falling back to the process default is allowed **only** on a confirmed `meta.ErrNotFound`. 8. Config updates affect only unfinished and future tasks. Revisions already terminally discarded or ACKed as `disabled`/`not_configured` are not replayed automatically; re-extract/backfill is a separate explicit feature, not implied by PUT. 9. Single-tenant / local server (`drive9-server-local`) is unchanged: env-based default config only, no resolver. 10. Native and shared-schema tenants use the same durable worker and runtime resolution path. Shared task and quota queries remain scoped by `fs_id`; enabling media extraction does not change shared-schema embedding policy. ### Emergency kill switch `DRIVE9_IMAGE_EXTRACT_GLOBAL_DISABLED=true` (default `false`), an execution-level switch independent of the default-config flag: - workers stop claiming new `img_extract_text` tasks; any already-claimed task halted before its external call is returned to the queue without consuming the claim's attempt; - file writes still create durable tasks, so nothing is permanently lost while the switch is on; - tasks are never ACKed `disabled` because of the switch — it is a temporary cluster state, not a tenant policy. ## Runtime resolution (workers) - `resolveExtractConfig(tenantID, type)`: meta read → build/reuse extractor. - In-memory cache keyed by (tenant_id, media_type) with a 60s retention budget: the effective cache TTL and the janitor sweep interval are both 30s (half the budget), so worst-case physical retention of an entry — including its decrypted provider key — is ≤ 60s. Config changes take effect within one effective TTL window (30s). - The janitor goroutine sweeps expired entries so decrypted provider keys are reaped rather than living for the process lifetime; `Close()` (wired into server shutdown) stops the janitor. Admin PUT writes and local tenant deleting/deleted transitions evict the tenant's entry on the local pod immediately. - Fail-closed: cached entries are not served past TTL when re-resolution fails; a resolution failure yields a retryable task outcome without calling any provider. - Concurrent misses for the same tenant on one pod are collapsed into one resolution. Resolution failures (meta read, decrypt, validation, or extractor construction) are cached fail-closed for 1s: callers receive the same retryable error and no provider is called. This short backoff covers the worker's initial retry burst without extending failure recovery to the 30s successful-resolution TTL. PUT and tenant deletion evict both successful and failed entries immediately. - The in-flight eviction fence uses one process-wide generation counter rather than a per-tenant generation map. Config writes are rare; accepting a retryable result for an unrelated in-flight resolution during an eviction is preferable to retaining an unbounded map of every evicted tenant ID. - Image extractor construction uses the same OpenAI-compatible implementation and basic-extractor fallback wrapper as the default path. Audio and video use the same protocol factories as their process defaults. The image OpenAI-compatible extractor treats empty provider output as an error (retryable); the fallback wrapper only covers a primary that succeeds with empty text. ### Existing tenant-pool activation contract This feature does not change tenant-pool or unified-worker activation: - request-driven `Pool.Acquire` and existing local/outbox worker kicks may open a tenant DB, as before; - periodic safety-net and storage-tier scans remain warm-only through `Pool.AcquireCached` and never open a cold tenant DB; - extract config is resolved only after a semantic task is claimed. Enqueue, outbox polling, and safety-net scanning never query extract config and no extract-specific tenant/config scan is added. This preserves the pre-existing event-driven behavior while ensuring tenant extract config introduces neither a new cold-DB activation path nor a fleet-wide MetaDB scan. Resolver caching, miss coalescing, and the 1s failure backoff bound the new per-task MetaDB read amplification. No distributed cache or cross-pod invalidation mechanism is introduced. ### SSRF controls for tenant-configured endpoints The extractor POSTs tenant file bytes and an Authorization header to the configured `api_base`, so tenant-controlled endpoints are an SSRF boundary. Tenant extractors use a **dedicated `http.Transport`** with: - `Proxy: nil` — environment proxies (`HTTP_PROXY`/`HTTPS_PROXY`/ `NO_PROXY`) are never consulted; - HTTPS-only `api_base` (plain HTTP rejected at validation and at dial time); - a custom `DialContext`: resolve the host and reject loopback, RFC1918, link-local (incl. cloud-metadata 169.254.169.254), and other non-public addresses; if any resolved address is non-public the dial fails (mixed DNS results are not trusted); the connection is pinned to a validated resolved IP (DNS-rebinding mitigation); - TLS certificate and SNI verification always against the **original hostname** — never against the pinned IP, never disabled; - redirects are not followed (3xx is an error). Applied to every tenant-configured image, audio, and video extractor. ## Admin API Type lives in the URL path; each type gets its own sub-path when supported. ``` GET /v1/admin/tenants/{tenant_id}/extract-config/image PUT /v1/admin/tenants/{tenant_id}/extract-config/image GET /v1/admin/tenants/{tenant_id}/extract-config/audio PUT /v1/admin/tenants/{tenant_id}/extract-config/audio GET /v1/admin/tenants/{tenant_id}/extract-config/video PUT /v1/admin/tenants/{tenant_id}/extract-config/video ``` - `image`, `audio`, and `video` are registered. `text` and unknown types return `400 {"error": "extract config for \"\" is not supported"}`. - Unknown tenant → 404 (uniform with existing admin anti-enumeration). ### Authorization (both dispatcher and legacy modes) Same two-step chain as existing admin tenant endpoints: 1. Authenticate + authorize the organization-level resource (innerauth dispatcher with the resource URIs below, or the legacy admin public/private-key path when the dispatcher is disabled); 2. `authorizedAdminTenant` → `resolveTiDBCloudAuthorizedTenant` binds the `{tenant_id}` in the URL to the authenticated organization (native org binding / shared pool ownership), returning 403 on cross-organization access and 404 on unbound/unknown tenants. No billing / isFree check. ### RBAC mapping Entries in `matchInnerControlPlaneRoute`: | Request | Resource URI | |---|---| | GET `…/extract-config/{image\|audio\|video}` | `/drive9/orgs/{org_id}/extract-config/view` | | PUT `…/extract-config/{image\|audio\|video}` | `/drive9/orgs/{org_id}/extract-config/modify` | Type is not part of the IAM resource URI, so enabling more types later does not require IAM policy changes. ### GET responses | State | Response | |---|---| | Custom row (`enabled=true`) | 200 config object, `api_key` masked, `"source": "custom"` | | Off-marker row (`enabled=false`) | 200 `{"enabled": false, "source": "custom"}` | | Row absent + image provider default | 200 default provider fields, `api_key` masked, `"enabled": true`, `"source": "default"` | | Row absent + runtime-only default | 200 `{"enabled": true, "source": "default"}`; audio also reports its effective protocol | | Row absent + no default at all | 200 `{"enabled": false, "source": "none"}` | Masking rules (apply to both custom and default): `api_key` longer than 4 chars keeps the first 4 chars with the rest replaced by `*`; keys of length ≤ 4 are fully masked. `api_base`, `model`, `prompt`, and `protocol` are returned as-is (not credentials). ### PUT semantics Body is the config object (no type wrapper). `enabled` is required in the request unless an enabled custom row already exists. State transition table (existing row state × request): | Existing state | Request | Result | |---|---|---| | absent | `enabled` omitted | 400 (`enabled` required) | | absent | `enabled: false` | create off-marker (no trio required) | | absent | `enabled: true` | 400 unless full trio (`api_base` + `api_key` + `model`) provided | | off-marker | `enabled` omitted | 400 (`enabled` required) | | off-marker | `enabled: false` | no-op, returns off-marker | | off-marker | `enabled: true` | 400 unless full trio provided (previous fields were cleared) | | enabled | `enabled` omitted | stays enabled; omitted fields unchanged | | enabled | `enabled: false` | becomes off-marker (provider fields cleared) | | enabled | `enabled: true` | omitted fields unchanged; changing any provider-trio member requires and replaces the full trio | Field rules: - Explicit JSON `null` for any field (`enabled`/`api_base`/`api_key`/ `model`/`prompt`/`protocol`) is invalid → 400. Omission and explicit null are never conflated. - `api_key` omitted = keep existing (enabled rows only); provided = replace (stored encrypted). A value containing `*` is rejected (masked echo-back). The server never copies the default api_key into tenant storage. - On an already-enabled row, providing any of `api_base`, `api_key`, or `model` requires providing all three. Treating the provider trio as one replacement unit lets the server validate exactly what it will persist without decrypting the existing key in an extra MetaDB read. Prompt-only updates remain partial. - `prompt` omitted = keep existing (enabled rows) / no custom prompt (create); `prompt: ""` explicitly clears the custom prompt — normalized to NULL, runtime falls back to the default prompt body; persisted NULL is a legal state (off-markers require NULL provider fields, and enabled rows use NULL to mean "no prompt override"). - `api_base`/`model`/`api_key` empty strings are rejected on enabled rows (the trio must stay complete); clearing happens only via `enabled:false`. - `protocol` omitted defaults to `openai` when creating or re-enabling a row, but preserves the stored protocol on an already-enabled row. An explicitly provided empty string normalizes to `openai`. Image and video support only `openai`; audio supports `openai` and `qwen-asr`. Changing the audio protocol (including an explicit empty string) requires the complete provider trio in the same request, so a protocol-only audio PUT is rejected and the exact candidate can be preflighted without an additional MetaDB read. Disabling clears `protocol` with the other provider fields. - Validation: `api_base` must be HTTPS (see SSRF controls); `prompt` length cap 8 KiB. - Concurrency: read-modify-write runs in one transaction with `SELECT ... FOR UPDATE` on the (tenant_id, media_type) row, so concurrent partial PUTs cannot lose each other's omitted fields. For an audio provider trio validated with an omitted protocol, the transaction-time effective protocol must still match the protocol used by the preflight. A concurrent protocol change aborts the PUT with 409 and asks the caller to retry; an unvalidated provider trio is never persisted under the new protocol. - Response: 200 with the updated masked config (same shape as GET). ### Mandatory provider validation PUT performs a real provider check before persisting a new provider trio. The check runs after authorization and tenant ownership validation, but before opening the MetaDB write transaction. It is required when creating or re-enabling a custom config and whenever an enabled config replaces its provider trio. Disabling a config and an enabled PUT that omits the provider trio (including a prompt-only update) do not call the provider. The check reuses the production provider request path and the tenant SSRF-safe HTTP client: - image and video send a small built-in PNG with a fixed short prompt and a small output-token limit through the OpenAI-compatible vision path. A non-empty valid response is required; - audio sends a bounded silent WAV through the selected `openai` or `qwen-asr` path. A typed successful empty transcript is valid because silence is a legitimate result. A model-list request is not used because compatible providers do not consistently implement it and listing a model does not prove the required media capability. The probe may incur a small charge on the tenant-supplied provider account. It is an admin configuration check, so it is not written to the media usage ledger and does not consume the runtime extraction cost budget. Provider I/O never runs while a MetaDB transaction or row lock is held. The API key is encrypted and written only after validation succeeds. A failed check leaves the existing row unchanged and returns a safe error beginning: ``` extraction provider validation failed: ``` The response never contains the API key, request body, probe media, complete provider response body, or provider-supplied headers. Stable mappings cover authentication rejection, endpoint/model not found, request or media-capability incompatibility, rate limiting, timeout/network failure, provider 5xx, and an invalid/empty success response. Provider rejections caused by the supplied configuration return 400; provider availability failures return 502, and a validation timeout returns 504. ### Provider failure retention and alerting The provider clients expose a typed, low-cardinality failure category and HTTP status to both the PUT validation path and the media workers. They do not expose the complete provider response as an error string. Workers keep the existing retry/backoff/dead-letter behavior. Runtime provider failures are retained without persisting exception details: - structured warning logs record tenant/task identity, media type, attempt, failure category, and HTTP status, but never credentials, payloads, media, or the provider response body; - existing tenant metrics use `_extract`, with operation `provider_validate` for PUT checks or `provider_request` for worker calls, and the stable failure category as `result`; no model, URL, API key, or free-form error is used as a metric label; - `semantic_tasks.last_error` receives only fixed media-specific codes such as `image_extract_provider_auth_rejected`, `audio_extract_failed`, or `video_extract_provider_auth_rejected`, never `err.Error()` or other provider-supplied text. Task status and attempt count continue to show the retry/dead-letter lifecycle. No validation-status column, validation history table, separate validation endpoint, background validation worker, periodic provider probe, or new tenant/config scan is added. A key revoked or model removed after PUT is therefore detected by the next real task, then surfaced through the same log, metric, fixed task error code, and existing retry path. ## Media execution and workspace ### Runtime and temporary workspace `DRIVE9_EXTRACT_TMP_DIR` configures the process-local media workspace and defaults to `os.TempDir()` (normally `/tmp` in Linux containers). Production deployments should point it at a disk-backed, size-limited ephemeral volume. Tasks that need staging or derived artifacts create a private `0700` directory, write files with restrictive permissions, and remove the directory on success, failure, cancellation, or timeout. Temporary workspace paths and media bytes never enter task payloads. `DRIVE9_EXTRACT_MAX_CONCURRENT_TASKS` controls the workspace semaphore and defaults to 4. Existing semantic-worker and per-tenant concurrency settings remain the primary scheduling controls; the workspace is an additional bound only around disk staging and heavy preprocessing. Waiting for a workspace slot is context-bound. Disk exhaustion and temporary resource contention remain retryable failures. S3 images whose declared size is within the provider limit are read directly through a bounded reader and do not create a task workspace. Oversized JPEG/PNG/GIF sources that are eligible for normalization stream directly from S3 into the task directory; the original staged file is inspected and decoded from disk, and only the bounded normalized JPEG enters the Go heap as the provider payload. The image task directory and workspace slot are released as soon as normalization completes, before provider network I/O. DB-inline images are already memory-backed and are not redundantly staged to disk. If workspace initialization is unavailable, S3 media may use the existing in-memory path only up to its provider-scale limit. Larger sources return a retryable workspace-required load error instead of allocating against the GiB source ceiling or being misclassified as terminal `too_large`. Source staging and provider payloads use separate process-level limits: | Media | Source staging limit | Default | Provider payload limit | Default | | --- | --- | ---: | --- | ---: | | image | `DRIVE9_IMAGE_EXTRACT_MAX_SOURCE_BYTES` | 1 GiB | `DRIVE9_IMAGE_EXTRACT_MAX_BYTES` | 8 MiB | | audio | `DRIVE9_AUDIO_EXTRACT_MAX_SOURCE_BYTES` | 5 GiB | `DRIVE9_AUDIO_EXTRACT_MAX_BYTES` | 32 MiB for OpenAI-compatible, 7 MiB for `qwen-asr` | | video | `DRIVE9_VIDEO_EXTRACT_MAX_SOURCE_BYTES` | 10 GiB | `DRIVE9_VIDEO_EXTRACT_MAX_PAYLOAD_BYTES` | 8 MiB | For compatibility, `DRIVE9_VIDEO_EXTRACT_MAX_BYTES` is used as the video source limit only when `DRIVE9_VIDEO_EXTRACT_MAX_SOURCE_BYTES` is unset. The old `DRIVE9_VIDEO_EXTRACT_MAX_FRAMES` cap is intentionally ignored because five-minute segmentation and duplicate-frame removal now bound visual work; startup logs a deprecation warning when it is still configured. The source limit bounds bytes streamed into one task workspace. The provider limit bounds one normalized image, audio segment, or visual request. Tenant configuration selects a provider but cannot enlarge either process-level resource boundary. `DRIVE9_MAX_UPLOAD_BYTES` remains an independent HTTP upload limit; a newly uploaded file must satisfy both the upload limit and its media source limit. Existing files are evaluated against the source limit when their extraction task runs. Declared object size is checked before staging, and the streaming copy remains bounded so incorrect metadata cannot bypass the source limit. A source over its configured limit is terminal `too_large`. Disk exhaustion, inode exhaustion, and temporary workspace contention are runtime resource failures and remain retryable; they must not permanently classify a valid source as `too_large`. ### Durable task routing and writeback The revision-scoped `video_extract_visual` task owns the composite video result. It runs visual and, when an audio runtime is available, audio phases, then writes `content_text` once after composing both outputs. An MP4 may also receive an `audio_extract_text` task when audio capability is present. If a video task was admitted, that audio task is marked as a fallback. After claim, the worker discards the fallback when the effective video runtime has an audio extractor; otherwise it runs independently. This preserves audio extraction when video is disabled, not configured, visual-only, or rejected by video quota without allowing two workers to compete for `content_text`. Standalone audio files use `audio_extract_text`. Large audio is staged, transcoded, split, and sent segment by segment; small audio may retain the existing direct provider path. All writebacks are gated by `file_id + revision`, so stale tasks cannot update a newer revision. After a successful writeback, existing embedding behavior is unchanged: TiDB auto-embedding tenants let the database generate the vector, while app-managed embedding mode queues its existing embed task. ### Checkpoint and retry contract Audio segment progress, video visual-segment progress, and video modality progress are stored in the existing semantic-task payload. Checkpoint writes are fenced by task ID, processing state, receipt, and unexpired lease. `payload_json` remains capped at 64 KiB. Standalone audio reserves at most 40 KiB of JSON-encoded checkpoint text. In-progress video visual extraction keeps only a bounded summary for each completed segment, the next segment cursor, the deterministic segment count, and bounded merged tags; it never stores frames or provider request bodies. Completed visual synthesis clears the intermediate segment summaries and retains the existing at-most-20-KiB visual result. The audio modality keeps its existing at-most-20-KiB checkpoint text. An audio or visual segment cursor advances only with the corresponding bounded checkpoint. Once a complete payload no longer fits, durable state remains at the last recoverable cursor and a retry may repeat provider calls after that point. JSON-budget checks are UTF-8 safe and account for escaping. Payloads contain only bounded cursors, phase state, transcript prefixes, and task metadata. They never contain API keys, raw media, frames, audio bytes, base64 payloads, workspace paths, or unbounded transcripts. Completed segments and modalities represented by a complete durable checkpoint are not sent to a provider again after retry. Visual batches remain an in-memory implementation detail inside one five-minute segment; they do not add another durable payload state machine. Successful media-provider calls are written to the existing usage ledger for each task attempt before a retryable error or writeback result is returned. A retry bills only provider calls made during that retry; resuming an already-complete audio or visual segment without a provider call creates no fallback charge. If a later visual batch fails, retry may repeat earlier batches from that same incomplete five-minute segment; each successful repeated call remains visible in usage accounting. This keeps the durable contract at the existing segment boundary instead of adding batch-level checkpoint state. Before the final attempt, a transient modality failure retries without writeback. On the final attempt: - video writes the successful modality as `partial_success` when only one phase succeeds; both phases failing dead-letters without empty writeback; - standalone audio writes `partial_success` when at least one segment succeeded; no successful segment dead-letters without empty writeback; - a video with no audio track writes its visual result with the audio phase marked absent, not failed. ### Video visual phase The video source is streamed to the task workspace and processed by FFprobe and FFmpeg. Visual extraction has an explicit two-hour support boundary. A longer duration returns terminal `too_long` before any visual-provider call; the worker never produces a misleading sparse summary for an unsupported timeline. Supported videos are processed as follows: 1. Probe the complete duration once. 2. Split the timeline into consecutive five-minute segments, at most 24. 3. Within every segment, one primary FFmpeg process samples one candidate every five seconds from the segment start, and one additional bounded FFmpeg call captures an explicit near-end boundary. There is no retained-frame-count cap. A five-minute segment produces at most about 61 candidates, and a two-hour video produces at most about 1464 candidates. 4. Decode bounded JPEG frames and reject frames over 1920 pixels on either dimension or over 1920x1920 total pixels. 5. Compute grayscale dHash values as candidates are read and remove near-duplicates within that segment while retaining distinct timeline boundaries. Only hashes and retained frames are kept; rejected candidates are removed immediately. 6. Build 2x2 contact sheets from every retained frame. Do not discard a distinct frame to satisfy an arbitrary frame or sheet count. Pack contact sheets into deterministic batches whose encoded bytes do not exceed `DRIVE9_VIDEO_EXTRACT_MAX_PAYLOAD_BYTES`. 7. Send one visual-provider request per batch and keep its bounded summary in memory. After all batches succeed, synthesize them into one bounded segment summary. A batch failure retries the incomplete five-minute segment; it does not create a second durable checkpoint protocol. 8. Checkpoint the segment summary, merged attributes, usage, and next segment cursor before continuing. Completed segments are not resent after retry. If FFmpeg succeeds but a later segment produces no frame files after at least one segment summary is durable, checkpoint only the advanced cursor and skip its provider call. This covers containers whose audio track extends past the decodable video stream. An empty first segment, FFmpeg failure, or emitted frames that all fail validation remains retryable and fail-closed. 9. After all segments complete, send one bounded text-only request through the same OpenAI-compatible provider to synthesize the ordered segment summaries into the final structured visual result. Contact sheets are not resent, and the final completion plus intermediate-summary cleanup use one checkpoint write. The configured visual timeout applies independently to duration probing, each batch request, segment synthesis, and final synthesis; it is not a wall-clock limit for the complete multi-request visual phase. A retry reconstructs deterministic segment and batch boundaries and resumes from the first incomplete segment. Derived frames live under the task directory and are removed with it. Only the bounded retained frames/contact sheets are materialized for the provider request. ### Audio phase Video audio tracks and standalone audio use the same bounded FFmpeg helpers. Derived audio is mono 16 kHz, 32 kbps MP3. Processing is limited to two hours, at most twelve segments, at most ten minutes per segment, and 40 MiB of total derived audio. Duration is probed before transcoding; a source over two hours returns terminal `too_long` and is never silently truncated. A duration-probe failure is retryable and does not call the provider. Segment duration is shortened when needed to satisfy the effective provider byte limit; `qwen-asr` uses a 7 MiB segment cap. The worker, rather than the ASR model, preserves coarse source time. It prefixes every transcript with an absolute source range such as `[Audio 00:10:00-00:20:00]`. A small unsplit file receives one range from zero to its probed duration; a split file receives one deterministic range per segment, with the final end clamped to the source duration. This applies equally to standalone audio and video audio tracks. Provider-native word or sentence timestamps are outside this contract; prompts are not treated as a reliable source of timing metadata. The effective audio runtime timeout bounds the complete audio phase, including FFprobe, FFmpeg segmentation, and all provider calls. Cancellation or deadline expiry terminates the local command and returns the context error through the normal retry path. The selected provider protocol is explicit. No model-name inference chooses a protocol, and runtime processing does not add a separate silence-detection provider call. Transcripts are assembled in source order. ### Image spill-to-disk Images at or below the effective provider payload limit (8 MiB by default) may use the existing in-memory path. A larger supported S3-backed source is streamed to the task workspace, reopened for `DecodeConfig` and normalization, and converted to a bounded JPEG. Only the normalized provider payload is then held in memory. The independently configured source cap defaults to 1 GiB. The 16-million decoded-pixel cap, provider payload cap, and single process-wide normalization slot remain in force. A non-positive source or provider limit is invalid, not unlimited. ### Shared-tenant, quota, and observability constraints - Shared semantic-task and quota queries always include `fs_id`; one shared tenant cannot consume another tenant's task admission or usage. - The existing general media quota applies to image/audio admission and the existing video quota applies to video-file admission. The monthly LLM cost budget is checked once at task entry, avoiding per-segment MetaDB reads; the hard 24-segment plus one-synthesis boundary limits one admitted task's exposure. Every successful segment and synthesis response is still recorded separately in the existing usage ledger before a later failure can return. - Fresh image, audio, and video tasks resolved as disabled or not configured are lease-fenced and discarded. Up to 1000 additional due, queued, never-attempted tasks of the same type are deleted through the tenant store; shared stores apply the existing `fs_id` scope. Future-due, leased, retried, and different-type tasks are untouched. Central video-file usage increments once per visual attempt at its first external HTTP call boundary after local frame extraction, validation, deduplication, contact-sheet construction, and payload validation succeed; later segments in the same attempt do not create repeated MetaDB counter writes. Local pre-provider failures and retries that only reuse a completed visual checkpoint do not increment it. Successful visual token/cost usage is recorded per provider response before checkpoint or writeback errors can return. - Bounded metrics cover task result, source/workspace/normalization failures, frame-count buckets, audio segment outcomes, provider categories, retries, partial success, and dead letters. Labels contain no credentials, raw error text, model, URL, or unbounded tenant-provided value. ## KV attribute tags (media extraction) ### Prompt composition Prompt composition follows each provider protocol's actual prompt semantics: - **Image:** the base is the tenant prompt when non-empty, otherwise `defaultImageExtractPrompt`. The effective prompt is `base + fixed image attributes suffix`. - **Video visual:** the base is the tenant prompt when non-empty, otherwise `defaultVideoExtractPrompt`. The effective prompt is `base + fixed video attributes suffix`. Frame batches, per-segment synthesis, and final timeline synthesis all receive this same effective prompt. The video default body does not maintain a second copy of the attributes rules. - **Audio and video audio tracks (`qwen-asr`):** `defaultAudioExtractPrompt` is the first system-message section. A non-empty tenant prompt is appended as context and vocabulary, followed by the fixed JSON `transcript` and `attributes` contract. Plain transcript text remains accepted and produces no generated attributes. - **Audio and video audio tracks (`openai` transcription):** the multipart `prompt` field is previous-text/style/vocabulary conditioning rather than an instruction channel. It therefore contains only the trimmed tenant prompt and is omitted when blank. The server does not put its transcription or attributes contract in that field; OpenAI-compatible providers returning plain text produce transcript text without generated attributes. The fixed attributes suffixes used by image, video visual, and instruction-capable audio providers use the same contract: at most 30 retrieval-valued scalar entries, key length at most 64 characters, rendered value length at most 255 characters, and no `drive9.` key prefix. They are not tenant-overridable. For `qwen-asr`, a persisted NULL or empty prompt removes only the tenant section and retains the built-in task and output contract. For `openai` transcription it omits the optional conditioning field. An explicit JSON `prompt: null` remains invalid. The same protocol-specific rules apply to process defaults. Video combines two independently configured provider paths. The video prompt controls visual extraction only. Its audio track uses the effective audio prompt from the tenant audio config (or audio process default). Final video tags merge visual and audio attributes through the existing bounded merge, with audio retaining precedence for identical keys. ### `file_tags.source` column (tenant schema change) ``` source VARCHAR(32) NOT NULL DEFAULT 'user' -- 'user' | 'image_extract' | 'audio_extract' | 'video_extract' ``` Primary keys stay as they are per provider: standalone TiDB/MySQL/DB9 keep `(file_id, tag_key)`; shared schema keeps `(fs_id, file_id, tag_key)`. The query protocol and `find&tag=k=v` behavior are unchanged. All source-conditional deletes, conflict checks, and inserts in the shared store always carry `fs_id` (via the existing scope helpers). Semantics: 1. Existing rows migrate to `source='user'` via the column default. 2. Normal writes / overwrite / upload-complete keep using `ReplaceFileTagsTx`; INSERTs do not set `source`, so they are `user`. 3. Media writeback (in the existing revision-gated transaction): a. delete all rows for that worker's source (`image_extract`, `audio_extract`, or `video_extract`); b. image writeback also deletes legacy `drive9.image.*` rows (old numbered format); c. insert new attributes in stable order with the corresponding media source; d. if a `tag_key` already exists with `source='user'`, the user value wins: skip the attribute, bump a metric, never overwrite, never retry; e. empty attributes (or non-structured fallback text) still delete that worker's old generated rows, so stale attributes never linger. 4. A user's next explicit tag write replaces the whole set as `user`, so a user can naturally take over a previously model-generated key. 5. Revision miss → neither tags nor source change (existing gate). Schema shape change checklist (per AGENTS.md): - `pkg/tenant/schema/tidb_auto.go` - `pkg/tenant/schema/tidb_app.go` - `pkg/tenant/schema/tidb_shared.go` - `pkg/tenant/schema/mysql_no_embedding.go` - `pkg/tenant/db9/schema.go` - schema diff / shared-schema parity / schema version bumps - `dump-init-sql` artifacts regenerated for `tidb_zero`, `tidb_cloud_native`, `db9`, `tidb_cloud_native_shared` ## Tenant deletion All transactions that mark a tenant deleted must delete the tenant's `tenant_extract_configs` rows **in the same transaction**: - `MarkTenantDeleted` (meta.go) — no-namespace path; - `FinalizeTenantDelete` (meta.go) — async delete path; - `FinalizeSharedTenantDeleteMetadata` (db_pool.go) — shared tenant path; - `DiscardUnclaimedSharedTenant` (db_pool.go) — defensive: unclaimed free tenants cannot hold configs today (admin auth rejects them), but deleting here keeps the invariant true if that ever changes. This is enforced in the meta layer (not only in individual server callers), so every deletion shape wipes the stored credential. ### In-flight tasks during deletion (lightweight contract) Deleting the tenant's rows prevents **new** claims, but tasks already claimed by a worker goroutine may run to completion — bounded by the task/provider timeout — and may issue one more provider request during or after deletion. Their writeback/finalize then fails or no-ops because the underlying data is gone. The cache never spawns new work on its own. Per-tenant in-flight cancellation (with cross-pod propagation and HTTP abort semantics) is explicitly **not** built this phase. Resolver caches are process-local; other pods may physically retain a decrypted extractor for up to the 60s retention budget after deletion (an entry stops being served after the 30s effective TTL and is reaped by the next 30s sweep). The writing/deleting pod evicts its own entry immediately (admin PUT, local lifecycle transitions), and every pod's janitor reaps the entry within the retention budget. No cross-pod invalidation is built this phase. ## Cleanup Remove the dead shared-wave auto-embedding profile plumbing (it always produces nil for `tidb_cloud_native_shared` because `UsesTiDBAutoEmbedding(shared) == false`): - `admin_tenant_pool.go`: the wave-member profile build call. - `meta.ManagedSharedDBPoolWaveMember.AutoEmbeddingProfile` field, its validation, and the `tenant_auto_embedding_profiles` INSERT branch in `shared_pool_wave.go`. The implementation also removes avoidable feature-local complexity: - parse the process-default image provider environment once and derive both its runtime and admin view from that result; - keep admin presentation state on `Server`, not on the runtime resolver; - keep the complete attributes prompt contract, including the 30-attribute limit, in the mandatory suffix instead of declaring `attributes` twice; - replace the exported MetaDB transaction test hook with a scoped failpoint in a `//go:build failpoint` test. ## Testing - meta: new-table CRUD (upsert/get/delete, encryption round-trip); all four tenant-deletion paths (`MarkTenantDeleted`, `FinalizeTenantDelete`, `FinalizeSharedTenantDeleteMetadata`, `DiscardUnclaimedSharedTenant`) wipe config rows in-tx; retry-safe deletion; deterministic claim → blocked extractor → tenant delete → release-extractor concurrency test verifying the lightweight in-flight contract (in-flight task completes or fails its writeback, no new claims after deletion). - schema: migration leaves existing tags as `source='user'`; shared-schema parity; dump-init-sql outputs updated. - resolver/worker: row absent → default; `enabled=false` → terminal `disabled` discard; absent + no default → terminal `not_configured` discard; fresh due peers are discarded in bounded type- and `fs_id`-scoped batches while attempted tasks remain durable; meta/decrypt/build failure → retryable without calling any provider; fallback only on `ErrNotFound`; concurrent same-tenant misses issue one MetaDB read; failures are served fail-closed from the 1s backoff cache; eviction fences in-flight resolution without retaining per-tenant generation state; TTL cache behavior (incl. two independent resolver instances observing the same tenant); kill switch (no new claims, in-flight pre-provider tasks deferred without consuming an attempt, tasks still enqueued). - API: masking (custom + default; keys of length ≤ 4 — including exactly 4 — fully masked); image/audio/video GET and PUT; text/unknown rejection; PUT state transition table (absent/off-marker/enabled × omitted/false/true); full-trio requirement on create/re-enable; masked api_key echo-back rejected; explicit JSON null for enabled/provider/prompt/protocol fields rejected; changing any provider-trio member requires the full trio; mandatory provider validation succeeds before persistence and each classified failure leaves the prior row unchanged; prompt-only/disable/unchanged PUTs do not probe; validation runs outside the MetaDB transaction; safe 400/502/504 responses; omitted protocol defaults to `openai` on create/re-enable and preserves the stored value on an enabled row; explicit empty protocol normalizes to `openai`; image/video reject non-OpenAI protocols; audio accepts `openai` and `qwen-asr`; protocol-only audio updates are rejected; a concurrent protocol change after provider preflight returns 409 without persisting the provider trio; image/video require a non-empty vision probe response while a typed successful empty audio transcript is accepted; concurrent partial PUTs do not lose omitted prompt fields; unknown tenant 404; cross-organization GET/PUT rejected for native and shared tenants (dispatcher and legacy key modes). - SSRF: literal and DNS-resolved private/loopback/link-local addresses rejected; mixed public/private DNS results rejected; redirects rejected; plain HTTP rejected; `HTTPS_PROXY` set in the environment does not reach the tenant endpoint (dedicated transport, `Proxy: nil`); TLS verified against the original hostname; the mandatory provider probe follows the same controls. - observability: validation and runtime provider failures log only safe structured fields, increment existing tenant metrics with stable categories, and store only a fixed code in `semantic_tasks.last_error`; tests assert that provider response text, credentials, URL, model, request body, and media bytes appear in none of those sinks. Key revocation/model removal keeps the existing retry and dead-letter behavior while emitting category metrics. - prompt composition: explicit `prompt: null` in PUT returns 400 and `prompt: ""` clears the tenant section (persisted NULL); image/video tenant prompt becomes tenant base plus media suffix; `qwen-asr` retains the default transcription instructions and audio suffix around optional tenant context; `openai` transcription sends only trimmed tenant conditioning and omits the multipart prompt field when blank; process defaults follow the same protocol-specific rules. - attributes: limits (count/length), reserved `drive9.` prefix, malformed JSON, `file_tags.source` writeback (re-extract deletes old generated rows, user same-key wins, user takeover via explicit write, empty attributes cleanup, revision miss leaves tags/source unchanged), attribute values present in `content_text`. - media workspace: default and explicit root, `0700` task directories, context-bound single-slot contention, bounded streaming, cleanup on every outcome, and disk/resource failures. - video: five-minute segmentation across a two-hour timeline, five-second candidate sampling with start/end coverage, dHash deduplication without a retained-frame cap, deterministic contact-sheet payload batching, visual segment checkpoint/resume, default/custom visual prompt composition with the fixed attributes suffix in batch and synthesis requests, frame dimension/pixel validation, configured provider-payload enforcement before each request, terminal `too_long` before provider access, bounded text-only segment/final synthesis, per-request timeouts, process-level source-limit inheritance, retry-safe usage accounting, visual/audio resume, no-audio behavior, final-attempt partial writeback, both-modalities-failed dead letter, and stale-revision fencing. - audio: configurable source limit, explicit `too_long` without truncation, two-hour/twelve-segment/40 MiB derived limits, ten-minute and provider-byte segment bounds, ordered checkpoint resume, default instructions plus tenant context/vocabulary plus fixed attributes suffix for both protocols, plain-text provider fallback, deterministic absolute time ranges for unsplit and segmented audio (including video audio), timeout coverage for blocking FFmpeg/FFprobe preprocessing, full final output beyond the durable checkpoint budget, retry-safe per-attempt usage accounting, UTF-8/JSON-safe checkpoint bounds, partial final-attempt writeback, and no empty writeback when every segment fails. - task routing: MP4 fallback audio is discarded only when video owns the audio modality; disabled/not-configured/visual-only/video-quota-rejected paths keep independent audio behavior; fresh disabled/not-configured video tasks do not consume future video quota. - media size configuration: independent image/audio/video source limits and provider payload limits, default values, invalid non-positive values, declared-size early rejection, bounded-reader enforcement for incorrect metadata, and retryable disk/inode exhaustion. - image spill: oversized supported S3 images stage to disk, enforce the configured source and decoded-pixel caps, normalize to the provider limit, serialize normalization, clean workspaces, and make no provider call on rejection. - shared mode: every task/quota operation is isolated by `fs_id`, including two tenants in one physical database; upload/finalize performs no extract config lookup and no extract-specific path wakes a cold tenant database.