# O11y TiDBGroupAgent Tactical Export Fix ## Current Issue Cluster Service creates one `TiDBGroupAgent` custom resource per TiDBGroup. The CR name is the TiDBGroup ID, but the O11y identity inside the spec is cluster-scoped: - `spec.tidbGroup = ` - `spec.clusterId = ` - `spec.o11y.tenantId = ` - `spec.o11y.applicationId = ` This means multiple TiDBGroupAgent CRs for the same TiDB cluster can share the same O11y backend identity. The global control plane currently creates one `DEFAULT` TiDBGroup for each TiDB cluster. During export tasks, it creates an extra `INTERNAL` TiDBGroup and deletes that internal group when the export task finishes. The export path can therefore create and then delete an internal TiDBGroupAgent whose `applicationId` is the TiDB cluster ID. If the O11y TiDBGroupAgent controller treats TiDBGroupAgent deletion as permission to delete backend O11y cluster metadata by `applicationId`, deleting the temporary internal agent can delete O11y metadata for the whole TiDB cluster even though the default TiDBGroup still exists. There is a similar shared-resource shape for TiProxy scrape resources: a TiDBGroupAgent can manage resources whose identity is cluster-level rather than TiDBGroup-level. ## Goal Stop export-created `INTERNAL` TiDBGroups from triggering deletion of cluster-level O11y metadata, without changing the O11y TiDBGroupAgent controller. The tactical lifecycle invariant is: - Only `DEFAULT` TiDBGroups create or repair TiDBGroupAgent CRs. - TiDBGroup deletion does not delete TiDBGroupAgent CRs. - TiDB cluster deletion owns cleanup of all TiDBGroupAgent CRs for that TiDB cluster. ## Non-Goals - Do not redesign the long-term O11y ownership model in this change. - Do not change O11y controller delete semantics. - Do not change `spec.o11y.applicationId`; it should remain associated with the TiDB cluster ID. - Do not delete existing internal TiDBGroupAgent CRs during backfill. ## Proposed Changes ### 1. Create TiDBGroupAgent only for DEFAULT TiDBGroups Update `CreateO11yTiDBGroupAgentAction`: - After unmarshalling the TiDBGroup from workflow args, check `tidbGroup.GetType()`. - If the type is `INTERNAL`, log and return nil. - If the type is `DEFAULT`, create or repair the TiDBGroupAgent. The action should add stable labels to the CR: ```text tidbcloud.com/cluster-id= tidbcloud.com/tidb-group-id= ``` These labels are required for efficient cluster-scoped cleanup without listing all TiDBGroupAgent CRs and filtering by spec. The current helper `k8s.CreateIfNotExist` is not enough because it does not repair existing CRs. Replace it with create-or-update or an explicit get-and-patch flow so an existing default TiDBGroupAgent gets the labels if they are missing. When the action creates or repairs a TiDBGroupAgent, it should load the shared pool and reconcile the O11y-managed annotations from the current shared pool state. If shared pool loading fails, return an error because the desired annotation state is unknown. ### 2. Stop scheduling TiDBGroupAgent deletion in TiDBGroup deletion Remove `DeleteO11yTiDBGroupAgentAction` from newly generated delete-TiDBGroup DAGs. TiDBGroup deletion should not delete a TiDBGroupAgent CR. Update `DeleteO11yTiDBGroupAgentAction` so it does not delete the CR if it is invoked. It should log that TiDBGroupAgent cleanup is intentionally deferred to TiDB cluster deletion. Keep the action registered. Persisted or in-flight delete-TiDBGroup DAGs may still reference the action name. Keeping the action harmless is safer than removing it immediately. This protects existing internal TiDBGroupAgent CRs that may already exist before the creation-side fix is deployed. When export finishes and deletes the internal TiDBGroup, the TiDBGroup deletion workflow will no longer delete the internal TiDBGroupAgent and therefore will not trigger controller-side backend deletion. ### 3. Delete TiDBGroupAgent CRs during TiDB cluster deletion Add a TiDB-cluster-scoped cleanup action, for example: ```text delete-o11y-tidbgroup-agents-by-cluster ``` The action should: - Get the TiDB cluster ID from workflow args. - List TiDBGroupAgent CRs in `entity.GetNamespace()` using label selector `tidbcloud.com/cluster-id=`. - Delete each matched CR. - Ignore not found errors. - Return errors for list/delete failures that are not not-found. Register this action in the TiDB cluster delete DAG, not the TiDBGroup delete DAG. Place this action as the leaf of the delete-TiDBCluster DAG chain. DAG children execute before parents, so leaf placement lets the lightweight O11y CR cleanup run before later dataplane/cert cleanup steps can fail and block it. ## Backfill Design Add a one-time deploy migration under `pkg/storage/mysql/migrate/deploy`. The migration should label existing TiDBGroupAgent CRs so cluster deletion can use label selectors. Required migrator wiring: - Extend the deploy migrator to accept the regional Kubernetes client used for O11y CR operations. - Pass the existing `kubeClient` from server startup into `deploy.NewMigrator`. Migration behavior: - List all TiDBGroupAgent CRs in `entity.GetNamespace()`. - For each CR: - If `tidbcloud.com/cluster-id` is missing and `spec.clusterId` is non-empty, set it to `spec.clusterId`. - If `tidbcloud.com/tidb-group-id` is missing and `spec.tidbGroup` is non-empty, set it to `spec.tidbGroup`. - Patch only metadata labels. - Do not delete any CRs. - Make the migration idempotent. Backfill should include both default and existing internal TiDBGroupAgent CRs. Existing internal agents are intentionally preserved until TiDB cluster deletion, because deleting them during export completion is the dangerous operation this tactical fix avoids. ## Existing Cluster Behavior For existing TiDB clusters: - Existing default TiDBGroupAgent CRs get cluster labels from the backfill migration. - Existing internal TiDBGroupAgent CRs also get cluster labels from the backfill migration. - Export completion no longer deletes internal TiDBGroupAgent CRs because TiDBGroup-level delete becomes a no-op. - When the TiDB cluster is later deleted, the new cluster-scoped cleanup action deletes all labeled TiDBGroupAgent CRs for that cluster. If the backfill migration has not run successfully, label-based cluster cleanup can miss old unlabeled CRs. Deployment should therefore ensure the backfill-capable server runs before relying on label-only cleanup. ## Validation Plan Unit tests: - `CreateO11yTiDBGroupAgentAction` skips `INTERNAL` TiDBGroups. - `CreateO11yTiDBGroupAgentAction` creates a `DEFAULT` TiDBGroupAgent with `tidbcloud.com/cluster-id` and `tidbcloud.com/tidb-group-id`. - `CreateO11yTiDBGroupAgentAction` repairs missing labels on an existing default TiDBGroupAgent. - `CreateO11yTiDBGroupAgentAction` removes stale managed annotations when shared pool state no longer requires them. - `CreateO11yTiDBGroupAgentAction` fails when shared pool loading fails. - New delete-TiDBGroup DAGs do not include `DeleteO11yTiDBGroupAgentAction`. - `DeleteO11yTiDBGroupAgentAction` remains registered as a no-op compatibility shim. - The TiDB cluster delete DAG makes cluster-scoped O11y cleanup the first ready action. - The cluster-scoped cleanup action deletes only TiDBGroupAgent CRs with the matching cluster label. - The backfill migration patches missing labels and is idempotent. Manual or integration validation: - Create a cluster with a default TiDBGroup and verify its TiDBGroupAgent has the cluster label. - Simulate export by creating an internal TiDBGroup and verify no TiDBGroupAgent is created for it after the fix. - For a pre-existing internal TiDBGroupAgent, run an old persisted TiDBGroup deletion task that still references `DeleteO11yTiDBGroupAgentAction` and verify the CR is not deleted. - Run TiDB cluster deletion and verify all TiDBGroupAgent CRs with the cluster label are deleted. ## Risks and Mitigations Risk: stale internal TiDBGroupAgent CRs remain after export tasks. Mitigation: this is intentional for the tactical fix. They are removed during TiDB cluster deletion. A later cleanup can remove stale internal agents after O11y controller semantics are fixed. Risk: label-only cleanup misses existing CRs if backfill fails. Mitigation: make the backfill migration idempotent and observable through logs. Optionally add a transitional fallback in the cluster cleanup action using DB-derived TiDBGroup IDs, but the primary cleanup path should be label selector. Risk: create action does not repair existing CRs. Mitigation: replace `CreateIfNotExist` with create-or-update or patch-on-exists for labels. Risk: TiDBGroup delete DAG still contains the old action. Mitigation: keep the action registered but make it no-op, so old DAG references remain safe. ## Follow-Up Durable Fix This tactical change avoids the export-triggered incident without touching the O11y controller. The durable model should still be addressed separately: - O11y backend metadata is TiDB-cluster scoped. - TiDBGroupAgent should not be the owner of cluster-level backend metadata deletion. - A future TiDBCluster-scoped O11y owner, or controller-side reference check by `applicationId`, should own backend metadata lifecycle.