# Diagnose Evidence Bundle Implementation Plan > **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development if subagents are available, or superpowers:executing-plans if they are not. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Replace the current `clinic diagnose -o json` MVP output with a deterministic `diagnose.v2` evidence bundle that records scope, exact seed matches, related logs, events, topology, groups, timelines, summaries, warnings, and query provenance. **Architecture:** Keep command behavior in `internal/cmd`, following the existing command package boundary. Build one in-memory bundle per invocation, with small helpers for scope/query planning, source collection, log parsing, expansion, grouping/timelines, summaries, and rendering. Reuse the existing Clinic client abstraction, Loki request helpers, cluster metadata resolution, activity events route, time parsing, JSON writer, and fake-client tests. **Tech Stack:** Go, Cobra command wiring, existing `internal/cmd` helpers, Clinic HTTP client abstraction, Loki `query_range`, ActivityHub events endpoint, cluster topology endpoints, `go test`, `make verify`. --- This ExecPlan is a living document. Keep `Progress`, `Surprises & Discoveries`, `Decision Log`, and `Outcomes & Retrospective` up to date as work proceeds. Reference: `PLANS.md` at repository root; this plan must be maintained according to it. The requirement source was `/tmp/clinic-diagnose-evidence-bundle-spec.md`, and the decisions in this plan are self-contained so implementation does not depend on that temporary file. ## Purpose / Big Picture After this change, a user can run: clinic diagnose "Request range exceeds bound" --since 48h -o json and receive an agent-ready evidence bundle with a stable top-level shape: { "schema_version": "diagnose.v2", "generated_at_utc": "...", "scope": {}, "query_plan": [], "seed_matches": [], "related_logs": [], "events": [], "topology": {}, "groups": [], "timelines": [], "summaries": {}, "warnings": [] } The command remains deterministic and auditable. It collects, parses, labels, groups, and summarizes factual evidence. It must not infer root cause, assign blame, decide whether behavior is benign, or emit causal conclusions. The current MVP JSON shape with `query_scope`, `incidents`, and string `warnings` is intentionally replaced for `-o json`. There is no backward-compatibility requirement for old `diagnose` JSON consumers. ## Execution Protocol (Mandatory) Implement this plan sequentially. Do not execute multiple subtasks at the same time, even if they look independent. For every implementation subtask, use a fresh non-inherited subagent session. Give that subagent only this plan path, the current subtask, and the repository context needed for that subtask. The parent agent reviews the result, integrates any needed adjustments, runs the subtask validation commands, commits the subtask changes, and updates the `Progress` section before starting the next subtask. Each subtask has one commit boundary. Do not start the next subtask until the previous subtask is reviewed, validated, committed, and reflected in `Progress`. If a subtask cannot be completed or committed cleanly, stop and resolve that subtask before moving on. The operator prompt template for runtimes that require explicit delegation authorization is: Implement docs/2026-04-22-diagnose-evidence-bundle-plan.md using fresh non-inherited subagents per subtask, sequentially, with parent review/integration, one commit per subtask, and Progress updates after each subtask commit. If subagents are unavailable in the execution environment, the current agent may execute subtasks directly, but it must preserve the same sequential order, review checkpoint, validation commands, one-commit-per-subtask boundary, and `Progress` updates. ## Progress - [x] (2026-04-22 Asia/Shanghai) Authored this implementation plan from the evidence-bundle requirement and repository context. - [x] (2026-04-22 Asia/Shanghai) Subtask 1: Added the V2 bundle skeleton, scope, and query-plan model behind JSON contract tests. Validation: `go test ./internal/cmd -run 'Diagnose.*(V2|Bundle|QueryScope|QueryPlan|JSON)' -v` and `go test ./internal/cmd -run Diagnose -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 2: Collected topology and events in `diagnose` with query provenance and partial-output warnings. Validation: `go test ./internal/cmd -run 'Diagnose.*(Topology|Events|Source)' -v`, `go test ./internal/cmd -run 'ClusterTopology|EventsList' -v`, and `go test ./internal/cmd -run Diagnose -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 3: Converted seed-match parsing to V2 log entries with anchors, raw preservation, redactions, parse warnings, and query IDs. Validation: `go test ./internal/cmd -count=1 -run 'Diagnose.*(SeedMatch|LogEntry|Anchor|Redaction|Parse)' -v` and `go test ./internal/cmd -run Diagnose -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 4: Added bounded related-log expansion and relation labeling. Validation: `go test ./internal/cmd -run 'Diagnose.*(Expand|Related|Relation|NoExpand|MaxRelated)' -v`, `go test ./internal/cmd -run Diagnose -v`, and `go test ./internal/cmd -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 5: Added grouping, timelines, and factual summaries. Validation: `go test ./internal/cmd -count=1 -run 'Diagnose.*(Group|Timeline|Summary|MultiKeyspace)' -v` and `go test ./internal/cmd -count=1 -run Diagnose -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 6: Rebuilt table rendering from the V2 bundle and updated user-facing docs. Validation: `go test ./internal/cmd -count=1 -run 'Diagnose.*(Table|Help|Render)' -v`, `make legibility-check`, and `go test ./internal/cmd -count=1 -run Diagnose -v` passed. - [x] (2026-04-22 Asia/Shanghai) Subtask 7: Added the fixture-style regression suite and completed final verification. Validation: `go test ./internal/cmd -count=1 -run DiagnoseV2FixtureSuite -v`, `go test ./internal/cmd -count=1 -run Diagnose -v`, `make test`, and `make verify` passed. ## Surprises & Discoveries - Observation: Profiles currently do not have a timezone field. Evidence: `internal/config/config.go` `ProfileRecord` contains cluster metadata and range defaults but no timezone field. V1 uses `time.Local`, honoring `TZ`, as the profile-local timezone. - Observation: The current diagnose command already has most input flags and a one-query MVP pipeline. Evidence: `internal/cmd/diagnose.go` registers `--cluster-id`, `--start`, `--end`, `--since`, `--around`, `--window`, `--limit`, and `-o/--output`; `runDiagnoseSearchWithOutput` builds one Loki seed query and renders the MVP result. - Observation: Existing event and topology command paths can be reused in-process but currently write directly to stdout. Evidence: `internal/cmd/events.go` has `runEventsList`, and `internal/cmd/cluster.go` has `runClusterTopology` plus endpoint helpers. Bundle implementation needs raw/typed helper variants rather than shelling out or writing command output. - Observation: Agent legibility checks enforce plan structure. Evidence: `scripts/check-agent-legibility.sh` requires every `docs/*-plan.md` file to include `## Execution Protocol (Mandatory)`, the phrase `fresh non-inherited subagent`, and an operator prompt template. ## Decision Log - Decision: Implement the full V1 evidence bundle, not only a phase-one subset. Rationale: The user explicitly selected Full V1 during planning. The largest diagnostic value comes from related-log expansion, grouping, and timelines. Date/Author: 2026-04-22 / Codex - Decision: Break `diagnose -o json` compatibility and remove top-level `incidents`. Rationale: The user confirmed there is no compatibility issue. The spec's JSON contract uses groups, timelines, summaries, seed matches, and related logs instead of incident records. Date/Author: 2026-04-22 / Codex - Decision: Use `time.Local` as V1 profile-local time. Rationale: The current profile schema has no timezone field. Adding profile timezone management would expand the feature into config UX. The existing diagnose MVP already reports local/profile timezone from `time.Local`. Date/Author: 2026-04-22 / Codex - Decision: Keep all implementation in `internal/cmd`. Rationale: The repository architecture assigns command parsing, validation, request shaping, and output formatting to `internal/cmd`; `internal/clinic` remains transport-only. Date/Author: 2026-04-22 / Codex - Decision: Do not shell out to `clinic` subcommands from `diagnose`. Rationale: Existing command tests use an in-process fake client, and shelling out would bypass request provenance, retries, testability, and the shared runtime context. Date/Author: 2026-04-22 / Codex - Decision: Keep factual relation strength, but never use it as causality. Rationale: Relation strength helps evidence ranking. The tool must not infer cause or mark evidence as benign. Date/Author: 2026-04-22 / Codex ## Outcomes & Retrospective - Subtask 1 replaced the `diagnose -o json` MVP shape with a `diagnose.v2` bundle skeleton for JSON output. The bundle now includes deterministic scope metadata, initialized V2 arrays/objects, and seed Loki query provenance as `q1`; seed log parsing and downstream evidence sections remain intentionally empty for later subtasks. Validation passed with `go test ./internal/cmd -run 'Diagnose.*(V2|Bundle|QueryScope|QueryPlan|JSON)' -v` and `go test ./internal/cmd -run Diagnose -v`. - Subtask 2 added in-process topology and ActivityHub event collection to the V2 bundle. The query plan now records `topology` and `events` Clinic API requests after the seed Loki query; source failures become structured `missing_source` warnings while preserving partial JSON output. Validation passed with `go test ./internal/cmd -run 'Diagnose.*(Topology|Events|Source)' -v`, `go test ./internal/cmd -run 'ClusterTopology|EventsList' -v`, and `go test ./internal/cmd -run Diagnose -v`. - Subtask 3 added V2 seed-match parsing. Seed Loki results now populate `seed_matches` with stable `log-###` IDs, raw line preservation, labels, fields, anchors, redaction metadata, query IDs, and structured `parse_failure` or `redaction` warnings where applicable. Validation passed with `go test ./internal/cmd -count=1 -run 'Diagnose.*(SeedMatch|LogEntry|Anchor|Redaction|Parse)' -v` and `go test ./internal/cmd -run Diagnose -v`. - Subtask 4 added bounded related-log expansion. The diagnose command now supports `--no-expand`, `--max-seed-expand`, and `--max-related-logs`; related Loki queries are recorded in `query_plan`, related logs carry relation labels and strengths, seed duplicates merge query IDs, and expansion failures or caps preserve partial bundle output with structured warnings. Validation passed with `go test ./internal/cmd -run 'Diagnose.*(Expand|Related|Relation|NoExpand|MaxRelated)' -v`, `go test ./internal/cmd -run Diagnose -v`, and `go test ./internal/cmd -v`. - Subtask 5 added deterministic V2 grouping, timelines, and factual summaries. Seed matches now classify into active profile keyspace, other keyspace, shared component, or unknown scope; related logs attach through seed relations; full-window events are included in group timelines; timeline entries are sorted and labeled relative to the seed window; summaries aggregate counts, affected components/instances, anchors, and query-limit state without causal fields. Validation passed with `go test ./internal/cmd -count=1 -run 'Diagnose.*(Group|Timeline|Summary|MultiKeyspace)' -v` and `go test ./internal/cmd -count=1 -run Diagnose -v`. - Subtask 6 rebuilt the default table output on top of the same `diagnose.v2` bundle used for JSON output. Table output now reports scope metadata, seed and related-log counts, warning types, and compact group rows without raw log lines. Help text and user-facing docs now describe the V2 evidence bundle and expansion flags. Validation passed with `go test ./internal/cmd -count=1 -run 'Diagnose.*(Table|Help|Render)' -v`, `make legibility-check`, and `go test ./internal/cmd -count=1 -run Diagnose -v`. - Subtask 7 added an end-to-end fixture suite for the full V2 evidence bundle contract. The fixtures cover no seed matches, seed matches with and without anchors, related-log expansion, active/other/shared grouping, query limits, redactions, topology partial output, and event timing. The suite asserts stable top-level schema, query provenance, raw preservation, grouping, warnings, absence of causal fields, and reproducible query-plan records. Final validation passed with `go test ./internal/cmd -count=1 -run DiagnoseV2FixtureSuite -v`, `go test ./internal/cmd -count=1 -run Diagnose -v`, `make test`, and `make verify`. - Final outcome: `clinic diagnose` now builds a deterministic `diagnose.v2` evidence bundle for JSON and table output. The bundle records scope, query provenance, structured seed matches, related logs, topology, events, groups, timelines, summaries, and structured warnings without root-cause inference. Default table output is a compact view over the same bundle, and user-facing docs describe the V2 JSON contract and expansion controls. ## Context and Orientation This repository builds `clinic`, a Go CLI for TiDB Cloud Clinic APIs. The binary entrypoint is `cmd/clinic/main.go`, which delegates to `internal/cmd.Run`. Top-level command routing and runtime setup live in `internal/cmd/root.go`. The `RuntimeContext` contains the selected environment, selected profile, config, client, verbose flag, and diagnostics writer. The command package uses Cobra through helpers in `internal/cmd/cobra_helpers.go`. `diagnose` currently lives across: - `internal/cmd/diagnose.go`: command flags, time scope, current one-query pipeline. - `internal/cmd/diagnose_types.go`: current MVP `Evidence`, `DiagnoseResult`, `Incident`, and related types. - `internal/cmd/diagnose_extract.go`: Loki stream normalization, field parsing, anchor extraction. - `internal/cmd/diagnose_correlation.go`: current score-based incident grouping. - `internal/cmd/diagnose_render.go`: current table and JSON renderers. - `internal/cmd/diagnose_window.go`: local/UTC window formatting. - `internal/cmd/diagnose_test.go`: current command, parsing, correlation, and rendering tests. Useful existing helpers: - `requiredFlagOrEnvWithContext` in `internal/cmd/flag_env.go` resolves `--cluster-id` by flag, `CLINIC_CLUSTER_ID`, then active profile. - `resolveClusterReferenceByIDWithMetadata`, `inferLokiNamespace`, and `buildDefaultLokiQuery` provide cluster metadata and Loki selector defaults. - `executeLokiQueryRangeRawWithRetry` executes Loki `query_range` with existing 504 retry behavior. - `runEventsList` shows the ActivityHub route shape: `/clinic/api/v1/activityhub/applications//targets//activities`. - `runClusterTopology` shows dedicated vs premium/byoc topology routes. - `recordingClinicClient` in `internal/cmd/cluster_test.go` records requests and returns queued fake responses for command tests. Terms used in this plan: - Seed: the required free-text argument passed to `clinic diagnose`. - Seed match: a Loki log entry returned by the exact seed query. - Anchor: an exact value extracted from labels or fields, such as `conn`, `txnStartTS`, `regionID`, SQL text, task ID, or instance. - Related log: a bounded follow-up Loki result found through one or more anchors or time proximity. - Query plan: the ordered audit trail of every request executed while building the bundle. - Group: a deterministic scope bucket: active profile keyspace, other keyspace, shared component, or unknown scope. - Timeline: sorted evidence records for one group. - Warning: structured metadata explaining incomplete, ambiguous, redacted, truncated, failed, or skipped evidence. ## Interfaces and Dependencies The implementation should create focused V2 helpers rather than expanding the current files indefinitely. Exact names can be adjusted only if the parent reviewer finds a clearer local fit, but the responsibilities must stay separate. Create these files: - `internal/cmd/diagnose_bundle_types.go`: V2 JSON model types. - `internal/cmd/diagnose_bundle.go`: orchestration from resolved options to final bundle. - `internal/cmd/diagnose_query_plan.go`: query ID allocation, query-plan records, duration/status/result-count handling. - `internal/cmd/diagnose_log_entry.go`: V2 log parsing, redaction detection, dedup keys, query ID merging. - `internal/cmd/diagnose_sources.go`: topology and event fetch helpers returning raw/typed data for bundle use. - `internal/cmd/diagnose_expand.go`: related-log query construction, cap enforcement, relation labeling. - `internal/cmd/diagnose_group_timeline.go`: groups, timelines, relative timing, summaries. - `internal/cmd/diagnose_bundle_render.go`: V2 JSON and table rendering. - `internal/cmd/diagnose_bundle_test.go`: V2 contract and fixture tests. It may grow large; split into focused `_test.go` files if it becomes hard to navigate. Modify these files: - `internal/cmd/diagnose.go`: add expansion flags, validation, and call the V2 bundle pipeline. - `internal/cmd/diagnose_render.go`: retire or delegate old render path once V2 table rendering exists. - `internal/cmd/diagnose_types.go`: keep old types only while still needed by tests or table transition; remove unused old JSON path by the final subtask. - `internal/cmd/diagnose_extract.go`: reuse field parsing helpers or move them behind V2 log parsing. - `internal/cmd/diagnose_test.go`: update existing tests that assert old JSON shape; preserve input/time/query tests where still valid. - `docs/COMMAND_SURFACE.md`, `README.md`, and `docs/INDEX.md`: update user-facing command contract and document this new plan. Stable V2 top-level Go model should encode the JSON keys exactly: type DiagnoseBundle struct { SchemaVersion string `json:"schema_version"` GeneratedAtUTC string `json:"generated_at_utc"` Scope DiagnoseBundleScope `json:"scope"` QueryPlan []DiagnoseQueryRecord `json:"query_plan"` SeedMatches []DiagnoseLogEntry `json:"seed_matches"` RelatedLogs []DiagnoseLogEntry `json:"related_logs"` Events []DiagnoseEvent `json:"events"` Topology DiagnoseTopology `json:"topology"` Groups []DiagnoseGroup `json:"groups"` Timelines []DiagnoseTimeline `json:"timelines"` Summaries DiagnoseSummaries `json:"summaries"` Warnings []DiagnoseWarning `json:"warnings"` } All slices in the public JSON contract must be initialized to empty slices, not `null`. `scope`, `topology`, and `summaries` must encode as objects even when unavailable. ## Plan of Work Implement the work in seven subtasks. Each subtask must begin with failing tests, must run the subtask validation command before commit, and must update this plan's `Progress` immediately after commit. ### Subtask 1: V2 Bundle Skeleton, Scope, and Query Plan Purpose: make `clinic diagnose "" --since 48h -o json` emit the V2 top-level schema with scope and the seed query plan entry. This subtask may still have empty `seed_matches`, `events`, `topology`, `groups`, and `timelines` beyond the seed query. Files: - Create `internal/cmd/diagnose_bundle_types.go`. - Create `internal/cmd/diagnose_bundle.go`. - Create `internal/cmd/diagnose_query_plan.go`. - Create `internal/cmd/diagnose_bundle_test.go`. - Modify `internal/cmd/diagnose.go`. - Modify old JSON-shape tests in `internal/cmd/diagnose_test.go`. Behavior to implement: - JSON output uses `schema_version: "diagnose.v2"`. - `generated_at_utc` uses the invocation clock passed to `runDiagnoseSearchWithOutput` in tests. - `scope` includes env, selected profile name, profile_active boolean, org ID, cluster ID, cluster name, deploy type, namespace, sharedpool ID when parent/resource-pool ID is available, timezone, UTC/local start and end, seed, seed_source, cluster_source, time_source, and time_interpretation_notes. - `cluster_source` values are `explicit_flag`, `environment_variable`, `active_profile`, or `unresolved`. - `time_source` values are `since_flag`, `start_end_flags`, `around_window_flags`, `profile_range`, or `default`. - V1 timezone is `time.Local`; notes should say local timezone is used because profiles do not yet store timezone. - `query_plan` includes the seed Loki query with ID `q1`, type `loki`, purpose `seed_exact_match`, query string, UTC bounds, limit, direction `backward`, status, result_count, hit_limit, and duration_ms. - Do not add top-level `incidents` to V2 JSON. Testing steps: - Write failing tests for: - top-level V2 keys exist and `incidents` does not exist, - empty arrays encode as `[]`, - scope source values for explicit cluster flag, env cluster value, active profile cluster, `--since`, `--start/--end`, `--around/--window`, profile range, and default range, - seed query plan entry is reproducible and has `hit_limit=true` when result count equals limit. - Run: go test ./internal/cmd -run 'Diagnose.*(V2|Bundle|QueryScope|QueryPlan|JSON)' -v Expected before implementation: new tests fail because V2 types and fields do not exist or old JSON is emitted. Expected after implementation: new tests pass, and existing diagnose input/time/query tests still pass. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "feat(diagnose): add v2 bundle scope and query plan" ### Subtask 2: Topology and Events Collection Purpose: add topology and ActivityHub events to the bundle with query provenance and partial-output warnings. Files: - Create `internal/cmd/diagnose_sources.go`. - Modify `internal/cmd/diagnose_bundle.go`. - Modify `internal/cmd/diagnose_query_plan.go`. - Add tests in `internal/cmd/diagnose_bundle_test.go` or `internal/cmd/diagnose_sources_test.go`. Behavior to implement: - Fetch cluster metadata once where possible and reuse it for namespace, topology route selection, and events route selection. - Topology query: - dedicated clusters use `/clinic/api/v1/orgs//clusters/` and prefer `components`, then `topology`, matching existing command behavior, - premium/byoc clusters use `/clinic/api/v1/orgs//clusters//resource-pool/components`, - failures become `missing_source` warnings and a failed query-plan entry, not hard failures, after cluster ID and namespace are resolved. - Events query: - use `/clinic/api/v1/activityhub/applications//targets//activities`, - send `begin_ts`, `end_ts`, and `limit=200`, - normalize event fields best-effort into event records with `query_ids`, - truncate large payloads above 20000 bytes and mark `payload_truncated=true`. - Relative timing uses first and last seed match when present; before seed parsing exists, `no_seed_matches` is acceptable for no seed matches. Final timing is completed in later subtasks. - Query plan records include topology and events query IDs and statuses. Testing steps: - Write failing tests for: - dedicated topology request and normalized topology object, - premium/byoc resource-pool topology request, - topology failure returns JSON bundle with warning and failed query plan, - events request params and event provenance, - event payload truncation. - Run: go test ./internal/cmd -run 'Diagnose.*(Topology|Events|Source)' -v Expected before implementation: tests fail because diagnose does not collect these sources. Expected after implementation: tests pass, and existing `ClusterTopology` and `EventsList` tests still pass. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "feat(diagnose): collect topology and events" ### Subtask 3: V2 Seed Log Entries, Anchors, Redactions, and Provenance Purpose: replace current MVP `Evidence` output in JSON with spec-shaped `seed_matches` log entries. Files: - Create `internal/cmd/diagnose_log_entry.go`. - Modify `internal/cmd/diagnose_extract.go` if reusing or moving parsing helpers. - Modify `internal/cmd/diagnose_bundle.go`. - Add tests in `internal/cmd/diagnose_log_entry_test.go` or `internal/cmd/diagnose_bundle_test.go`. Behavior to implement: - Parse Loki streams into `DiagnoseLogEntry` records with: - stable IDs like `log-001`, - `kind: "log"`, - `time_utc` in RFC3339/RFC3339Nano as appropriate, - `time_local` in local timezone with offset, - component, instance, node, stream, level, file, line_no, message when parseable, - raw line under `raw`, - stream labels under `labels`, - parsed bracket/KV fields under `fields`, - anchor arrays or fields under `anchors`, - `query_ids`. - Extract exact anchors first from fields and labels: `conn`, `conn_id`, `txn_start_ts`, `timestamp_or_start_ts`, `region_id`, `region_version`, `store_addr`, `peer_addr`, `keyspace`, `cluster_id`, `tenant_id`, `namespace`, `instance`, `node`, `component`, `sql`, `sql_digest`, `database`, `table`, `job_id`, `task_id`, `subtask_id`, `request_id`, `trace_id`, `stack_frames`, and `error_text`. - Keep SQL normalization conservative. If no existing digest/normalizer exists, store raw SQL and table anchors, and leave `normalized_sql` unavailable rather than inventing lossy normalization. - Preserve raw logs even if structured parsing fails; emit `parse_failure` warnings with source IDs. - Detect redaction indicators such as `?`, `[?, ?)`, `[?, ?]`, and quoted `"[?,?]"`. Add per-log `redactions` and a bundle-level `redaction` warning. - Deduplicate logs by timestamp nanoseconds, component, instance, stream labels, and raw line hash. If a duplicate appears from multiple queries later, merge `query_ids`. Testing steps: - Write failing tests for: - seed matches appear under `seed_matches`, - raw line and labels are preserved, - TiDB-style bracket fields parse without dropping unknown fields, - anchors are extracted from both fields and labels, - parse failures keep raw line and warning, - redactions are preserved and warned, - duplicate seed results produce one log entry with merged query IDs. - Run: go test ./internal/cmd -run 'Diagnose.*(SeedMatch|LogEntry|Anchor|Redaction|Parse)' -v Expected before implementation: tests fail because current JSON has no `seed_matches` array. Expected after implementation: tests pass. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "feat(diagnose): emit structured seed log entries" ### Subtask 4: Bounded Related-Log Expansion Purpose: run deterministic follow-up Loki queries based on extracted seed anchors and add `related_logs` with relation labels. Files: - Create `internal/cmd/diagnose_expand.go`. - Modify `internal/cmd/diagnose.go`. - Modify `internal/cmd/diagnose_bundle.go`. - Modify `internal/cmd/diagnose_query_plan.go`. - Add tests in `internal/cmd/diagnose_expand_test.go`. Behavior to implement: - Add flags: - `--no-expand`: skip related-log expansion, - `--max-seed-expand `: default 20, must be positive, - `--max-related-logs `: default 1000, must be non-negative; zero means no related logs are kept and must warn if expansion would otherwise run. - Keep existing `--limit` as the seed query limit. Use internal default 100 logs per related query, capped by `lokiMaxLimit`. - For each seed match up to `--max-seed-expand`, run at most 8 related queries: - same namespace WARN/ERROR within plus/minus 5 minutes, - same instance within plus/minus 5 minutes, - same connection ID within plus/minus 10 minutes, - same transaction/start timestamp within plus/minus 10 minutes, - same region ID within plus/minus 30 minutes, - same store or peer address within plus/minus 10 minutes, - same SQL text or table within plus/minus 30 minutes, - same task, job, or subtask ID within plus/minus 30 minutes. - Related queries must include deterministic query-plan purposes: `nearby_warn_error_same_namespace`, `same_instance_nearby`, `same_connection`, `same_transaction_ts`, `same_region`, `same_store_address`, `same_sql_or_table`, `same_task_or_job`. - Use exact string matching with `strconv.Quote` for user/log values. Do not build unbounded Loki regexes. - Deduplicate related logs against seed matches and against other related logs. A seed match returned by an expansion query remains in `seed_matches` and gains the expansion query ID; do not duplicate it in `related_logs`. - Add `relations` to related logs. Strengths: - `strong`: same `conn`, `txn_start_ts`, `request_id`, `trace_id`, task/job/subtask ID, - `medium`: same `region_id`, exact SQL text, SQL digest, store/peer address, table plus near time, - `weak`: same instance, same component, same error text, same namespace, near time only. - If `--no-expand` is used, seed matches exceed `--max-seed-expand`, a per-query limit is hit, global related-log cap is reached, or a related query fails, keep partial output and emit structured warnings. Testing steps: - Write failing tests for: - `--no-expand` runs no related Loki queries and emits `expansion_skipped`, - seed count above `--max-seed-expand` skips expansion and warns, - same `conn` seed produces a `same_connection` query and related log relation, - same instance and same region produce correct purpose and strength, - related query hitting limit marks `hit_limit=true` and emits `limit`, - related query failure emits `query_error` and keeps bundle output, - global `--max-related-logs` caps returned related logs and warns. - Run: go test ./internal/cmd -run 'Diagnose.*(Expand|Related|Relation|NoExpand|MaxRelated)' -v Expected before implementation: tests fail because related expansion does not exist. Expected after implementation: tests pass. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "feat(diagnose): add bounded related log expansion" ### Subtask 5: Groups, Timelines, and Summaries Purpose: classify evidence into scope groups, build deterministic timelines, and compute factual summaries. Files: - Create `internal/cmd/diagnose_group_timeline.go`. - Modify `internal/cmd/diagnose_bundle.go`. - Add tests in `internal/cmd/diagnose_group_timeline_test.go`. Behavior to implement: - Group kinds: - `active_profile_keyspace`, - `other_keyspace`, - `shared_component`, - `unknown_scope`. - Grouping rule order: - if parsed `keyspaceName` or keyspace anchor equals active profile cluster ID, active profile keyspace, - else if parsed keyspace exists, other keyspace, - else if label `cluster_id` or `tidb_cluster_id` equals active profile cluster ID, active profile keyspace, - else if label `cluster_id` is `unknown` and namespace or sharedpool matches active profile, shared component, - else unknown scope. - If multiple keyspaces appear in seed matches, emit `multi_keyspace`. - Groups list seed match IDs, related log IDs, event IDs, first and last seen UTC times, namespace, keyspace/cluster ID when known, and sorted component names. - Timelines: - one timeline per group, - include grouped seed matches, - include related logs connected to grouped seed matches, - include full-window events, - sort by `time_utc`, - cap entries at 500 per group and warn on truncation, - classify entries as `before`, `during`, `after`, or `no_seed_matches`. - Summaries include: - total seed match counts, - counts by component and keyspace, - first and last seed match UTC, - active/other/shared seed match counts, - affected components and instances, - extracted anchors, - query limit summary. - Summaries must not include `root_cause`, `likely_cause`, `culprit`, or `benign`. Testing steps: - Write failing tests for: - active keyspace, other keyspace, shared component, and unknown grouping, - multi-keyspace warning, - timeline sort order and relative timing, - timeline truncation warning, - summaries counts and anchor aggregation, - forbidden root-cause fields absent from serialized JSON. - Run: go test ./internal/cmd -run 'Diagnose.*(Group|Timeline|Summary|MultiKeyspace)' -v Expected before implementation: tests fail because groups/timelines/summaries are empty or incomplete. Expected after implementation: tests pass. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "feat(diagnose): group evidence and build timelines" ### Subtask 6: Table Rendering and Documentation Purpose: render table output from the same V2 bundle and update user-facing docs. Files: - Create or modify `internal/cmd/diagnose_bundle_render.go`. - Modify `internal/cmd/diagnose_render.go`. - Modify `internal/cmd/diagnose.go` help text. - Modify `README.md`. - Modify `docs/COMMAND_SURFACE.md`. - Modify `docs/INDEX.md`. Behavior to implement: - Table output remains the default output. - Table output is derived from `DiagnoseBundle`, not from the old `Incident` model. - Table output must show: - searched cluster, namespace, time window UTC/local, and timezone, - seed match count and related log count, - warning count and concise warning types, - compact group rows with group kind, keyspace/cluster, components, seed/related/event counts, and first/last seen times. - Table output must not print raw log lines. - JSON output must print the full bundle. - Help text documents the V2 JSON bundle and expansion flags. - README examples show `diagnose -o json` as the evidence bundle path. - `docs/COMMAND_SURFACE.md` documents the new flags and JSON contract. - `docs/INDEX.md` lists this plan under historical execution artifacts. Testing steps: - Write failing tests for: - table output includes bundle scope and group counts, - table output does not include raw logs, - command help includes `--no-expand`, `--max-seed-expand`, `--max-related-logs`, and V2 JSON wording, - JSON output still includes full bundle after render refactor. - Run: go test ./internal/cmd -run 'Diagnose.*(Table|Help|Render)' -v make legibility-check Expected before implementation: tests fail because table still uses old incident rendering and help lacks new flags. Expected after implementation: tests pass and legibility check passes. Commit: git add internal/cmd README.md docs scripts docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "docs(diagnose): document evidence bundle output" ### Subtask 7: Fixture Suite and Final Verification Purpose: lock down the full V1 contract with end-to-end fixture-style tests and run repository verification. Files: - Add fixture helpers in `internal/cmd/diagnose_bundle_test.go` or split into `internal/cmd/diagnose_fixture_test.go`. - Modify any diagnose tests still asserting old JSON behavior. - Update this plan's `Outcomes & Retrospective`. Fixture scenarios: - no seed matches, - one seed match with no anchors, - one seed match with `conn` and same-connection related logs, - multiple seed matches in active keyspace, - seed matches in active keyspace and another keyspace, - shared component logs with `cluster_id=unknown`, - query hits result limit, - redacted fields in logs, - topology query failure with partial output, - events after seed matches, verifying timing classification only. Each fixture asserts: - stable top-level schema, - all seed matches, related logs, and events reference query IDs present in `query_plan`, - every log entry preserves `raw`, - correct grouping, - correct warnings, - no root-cause fields, - reproducible query plan fields. Validation commands: go test ./internal/cmd -run Diagnose -v make test make verify Expected result: all commands pass. If `make verify` fails outside changed diagnose behavior, investigate before committing; do not hide or skip failures. Commit: git add internal/cmd docs/2026-04-22-diagnose-evidence-bundle-plan.md git commit -m "test(diagnose): cover evidence bundle fixtures" ## Validation and Acceptance The final implementation is acceptable when these observable behaviors hold: - `clinic diagnose "some error" --since 48h -o json` returns JSON with the V2 top-level keys and no top-level `incidents`. - Every successful run includes a reproducible `query_plan` with entries for seed Loki query, events query, and topology query; related query entries appear when seed matches have expandable anchors and expansion is not skipped. - Every `seed_matches`, `related_logs`, and `events` record references query IDs present in `query_plan`. - Every log entry preserves `raw`. - If seed matches span multiple keyspaces, active and other keyspace evidence are distinguishable and a `multi_keyspace` warning is emitted. - If any query returns exactly its limit or results are otherwise capped, `query_plan[].hit_limit` is true for the relevant query and a `limit` warning is emitted. - Topology, events, and related-query failures produce partial JSON output with structured warnings. - JSON does not contain `root_cause`, `likely_cause`, `culprit`, or `benign`. - Table output is compact, derived from the same bundle, and does not print raw logs. - `make verify` passes. ## Idempotence and Recovery All test commands are safe to rerun. The fake-client tests should avoid real network access. If a subtask partially changes files and tests fail, inspect the diff, fix forward, and rerun the subtask validation. Do not revert unrelated user changes. If a subtask cannot reach a passing state, stop before committing and record the blocker in `Surprises & Discoveries`. If a commit is created with an incomplete plan progress update, amend only the plan progress before starting the next subtask. ## Artifacts and Notes Before implementation, focused diagnose tests pass against the MVP behavior: go test ./internal/cmd -run Diagnose -v PASS The new tests in Subtask 1 are expected to fail first because the MVP JSON shape is still `query_scope`, `incidents`, and string `warnings`. Keep all generated sample JSON in tests small and deterministic. Prefer asserting decoded maps and structs over brittle full-string comparison unless key order or formatted output is the behavior under test.