# Add Loki Instance Mapping Command (`loki instance-map`)

This ExecPlan is a living document. Keep `Progress`, `Surprises & Discoveries`, `Decision Log`, and `Outcomes & Retrospective` up to date as work proceeds.

Reference: `PLANS.md` (repository root). This plan must be maintained according to that document.

## Execution Protocol (Mandatory)

This plan must be executed sequentially, one subtask at a time.

For each subtask, use a fresh subagent session that does not inherit parent-agent context. Give that subagent only the current subtask instructions plus the minimum repository context needed for the subtask.

After each subtask is implemented, the parent agent must review and integrate the changes, run the subtask's validation commands, and create exactly one commit for that subtask before proceeding.

Update the `Progress` checklist immediately after every subtask commit, including timestamp and a short evidence note. If a subtask cannot be completed cleanly, stop and resolve it before starting the next subtask.

operator prompt template (required for runtimes that need explicit delegation authorization):

`Implement docs/2026-04-07-loki-instance-map-plan.md using fresh non-inherited subagents per subtask, sequentially, with parent review/integration, one commit per subtask, and Progress updates after each subtask commit.`

## Purpose / Big Picture

Users querying node-level metrics through `metrics range --data-server` often need a metric `instance` label value such as `cn-hongkong.10.0.7.198`, while Loki streams expose both `node` and a pod-oriented `instance` label such as `db-4b0d6fe-worker-tidb-u12t8g`.

After this change, users can run one command (`loki instance-map`) to list the mapping between these identifiers, with a human-readable default table and machine-readable JSON output for scripting.

Observable outcome: running `clinic loki instance-map ...` returns deterministic rows containing `node`, `data_server_instance`, `loki_instance`, and derived `ip`.

## Progress

- [x] (2026-04-07 10:47 +0800) S1: Add `loki instance-map` command wiring, flag parsing, and Loki request construction (cluster/range fallback parity with `loki query`). Evidence: parent rerun `go test ./internal/cmd -run 'Loki.*(InstanceMap|QueryRange|Labels|LabelValues)' -v` PASS; spec + quality reviewer subagents approved.
- [x] (2026-04-07 10:58 +0800) S2: Add mapping extraction, deterministic dedupe/sort, IP derivation, and dual output renderers (`table` default, `json` optional). Evidence: parent rerun `go test ./internal/cmd -run 'Loki.*InstanceMap' -v` PASS; spec + quality reviewer subagents approved.
- [x] (2026-04-07 11:06 +0800) S3: Add/adjust tests for command validation, request shape, mapping behavior, output behavior, and retry behavior. Evidence: added `TestLokiInstanceMapRetriesGatewayTimeout` and request-shape method assertion; parent rerun `go test ./internal/cmd -run 'Loki.*(InstanceMap|QueryRange|Labels|LabelValues)' -v` PASS; spec + quality reviewer subagents approved.
- [x] (2026-04-07 11:13 +0800) S4: Update user-facing/docs surfaces (`help`, command surface doc, README, shell completion map where needed). Evidence: parent rerun `go test ./internal/cmd -run 'Help|Completion' -v` PASS; spec + quality reviewer subagents approved.
- [x] (2026-04-07 11:13 +0800) S5: Run verification ladder (`go test` focused -> `make verify`) and capture evidence in `Artifacts and Notes`. Evidence: parent rerun `go test ./internal/cmd -run Loki -v` PASS; `make verify` PASS after plan wording fix required by legibility check.

## Surprises & Discoveries

- Observation: repository working tree is already dirty before this plan starts.
  Evidence: `git status --short` reports many modified files not scoped to this feature.

- Observation: Loki `labels` and `label-values` APIs alone cannot guarantee co-occurrence mapping between `node` and `instance`; mapping requires stream-level data.
  Evidence: existing command paths call `/api/v1/labels` and `/api/v1/label/<label>/values`, which return independent label sets.

- Observation: legibility check enforces lowercase literal `operator prompt template` in plan files.
  Evidence: initial `make verify` failed with `docs/2026-04-07-loki-instance-map-plan.md must include an operator prompt template for explicit delegation authorization.`; rerun passed after changing the heading text casing.

## Decision Log

- Decision: Introduce new subcommand under `loki` family: `instance-map`.
  Rationale: keeps Loki label-discovery/mapping behavior in the same command family and avoids coupling `metrics` family to Loki-specific semantics.
  Date/Author: 2026-04-07 / user+agent.

- Decision: Default output is human-readable table; `-o/--output json` provides machine-readable rows.
  Rationale: optimize for interactive use while preserving scripting support.
  Date/Author: 2026-04-07 / user+agent.

- Decision: Time window semantics mirror existing required-range commands (`--start`/`--end` with fallback to env/profile/default now-5m..now).
  Rationale: consistent UX and predictable behavior across command families.
  Date/Author: 2026-04-07 / user+agent.

- Decision: Do not accept custom LogQL selector in v1.
  Rationale: keep command purpose narrow and predictable for mapping lookup.
  Date/Author: 2026-04-07 / user+agent.

- Decision: If one node maps to multiple Loki instances in range, return all unique pairs.
  Rationale: avoid data loss during rollouts/restarts; downstream consumers can choose filtering policy.
  Date/Author: 2026-04-07 / user+agent.

- Decision: Default `--limit` is `1000` with max `1000`.
  Rationale: maximize mapping coverage while matching existing server cap constraints.
  Date/Author: 2026-04-07 / user+agent.

## Outcomes & Retrospective

Delivered user-visible behavior:

- Added `clinic loki instance-map` with required range/cluster resolution parity, deterministic table output by default, and `-o/--output json`.
- Implemented deterministic mapping normalization (`node`, `data_server_instance`, `loki_instance`, `ip`) with dedupe and stable sort.
- Added focused coverage for instance-map validation, request shape, mapping/output behavior, and retry behavior.
- Updated user-facing surfaces (help coverage, completion map/tests, README, command-surface doc).

Deviations from original plan:

- `help.go` instance-map text was introduced during S1 quality-fix loop instead of waiting for S4, to avoid a known user-facing mishelp regression.
- `make verify` initially failed due a legibility string-case check; fixed in-plan wording and reran verification in S5.

Remaining risks/gaps:

- IP derivation currently targets trailing IPv4 in `node`; non-IPv4/non-trailing formats return empty `ip`.
- Mapping rows with missing `node` or `instance` are intentionally skipped.

Follow-up opportunities:

- Add explicit integration/e2e coverage against a real Loki payload fixture if/when stable fixture inputs are available.
- Consider optional diagnostics for skipped streams missing required labels.

## Context and Orientation

Relevant current code and docs:

- `internal/cmd/loki.go`: Loki command graph (`query`, `labels`, `label-values`), shared retry helper, and Loki response parsing for logs.
- `internal/cmd/loki_test.go`: existing Loki command contract tests and request-shape assertions.
- `internal/cmd/help.go` + `internal/cmd/help_test.go`: user-facing help text contract.
- `internal/cmd/completion.go`: static family subcommand completion map.
- `docs/COMMAND_SURFACE.md`: command-family responsibilities and update checklist.
- `README.md`: user examples and behavior notes.

Non-obvious terms used in this plan:

- data-server instance: the value expected by node-level metric queries (`metrics range --data-server`), represented by Loki `node` label.
- Loki instance: pod/workload identifier in Loki stream label `instance`.
- mapping row: one unique tuple (`node`, `data_server_instance`, `loki_instance`, `ip`) derived from stream labels.

Scope boundaries:

- In scope: add new Loki subcommand and associated tests/docs.
- Out of scope: modifying `metrics` command behavior, adding custom selector support, or changing existing Loki query output contracts.

## Milestones

### Milestone 1: Command Surface and Request Path

Add new `loki instance-map` subcommand and parse flags (`--cluster-id`, `--start`, `--end`, `--limit`, `-o/--output`).

Use existing range resolution helpers and existing Loki retry helper to issue `GET /data-proxy/loki/<cluster-id>/api/v1/query_range`.

Acceptance: command builds request with expected path/query values and validation errors for bad flags.

### Milestone 2: Mapping Extraction and Rendering

Decode stream results, extract unique `node` + `instance` pairs, derive IPv4 from `node`, and sort deterministically.

Render default table output and JSON array output from the same normalized row model.

Acceptance: deterministic rows, no duplicate pairs, and correct default/JSON output selection.

### Milestone 3: Tests and Docs

Add focused tests in `internal/cmd/loki_test.go`; update help, completion, README, and command-surface docs.

Acceptance: focused tests pass and docs/help reflect exact command contract.

### Milestone 4: Full Verification

Run repository verification ladder and capture concise evidence.

Acceptance: `make verify` passes with no regressions.

## Plan of Work

Implement this work as four commit-bounded subtasks.

Subtask S1 edits `internal/cmd/loki.go` to register and dispatch `instance-map`, parse flags, resolve cluster/range values using existing helpers, and call the existing query_range transport/retry path. Keep request-shaping logic explicit in the command handler.

Subtask S2 extends `internal/cmd/loki.go` with small focused helpers:

- parse query_range response into stream label sets,
- derive mapping rows from stream labels,
- compute IPv4 from `node` where possible,
- dedupe and stable-sort rows,
- render table and JSON output.

Subtask S3 adds contract tests in `internal/cmd/loki_test.go` for:

- required flags and fallback behavior,
- request path/query validation,
- limit and output validation,
- dedupe/conflict behavior,
- retry behavior for 504s,
- default table and JSON output behavior.

Subtask S4 updates documentation/help/completion surfaces in `internal/cmd/help.go`, `internal/cmd/help_test.go`, `internal/cmd/completion.go` (plus tests if affected), `README.md`, and `docs/COMMAND_SURFACE.md`.

Keep each subtask near ~500 LOC changed (or less) and independently verifiable.

## Concrete Steps

Run all commands from repository root: `/Users/jujiajia/code/tidbcloud/clinic-cli`.

1. Baseline Loki tests before edits:

   `go test ./internal/cmd -run Loki -v`

   Expected: PASS on current baseline.

2. Execute S1 (wiring + parsing + request-shape scaffolding), then run:

   `go test ./internal/cmd -run 'Loki.*(InstanceMap|QueryRange|Labels|LabelValues)' -v`

   Expected: new instance-map tests may fail until S2/S3; no unrelated regressions in existing Loki tests.

3. Execute S2 (mapping and rendering), then run targeted tests:

   `go test ./internal/cmd -run 'Loki.*InstanceMap' -v`

   Expected: PASS for mapping-specific tests including output and dedupe/conflict behavior.

4. Execute S3 (remaining tests) and rerun full Loki package focus:

   `go test ./internal/cmd -run Loki -v`

   Expected: PASS.

5. Execute S4 (help/docs/completion), then run:

   `go test ./internal/cmd -run 'Help|Completion' -v`

   Expected: PASS with updated help/completion expectations.

6. Final repo verification:

   `make verify`

   Expected: PASS (legibility check + all tests).

7. After each subtask pass, commit once with descriptive message and update `Progress` with timestamp and evidence.

## Validation and Acceptance

Behavioral acceptance criteria:

1. `clinic loki instance-map --cluster-id <id> --start <t1> --end <t2>` returns a table with columns for node/data-server-instance/Loki-instance/IP.
2. `clinic loki instance-map ... -o json` returns valid JSON array of objects with keys: `node`, `data_server_instance`, `loki_instance`, `ip`.
3. Default output is table when `-o/--output` is omitted.
4. Invalid `--output` returns deterministic validation error.
5. `--limit` default is `1000`, rejects `<=0`, rejects `>1000`.
6. Missing/invalid time inputs follow existing Loki required-range behavior and error style.
7. Duplicate stream pairs are deduped; conflicting node->multiple-instance mappings preserve all unique pairs.
8. Request retries 504 up to existing policy, then returns last 504 if still failing.
9. Existing commands (`loki query`, `loki labels`, `loki label-values`, metrics/data/events families) remain behaviorally unchanged.

## Idempotence and Recovery

- All test and verify commands in this plan are safe to rerun.
- If a subtask fails validation, do not commit; continue editing within that subtask boundary until tests pass.
- If documentation updates drift from implemented behavior, treat tests/help assertions as source of truth and realign docs before commit.
- If unexpected unrelated diffs appear during execution, pause and resolve scope before proceeding to next subtask.

## Artifacts and Notes

Key evidence snippets from execution:

- Focused Loki rung:
  - Command: `go test ./internal/cmd -run Loki -v`
  - Result: `PASS` with `ok   github.com/tidbcloud/clinic-cli/internal/cmd ...`
- Help/completion rung:
  - Command: `go test ./internal/cmd -run 'Help|Completion' -v`
  - Result: `PASS`
- Repo verification:
  - First run: failed at legibility check due missing lowercase `operator prompt template` phrase.
  - Final run: `make verify` => `agent legibility checks passed` and `go test ./... -v` all PASS.

Suggested sample output shape for manual spot check:

    NODE                        DATA_SERVER_INSTANCE        LOKI_INSTANCE                           IP
    cn-hongkong.10.0.7.198      cn-hongkong.10.0.7.198      db-4b0d6fe-worker-tidb-u12t8g          10.0.7.198

    [
      {
        "node": "cn-hongkong.10.0.7.198",
        "data_server_instance": "cn-hongkong.10.0.7.198",
        "loki_instance": "db-4b0d6fe-worker-tidb-u12t8g",
        "ip": "10.0.7.198"
      }
    ]

## Interfaces and Dependencies

CLI interface to add:

- `clinic [--env <env>] [--profile <name>] loki instance-map --cluster-id <cluster-id> --start <time> --end <time> [--limit <n>] [-o <table|json>]`

Expected request contract:

- Method: `GET`
- Path: `/data-proxy/loki/<cluster-id>/api/v1/query_range`
- Query params: `query`, `start`, `end`, `limit`
- Query value source:
  - `query`: existing default Loki selector inference (`buildDefaultLokiQuery`)
  - `start`/`end`: existing required range resolution and Unix-nanoseconds formatting

Expected row model (internal/output contract):

    type lokiInstanceMapRow struct {
        Node               string `json:"node"`
        DataServerInstance string `json:"data_server_instance"`
        LokiInstance       string `json:"loki_instance"`
        IP                 string `json:"ip"`
    }

Dependencies and reuse expectations:

- Reuse existing Loki retry helper (`executeLokiQueryRangeRawWithRetry`) and response types where practical.
- Use standard library only for new helpers (`sort`, `regexp`, `encoding/json`, `text/tabwriter`, etc.); avoid introducing new module dependencies.
- Preserve `clinic.HTTPError` behavior and existing error-normalization conventions.

---

Plan created on 2026-04-07 and aligned with user-confirmed decisions for command naming, output modes, conflict handling, and range semantics.