---
name: investibook-regression-add-case
description: Add or update Investibook regression cases under evaluation/regression/cases and suites. Use when creating a new regression case, migrating a workspace export into case.yaml plus judge.yaml, anonymizing case metadata, or updating suite membership.
---

# Investibook Regression Add Case

Use this skill when adding or updating regression cases in this repo.

## Scope

The current case layout is:

```text
evaluation/regression/cases/<case_id>/
  case.yaml
  judge.yaml
  expected.md
```

Suites live under:

```text
evaluation/regression/suites/
```

`case.yaml` is run-visible input.
`judge.yaml` is judge-only policy and ground truth.

## Hard Rules

- Keep case metadata anonymized.
- Do not leak the actual root cause, diagnosis direction, dominant query pattern, or known-bug label in run-visible fields.
- Treat these fields as run-visible and therefore sanitized:
  - `case.yaml:id`
  - `case.yaml:title`
  - `case.yaml:tags`
  - `case.yaml:source`
  - suite descriptions and case ids referenced from suite files
- Do not put real provenance in `case.yaml`, including:
  - Jira issue ids such as `O11Y-...`
  - Lark or Investibook workspace ids
  - GitHub issue numbers
  - original customer or reporter identifiers
  - `golden_run` records that include source workspace ids
- `case.yaml.source` may keep only a coarse, non-identifying source type such as
  `jira` or `workspace_export`; source ids belong outside formal regression cases.
- Prefer neutral names such as:
  - `service-degradation-during-incident-window`
  - `component-restart-during-incident-window`
  - `latency-regression-during-incident-window`
- Do not use names like:
  - `tidb-cpu-hotspot-from-stats-misestimation`
  - `known-bug-63956`
  - `missing-index-root-cause`
- Keep judge concept names generic and reusable.
- Do not invent new `judge.*.checks[].id` concepts that are narrowly phrased around one case's root cause unless the concept is clearly reusable across multiple cases.
- Prefer stable evidence-category names, and put case-specific detail in `rules`, not in the concept name.
- Do not add global `output_fact` ids for one-off issue ids, GitHub issue numbers,
  error codes, SQL digests, alert labels, or exact panic strings. These are
  case-local string matching requirements and belong in `judge.yaml` rules with
  `source: output_text`.
- Add a new global `output_fact` only when it represents a reusable diagnosis
  semantic across multiple cases, not merely a literal mention. Good candidates
  describe an asserted diagnosis path or a reusable misdiagnosis class; bad
  candidates describe a specific ticket, bug number, query digest, or error code.
- Prefer names like:
  - `primary_symptom_evidence_present`
  - `primary_query_evidence_present`
  - `root_cause_mechanism_evidence_present`
- Avoid names like:
  - `stats_misestimation_evidence_present`
  - `tidb_cpu_hotspot_root_cause`
  - `known_bug_63956_evidence_present`

## Allowed Detail Placement

- Put execution inputs in `case.yaml`:
  - coarse source type only
  - cluster id
  - version
  - time window
  - prompt text
- Put answer-side information in `judge.yaml`:
  - `ground_truth`
  - `judge.*`
- `expected.md` may describe the expected diagnosis, but should still avoid unnecessary customer-identifying wording.

## Authoring Checklist

- Create or update `case.yaml`, `judge.yaml`, and `expected.md` together.
- Verify `case.yaml` contains no `judge` or `ground_truth`.
- Verify `judge.yaml.case_id` exactly matches `case.yaml.id`.
- Sanitize `case.yaml.id`, `case.yaml.title`, `case.yaml.tags`, and suite descriptions.
- Verify `case.yaml.source` does not contain `issue_id`, `workspace_id`, or other
  real provenance, and that `case.yaml` does not contain `golden_run`.
- Review new judge check ids and descriptions for concept sprawl; reuse generic evidence-type concepts before adding a new one.
- Keep the prompt faithful to the incident, but avoid embedding the expected answer.
- For `evaluation.mode: adhoc_evidence`, keep the prompt colloquial and
  output-focused; do not hardcode an investigation script unless the script
  choice itself is part of the test.
- Update the target suite file when the case should run in an existing suite.
- Run targeted registry and judge tests after changes.

## Minimum Validation

Run:

```bash
backend/.venv/bin/python -m pytest backend/tests/test_eval_registry.py -q
backend/.venv/bin/python -m pytest backend/tests/test_eval_judge.py -q -k "<targeted selectors>"
```

## Ad-hoc Evidence Cases

Use `evaluation.mode: adhoc_evidence` when the case is closer to a real user
asking for a concrete data answer than to a full root-cause analysis. Typical
examples are ranking, top-N, point lookup, or structured evidence extraction
requests.

### Prompt shape

- Write the prompt in natural user language.
- State the final output contract clearly.
- Do not prescribe a fixed method unless the method choice is intentionally
  under test.
- Prefer constraints like:
  - exact output schema
  - exact required fields
  - whether extra prose is allowed
  - what to do when evidence is insufficient

Good pattern:

```yaml
evaluation:
  mode: adhoc_evidence
prompt:
  text: |
    帮我看一下集群 {{environment.cluster_id}} 在 {{runtime.resolved_window.start}}
    到 {{runtime.resolved_window.end}} 这段时间里，哪些节点最忙。

    我只关心最后结果。最终只输出一个 `json` 代码块，字段必须严格为 ...
```

Avoid prompts that expose the intended implementation path unless that path is
the test target. For example, do not tell the agent to run a specific script,
call `collect_evidence.py`, or use a fixed metric query template unless the case
is specifically validating that behavior.

### Runtime-resolved windows and references

Ad-hoc cases often need a time window or expected answer that must be computed
at runtime. Prefer runtime hooks over hardcoding stale values.

- Use a `before_acp` hook to resolve the actual window and expose it through a
  runtime output such as `resolved_window`.
- Use a `before_judge` hook to build a runtime reference answer from the same
  environment and resolved window.
- Point `time_window.resolved_by_output` at the runtime window output when the
  final evaluated window is not static.

Typical pattern:

```yaml
time_window:
  resolved_by_output: resolved_window
runtime:
  hooks:
  - id: resolve_runtime_window
    stage: before_acp
    output_id: resolved_window
    ...
  - id: build_runtime_reference
    stage: before_judge
    output_id: reference_answer
    ...
```

### Judge design for ad-hoc outputs

When the task is about objective output correctness rather than diagnosis
reasoning, put the important checks in `judge.critical_evidence`.

- Use `source: output_json` matchers such as:
  - `json_path_equals_reference`
  - `json_array_matches_reference`
  - `json_list_nonempty`
  - `json_path_unique_count_gte`
- Keep evidence-channel checks in `evidence_collection` or workflow sections.
- Omit `judge.diagnosis` entirely when the case does not truly require a
  diagnosis narrative and only needs correct evidence-backed output.

This keeps the case focused on what the user actually asked for.

### Optional hard check: forbid `collect_evidence.py`

Some ad-hoc cases are intended to remain direct-query requests rather than full
evidence-collection workflows. In those cases, add a case-local hard check that
invalidates the run if `collect_evidence.py` is invoked.

Use the existing reusable concepts:

```yaml
judge:
  workflow:
    checks:
    - id: collect_evidence_not_used
      required: true
      match: all_of
      rules:
      - source: workflow_fact
        matcher: bool_false
        path: collect_evidence_invoked
  execution_contract:
    checks:
    - id: collect_evidence_not_used
      required: true
      match: all_of
      rules:
      - source: workflow_check
        matcher: subset_observed
        paths:
        - collect_evidence_not_used
```

Important boundaries:

- This is opt-in per case, not a global default.
- Only add it when the case is explicitly testing an ad-hoc direct-query path.
- Do not add it to legacy diagnosis cases that are expected to use the normal
  evidence collection workflow.
- Do not try to force the agent away from other valid direct metric/query paths
  in the prompt; the hard check should only ban `collect_evidence.py` itself.

## Notes

- If the case comes from a workspace export, derive prompt, environment, and time window from the export artifacts, but keep the public-facing case identity anonymized.
- If a new case needs diagnosis semantics different from existing cases, prefer extending case-driven `judge.yaml` policy rather than reintroducing global hardcoded judge logic.
- When a case needs case-specific matching, bind the specificity in `rules` such as digest/keyword/artifact matching, not by minting a one-off check concept name.
- For answer-text requirements, prefer:

```yaml
- source: output_text
  matcher: keyword_any
  keywords:
  - "[executor:3024]"
  - "maximum statement execution time exceeded"
```

or, when phrasing needs more flexibility:

```yaml
- source: output_text
  matcher: regex_any
  patterns:
  - "known bug.{0,40}63956"
```

Use `source: output_fact` only for reusable semantic facts already supported by
the deterministic judge, such as an asserted diagnosis path or a reusable
negative/failure classification.