# Investibook Scenario Rule Isolated Evaluation

## Outcome

Five Investibook-derived scenario-rule candidates were evaluated independently
on 20 additional Jira cases. None passed admission:

- target cases materially improved: `0 / 10`;
- control cases regressed: `0 / 10`;
- candidates admitted: `0 / 5`.

All five candidate paragraphs were removed from the scenario references after
the evaluation. Their intended causal distinctions were already produced by
the existing scenario contract, so retaining the extra prose would add prompt
weight without demonstrated diagnostic gain.

## Method

This was a mechanism-isolated, manual A/B comparison rather than an end-to-end
provider replay or API answer comparator.

For each candidate:

1. Select two target cases where the proposed distinction is load-bearing and
   two controls where the current diagnosis should remain stable.
2. Freeze the same question and evidence facts for both arms.
3. Run the same model, common contract, scenario contract, output schema, and
   reasoning effort in both arms.
4. Baseline removes exactly the candidate paragraph. Candidate adds exactly
   that paragraph. No other prompt or evidence difference is allowed.
5. Manually compare causal status, primary mechanism, strongest alternative,
   unsupported claims, and action direction case by case.

The frozen packets are in
[`docs/evaluations/2026-08-06-investibook-scenario-cases.json`](evaluations/2026-08-06-investibook-scenario-cases.json).
They were frozen from the Jira descriptions, timelines, existing auto-oncall
answers, and later verified support conclusions available on 2026-08-06.

## Result By Candidate

| Candidate | Target cases | Control cases | Target improvement | Control regression | Decision |
|---|---|---|---:|---:|---|
| SQL causal admission | TCID-13001, TCID-12986 | APID-29251, TCID-12987 | 0 / 2 | 0 / 2 | Reject as redundant |
| OOM / GC distinction | ST-2847, ST-2865 | ST-2862, APID-29041 | 0 / 2 | 0 / 2 | Reject as redundant |
| Capacity vs downstream pressure | TCID-12926, TCID-12985 | ST-2864, APID-29277 | 0 / 2 | 0 / 2 | Reject as redundant |
| TiKV workload-induced vs independent IO | TCID-12948, APID-29259 | APID-29211, APID-29298 | 0 / 2 | 0 / 2 | Reject as redundant |
| TiFlash workload-induced vs independent IO | APID-29265, APID-12588 | APID-12208, ST-2766 | 0 / 2 | 0 / 2 | Reject as redundant |

## Per-case Manual Judgment

### SQL causal admission

| Case | Baseline | Candidate | Judgment |
|---|---|---|---|
| TCID-13001 | `unresolved`: confirms aligned system pressure but does not bind a TopSQL candidate to the DDL | Same causal status, primary mechanism, and missing-link boundary | No material change |
| TCID-12986 | `correlated`: bad plan is a partial contributor explaining about one third of CPU growth | Same partial attribution and same unresolved remainder | No material change |
| APID-29251 | `confirmed`: aligned ANALYZE scan dominates the affected instance and ends with CPU recovery | Same direct-cause conclusion | Control preserved |
| TCID-12987 | `confirmed`: per-execution keys and burst volume increased; reproduction and remediation align | Same direct-cause conclusion | Control preserved |

The existing alignment and evidence-ref rules already prevent cross-instance,
cross-window, or non-dominant TopSQL from being promoted to a root cause.

### OOM / GC distinction

| Case | Baseline | Candidate | Judgment |
|---|---|---|---|
| ST-2847 | Treats the event as a stable threshold alert and explicitly says it is not an observed OOM | Same conclusion | No material change |
| ST-2865 | Treats high allocator memory as unresolved and does not infer OOM, leak, or Region causality | Same conclusion | No material change |
| ST-2862 | Confirms a query-level 10 GiB quota failure and distinguishes it from process OOM | Same conclusion | Control preserved |
| APID-29041 | Confirms a real process OOM but leaves SQL/allocator cause unresolved | Same conclusion | Control preserved |

The existing scenario already requires restart reason, memory consumers,
limit/headroom, workload alignment, and a strongest alternative. The proposed
paragraph restated those boundaries without changing the answer.

### Capacity vs downstream pressure

| Case | Baseline | Candidate | Judgment |
|---|---|---|---|
| TCID-12926 | Keeps scale-in as correlated background and hotspot/Leader concentration as the stronger mechanism | Same ordering | No material change |
| TCID-12985 | Rules out RCU throttling from quota and wait evidence; keeps burst/lock pressure as the leading alternative | Same ordering | No material change |
| ST-2864 | Confirms capacity headroom risk while ruling out an acute IO/apply incident | Same conclusion | Control preserved |
| APID-29277 | Confirms low disk headroom as the operational boundary while not overclaiming the full TiCDC-lag share | Same conclusion | Control preserved |

The baseline already compares quota/saturation, wait paths, and workload
volume before assigning capacity as primary.

### TiKV workload-induced vs independent IO

| Case | Baseline | Candidate | Judgment |
|---|---|---|---|
| TCID-12948 | Makes one-row writes and index-maintenance amplification primary; disk headroom is not the limiting boundary | Same conclusion | No material change |
| APID-29259 | Locates the write-path/latch mechanism but preserves key contention vs slow storage as unresolved | Same uncertainty boundary | No material change |
| APID-29211 | Makes the independently aligned device/WAL/raft delay primary | Same conclusion | Control preserved |
| APID-29298 | Explains Slow Score semantics and limits the result to local write-path correlation, not a proven cloud-disk fault | Same conclusion | Control preserved |

The baseline scenario already requires matching internal storage/raft and
disk/engine evidence, separates producer backlog from consumer latency, and
checks workload/hotspot change as the strongest alternative.

### TiFlash workload-induced vs independent IO

| Case | Baseline | Candidate | Judgment |
|---|---|---|---|
| APID-29265 | Makes the mischosen TiFlash query plan primary based on the binding intervention and multi-day non-recurrence | Same conclusion | No material change |
| APID-12588 | Makes excessive query reads primary and independent disk failure unsupported | Same conclusion | No material change |
| APID-12208 | Makes broad host/device jitter primary because no single query explains the simultaneous behavior | Same conclusion | Control preserved |
| ST-2766 | Keeps the MPP inter-node connection failure primary and task accumulation downstream | Same conclusion | Control preserved |

The baseline already separates producer and consumer, rejects apply-wait-only
IO attribution, and requires a disk/engine counterpart.

## Boundary Of This Result

This evaluation answers whether the five paragraphs improve diagnosis logic
when evidence is held constant. It does not validate the newer Clinic
multi-source evidence adapter, provider availability, or an end-to-end deployed
replay. Those are separate capability and rollout questions.

If a future case shows one of these exact distinctions failing under the
existing contract, it should become a new bounded target signal. Until then,
the rejected paragraphs should not be reintroduced from Investibook by default.