# Investibook Mechanism Candidate Evaluation

## Post-evaluation result

The five scenario-local candidate paragraphs were subsequently tested as
single-variable A/B changes on 20 additional cases. None improved either of
its two target cases, and none changed a control conclusion. They were removed
as redundant rather than admitted on design plausibility alone. See
[`2026-08-06-investibook-scenario-rule-isolated-evaluation.md`](2026-08-06-investibook-scenario-rule-isolated-evaluation.md).

## Decision boundary

Investibook family rules may be adapted inside the matching auto-oncall
scenario with scenario-specific contract tests. A cross-scenario diagnosis
mechanism must not enter the common scenario contract or final synthesis only
because it exists in Investibook.

The current four-case comparison does not establish a shared mechanism gap:

- `ST-2864` failed at component/recipe selection, before diagnosis reasoning.
- `ST-2863` and `ST-2865` favored the existing Clinic diagnosis path.
- only `TCID-13001` showed materially deeper Investibook investigation, while
  also showing over-attribution from incompletely aligned evidence.

Therefore no Investibook-wide candidate ranking, mandatory source count,
numeric dominance threshold, profiling count, or statistical workflow is
admitted as a common mechanism by the existing comparison.

## Existing four-case pre-screen

This is a judgment from the already completed manual comparison, not a
post-change replay:

| Common candidate | ST-2865 | ST-2864 | ST-2863 | TCID-13001 | Pre-screen |
|---|---|---|---|---|---|
| Always rank 2-3 root-cause families | No established benefit; may add unsupported bug/query hypotheses | Does not fix the wrong TiKV recipe | Existing result is already precise; extra hypotheses can dilute it | May preserve memory/query/DDL alternatives and reduce premature attribution | Reject as common; only 1/4 plausibly benefits |
| Require two or more source families | Adds work without a demonstrated missing causal gate | Correct component metrics/topology are sufficient to fix the main gap | Can turn an already sufficient metric chain into unnecessary queries | Multi-source evidence is useful | Reject as mandatory; choose sources per scenario |
| Require previous-day same-clock baseline | No periodicity requirement established | Useful only for capacity trend, not incident diagnosis by default | No long-period pattern established | Potentially useful if the reported behavior is periodic | Keep conditional inside the relevant scenario |
| Apply hard TopSQL dominance thresholds | Not the bounded memory question | Not applicable | Not required for the confirmed limit comparison | Can help rank query candidates but cannot repair DDL/window misalignment | Query/hotspot scenario heuristic only |
| Require about 50 profiles plus PCA-style analysis | Disproportionate to the current evidence gap | Not applicable | Not applicable | Could deepen a later CPU investigation but is not required for the current bounded answer | Reject as a common auto-oncall workflow |

The four-case evidence therefore supports the evidence-adapter and
scenario-local changes, but does not pass the minimum two-target bar for a
shared diagnosis-mechanism change.

## Candidate procedure

Evaluate one common-mechanism candidate at a time. Freeze identical Jira
fields, comments, attachments, cutoff, provider evidence, and provider failures
for both arms:

- baseline: current common diagnosis contract plus scenario-local rules;
- candidate: baseline plus exactly one proposed common-mechanism change.

Use manual blinded comparison. Randomize arm labels before review and do not
show the reviewer which output came from Investibook-derived logic. Do not build
an API-based answer comparator.

## Required cohort

- at least two target cases reproducing the same bounded mechanism problem;
- at least two controls where the current answer is already acceptable;
- include both Cloud and OP controls when the candidate changes their shared
  diagnosis contract;
- use additional cases beyond the four-case Clinic/Investibook report when that
  report contains fewer than two matching target gaps.

## Review dimensions

For each arm, record:

1. evidence fidelity: no invented, leaked, or misbound fact;
2. entity/time alignment: same instance and incident window where required;
3. causal calibration: observed, correlated, and confirmed are not conflated;
4. diagnostic specificity: the answer names the useful mechanism and strongest
   remaining alternative;
5. question coverage: the customer's actual question is answered;
6. evidence economy: no irrelevant source, profile, or broad query sweep;
7. action safety: recommendations follow from the admitted mechanism.

## Admission rule

Admit the common mechanism only when both target cases improve on the bounded
problem, neither control regresses on a load-bearing dimension, and the change
does not rely on provider-specific data unavailable to the other shared
environment. Mixed or sample-limited results keep the mechanism as a rejected
or unresolved candidate; they may still support a narrower scenario rule.

Record the per-case baseline/candidate evidence and manual judgment before
editing the shared contract. Deployment, replay, and Jira refresh remain
separate later verification steps.