# Troubleshooting Skill Time-Decayed Dry-run Plan

## Current Design

- The Confluence troubleshooting collection contains broad procedural guidance,
  historical ticket examples, fixed-version notes, internal source links, and
  production workarounds in the same pages.
- Auto-oncall currently routes evidence before the support agent runs:
  Investibook is primary for eligible Cloud performance cases, Wetech and Manus
  use account-specific read-only skills, and unsupported OP accounts stop
  before diagnosis.
- `npm run evaluate:recent` is the canonical read-only replay path for the one
  Active Focus Issue. It filters future support replies from agent input and
  uses the hidden historical reply only for evaluation.
- The current Active Focus contains three recent performance-diagnosis bad
  cases and one recent control case.

## Risks

- Loading the complete troubleshooting collection would consume excessive
  context and expose the agent to unrelated scenarios.
- Old cases can encode fixed bugs, obsolete defaults, retired dashboards, or
  unsupported operational workarounds.
- A recent ticket can still be version-incompatible with the current target;
  recency alone does not make its conclusion transferable.
- Case examples can leak expected answers into replay and produce a false
  improvement.
- A broad skill can regress a working control case while improving one target.
- Read-only diagnosis must not turn documented mitigations into production
  actions.

## Target Contract

### Case freshness

Use the incident time when it is explicit and reliable; otherwise use Jira
ticket creation time.

- Age `<= 90 days`: eligible as a trusted candidate example only when product,
  deployment type, major/minor version behavior, symptom, and evidence path are
  compatible.
- Age `> 90 days && <= 365 days`: reference-only. It may generate a hypothesis
  or a missing-evidence check, but cannot establish the current root cause,
  product behavior, threshold, default, workaround, or fixed version.
- Age `> 365 days`: discard. Do not load it into agent context, skill
  references, evaluation expectations, or generated recommendations.
- Version incompatibility overrides freshness. A recent case about behavior
  already fixed or removed for the target version is not trusted evidence.
- Official current documentation and release notes are product contracts, not
  case examples. Validate version-dependent behavior against them separately.

### Skill content

- Do not include historical ticket bodies, customer identifiers, case IDs,
  screenshots, human answers, or bug catalogs in the skill.
- Keep `SKILL.md` limited to routing, evidence ranking, freshness checks,
  read-only boundaries, and the fixed RCA output contract.
- Load at most one symptom reference initially:
  - storage latency;
  - PD/TSO wait;
  - connection/network latency.
- If a ticket artifact already names a direct wait or latency mechanism and
  provider evidence is still required, that mechanism selects the branch. The
  first provider batch is limited to three checks: the mechanism, its
  client/server counterpart, and the strongest alternative. Generic CPU, GC,
  overall P99, TopSQL, and baseline sweeps come later.
- Do not add OOM, scaling, merge-region, TiCDC, Kill/Cancel, or transaction
  internals until a separate target/control dry-run validates that module.
- Existing provider skills remain transports. The candidate skill must not add
  provider fallback, authentication, or persistence behavior.
- Provider-specific diagnosis must honor the selected branch and stop
  condition. Its generic completeness table must not force CPU, TopSQL,
  workload attribution, metric inventory, or extra baselines when the direct
  mechanism, counterpart, and strongest alternative already establish a
  bounded preliminary conclusion.

### Evaluation

- Baseline and challenger use the same Active Focus cases, Jira replay cutoff,
  model/reasoning settings, provider credentials, and installed transports.
- Case examples and expected answers are never supplied to the support agent.
- Compare all acceptance criteria, not only the aggregate finding:
  evidence route, direct evidence first, minimal follow-up, and reply
  convergence.
- Any target improvement accompanied by a control regression is retained as
  evidence but not accepted as a general skill improvement.

## Implementation Steps

1. Record Jira creation/incident time and version for every selected case.
2. Drop cases outside the freshness and version-compatibility contract.
3. Run the current `main` baseline in a staging-parity local skill environment.
4. Create one minimal diagnostic-reasoning candidate with only the three
   validated symptom references.
5. Add focused contract tests for freshness, conditional reference loading,
   missing evidence, provider isolation, and read-only behavior.
6. Run the same Recent Evaluation as challenger.
7. Keep only rules that improve a target without regressing the control.
8. If the first challenger still performs a broad sweep before direct evidence,
   tighten the provider-query ordering contract and rerun only the failed target
   before another full four-case evaluation.

## Verification Checklist

- [x] The selected Active Focus cases are within 90 days.
- [x] Jira write mode is `none`.
- [x] Baseline completes with required provider skills available.
- [x] Candidate `SKILL.md` stays below 500 lines and does not contain case IDs.
- [x] Tests cover 90-day, 365-day, expired, and version-incompatible cases.
- [x] Challenger reruns the identical target/control set.
- [x] First challenger preserved the control but APID-29166 still timed out
      after broad and duplicate metrics; the candidate is not accepted yet.
- [x] Tightened query ordering made TSO wait, TSO RPC, and PD TSO handling the
      first three checks, but the target still timed out because the provider
      workflow kept filling generic completeness dimensions.
- [x] The failed target loads only the PD/TSO reference and queries direct
      TSO/PD evidence before generic metrics.
- [x] The stop-condition candidate completes APID-29166 within the same
      timeout without metric inventory or unrelated workload queries.
- [x] Target and control results are compared criterion by criterion:
      APID-29166 meets direct-evidence and minimal-follow-up criteria; the
      APID-29178 control meets all four criteria.
- [x] No Jira, Feishu, deployment, configuration, or production operation is
      performed.

## Dry-run Decision

Keep these candidate modules:

- historical case freshness and version compatibility;
- one symptom branch at a time;
- direct artifact to first provider query ordering;
- branch-specific completeness and an explicit stop condition;
- missing evidence recorded as unverified.

Do not claim that this candidate resolves the complete Active Focus:

- APID-29166 classification still reports missing cluster context even though
  the Wetech provider can resolve and query the cluster. This is an upstream
  classifier/lookup-key contract gap, not missing troubleshooting knowledge.
- The final target draft puts the bounded conclusion in its second paragraph.
  Reply-first-paragraph convergence remains a separate output-style gap.

## Not Doing

- Do not copy the full Confluence collection into a skill.
- Do not treat historical human replies as prompts or automatic goldens.
- Do not add a second bad-case source alongside the Active Focus Issue.
- Do not merge, deploy, write Jira comments, or change customer-visible
  automation in this task.