# 分层诊断 Evaluation 结果记录

这份文档记录 PR #549 本地运行分层诊断 skill 后上传的 evaluation artifacts，方便 reviewer 直接查看结果，不需要重新跑完整评测。

## 运行环境

- 分支：`multi-agent`
- Agent：`opencode`
- Model：`chat/investibook-regression`
- Suite：`core-daily-demo @ e3571509e8ac`
- Baseline reference：`baseline-evalrun-b92aea0782574c129e3a933d662adcdd`
- 说明：自动 baseline comparison 被标记为 `resolved_model_mismatch`，因为 promoted baseline 使用的是 `chat/claude-standard`；下面的对比是基于 promoted baseline 分数做的人工汇总。

## 上传的 Evaluation Runs

| Run | 范围 | 结果 | Report |
| --- | --- | --- | --- |
| `evalrun-85ca047c679b446d890336ccf19f5de4` | 4 个 baseline 已通过 case | 4 pass / 0 fail / 0 invalid，平均分 77 | `backend/data/eval_audits/evalrun-85ca047c679b446d890336ccf19f5de4/report_detail.txt` |
| `evalrun-5f17b24f2b4c4519a2f35964240f2b08` | 5 个 baseline 低分 case | 0 pass / 4 fail / 1 invalid，平均分 31.8 | `backend/data/eval_audits/evalrun-5f17b24f2b4c4519a2f35964240f2b08/report_detail.txt` |
| `evalrun-44b6e57c48724404bb75753d4ccede10` | panic case 补跑 | 1 pass / 0 fail / 0 invalid，分数 75 | `backend/data/eval_audits/evalrun-44b6e57c48724404bb75753d4ccede10/report_detail.txt` |

## 10 个 Case 的分数对比

Bucket 顺序为：`workflow / evidence collection / critical evidence / routing / diagnosis`。
满分为 `10 / 5 / 30 / 5 / 50`。

| Case | Baseline | 分层后 | 变化 | Baseline buckets | 分层后 buckets |
| --- | ---: | ---: | ---: | --- | --- |
| `component-restart-during-incident-window` | 28 fail | 20 fail | -8 | 10/5/0/5/8 | 10/5/0/5/0 |
| `cpu-spike-during-incident-window` | 30 fail | 47 fail | +17 | 10/5/10/5/0 | 10/5/10/5/17 |
| `expensive-query-alert-correlation-during-incident-window` | 70 pass | 78 pass | +8 | 10/5/30/5/17 | 10/5/30/5/28 |
| `expensive-query-spike-during-incident-window` | 72 pass | 72 pass | 0 | 10/5/30/5/22 | 10/5/30/5/22 |
| `panic-in-recoverable-goroutine` | 70 pass | 75 pass | +5 | 10/5/30/5/17 | 10/5/30/5/25 |
| `service-degradation-during-incident-window` | 75 pass | 88 pass | +13 | 10/5/30/5/25 | 10/5/30/5/38 |
| `sql-tail-latency-spike-during-incident-window` | 48 fail | 55 fail | +7 | 10/5/20/5/8 | 10/5/20/5/15 |
| `sustained-cpu-pressure-during-incident-window` | 69 fail | 0 invalid | -69 | 10/5/16/5/35 | invalid |
| `system-unavailability-during-incident-window` | 54 fail | 37 fail | -17 | 10/5/20/5/14 | 10/5/0/5/17 |
| `tail-latency-surge-during-incident-window` | 100 pass | 70 pass | -30 | 10/5/30/5/50 | 10/5/30/5/17 |

## 汇总

| 指标 | Baseline | 分层后 |
| --- | ---: | ---: |
| Pass / Fail / Invalid | 5 / 5 / 0 | 5 / 4 / 1 |
| 平均分 | 61.6 | 54.2 |

这轮结果说明，分层结构在部分 query、panic、service degradation case 上有明显改善，但对需要更强 evidence routing 和专项诊断模板的 case 仍然不稳定。

- `sustained-cpu-pressure-during-incident-window` 在输出可比较诊断前超时，导致 invalid。
- `component-restart-during-incident-window` 仍然缺少 restart / panic / query interruption 证据链，诊断质量下降。
- `system-unavailability-during-incident-window` 漏掉了 coprocessor worker capacity collapse 相关证据链。
- `tail-latency-surge-during-incident-window` 保住了 critical evidence 满分，但 diagnosis 分数下降，说明证据被收集到了，但后续推理没有稳定用好。

因此，这批 artifacts 的价值不在于证明分层方案已经全面优于 baseline，而是暴露了下一步应该补强的位置：显式 signal classifier、强制 evidence checklist，以及更明确的 specialist diagnosis routing。