# Diagnosis Quality Evaluation — 诊断质量闭环评估系统 > **Status**: Production > **Author**: Investibook Team > **Date**: 2026-04-21 > **Scope**: Postmortem workflow, regression testing, issue lifecycle management --- ## 目录 - [1. 问题定义](#1-问题定义) - [2. 现状与成果](#2-现状与成果) - [3. 系统架构](#3-系统架构) - [4. 执行流程详解](#4-执行流程详解) - [5. 数据模型](#5-数据模型) - [6. 关键设计决策](#6-关键设计决策) - [7. 目录结构](#7-目录结构) - [附录 A: Issue 分布分析](#附录-a-issue-分布分析) - [附录 B: Case 库概览](#附录-b-case-库概览) - [附录 C: 远程部署方案 — Evolver](#附录-c-远程部署方案--evolver) --- ## 1. 问题定义 ### 1.1 核心挑战 Investibook 的 AI 诊断 agent 持续接收真实故障，但缺少系统化的质量评估机制： - agent 诊断对了还是错了？错在哪里？ - 修了一个问题后，有没有引入新的退化？ - 相同类型的错误是否在多个 case 中反复出现？ 传统做法靠人工抽查，覆盖率低、无法追踪改进、问题容易被遗忘。 ### 1.2 Replayer 解决什么 Replayer 建立了一套**从用户反馈到回归验证的完整闭环**： ``` Incident --> Agent diagnosis --> User feedback --> Replayer postmortem ^ | | Find issues --> File issue --> Fix | | +-------- Regression verifies fix, no regression <-------------+ ``` 核心价值：**让诊断质量的提升可度量、可回溯、可自动化验证。** ### 1.3 两种核心能力 | 能力 | 目的 | 输入 | 输出 | |------|------|------|------| | **Postmortem** | 从已有诊断中提取教训 | workspace 对话记录 | postmortem.md + expected.md + issues | | **Regression** | 验证修复后诊断不退化 | case ID + expected.md | pass/fail + regression report | Postmortem 产出 case 和 issue；regression 消费 case 验证修复。两者构成闭环。 --- ## 2. 现状与成果 ### 2.1 数据规模 | 指标 | 数量 | 说明 | |------|------|------| | Case 库 | 12 cases | 覆盖 expensive-query、panic、latency-spike、CPU-spike、OOM 等场景 | | Issue 库 | 34 open issues | 12 个有详细 plan.md 的修复计划 | | GitHub Issues | 15+ filed | 自动创建，带 `replayer-triaged` 标签 | | 评分范围 | 1.8 ~ 4.0 / 5 | 最差：panic 误判为 FAKE ALARM；最好：根因正确但路径不优 | ### 2.2 Issue 类型分布 ``` FLOW ████████████ 12 Flow gaps (wrong route, missing steps) UX ██████████ 10 UX issues (msg lost, slow response, UI bug) REASONING █████ 5 Reasoning flaws (premature conclusion) KNOWLEDGE ████ 4 Domain knowledge gaps (TiDB patterns) TOOLING ████ 4 Tooling failures (API broken, empty logs) CROSS-REF █ 1 Cross-validation missing ``` **核心发现**：FLOW（流程缺陷）和 UX（体验问题）占总量的 65%，说明最大的改进空间在于"agent 没做该做的事"和"用户看到的体验不好"，而非"agent 推理能力不够"。 ### 2.3 典型 case 示例 | Case | Score | 问题 | 根因 | |------|-------|------|------| | O11Y-2429368 | 1.8 | Agent 把正常 panic 判为 FAKE ALARM | 不认识 executor:3024 超时中断模式 | | TCOC-5140 | 2.8 | Agent 判 OOM，实际是 ScaleResourcePool | 不查 Activity Hub、不验证 SIGTERM | | GH-229 | 2.6 | Agent 归因 plan cache miss | 不认识 prepare:exec:close=1:1:1 模式 | | O11Y-2426677 | 4.0 | 根因正确（缺索引），但路径不对 | 应查 slow query logs 而非 TopSQL | --- ## 3. 系统架构 ### 3.1 端到端流程 ``` +-----------------------------------------------------------+ | Input Layer | | | | GitHub Issue (#248) Workspace ID Case ID | | label: [diagnosis, "ws-xxx" "O11Y-xxx" | | auto-created] (direct) (regression) | +----------+-----------------------+------------------+------+ | | | v v v +----------------+ +----------------+ +----------------+ | Phase 0 | | Mode B/D | | Mode A/F | | Discover | | Postmortem | | Regression | | | | | | | | * Pull GitHub | | * Pull msgs | | * Create/rerun | | issue list | | * Evaluate | | * Check rubric | | * Parse title | | * Gen report | | * Follow-up | | * Route decide | | | | (max 3 rnd) | +-------+--------+ +-------+--------+ +-------+--------+ | | | +----------------------+---------------------+ | v +-----------------------------------------------------------+ | Evaluate (Phase 4-6) | | | | Phase 4: Extract Phase 5: Evaluate Phase 6: Gen | | * Messages * 5-dim score 1-5 * postmortem.md | | * Tool calls * Data src coverage * expected.md | | * Diagnosis text * Human intervention * cases.yaml | +---------------------------+-------------------------------+ | v +-----------------------------------------------------------+ | Issue Mgmt (Phase 7-8) | | | | Phase 7: Triage Phase 8: File | | * Scan msgs + title * Create GitHub issue | | * Dedup vs issues.yaml * [ISS-NNNN] prefix | | * Assign ISS-NNNN ID * label: replayer-triaged | | * Write plan.md * Write-back github_issue | | * Comment on source issue | +-----------------------------------------------------------+ ``` ### 3.2 闭环机制 Replayer 的价值不在于单次 postmortem，而在于**闭环**——issue 被追踪直到修复并回归验证： ``` Case Library (12 cases) | | postmortem finds issues v Issue Registry (34 issues) | | developer fixes v Regression (Mode F) | | verify expected.md rubric all pass v Issue status: fixed --> next postmortem no longer finds same issue ``` ### 3.3 Anti-loop 设计 Replayer 自己也会创建 GitHub issues（Phase 8）。为防止自己处理自己创建的 issue，通过 label 机制隔离： ``` User submitted: label = [diagnosis, auto-created] Replayer created: label = [replayer-triaged] Phase 0 query: -label:replayer-triaged --> only picks up user-submitted ``` 这是一个简单但关键的设计——没有它，系统会进入无限循环。 --- ## 4. 执行流程详解 ### 4.1 Phase 0: Discover — 从 GitHub Issue 到执行路径 Phase 0 是可选的入口。当用户通过 `/issue` 命令提交问题后，GitHub issue 成为 replayer 的输入源。 ``` GitHub Issue #248 title: "[Investibook] lark-ws-xxx: no useful analysis, 2nd response lost" labels: [diagnosis, auto-created] body: workspace URL + metadata | v +------------------------------------+ | Step 1: Fetch candidate issues | GitHub Search API | q: is:open label:diagnosis | exclude replayer-triaged | label:auto-created | | -label:replayer-triaged | +----------------+-------------------+ | +----------------v-------------------+ | Step 2: Parse issue | | * workspace ID <-- body | | * environment <-- URL domain | | * ticket ID <-- title | | * user feedback <-- after colon | +----------------+-------------------+ | +----------------v-------------------+ | Step 3: Title analysis | | | | Analyze title --> decide route: | | | | UI/UX issue -------> Phase 7 | "button broken" | (title-only triage) | "msg lost on refresh" | | | Diagnosis quality --> Phase 1 | "wrong diagnosis" | (full postmortem) | "no analysis found" | | | Both ------------> Phase 1 | "analysis + response lost" | (postmortem + extra triage) | +------------------------------------+ ``` **设计意图**：Title analysis 避免对纯 UI/UX 问题拉取 workspace 对话——这类问题不需要分析诊断过程，直接从 title 创建 issue 即可。 **关键规则**：Title 中的用户反馈**必须**被 triage 为 issue，无论是否走 postmortem 流程。Title-derived issues 和 workspace-derived issues 是**叠加**关系，不是二选一。 ### 4.2 Phase 1: Authenticate — 双 Token 体系 ``` +-------------------------------------------------+ | Token 1: Investibook API | | | | Local -> Python one-liner create dev token | | Remote -> ~/.investibook/config.json profiles | | +- dev: investibook.dev.xxx.com | | +- prod: investibook.tidbcloud.com | +-------------------------------------------------+ | Token 2: GitHub API | | | | Priority: | | 1. tools/replayer/.github_token (local file) | | 2. $GITHUB_TOKEN / $GH_TOKEN (env var) | | 3. kubectl exec (from K8s pod, last resort) | | | | No token -> skip Phase 0/8, postmortem only | +-------------------------------------------------+ ``` ### 4.3 Phase 2-3: Create + Run — 六种模式 根据场景选择不同的执行模式： | 模式 | 场景 | 环境 | 流程 | |------|------|------|------| | **D** | 已有远程 workspace 的 postmortem | Remote | Auth → Extract → Evaluate → Generate | | **B** | 已有本地 workspace 的 postmortem | Local | 同 D，但连本地 backend | | **F** | 交互式回归测试（推荐） | Local | Auth → Run → Check → Rubric → Follow-up loop | | **A** | 新 case 全量回归 | Local | Auth → Create → Run → Extract → Evaluate | | **C** | 已有 workspace 重跑 | Local | Auth → Clear → Re-run → Evaluate | | **E** | 远程全量回归 | Remote | 同 A，但在远程环境 | **Mode F 的交互式 follow-up 循环**： ``` start ─→ run diagnosis ─→ --check ─→ rubric verify │ all pass? ─── yes ──→ Phase 4 │ no │ round < 3? ─── no ──→ Phase 4 (evaluate as-is) │ yes │ --followup (targeted hint) │ --check ─→ rubric verify ─→ ... ``` Follow-up 的关键：**不是泛泛地说"再想想"，而是引用具体的 rubric 失败项，告诉 agent 该查什么数据。** ### 4.4 Phase 4-6: Extract → Evaluate → Artifacts ``` +---------------+ +--------------------+ +--------------------+ | Phase 4 | | Phase 5 | | Phase 6 | | Extract |--->| Evaluate |--->| Generate | | | | | | | | From messages | | 5-dim scoring: | | postmortem.md | | API pull: | | | | - issue summary | | * Chat text | | Root Cause 1-5 | | - actual root | | * Tool calls | | Evidence 1-5 | | cause | | * Human | | Causal Chain 1-5 | | - agent summary | | interventn | | Actionability 1-5 | | - scores | | | | Efficiency 1-5 | | - human guidance | | | | | | - deficiencies | | | | Data src coverage: | | | | | | [x] Metrics | | expected.md | | | | [x] TopSQL TiDB | | - root cause | | | | [x] TopSQL TiKV | | - evidence chain | | | | [x] SlowQuery | | - scoring rubric | | | | [x] Logs | | | | | | [ ] Activity Hub | | cases.yaml entry | +---------------+ +--------------------+ +--------------------+ ``` **评分维度说明**： | 维度 | 1 分 | 3 分 | 5 分 | |------|------|------|------| | Root Cause Accuracy | 完全错误 | 方向对但不精确 | 精确命中 | | Evidence Completeness | 只查了 1 个数据源 | 查了主要数据源 | 所有相关数据源 | | Causal Chain | 无因果链 | 有链但缺环节 | 完整 A→B→C | | Actionability | "请检查" | 有具体方向 | 可执行的 SQL/配置 | | Efficiency | 30+ 轮人工干预 | 5-10 轮 | 0-1 轮自主完成 | ### 4.5 Phase 7: Issue Triage — 问题发现的核心环节 Phase 7 是 Replayer 最独特的价值：**从每个 case 中系统性地提取可改进项**。 ``` Input: +-- workspace messages (from Phase 4) +-- issue title (from Phase 0) | v +------------------------------------------+ | Scan for issues: | | | | Missing evidence "no Activity Hub" | | Tool failure "Loki returned empty" | | Flow gap "no node restart chk" | | Reasoning error "plan cache != cause" | | UX feedback "2nd response lost" | +-------------------+----------------------+ | v +------------------------------------------+ | Dedup (issues.yaml): | | | | Match by title similarity | | Match by affects (skill+file) | | | | +- Exists -> append found_in | | | ISS-0030 += O11Y-2426677 | | +- New -> assign ISS-NNNN | | ISS-0043 (new) | +-------------------+----------------------+ | v +------------------------------------------+ | Write (two tiers by complexity): | | | | Simple -> inline in issues.yaml | | fix_plan: > one paragraph | | Complex -> ISS-NNNN/plan.md | | fix_plan: ISS-NNNN/plan.md | | * Full context | | * Root cause analysis | | * Actionable fix steps | | Goal: fixer needs no original workspace | +------------------------------------------+ ``` **关键设计**：`plan.md` 的目标是**自包含**——修复者读完 plan.md 就能实现修复，不需要重新拉取原始 workspace 对话。因为 workspace 可能已过期或不可访问。 ### 4.6 Phase 8: File to GitHub — 闭环追踪 ``` issues.yaml | | filter: status=open, no github_issue, has fix_plan | v +------------------------------------------------------+ | For each issue to file: | | | | 1. GitHub Search: [ISS-NNNN] --- exists -> skip | | write-back URL | | 2. Create issue: | | title: [ISS-0043] description | | labels: [replayer-triaged, P2] | | body: description + plan.md full text | | | | 3. Write-back github_issue URL to issues.yaml | | | | 4. Comment on source issue: | | "Replayer postmortem completed" | | "Score: 4.0/5" | | "Issues: ISS-0043 (#257)" | +------------------------------------------------------+ ``` --- ## 5. 数据模型 ### 5.1 Case (cases.yaml) Case 是一次真实故障诊断的记录，包含 ground truth。 ```yaml - issue_id: "TCOC-5140" # Ticket ID (unique) prompt: "Investigate latency spike..." # Diagnosis prompt cluster_type: "premium" # dedicated / premium tidb_version: "v8.5.4" # TiDB version tags: ["latency-spike", "rolling-restart"] # Searchable tags expected_root_cause: "Human-initiated..." # One-line ground truth date_added: "2026-04-20" reviewer: "claude-code" ``` 每个 case 对应 `cases/{issue_id}/` 目录： | 文件 | 用途 | 谁写 | |------|------|------| | `postmortem.md` | 诊断质量分析报告 | Postmortem 生成 | | `expected.md` | 回归测试基线（含 scoring rubric） | Postmortem 生成，人工审核 | ### 5.2 Issue (issues.yaml) Issue 是从 case 中提取的可改进项，跨 case 去重。 ```yaml - id: ISS-0039 # Unique ID, monotonic title: Agent should check Activity Hub on node restart type: FLOW # 7 types severity: P1-high # P0~P3 status: open # open / in_progress / fixed / wontfix github_issue: https://github.com/.../issues/251 # Written back by Phase 8 affects: # Affected code locations - skill: tidb-clinic-diagnosis file: agentkit/skills/builtin/tidb-clinic-diagnosis/SKILL.md description: > # Issue description 4 TiKV nodes restarted, Agent analyzed metrics/logs but never checked Activity Hub for human operations... fix_plan: ISS-0039/plan.md # Fix plan (inline or file) found_in: # Cross-case tracking - case: TCOC-5140 ws: ws-1776646913272-b0d2b9b99 ``` **Issue 类型**： | Type | 含义 | 例子 | |------|------|------| | FLOW | 诊断流程缺陷 | 没检查节点重启 | | UX | 用户体验问题 | 消息丢失、响应慢 | | TOOLING | 工具链故障 | API 废弃、查询返回空 | | KNOWLEDGE | 领域知识缺失 | 不认识 panic 模式 | | REASONING | 推理逻辑缺陷 | A/B 对比不验证 | | CROSS-REF | 交叉验证缺失 | 单一数据源下结论 | | TEMPLATE | 模板/格式问题 | SQL 列名错误 | ### 5.3 数据关系 ``` cases.yaml issues.yaml +--------------+ +--------------+ | TCOC-5140 |--postmortem--------> | ISS-0002 | | | found 3 issues | ISS-0038 | | | | ISS-0039 | +--------------+ +--------------+ | O11Y-2429368 |--postmortem--------> | ISS-0035 | | | found 3 issues | ISS-0036 | | | | ISS-0037 | +--------------+ +--------------+ | GH-229 |--postmortem--------> | ISS-0041 | | | found 2 issues | ISS-0042 | +--------------+ +--------------+ | found_in back-reference (one issue can appear in multiple cases) ``` --- ## 6. 关键设计决策 ### 6.1 为什么用 YAML 而非数据库 issues.yaml 是 **单文件注册表**，不是数据库。选择理由： 1. **Git 追踪**：每次变更有 commit 历史，谁在什么时候改了什么一目了然 2. **Claude Code 友好**：agent 可以直接 `Read` + `Edit`，无需 SQL 3. **去重简单**：读整个文件，按 title 或 affects 字段匹配 4. **规模可控**：预计 100-200 个 issues，YAML 完全可以承载 当 issue 数量超过 500 时可能需要迁移，但当前阶段这不是问题。 ### 6.2 为什么 plan.md 是自包含的 Workspace 对话有时效性——远程 workspace 可能被清理，dev 环境数据不持久。如果 plan.md 依赖"去 workspace 看第 5 轮对话"，修复者可能看不到。 因此 plan.md 的设计目标是：**读完就能实施修复，无需回溯原始对话。** 包含完整的上下文、错误消息、代码片段、修复步骤和验证方法。 ### 6.3 为什么 expected.md 和 postmortem.md 分开 - `postmortem.md` 记录**agent 做了什么**——是对诊断过程的分析 - `expected.md` 定义**agent 应该做什么**——是回归测试的基线 Regression 只消费 `expected.md`。两者分开确保回归测试不受 postmortem 分析措辞的影响。 ### 6.4 为什么 Phase 0 有 title analysis 早期版本对每个 GitHub issue 都拉取 workspace 对话。但实践中发现： - 约 30% 的 issue 是纯 UI/UX 问题（"下载按钮无反应"、"消息丢失"） - 这些 issue 的 workspace 对话与 title 描述的问题无关（workspace 可能是一个正常诊断） - 拉取和分析对话浪费时间，且可能引入误导信息 因此加入 title analysis：**title 描述 UI/UX → 跳过对话，直接从 title triage issue。** 节省约 30% 的处理时间。 ### 6.5 Scoring rubric 的设计理念 `expected.md` 中的 scoring rubric 是**二值 checklist**（通过/不通过），不是模糊评分： ```markdown - [ ] Query TiDB slow query logs as primary data source - [ ] Identify the specific expensive SQL - [ ] Filter out ANALYZE-related queries ``` 这个设计确保回归测试是**确定性的**——不存在"这次给 3 分还是 4 分"的主观判断。要么全过，要么有明确的失败项。 --- ## 7. 目录结构 ``` tools/replayer/ | +-- SKILL.md # Agent instructions - Phase 0-8 definitions | # Entry point for replayer as Claude Code skill | +-- README.md # This file - system design doc | +-- replay.sh # CLI tool | # replay.sh rerun | # replay.sh --check structured check | # replay.sh --followup send follow-up | # replay.sh --eval-only evaluate only | +-- cases.yaml # Case registry (12 entries) +-- cases/ | +-- TEMPLATE.md # Postmortem template | +-- O11Y-2328599/ | | +-- postmortem.md # Diagnosis quality report | | +-- expected.md # Regression baseline (scoring rubric) | +-- TCOC-5140/ | | +-- postmortem.md | | +-- expected.md | +-- ... (12 cases total) | +-- issues/ | +-- issues.yaml # Issue registry (34 open entries) | | # Single source of truth for cross-case dedup | +-- ISS-0002/plan.md # Detailed fix plan | +-- ISS-0039/plan.md | +-- ... (12 plan files) | +-- references/ | +-- api-cookbook.md # API call reference (Phase 1-4) | +-- evaluation-guide.md # Scoring template & checklist (Phase 5-6) | +-- troubleshooting.md # Common issues troubleshooting | +-- results/ # Regression run output ``` --- ## 附录 A: Issue 分布分析 ### 按严重度 ``` P1-high ████████████████ 16 Fix ASAP P2-medium ████████████████ 16 Next iteration P3-low ██ 2 Backlog ``` ### 按影响面 (affects) ``` tidb-clinic-diagnosis SKILL.md ████████████████████ 20+ Core diagnosis flow investibook-frontend ██████████ 10 Frontend UX collect_evidence.py ████ 4 Evidence collection investibook-core backend ████ 4 Backend logic jira scripts ██ 2 Jira integration ``` **洞察**：超过 60% 的 issue 指向 `tidb-clinic-diagnosis/SKILL.md`。这个文件是诊断质量的核心，也是改进投入产出比最高的位置。 --- ## 附录 B: Case 库概览 | # | Case ID | Cluster | Score | 核心问题 | Date | |---|---------|---------|-------|----------|------| | 1 | O11Y-2328599 | dedicated | — | 缺索引热点 | 2026-04-07 | | 2 | O11Y-2179140 | premium | — | 锁竞争热点 | 2026-04-07 | | 3 | TCOC-5011 | dedicated | — | Worker 下线级联 | 2026-04-10 | | 4 | TCOC-5019 | premium | — | 已知 bug #66096 | 2026-04-15 | | 5 | TCOC-4964 | dedicated | — | IN-list plan cache | 2026-04-15 | | 6 | O11Y-2298309 | dedicated | — | Kill query panic | 2026-04-07 | | 7 | O11Y-2392287 | dedicated | — | 自动恢复 expensive query | 2026-04-16 | | 8 | O11Y-2429368 | dedicated | 1.8 | Panic 误判 FAKE ALARM | 2026-04-20 | | 9 | TCOC-5140 | premium | 2.8 | OOM 误诊 (实为 ScaleResourcePool) | 2026-04-20 | | 10 | GH-229 | dedicated | 2.6 | Plan cache 误诊 (实为节点缩容) | 2026-04-20 | | 11 | O11Y-2426677 | dedicated | 4.0 | 根因对但路径不优 | 2026-04-20 | | 12 | O11Y-2328599-ws2 | dedicated | — | 第二轮 postmortem | 2026-04-20 | 场景覆盖：expensive-query, panic, latency-spike, CPU-spike, OOM, lock-contention, worker-failure, rolling-restart, plan-cache, known-bug --- ## 附录 C: 远程部署方案 — Evolver ### C.1 设计目标 将 Replayer 从本地手动执行迁移到 K8s 自动执行，pod 命名为 **Evolver**（强调诊断能力的持续进化）： 1. **每日自动 postmortem**：定时发现新 issue、生成 postmortem、归档问题 2. **零人工触发**：不需要人登录跑命令，结果通过 GitHub issue + PR 交付 3. **可观测**：日志、执行结果、commit 历史都可追溯 4. **可交互**：人工在 GitHub issue 中 @evolver-bot 对 Claude 的分析进行修正 ### C.2 架构 ``` K8s investibook namespace ├── backend (Deployment, 常驻) ← 提供 workspace/messages API ├── frontend (Deployment, 常驻) ← Web UI └── evolver (Deployment, 常驻) ← 新增 │ │ 常驻 Python 进程 │ ├── HTTP server (:8080) ← GitHub webhook receiver │ └── 内部 cron loop ← 每天 02:00 UTC 触发 │ ├── POST /evolver/webhook/github ← GitHub issue_comment event │ └── @evolver-bot 触发 Claude 交互 │ └── 定时 02:00 UTC └── claude -p "Run replayer Phase 0-8" ├── Phase 0: GitHub API → discover issues ├── Phase 1: Auth (in-cluster backend) ├── Phase 2-6: Postmortem + evaluate ├── Phase 7: Triage → issues.yaml └── Phase 8: File GitHub issues ``` 外部连接： ``` GitHub.com ──webhook──→ ALB ──/evolver/*──→ evolver:8080 │ ┌───────────────────────────┘ │ HTTP (in-cluster) │ HTTPS v v backend:8000 github.com/tidbcloud/investibook (workspace API) (issues + git push) ``` ### C.3 组件清单 | 组件 | 文件 | 说明 | |------|------|------| | Dockerfile | `deployment/evolver/Dockerfile` | 基于 backend-base + Node.js 20 + Claude CLI | | Webhook Server | `deployment/evolver/webhook_server.py` | HTTP server + 内部 cron + GitHub 交互处理 | | Entrypoint | `deployment/evolver/entrypoint.sh` | 启动 webhook_server.py | | Deployment | `deployment/kustomize/envs/dev/evolver-deployment.yaml` | 常驻 1 replica | | Service | `deployment/kustomize/envs/dev/evolver-service.yaml` | ClusterIP :8080 | | Secret | `deployment/kustomize/envs/dev/evolver-external-secret.yml` | Investibook API Token | ### C.4 镜像构建 ```dockerfile # deployment/evolver/Dockerfile FROM investibook-backend-base:latest # ubuntu:24.04 + git, python, curl, jq, gh, uv, AWS CLI + Node.js 20 # Claude CLI 依赖 + @anthropic-ai/claude-code # Claude CLI + COPY repo from CI checkout # 构建时复制，运行时 git pull 更新 + webhook_server.py # HTTP server ``` ### C.5 常驻进程设计 Evolver 以 Deployment（1 replica）运行，内部包含： **1. HTTP webhook server（:8080）** - `GET /healthz` — K8s 健康检查 - `POST /evolver/webhook/github` — 接收 GitHub webhook **2. 内部 cron loop** 每分钟检查当前 UTC 时间，在 `EVOLVER_CRON_HOUR_UTC`（默认 2）时触发 daily postmortem。 **3. 并发控制** 使用 threading Lock 保证同一时刻只有一个 Claude session 运行。如果 daily cron 正在跑，webhook 触发的交互会返回 "busy" 消息。 关键设计决策： - **单 replica + Lock**：Claude session 操作 issues.yaml 和 GitHub issues，并发会导致冲突 - **1h 超时**：每个 Claude session 最长 3600s - **Recreate strategy**：更新镜像时先杀旧 pod 再起新 pod，避免双跑 ### C.6 凭据管理 与之前一致。Claude CLI 通过 **Bedrock bearer token** 认证，不需要 Anthropic API key。 | 凭据 | 来源 | 用途 | |------|------|------| | `AWS_BEARER_TOKEN_BEDROCK` | agent-provider-secret (复用 backend 同一 secret) | Bedrock Claude CLI 认证 | | `INVESTIBOOK_API_TOKEN` | evolver-secret (从 `investibook-dev/config`) | 调用 backend workspace/messages API | | `SKILL_GITHUB_TOKEN` | investibook-config-secret (复用) | GitHub API + git push | | `EVOLVER_WEBHOOK_SECRET` | evolver-secret (可选) | 验证 GitHub webhook 签名 | ### C.7 @ 交互流程 ``` 用户在 GitHub issue comment 中写: "@evolver-bot 这个诊断有误，应该是 OOM 导致的" → GitHub 发送 issue_comment webhook → ALB → evolver pod /evolver/webhook/github → webhook_server.py 解析 payload → 检查 body 是否包含 "@evolver-bot" → 获取 session lock（如果 busy 则回复等待） → git pull latest main → 收集 issue body + comment history → claude -p "Address user feedback: ..." → 如有文件修改，commit + push 到 evolver/issue-{N} 分支 → 将 Claude 输出作为 comment 回复到 issue ``` 配置 GitHub webhook（需手动在 repo settings 中添加）： - **Payload URL**: `https://dev.investibook.com/evolver/webhook/github` - **Content type**: `application/json` - **Secret**: （可选，对应 `EVOLVER_WEBHOOK_SECRET`） - **Events**: 勾选 `Issue comments` ### C.8 数据流 ``` ┌─── backend:8000 (HTTP) ───┐ │ │ GitHub Issues ──(API)─→ evolver pod ──(API)──→ Workspace messages ^ │ │ │ ┌─────────────────┤ │ │ │ webhook │ claude -p (Bedrock) │ │ │ (issue_comment) │ │ │ v v │ │ @evolver-bot Postmortem + Triage │ │ interaction │ │ │ │ │ │ │ │ reply comment │ Phase 7-8 │ │ └────────────────→│ (issues + yaml) │ │ │ │ └─────────────────────┘ │ │ │ │ git push │ v │ evolver/* branches │ (issues.yaml, postmortem.md, etc.) │ ``` ### C.9 监控与告警 - **健康检查**：K8s liveness/readiness probe 通过 `/healthz` - **状态查询**：`GET /healthz` 返回 `session_busy` 字段 - 建议后续添加： - 每日 postmortem 结果汇总通知（Lark/Slack webhook） - Claude session 失败时自动通知