schema_version: 1 case_id: tikv-node-stall-during-incident-window ground_truth: root_cause: A RocksDB default-CF compaction burst on one TiKV instance, triggered around Raft leadership or SST ingestion, saturated the read/unified thread pool and caused a short SQL latency stall with connection-kill fallout judge: semantic_strategy: undecided contract: checks: - id: time_window_match required: true description: Actual diagnosis time window matches one allowed case window. match: all_of rules: - source: trace_fact matcher: bool_true path: time_window_match - id: cluster_scope_match required: true description: Actual diagnosis cluster scope matches expected cluster id. match: all_of rules: - source: trace_fact matcher: bool_true path: cluster_scope_match workflow: checks: - id: architecture_context_loaded weight: 5 required: true description: Architecture context is loaded. match: all_of rules: - source: workflow_fact matcher: bool_true path: architecture_context_loaded - id: misdiagnosis_guardrails_loaded weight: 5 required: true description: Misdiagnosis guardrails are loaded. match: all_of rules: - source: workflow_fact matcher: bool_true path: misdiagnosis_guardrails_loaded - id: diagnosis_run_initialized weight: 5 required: true description: Diagnosis run is initialized. match: all_of rules: - source: workflow_fact matcher: bool_true path: diagnosis_run_initialized - id: evidence_collection_completed weight: 10 required: true description: Evidence collection completes. match: all_of rules: - source: workflow_fact matcher: bool_true path: evidence_collection_completed - id: summary_read_completed weight: 5 required: false description: Summary artifact is read when available. match: all_of rules: - source: workflow_fact matcher: bool_true path: summary_read_completed - id: final_answer_emitted weight: 0 required: false description: Final answer is emitted. match: all_of rules: - source: workflow_fact matcher: bool_true path: final_answer_emitted execution_contract: checks: - id: required_workflow_present required: true description: Required workflow milestones are present. match: all_of rules: - source: workflow_check matcher: subset_observed paths: - architecture_context_loaded - misdiagnosis_guardrails_loaded - diagnosis_run_initialized - evidence_collection_completed - id: required_artifacts_present required: true description: Required evidence artifacts are present. match: all_of rules: - source: artifact_fact matcher: subset_observed paths: - core_evidence_bundle_present - id: preferred_channels_minimum_observed required: true description: At least one preferred evidence channel is observed. match: all_of rules: - source: channel_fact matcher: count_gte paths: - query_or_metric threshold: 1 - id: preferred_channels_all_observed required: false description: Preferred evidence channels are all observed. match: all_of rules: - source: channel_fact matcher: subset_observed paths: - query_or_metric evidence_collection: weight: 5 checks: - id: required_artifacts_present weight: 4 required: true description: Required evidence artifacts are present and readable. match: all_of rules: - source: artifact_fact matcher: subset_observed paths: - core_evidence_bundle_present - id: preferred_channels_minimum_observed weight: 0 required: true description: At least the minimum number of preferred evidence channels are observed. match: all_of rules: - source: channel_fact matcher: count_gte paths: - query_or_metric threshold: 1 - id: preferred_channels_all_observed weight: 6 required: false description: Preferred evidence channels are all observed. match: all_of rules: - source: channel_fact matcher: subset_observed paths: - query_or_metric critical_evidence: weight: 30 checks: - id: node_local_stall_signal_present weight: 8 description: TiKV node-local stall or queueing evidence is present for the incident window. match: any_of rules: - source: artifact_json artifacts: - queries matcher: json_text_contains_any keywords: - read_pool.rs - read pool - estimated_wait_ms - request rejected - queue overflow - source: artifact_json artifacts: - metrics matcher: json_text_contains_any keywords: - read_pool.rs - unified read-pool - read pool - estimated_wait_ms - pending tasks - scheduler worker - scheduler command duration - grpc - coprocessor - source: artifact_json artifacts: - summary_json matcher: json_text_contains_any keywords: - node-level freeze - read pool - queue overflow - request rejected - single TiKV - id: single_tikv_scope_evidence_present weight: 8 description: Evidence isolates the stall or hotspot to a single TiKV instance. match: any_of rules: - source: artifact_json artifacts: - queries matcher: json_text_contains_any keywords: - db-tikv-19 - source: artifact_json artifacts: - metrics matcher: json_text_contains_any keywords: - db-tikv-19 - one TiKV instance - single TiKV - hotspot TiKV - source: artifact_json artifacts: - summary_json matcher: json_text_contains_any keywords: - db-tikv-19 - single TiKV instance - hotspot instance - one TiKV hotspot - id: compaction_trigger_evidence_present weight: 9 description: Evidence identifies the compaction or ingestion trigger instead of stopping at generic slow SQL. match: any_of rules: - source: artifact_json artifacts: - queries matcher: json_text_contains_any keywords: - LevelMaxLevelSize - compaction - RocksDB - IngestExternalFile - SST - MsgTimeoutNow - Raft leader - source: artifact_json artifacts: - metrics matcher: json_text_contains_any keywords: - LevelMaxLevelSize - compaction - RocksDB - IngestExternalFile - SST - MsgTimeoutNow - Raft leader - source: artifact_json artifacts: - summary_json matcher: json_text_contains_any keywords: - LevelMaxLevelSize - compaction burst - SST ingestion - Raft leader transfer - snapshot ingestion evidence_routing: weight: 5 checks: - id: analysis_patch_valid_json weight: 1 required: true description: Analysis patch is valid JSON whenever the patch exists. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_patch_valid_json - id: analysis_items_shape_valid weight: 2 required: true description: Analysis patch items are structurally valid whenever the patch exists. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_items_shape_valid - id: analysis_key_evidence_present weight: 2 required: true description: Analysis patch includes key evidence whenever the patch exists. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_has_key_evidence - id: metric_refs_resolvable weight: 1 required: true description: Metric references resolve whenever the patch requires them. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_metric_refs_resolvable - id: query_refs_resolvable weight: 1 required: true description: Query references resolve whenever the patch requires them. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_query_refs_resolvable - id: supporting_evidence_present weight: 0 required: false description: Analysis patch includes supporting evidence whenever the patch exists. match: any_of rules: - source: routing_fact matcher: bool_false path: analysis_patch_present - source: routing_fact matcher: bool_true path: analysis_has_supporting_evidence - id: analysis_log_or_event_channel_resolvable weight: 0 required: false description: Routed log or event items stay resolvable against observed channels. match: all_of rules: - source: routing_fact matcher: bool_true path: analysis_log_or_event_channel_resolvable diagnosis: checks: - id: identify_compaction_read_pool_stall_as_primary_path mode: blended weight: 3 required: true description: Identifies a RocksDB compaction-driven TiKV read/unified-pool stall as the main incident path. deterministic: match: all_of rules: - source: output_text matcher: keyword_any keywords: - compaction - RocksDB - LevelMaxLevelSize - SST ingestion - snapshot ingestion - Raft leader - raft leader transfer - 压缩 - RocksDB - SST - Raft leader - source: output_text matcher: keyword_any keywords: - read pool - unified read-pool - thread pool - queue - rejected - request rejection - queue overflow - 读线程池 - 线程池 - 队列 - 拒绝 semantic: id: 1 title: 主路径判断 description: | 判断回答是否把主路径识别为 TiKV compaction/read-pool stall，而不是 generic slow SQL、 TiDB CPU 或应用侧问题。 - id: identify_single_tikv_stall_scope mode: blended weight: 3 required: false description: Identifies that the stall was concentrated on one TiKV instance rather than cluster-wide. deterministic: match: any_of rules: - source: output_text matcher: keyword_any keywords: - db-tikv-19 - single TiKV instance - hotspot TiKV - one TiKV hotspot - 单个 TiKV 实例 - 热点 TiKV semantic: id: 2 title: 热点范围识别 description: | 判断回答是否识别 stall/hotspot 集中在单个 TiKV 实例，并将 compaction、read-pool queueing 或 raftstore/scheduler stall 机制与该范围对应起来。 - id: identify_specific_hotspot_node_scope mode: deterministic_only weight: 3 required: true description: Names the specific hotspot TiKV node in the final diagnosis. deterministic: match: any_of rules: - source: output_text matcher: keyword_any keywords: - db-tikv-19 - tikv-19 - id: recommend_tikv_stall_mitigation mode: blended weight: 3 required: false description: Recommends mitigation targeted at the TiKV compaction or stall path. deterministic: match: any_of rules: - source: output_text matcher: keyword_any keywords: - rebalance Raft leaders - rebalance leaders - reduce batch DELETE - reduce batch delete - RocksDB - compaction - Titan BlobDB GC - TopSQL - profiling - leader rebalancing - leader 调度 - 调整 leader - 降低批量 DELETE - 减少 tombstone semantic: id: 4 title: 修复建议对齐 description: | 判断回答是否给出 TiKV stall/compaction path 的缓解建议，例如 leader rebalancing、 降低 batch delete/tombstone 压力、RocksDB/Titan 调优、TopSQL 或 profiling follow-up。