# Control-plane Causal Chain - Optimization ID: `control-plane-causal-chain-2026-07-27` - Owner: auto-oncall-services - Source reference: `APID-29223` - Implementation status: deployed to staging - Effective status: `optimized_verified` ## Bounded problem Control-plane diagnosis can correctly observe a short PD/etcd service-path disturbance yet overstate the causal chain. Window maxima may not include the maximum's timestamp, a generic TiKV-client backoff does not identify `pdRPC`, and current `up` or uptime establishes recovery or process continuity rather than the historical trigger. Without a TiDB/PD counterpart and a same-window storage or infrastructure alternative, the supported conclusion is correlation around the service path, not a confirmed PD/TSO or network root cause. The baseline read-only historical replay `855a5c84-7d09-4de6-bcb0-93e0a474644a` finished successfully but remained `PARTIAL`. It used a twenty-minute window whose compact results contained per-series maxima but not their sample timestamps. It found elevated PD peer RTT, short scrape loss, continuous PD process uptime, and customer-supplied backoff and Duration evidence. It did not align TiDB TSO or exact `pdRPC` with the PD server counterpart, check PD leader/term, or test the same-window etcd WAL/backend/disk alternative. ## Candidate change - Start from one direct alert/result tuple: artifact, affected entity, timestamp or interval, timezone, and sample period. - Use a 2-5 minute causal window around that anchor. Treat wider ranges as discovery only. - Call a maximum without its timestamp `window-aligned`; do not claim exact simultaneity, order, or causality. - Require TiDB TSO client versus PD handler/server, or exact TiDB `pdRPC` versus PD handler/server, before confirming a PD/TSO mechanism. - Check PD leader/term and process continuity separately from scrape state. - Compare the strongest same-window etcd WAL/backend/disk or direct host/network alternative before naming the deeper trigger. - Keep cross-ticket or cross-cluster matches out of direct and Key Evidence. - If one outcome-changing signal remains available through the current read-only provider, return it as `diagnosticFollowup`; otherwise preserve the exact bounded gap and human handoff. - Give every provider query a unique `evidence/scenarios///` output directory. ## Local implementation evidence The local skill and contract tests cover the narrow anchor window, `window-aligned` wording, the required control-plane counterpart, leadership and process continuity, the strongest alternative, recovery boundaries, cross-ticket exclusion, load-bearing follow-up, and query-specific evidence directories. ## Branch-local forward replay The first isolated post-change replay `3444caa4-c5b3-4003-8050-25f232e792fd` completed in 481.87 seconds. It found exact `pdRPC` backoff, a TSO client/server counterpart, continuous PD process uptime, and point-level etcd storage latency, but remained `PARTIAL`: the agent did not test leader/term availability, transport readiness was not stored under its scenario-specific output directory, and the final wording did not consistently distinguish window maxima from timestamped samples. One bounded generic correction was then made. It requires scenario-specific directories for readiness queries, requires a leader/term/campaign-equivalent test when Prometheus is available, and requires both the internal finding and customer draft to use literal `window-aligned` wording instead of causal or simultaneous language when a load-bearing maximum has no sample timestamp. The second isolated replay `4dbcb13c-42e5-4b33-9cc7-2b8ce267ac95` completed in 478.03 seconds. It issued ten provider metric queries into ten distinct scenario/query directories and found: - exact `pdRPC` backoff on all three TiDB instances and `tso-failed` client P99 up to 2.048 seconds; - PD/etcd leader changes, `has_leader=0` samples on two PD instances, no process-start change, and bounded scrape gaps; - etcd backend commit and WAL fsync P99 window maxima of about 3.95 and 3.83 seconds; - no TiDB scrape loss, connection loss, or increased execute-error rate, and recovery of the Duration spike by the end of the window. The internal replay context correctly labels the load-bearing maxima `window-aligned`, keeps the chain `suspected`, rejects PD process restart as the root cause, and preserves the unresolved host/disk/network trigger. The strict result is nevertheless still `PARTIAL`: the customer draft says the PD/etcd disturbance `导致` the client retries and presents the maxima without the required `window-aligned` boundary. The compact TSO counterpart summary also retained only the highest client-side series, so the checked server-side series is not visible in the final evidence summary. The staging runtime started the deployed image at `2026-07-27T09:52:53.000Z`. Deployed read replay `60c79118-5c05-4d94-a214-4749535a4001` completed successfully with `ai_better`. It used exact `pdRPC` and TSO evidence, checked leader and process continuity, preserved storage and network alternatives, and explicitly left the lower-level trigger unresolved instead of promoting an unsupported root cause. Deployed write replay `5c384f21-3d4a-44aa-a20b-d4305020dd45` updated the existing Jira internal comment `23266042` in place. Jira readback verified the exact comment ID and updated body. The result remained `human_required`, which is compatible with this diagnostic criterion: the causal boundary was correct and no wrong root cause was asserted. ## Unresolved gaps - The behavior is prompt/skill-enforced; the runtime does not validate causal tuple completeness or reject unsupported causal verbs. - The compact provider summary can rank client-side maxima above the matching server-side counterpart, leaving that checked counterpart absent from the agent-visible top-ten evidence. - The external provider still controls which per-point fields and control-plane metrics are available. This candidate does not modify that provider. ## Next validation action Treat deterministic causal-verb validation and compact-summary counterpart retention as separate hardening problems if they reproduce. Monitor later matching signals as regressions without suppressing unrelated gaps on `APID-29223`.