# Time And Entity Causal Alignment

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "time-entity-causal-alignment-2026-07-24",
  "problems": [
    {
      "problemId": "op-diagnosis-cross-time-or-entity-causal-attribution",
      "summary": "Auto-oncall can attribute an incident to a metric or workload from another instance or timestamp by combining window aggregates instead of validating the user-visible result and candidate mechanism on aligned samples.",
      "status": "optimized_unverified",
      "issueKeys": ["APID-29220", "APID-29231"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29220#problem:ao_feedback:unaligned-network-causality",
        "https://tidb.atlassian.net/browse/APID-29231#problem:ao_feedback:cross-instance-window-maxima"
      ],
      "signalTextIncludesAny": [
        "leader 变化直接归因为迁移",
        "未用网络指标证明因果",
        "选择了错误实例",
        "拼接不同时间点的峰值",
        "topsql 累计排名推断因果"
      ],
      "observedThrough": "2026-07-24",
      "effectiveAt": "2026-07-27T12:41:29Z",
      "verifiedAt": null,
      "attemptedAction": "Require OP latency diagnosis to locate the user-visible result and affected entity first, compare mechanisms on the same entity and sample time, treat window aggregates only as discovery evidence, and keep conclusions provisional when bounded per-point per-instance evidence is unavailable.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/pull/498"
    }
  ]
}
-->

- Optimization ID: `time-entity-causal-alignment-2026-07-24`
- Owner: auto-oncall-services
- Source references: `APID-29220`, `APID-29231`
- Implementation status: merged through `tidbcloud/aisupport#498` and available
  in the staging runtime by the `2026-07-27` Focus baseline
- Effective status: `optimized_unverified`; deployed replay still exposes an
  `APID-29220` causal-gate regression, so the shared problem is not verified

## Bounded problem

The two cases expose one reusable diagnostic error: an aggregate or top-ranked
signal can be selected from the wrong instance or from a different point in the
incident window, then presented as if it occurred with the user-visible result.
This can turn correlation into a false root cause.

The bounded change requires diagnosis to start with the user-visible result,
identify the actually affected entity and incident samples, and test each
candidate mechanism on the same entity and sample time. Window maxima and
cumulative workload ranks remain useful for discovery, but cannot prove
causality.

## Implementation evidence

- `skills/op-diagnosis/SKILL.md` now requires load-bearing evidence tuples with
  entity labels, sample timestamps, query step or interval, and values.
- `skills/tidb-latency-diagnosis/SKILL.md` applies the same alignment rule to
  latency branches.
- Contract tests cover cross-instance and cross-time maximum attribution,
  cumulative TopSQL or `ANALYZE` attribution, and provisional conclusions when
  bounded per-point evidence is unavailable.

## Branch-local historical replay

Validation ran at `2026-07-24T03:28:52Z` from implementation commit
`4d55558943f874b19a1e36e799df483d8e63a467`. An isolated `CODEX_HOME`
installed the bundled skills from this worktree. The installed and source
SHA-256 values matched:

- `op-diagnosis`: `f66f1cde522edec11b3a1acfdcc4207767f52d3e8d54ac156a3e95d02c449802`
- `tidb-latency-diagnosis`:
  `665dc7c55c309b1c9402554d40dac7176b5ee5c5ff8ffbf6376e9b06ff465490`

The normalized replay command was:

```text
CODEX_HOME=<isolated-codex-home> \
AUTO_ONCALL_RUNS_ROOT=<validation-root>/runs \
AUTO_ONCALL_CODEX_MODEL=gpt-5.6-sol \
node <validation-root>/run-historical-replay.mjs \
<issue-key> <source-job-id> <source-run-db-id> <first-run-cutoff>
```

No Jira, database, deployment, or provider configuration writes were enabled.
The final `APID-29220` attempt received only the Manus Grafana credentials in
its process environment for a read-only metrics query; credential values were
not logged. No Manus Dashboard credentials were supplied because that
transport's log search creates temporary provider task state.

### APID-29220

- Run ID: `208c64fd-4809-478c-8fef-04c9a941228d`
- Source: job `60171`, run DB id `60089`
- Cutoff: `2026-07-22T12:11:24.286Z`
  (`first_run_created_at`)
- Workspace:
  `/tmp/auto-oncall-causal-alignment-validation.JkhwiT/runs/208c64fd-4809-478c-8fef-04c9a941228d`
- Core output: `output/replay-context.md`
- Execution trace: `trace/context-trace.json`
- Input boundary: `input/input-manifest.json`

| Criterion | Result | Evidence |
|---|---|---|
| Start from the user-visible result and affected entity | PASS with provider limitation | The replay started from `tidb_tikvclient_backoff_seconds_count` on `172.33.173.113:10080`; provider access did not independently confirm the entity. |
| Do not combine cross-time or cross-instance maxima | PASS | No aggregate maxima were joined into a cause. The output stopped before ranking a mechanism. |
| Do not infer cause from cumulative TopSQL or `ANALYZE` rank | PASS | Neither workload ranking nor `ANALYZE` was used for attribution. |
| Keep the result provisional when aligned points are unavailable | PASS | The RCA says the root cause is unconfirmed and names the missing per-type time series, server counterpart, normal baseline, and logs. |

The branch behavior passed the bounded guard, but the case remains
evidence-inconclusive. The exact Manus `vida` Grafana query and health check
returned HTTP 403 from this host, so the replay could not confirm the actual
historical network or Region mechanism. This is not case-specific proof that
the root cause is corrected.

At `2026-07-24T04:06:27Z`, a current transport-layer comparison isolated the
403 boundary without another replay. The same bundled Manus skill and `vida`
target returned a fully healthy Dashboard, authenticated user, Grafana, and
Prometheus datasource from the running staging pod
`auto-oncall-services-5fd66bc499-h7f6r`. From this worktree host, the same
unauthenticated Dashboard info endpoint returned APISIX HTTP 403 with
`{"message":"Your IP address is not allowed"}` before credential use. The
endpoint, credentials, and pod transport are therefore healthy; the remaining
blocker is the external source-IP allowlist for the worktree-host egress.

#### Isolated staging-egress replay

With explicit authorization, one isolated temporary Pod reused the current
staging image and allowed staging egress without changing the active
Deployment:

- Pod:
  `auto-oncall-apid-29220-replay-20260724-051030`
- Pod UID: `65d62d50-0202-4ba1-a31c-2f452cb2af31`
- Image:
  `552185537300.dkr.ecr.us-west-2.amazonaws.com/aisupport/staging/auto-oncall-services:c64cddad242b5d443d1c19273f3bd19142983204`
- Image digest:
  `sha256:abbe2079da535a2e810680278d66d1a8db019675dad1ad406b70d2c82cb4a290`
- Sanitized manifest:
  `docs/opts/evidence/2026-07-24-apid-29220-staging-replay-pod.yaml`

The Pod had no Service or consumer. It did not receive database, Lark, GitHub,
or Clinic credentials. Its rendered guard values were
`AUTO_ONCALL_AUTOMATION_ROLE=disabled`,
`AUTO_ONCALL_POLL_ENABLED=false`, continuation/report/Base/notification/Jira
write/customer-reply flags all `false`, and no database URL. Manus `vida`
health passed before the replay.

The current branch skills were copied into an isolated `CODEX_HOME`. Source and
Pod SHA-256 values matched:

- `op-diagnosis/SKILL.md`:
  `1041912e285d3abefc0e8bb122b1195852df636c560d9a5382ceb2e43c2f0e63`
- `tidb-latency-diagnosis/SKILL.md`:
  `665dc7c55c309b1c9402554d40dac7176b5ee5c5ff8ffbf6376e9b06ff465490`
- `tidb-latency-diagnosis/references/storage-latency.md`:
  `4a410fc06921ec05b25fd2457db388f22d7d53ad55cc8b92fa8a1f569e6157c1`

Exactly one historical replay ran:

- Run ID: `db6a4a4d-ecc8-4478-9755-c6059eebd1e0`
- Source: job `60171`, run DB id `60089`
- Cutoff: `2026-07-22T12:11:24.286Z`
- Started: `2026-07-24T05:13:49.994Z`
- Completed: `2026-07-24T05:16:52Z`
- Pod workspace:
  `/tmp/apid-29220-runs/db6a4a4d-ecc8-4478-9755-c6059eebd1e0`
- Preserved local workspace:
  `/tmp/apid-29220-staging-replay-20260724-051030/workspace`
- Core output: `output/replay-context.md`
- Execution trace: `trace/context-trace.json`
- Decision: `human_required`

The aligned Manus metrics established a bounded alert mechanism on TiDB
`172.33.173.113:10080`. Across the exact 19:57–19:59 UTC+8 alert interval,
aggregate backoff rate was `57.60`–`192.42` events/s and region-error rate was
`68.51`–`263.27` errors/s across the same five 30-second samples. The replay
therefore limited the supported chain to region errors causing retries and
client backoff. It did not claim a region-error subtype, server-side TiKV/PD
event, or workload as the deeper cause.

| Criterion | Result | Evidence |
|---|---|---|
| Start from the user-visible result and affected entity | PASS | The replay started from `tidb_tikvclient_backoff_seconds_count` on the alerted TiDB `172.33.173.113:10080`, then queried that exact instance. |
| Do not combine cross-time or cross-instance maxima | PASS | Backoff and region-error rates were compared across the same five 30-second samples in the exact two-minute alert interval. The wider envelope was used only to establish recovery. |
| Do not infer cause from cumulative TopSQL or `ANALYZE` rank | PASS | No TopSQL, Slow Query, SQL Digest, or `ANALYZE` attribution was used. |
| Keep unsupported causal depth provisional | PASS | The output confirms only region error to retry/backoff and leaves subtype, TiKV/PD cause, and workload trigger unverified. |

The intended replay boundary was read-only, but the Agent invoked the bundled
Manus `logs` command once. That command performs a Dashboard
`PUT /api/logs/taskgroup`; it returned task group id `2` and zero matching
records, and the bundled client does not delete the task group. This provider
task-state write exceeded the intended pure read-only boundary. No second
replay, provider cleanup attempt, Jira write, database write, Feishu message,
configuration change, or customer-workload operation was performed. The
aligned metrics remain valid causal evidence, but this run is not proof that
the provider transport is side-effect-free.

The temporary Pod was deleted immediately after preserving the workspace.
`kubectl get` confirmed no replay-labeled Pod remained. Before and after the
replay, the active Deployment stayed at generation `208`, the active Pod UID
remained `8be889b1-cc31-4116-8f57-26403396554b`, its image digest and
`role=active` / `poll=true` values were unchanged, and queued/running job
counts remained zero.

### APID-29231

- Run ID: `4bb91454-d1ed-458e-a70c-6d8a2584364d`
- Source: job `90007`, run DB id `90003`
- Cutoff: `2026-07-23T19:46:48.509Z`
  (`first_run_created_at`)
- Workspace:
  `/tmp/auto-oncall-causal-alignment-validation.JkhwiT/runs/4bb91454-d1ed-458e-a70c-6d8a2584364d`
- Core output: `output/replay-context.md`
- Execution trace: `trace/context-trace.json`
- Input boundary: `input/input-manifest.json`
- Point evidence: `evidences/client-commit/metrics.json`,
  `evidences/grpc-commit/metrics.json`, and
  `evidences/tikv-cpu/metrics.json`
- Narrow workload evidence:
  `evidences/topsql-tikv-3-2-status/dashboard.json`

| Criterion | Result | Evidence |
|---|---|---|
| Start from the user-visible result and affected entity | PASS | The replay first located TiDB Commit/Prewrite p99, then the actually slow TiKV `10.107.3.2:20180`. |
| Do not combine cross-time or cross-instance maxima | PASS | TiKV `kv_prewrite`/`kv_commit` p99 and process CPU were compared on `10.107.3.2:20180` at overlapping 15-second samples around 03:26:30–03:28:30 UTC+8. |
| Do not infer cause from cumulative TopSQL or `ANALYZE` rank | PASS | The replay explicitly rejected the 30-minute screenshot ranking as proof. The instance-scoped 03:23–03:33 TopSQL check did not contain `ANALYZE` in its top five and was retained only as bounded workload context. |
| Keep the result provisional when aligned points are unavailable | PASS | The output limits the supported chain to a busy Commit/Prewrite path and leaves the deeper workload trigger unconfirmed. Missing logs are marked unavailable rather than healthy. |

The replay removed the original `ANALYZE` / unified-read attribution and
identified the correct affected store and transaction path. It did not query
the scheduler latch wait series, so it did not reach the known, more specific
write-side scheduler latch contention mechanism. The alignment guard therefore
passes. The distinct scheduler-latch omission was subsequently addressed and
branch-replayed under optimization record
`write-path-scheduler-latch-evidence-2026-07-24`; it remains separate from this
problem's status.

The agent attempted the externally installed transport's legacy
`browser-query-logs` command even though the current Wetech skill contract says
component logs are disabled. The command failed before a log query because no
matching Loki datasource exists; no provider write was performed. This
transport inconsistency is outside this repository change and remains a
follow-up.

## Unresolved gaps

- The change is deployed to staging, but the shared problem remains unverified
  because `APID-29220` still promoted an externally supplied network
  conclusion without completing current-run fault-domain evidence.
- Branch-local historical-cutoff replay has run for both source cases, but
  neither result is deployment proof.
- `APID-29220` now has aligned branch evidence through an isolated staging
  egress Pod. Its direct region-error to retry/backoff mechanism is supported,
  while the underlying subtype and server-side cause remain unresolved.
- The current Manus log command creates provider task-group state and does not
  delete it. It must not be used by a future read-only replay until the
  transport is made side-effect-free or that write is explicitly approved.
- `APID-29231` received a branch-local scheduler latch check under a separate
  bounded optimization record; that narrower problem has its own deployment
  and verification status.
- The repository does not change the external Wetech transport. If a transport
  cannot provide bounded per-point, per-instance data, the diagnosis must state
  that gap and remain provisional.
- The existing Jira internal comments have not been refreshed or verified.

## Next validation action

Deploy the current transient fault-domain reinforcement through the normal
rollout path. Then rerun `APID-29220` and `APID-29231` with their historical
cutoffs. Verify that the
draft first locates the affected result and entity, does not combine
different-time maxima or cumulative workload ranks, and keeps unsupported
causal claims provisional. For `APID-29231`, preserve the write-side scheduler
latch series in the bounded aligned comparison. For `APID-29220`, keep provider
log task-group creation disabled unless its write boundary is explicitly
approved. If each deployed replay passes, update its existing stable internal
comment in place, verify by comment ID or idempotency key, and promote this
problem record only with that case-specific evidence.