# Wetech Pzero Application Session Completion

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "wetech-pzero-application-session-2026-09-02",
  "problems": [
    {
      "problemId": "managed-wetech-pzero-application-session-incomplete",
      "summary": "Managed Wetech completed P81 identity authentication but did not complete the Pzero application connection, so Grafana readiness requests carried no Pzero application cookie and redirected back to /connect.",
      "status": "candidate",
      "applicability": "Agent9-backed Wetech OP Grafana evidence acquisition that must establish a Pzero application session after P81 authentication.",
      "issueKeys": ["APID-29437"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/APID-29437#problem:managed-wetech-pzero-application-session-incomplete"],
      "signalTextIncludesAny": [
        "pzero_application_session_not_ready",
        "grafana_session_not_ready"
      ],
      "observedThrough": "2026-09-03",
      "effectiveAt": null,
      "verifiedAt": null,
      "attemptedAction": "Record the allowlisted native browser identity and applications/connect request identity from both Playwright contexts so the unchanged Cloudflare Turnstile INCONSISTENT_USER_AGENT failure can be attributed to an exact remaining surface before changing execution behavior. Do not add a retry, change credentials, or broaden the Provider query.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29437"
    }
  ]
}
-->

- Owner: Wetech Managed Tool Provider Executor
- Source: APID-29437 v15, v22, v27, and v28 read replays, exact
  E2B-versus-local cookie comparison, and the internal-prod v16 through v28
  canaries
- Implementation status: Wetech Managed Tool v31 bounded browser-identity
  diagnostic candidate passed local validation and awaits internal-production canary
- Effective status: `candidate`

## Bounded problem

The v15 E2B browser completed P81 username, password, and MFA submission, but
its persisted state contained only identity-provider cookies. A direct request
to the selected Pzero Grafana `/api/user` endpoint therefore carried no
applicable application cookie and returned HTTP 307 to `/connect`. The same
request headers did not change that result. A working local browser request
contained both the Pzero application session and Grafana session cookies.

The first v16 canary reached `/connect` with a valid persisted P81 identity,
but the page exposed only `Close`. Because v16 retried the original Grafana URL
only after submitting P81 credentials or MFA in the same process, it neither
dismissed that bounded notice nor retried the application entry. The readiness
probe stayed at HTTP 307 and no Grafana query was sent.

The v17 canary added that bounded notice dismissal and unconditional one-time
resume, but the Provider Executor was killed by Agent9's 120-second outer Tool
timeout. E2B recorded a 121-second process lifetime, signal kill, empty stdout
and stderr, and `context canceled`. The retry navigation still had its own
fixed 60-second timeout and the login deadline started only after the initial
navigation, so the Tool could cross the outer timeout before it wrote its safe
network diagnostic or structured Provider result.

The v18 canary used package digest
`5a18d23a265d7e65b76f1428a04046394cce6ef577b4205198340d133886790f`.
Its one Managed Tool operation completed in 101.012 seconds instead of being
killed by the outer timeout, but the Provider result was still
`pzero_application_session_not_ready` with `requestCount=0`. The safe sandbox
diagnostic remained at the P81 `/connect` page with only `Close`, and the
recorded actions were ordered `pzero_application_resume_attempted` followed by
`pzero_application_notice_closed`. The single resume was therefore consumed
before the dynamically rendered notice could be dismissed; after dismissing
the notice, no application-entry retry remained.

The v19 canary used package digest
`2198e97fd2c4a1a9712f3fdca2cdf68654bc460ab368a28551f467cdc09ac1d6`
in session `sess_2ee8870e629e4549aa98ac383f25d9b4`, turn
`turn_9fb431bb4b8d4f2c980fdfcf460c001b`, and Managed Tool operation
`op_354ee4a30a3a47f381ae0c3aaeb6d62a`. The Provider operation completed in
103.196 seconds, but its result remained
`pzero_application_session_not_ready` with `requestCount=0`. Its safe action
order proved the intended timing change ran: `pzero_application_render_wait_started`,
`pzero_application_notice_closed`, then `pzero_application_resume_attempted`.
The cookie probe still returned HTTP 307 from `/api/user` to `/connect`.

The v20 canary used package digest
`a7b80dd9132e77e159a51e552d91073a585ce2201d10d2c2a6b423fcdfdab369`
in session `sess_07d398f1c48247d788ad12a9abd09ff7`, turn
`turn_28eb55163a9f463bb7e4c51a5707c065`, and Managed Tool operation
`op_e43cb38f273c42caa2a30da8d5cd4aa3`. The Provider operation completed in
100.500 seconds and returned `pzero_application_session_not_ready` with
`requestCount=0`. The safe action sequence proved the new branch ran:
`pzero_authorization_reauthentication_started`, `p81_credentials_submitted`,
and `p81_totp_submitted`, followed by a second bounded application render wait.
The page nevertheless returned to `https://cgxxx.perimeter81.com/connect` with
only `Close`, while the cookie probe still returned HTTP 307 from `/api/user`
to `/connect`. There were no failed HTTP responses; only optional Intercom and
Google Tag Manager hosts were blocked by the restricted network policy.

The post-deployment v22 APID-29437 read replay completed as run
`d6472eba-e087-4942-8830-12a30b8e1751`, selected the exact `bet-order` target in
`middle-east-prod`, and attempted 16 Wetech operations across three scenarios.
All 16 operations had `requestCount=0`: five ended at the Pzero application
session boundary, one at Grafana readiness, and ten were rejected during local
Provider argument validation. A separate bounded `vector(1)` canary in session
`sess_32eea6eab62049149170533ac82d4c56` and operation
`op_ab43ff33c2f44c0187fcc9ad3978c159` reproduced
`pzero_application_session_not_ready` with no Provider request. Its safe
diagnostic showed HTTP 307 from Grafana `/api/user` to Pzero `/connect`, an
authorization-error page, and only the `Close` action. This proves the v22
failure is independent of Evidence Agent planning.

The shared Drive9 state can contain a stale Pzero application authorization in
browser storage as well as cookies. Clearing cookies and navigating again in
the same context therefore preserves the failed application state. The v23
candidate treats this exact error as a one-time context replacement: it closes
the persisted context, creates a fresh empty Playwright context, completes the
normal P81/Pzero/Grafana flow once within the original 90-second deadline, and
persists only that replacement context. It also exposes only allowlisted local
validation messages as `failureDetail`; unknown argument values, credentials,
cookies, and response bodies remain excluded.

Inspection of the currently served Pzero frontend narrowed the meaning of that
page. The successful connection path posts the encoded application identity to
`applications/connect`, receives an application endpoint plus one-time token,
and replaces the current location with that endpoint. The visible `Close`
button belongs to its error-state container, so closing it and retrying the
same application URL cannot repair the failed authorization state. The existing
local Wetech implementation already handles the corresponding explicit
authorization and login-connection error markers by returning once to
`/sign-in` with the exact current page encoded as `from`.

## Intended change

- On `/connect`, prefer `Open`, `Launch`, `Continue`, `Connect`, or the target
  application action; do not unconditionally dismiss the page.
- If `/connect` exposes only `Close` or `OK`, dismiss that notice at most once,
  then retry the original Grafana entry URL once. Do this for both a fresh P81
  login and a persisted P81 session.
- Before using that one retry, allow the `/connect` page five bounded seconds to
  render its positive application action or `Close`/`OK`; if the notice appears,
  dismiss it before consuming the retry.
- If the Pzero page explicitly reports authorization failure, a login-again
  connection error, or denied access, close the persisted browser context and
  retry once from a fresh empty context. The replacement uses only the original
  login deadline and becomes the sole state persisted after the operation.
- Handle the actual Grafana `/login` form before the broad Pzero-host branch.
  Both pages use a `*.pzero.perimeter81.com` hostname and expose password
  inputs, so hostname and input type alone cannot identify the P81 owner.
- Probe `/api/user` without following redirects. If it still redirects to the
  P81 `/connect` route at the deadline, report
  `pzero_application_session_not_ready`; reserve `grafana_session_not_ready`
  for failures after the application boundary.
- Keep cookie values, authorization headers, and response bodies out of Tool
  results and diagnostics.
- Return only allowlisted validation messages as `failureDetail`; never include
  the rejected argument value, credentials, cookies, or response bodies.
- Start one absolute login deadline before the first Grafana navigation. Cap
  the initial navigation, every readiness probe, and the one resume navigation
  by the remaining time; use a shorter commit-only resume navigation and retain
  only a safe action marker if it times out.
- Reserve 30 seconds between the 90-second login budget and Agent9's 120-second
  Tool timeout for diagnostics, browser-state persistence, browser close, and
  result emission.

## Validation result and next action

The v18 deadline change is live and verified only for bounded failure
observability: it returned a structured Provider result before Agent9's outer
timeout. It did not establish the Pzero application session, so this problem
remains `candidate` and APID-29437 must not be replayed as a success check.

The v19 timing change is live and its single canary proved that the retry is no
longer consumed before the error-state button renders. It did not establish the
Pzero application session, so the problem remains `candidate` and APID-29437
must not be replayed as a success check.

The v20 authentication change is live. Its one Provider canary proved that the
explicit Pzero authorization failure triggered one fresh P81 credential and
MFA submission, but that authentication still did not establish the Pzero
application session. The record therefore remains `candidate`, and APID-29437
must not be replayed as a success check. The next investigation must identify
why Pzero's `applications/connect` flow returns to its error container after a
fresh identity login; it must not add another blind retry or broaden the
Provider query.

The v22 fresh local-login path is not sufficient as deployment proof. Its
internal-prod state reuse still reached the stale Pzero authorization error,
and the post-deployment APID replay did not retrieve data. The record remains
`candidate` until the v23 package first returns actual `vector(1)` Provider data
from one controlled internal-prod canary. Only after that can APID-29437 be
replayed to verify Evidence Agent planning and ticket-level integration.

The v23 internal-prod canary used package digest
`0078ed36f8bbc365ed796d751f12fd366e5604f66a3fa023f1e92326578638b3`
in session `sess_64dd0c6519f34f65bac72ce823ef61c3`, turn
`turn_b6303565482b4acfa54461462b013207`, and operation
`op_c112080f385e4411b98063890346b9ad`. The safe action order proves the
persisted context was discarded, P81 credentials and MFA succeeded in the
fresh context, and that fresh context independently returned to the same
Pzero authorization-error page. The result remained
`pzero_application_session_not_ready` with `requestCount=0`. The v24 candidate
therefore adds only a bounded, safe trace of the Pzero `applications/connect`
response status, shape, allowlisted failure fields, and exact visible failure
marker. It does not add another login retry or change evidence queries.

The v25 internal-prod canary used package digest
`5cd5472df65685ed414c9ca1df8cb2d51db90fb572bb99b7a62acfa5b1f20ca9`
in session `sess_2c9588e616a445459766be17ac8c456f`, turn
`turn_46af35c1bec54e7a95fe91c8aa541f38`, and Managed Tool operation
`op_50d2deec3c8d4834a07cba600f649158`. It still returned
`pzero_application_session_not_ready` with `requestCount=0`, but its bounded
diagnostic recorded the exact Pzero `applications/connect` result: outer HTTP
200 with nested error status 403 and message `INCONSISTENT_USER_AGENT`. This
locates the current internal-prod failure at Pzero's application-token binding,
after successful P81 username/password and MFA authentication and before any
Grafana query. The v26 candidate therefore gives the persisted and fresh
Playwright contexts one identical explicit browser User-Agent; it does not add
another retry or broaden the evidence request.

The local Docker simulator runs the same Managed Tool `index.mjs`, the
Playwright 1.61.1 browser revision used by Agent9, the same stdin payload and
secret environment names, and a persistent local replacement for the Drive9
state file. It does not simulate Agent9 authorization, audit, or restricted
egress. The unmodified v20 source reproduced the P81 and Pzero path and reached
the Grafana `/login` page, where it returned `grafana_session_not_ready`: the
broad Pzero-host branch consumed the Grafana password form before the later
Grafana branch could submit credentials. With the v21 local candidate, the
same bounded `vector(1)` simulation returned one series, six points, and peak
value 1. A second local invocation reused the persisted browser state and
returned the same result in 7.9 seconds without another login. This is real
Provider data through the local container, but it is not post-deployment
Agent9 proof. The later clean-state regression below supersedes that cached
validation as the pre-release result.

A later controlled empty-state v21 validation reached P81 MFA, emitted
`p81_totp_submitted` before submission was confirmed, then remained on `/mf`
until the Tool mislabeled the failure as `pzero_application_session_not_ready`.
The v22 candidate waits out a nearly expired TOTP window, completes the form
submission, records `p81_totp_submission_attempted` only after that action,
records `p81_totp_accepted` only after the MFA page advances, and returns
`p81_totp_failed` when the page rejects or does not advance. One isolated
empty-state v22 Docker run completed P81, Pzero, Grafana, and the bounded
five-minute `vector(1)` query in 30.86 seconds, returning one series, six
points, and peak value 1. This proves the fresh local login path but is not
post-deployment Agent9 evidence; the record remains `candidate` until one v22
internal-prod canary returns Provider data.

The v26 exact-source local Docker regression started from an isolated empty
storage state and completed P81 username/password, MFA, Pzero application
connection, Grafana login, and the bounded five-minute `vector(1)` query. It
returned one series, six points, and peak value 1 with exit code 0 in about 41
seconds. This proves the explicit User-Agent works across the full fresh local
login path without cached cookies. It remains pre-release evidence only; the
optimization stays `candidate` until the immutable v26 package returns actual
Provider data from the controlled internal-prod canary.

The immutable v26 source and release changes merged through AOS PRs #1266 and
#1267. Internal production loaded bundle
`dccb19dec5d9dd3ff7eaeca9b0111d681e3d32866060f0b129ce6d6c37728967`
after PR #1268 corrected the bundle VersionId for the loader's release-prefixed
object path. Both Agent9 pods became Ready with loader exit code 0 and no
restarts, `/livez` and `/readyz` returned HTTP 200, and the Agent9 image did not
change. Both target Agents read back Wetech package v26 digest
`118fba86a71738e5b32203ca763840dc21077b0fee9560d470918c17ed37487d`
with their existing Auth binding and policy preserved.

The first v26 canary session `sess_ee376dca5c8f42d6a4dc751a4c91dc30`
and bounded retry session `sess_1502ed87a12d4d3f8f79a4c865d176d4`
both failed before any `external_tool.execute` operation: each recorded one
`llm_provider_result_unknown` followed by an immediate
`LLM provider request failed`, with no Assistant response or Managed Tool
operation. A direct read-only E2B `Sandbox.list` from the live Agent9 pod and a
direct request to the official `/v2/sandboxes?limit=1` endpoint both returned
HTTP 500 `Error when getting sandboxes`. The shared Codex Organization Auth was
active with no runtime claim, and the prior 24-hour operation history contained
96 successful LLM operations out of the latest 100. This is an Agent9 runtime
preflight blocker, not a v26 Wetech result. Do not run APID-29437 or mark this
optimization effective until E2B accepts a sandbox request and the exact live
v26 canary returns Provider data.

The v28 read replay completed as run
`88954bbd-5a18-46f2-ad51-8118f728a7f5` on the merged staging image. It proved
the Provider-wide stop contract by materializing one Wetech attempt instead of
the previous ten redundant attempts, but it still returned no Provider data.
The exact v28 browser version and explicit User-Agent agreed on Chromium
`149.0.7827.55`; a fresh P81 username/password and MFA flow succeeded, then
Pzero `applications/connect` returned outer HTTP 200 with nested HTTP 403
`INCONSISTENT_USER_AGENT`.

That nested code is Cloudflare Turnstile's browser-identity consistency failure,
not a Wetech query or credential error. The public P81 frontend sends only the
encoded application identity plus optional `X-Platform-Version` to
`applications/connect`; it does not have another missing user-agent parameter.
Cloud-hosted headless execution can trigger the challenge even when the same
browser revision succeeds from the local Docker harness. The v29 candidate
therefore runs that exact Chromium revision headfully under one private Xvfb
display, keeps one explicit User-Agent across persisted and replacement
contexts, and allows only the exact `challenges.cloudflare.com` host required
by Turnstile. It does not add a second login retry or broaden Provider queries.

The immutable v29 package digest
`2d411227c93df92dfeeead2628f8082fc36a76edaedeff74447a9146318c744d`
was installed on the exact Evidence Agent with its existing Auth binding and
policy preserved. Its controlled canary ran in session
`sess_b6b91637f52640d99ce27785be99a351`, turn
`turn_f62680af2f4b404f84a58e0059720e35`, and Managed Tool operation
`op_60e71821058a4977814805bb2acdc1dc`. The single bounded `vector(1)` call
completed in 77.33 seconds but returned no Provider data. The safe diagnostic
confirmed Xvfb-backed headful Chromium, successful P81 credentials and MFA,
then the same nested HTTP 403 `INCONSISTENT_USER_AGENT` from
`applications/connect`. Thus headful execution alone is insufficient.

Chromium 149 headful mode natively emits a reduced `navigator.userAgent` with
`Chrome/149.0.0.0` while its native User-Agent Client Hints carry the browser's
full `149.0.7827.55` version. The v29 explicit override changed only the legacy
User-Agent to the full version; it did not replace Chromium's native Client
Hints. The v30 candidate removes that override from both persisted and fresh
contexts so Chromium owns both identity surfaces consistently. It preserves
the same Xvfb headful process, allowlist, one fresh-context recovery, absolute
deadline, credential flow, and evidence request.

The v30 exact-source local Docker regression reused the dedicated local browser
state and completed the bounded SaaS01 five-minute `vector(1)` query with exit
code 0. It returned one series, six points, and peak value 1. This proves the
native browser identity does not regress the full local Grafana path, but it is
not Agent9 deployment evidence. The record remains `candidate` until the
immutable v30 package returns Provider data from one controlled internal-prod
canary.

The immutable v30 package digest
`cc9db5493de0b65efb9bfeb0e8b19e3b4bfa03be8c51b3a268bca4f622c47a48`
was installed on the exact Evidence Agent with its existing Auth binding and
empty policy preserved. Both Agent9 pods loaded bundle
`79dac0c9b77d4c63bcb5ed6ce679f038b3b1d0913ce9afa7696614ab71db85e4`
with loader exit 0, Ready true, no restart, unchanged image, and HTTP 200 from
both health endpoints. The controlled canary used session
`sess_9cdc15095e6740c98adb3554dd6b3bca`, turn
`turn_a174233ec4384773ba3ab2e813bbdb29`, and Managed Tool operation
`op_00f47b5a8f794cf083b476aa1826146b`. It made one Provider call and returned
no data. The safe log confirmed successful P81 credentials and MFA with
`userAgentOverride=false`, followed by the same nested HTTP 403
`INCONSISTENT_USER_AGENT` from `applications/connect` in both persisted and
fresh contexts.

This disproves the v29 explicit legacy User-Agent override as the cause, while
the same exact v30 source still returns real Provider data in the local Docker
harness. The v31 candidate changes no login or query behavior. It records only
allowlisted, non-secret browser identity fields and the exact User-Agent Client
Hint fields sent by `applications/connect`, so the remaining browser-versus-
cloud-network distinction can be resolved from one bounded canary.

The v31 exact-source local Docker regression reused the dedicated browser state
and returned the same real SaaS01 `vector(1)` Provider data: one series, six
points, and peak value 1 with exit code 0. All 51 package and release tests
passed. The diagnostic fields are written only to the sandbox log on failure;
they do not enter the Provider result or upstream Agent session.
