# Wetech fixed egress through an AOS WSS tunnel

Deployed on 2026-09-08 CST as Wetech v61. Four real E2B canaries passed; metrics
and Dashboard reused sessions across fresh sandboxes without browsers. See the
[version-bound live verification](opts/2026-09-08-wetech-wss-fixed-egress.md).
The staging overlay adds `-agent9` to proxy Deployment/Service/Ingress and
ExternalSecret resource names; the generated Secret remains `wetech-egress-proxy`.

The Wetech browser, credentials, login, cached session and API queries remain
inside the E2B Managed Tool sandbox. The AOS proxy only forwards TCP streams.
Agent9 retains its existing sandbox and Organization Auth responsibilities;
its application backend does not relay query traffic.

## Transport

```text
E2B Playwright browser / APIRequestContext
  -> sandbox-local HTTP CONNECT listener on 127.0.0.1 (ephemeral port)
  -> wss://oncall.ops-staging.pingcap.net/aos-wetech-proxy
  -> existing devops-web-ingress-staging ALB
  -> separate AOS wetech-egress-proxy Pod
  -> existing private-subnet NAT / 44.245.243.63
  -> Wetech / P81 HTTPS origin
```

The existing hostname, DNS and wildcard ACM certificate are reused. The ALB
routes the exact path directly to the proxy Service with group order -20,
ahead of the existing oncall site's root path. It does not pass through the
oncall application. TLS for the outer WSS connection terminates at the ALB;
the ALB-to-Pod hop is HTTP within the VPC, as for the existing ingress.
The inner browser-to-Wetech HTTPS remains encrypted across the entire tunnel.
The proxy does not terminate or inspect that inner TLS connection.

Each CONNECT uses one WSS stream, negotiated with `aos-wetech-tcp-v1`.
`X-Egress-Target` carries a hostname and port 443; the connection upgrades only
after the proxy has connected to the checked public IP. There is no generic
HTTP fetch API, URL rewriting, or cookie handling on the proxy. HTTP port 80,
UDP and arbitrary TCP ports are not provided: current Wetech/P81 targets use
HTTPS. There is no destination hostname allowlist.

The client uses bundled `ws`, so the immutable Managed Tool needs no npm install
and no change to the Agent9 E2B template. API-only probes, browser navigation,
fresh browser contexts and browser-context API queries all receive the same
local proxy. Drive9 state storage remains on its existing direct route.
A tunnel failure is an execution error; the Tool does not retry via a direct
Wetech connection or treat it as evidence of an incorrect Dashboard password.

## Authentication and destination boundary

- A random proxy token authenticates the WSS upgrade in the Authorization
  header. It is never placed in a URL or forwarded to a Wetech origin.
- The proxy rejects unauthenticated connections before DNS or TCP access.
- All resolved addresses must be public unicast. Private, loopback, link-local,
  metadata, multicast and reserved addresses, including mapped IPv4 and IPv6
  transition ranges, are rejected. TCP connects to the validated numeric IP
  without a second DNS lookup.
- Connection logs contain only status codes, host, duration and byte counts.
  They do not contain credentials, cookies, SQL, or response bodies. The Tool
  includes bounded WSS failure/status/timing diagnostics in its existing
  stderr and workspace diagnostic file.
- Token rotation automation, quotas and post-leak response are deferred per
  the operator's scope. Existing Provider credentials are not migrated.

## Credentials and deployment order

1. Store `{ "token": "<random high-entropy token>" }` in AWS Secrets Manager
   `auto-oncall/staging/wetech-egress-proxy` in account `552185537300`, region
   `us-west-2`. Do not put the value in Git, shell arguments or tool output.
2. The new ExternalSecret uses the existing `default` ClusterSecretStore and
   creates Kubernetes Secret `wetech-egress-proxy`, key `token`, in
   `auto-oncall-services-agent9`. Verify the controller can read this exact
   secret before enabling the Tool version.
3. Create a separate Wetech Organization Auth for v61, retaining the existing
   credential values and adding `WETECH_EGRESS_PROXY_TOKEN` with this token.
   Do not add a field to the old Auth in place: Agent9 checks exact environment
   key equality, so that would invalidate v59 operations still using it.
   Merely storing an AWS secret does not make it available inside E2B.
4. Build the AOS image containing `dist/src/run-wetech-egress-proxy.js`. Sync
   this repository's app-delivery templates through the existing registration
   workflow; the proxy has its own Deployment, Service and Ingress. Its command
   starts only the TCP relay, never AOS automation.
5. Verify its Ready state, exact path routing, rejection of invalid credentials
   and internal destinations, and its real public egress IP. The Pod is limited
   to the three existing private admin node groups. Recheck routes whenever
   those groups or NAT infrastructure change.
6. Build and publish the immutable Wetech v61 candidate through the AOS-owned
   package release flow. Atomically update the Evidence Agent installation to
   the v61 reference and new Auth ID through the installation API, preserving
   its scope/policy. The routine release script preserves the existing Auth ID;
   do not use it to perform this one-time Auth-shape migration. Do not activate
   v61 before the proxy and new Organization Auth are ready. Retain the v59
   reference and its old Auth for in-flight operations and rollback.
7. Run bounded metrics and Dashboard canaries from two fresh E2B sandboxes.
   Require real Provider data, the same proxy egress, and inspect whether the
   second sandbox can use the persisted session without browser login. This
   is the criterion for resolving the cross-sandbox auth problem.

Rollback the Tool installation to its previous immutable version before
removing the proxy. Existing Wetech versions continue to use their original
direct transport. No new ticket replay or Jira write is required merely to
verify this transport; preserve the published APID-29464 answer.

## Validation

`npm test` covers auth-before-DNS, public address validation, numeric-IP
connection after resolution, TLS verification, no direct fallback, concurrent
multi-megabyte HTTPS streams and cleanup. The existing API-session tests also
assert that every Playwright context uses the supplied proxy.

Run the browser integration in the repository's existing pinned local image:

```bash
npm run build:agent9-tools
npm run build
docker run --rm --init --ipc=host \
  --volume "$PWD:/work:ro" --workdir /work \
  --env WETECH_PROXY_PLAYWRIGHT_TEST=1 --entrypoint node \
  auto-oncall-managed-wetech-local:playwright-1.61.1 \
  --test dist/test/wetech-egress-proxy.test.js
```

This uses isolated TLS fixtures and a real Playwright browser/API client. It
does not contact P81 or prove E2B ingress, real NAT identity, Organization Auth,
or cross-sandbox session reuse. The regular `simulate:wetech-managed-tool`
runner now requires `WETECH_EGRESS_PROXY_TOKEN` and mounts the bundled client;
use it for a bounded real query only after the proxy endpoint is available.