# Hosted MCP recovery assessment Status: ongoing independent L1 Worker rewrite; not deployed or accepted. Decision date: 2026-09-08. This document owns the L2 incident evidence and rollout tracking. The product architecture and implementation authority is now [L1 Worker PR #869](https://github.com/pingcap-inc/nexuscrm-app-packs/pull/869), including `docs/design/mcp-l1-worker.md` in that PR. The user explicitly superseded the embedded SFCRM AppService recovery and Connected App/resource-profile prerequisites: MCP is an independent L1 Worker, consumes only Runtime CRM L0 APIs, and acts with the authenticated user's current permissions, including writes that Runtime authorizes. It must not require a Connected App, an MCP-specific scope/profile, or a connector service. The historical assessments below retain their dated evidence. Their read-only recovery sequence, OAuth/Connected App acceptance requirements and implementation PR lists are **SUPERSEDED**, not active merge/deployment gates. #867, Contract #237, Core #4826 and API-host #116 have been withdrawn in favor of #869. Core #4823/#4825 remain landed generic improvements; their tests do not prove a Worker. Core #4824's embedded/resource-profile design is superseded by the user decision. L2 next consumes the published independent Worker artifact, declares its separate workload against the CRM installation through the existing deployment authority, routes `/mcp` to it, and verifies target/public user-permission-equivalent reads and writes. The chosen Runtime/Worker artifacts, user credential path and actual business readback must be recorded before any live recovery claim. Client login, ordinary-user object metadata coverage, release registration and rollout remain tracked in #869; do not copy their full checklist here. Latest native-Worker entry probes: 2026-09-08 09:57:28–29 UTC. Both `/mcp` and protected-resource metadata still return 404 on Staging and Prod. Read/write acceptance and deployed-image identity remain UNVERIFIED. The earlier three-path OAuth probes below are historical observations. No environment or CRM mutation was performed. ## Evidence and ownership The original historical PR revision reviewed was #5506 at `4e89f52213bcd0994ef147ddb839605b4c3d11b9`. The following source heads were independently checked against GitHub main on 2026-09-08; they are source identities, **not deployed image identities**: | Repository | Source SHA | | --- | --- | | nexuscrm-pingcap-pack | `9eaefb641cb7d5dc3ecace8b0b611e863e83e3e4` | | NexusCRM | `9d71b44d4fdd6ba61d5a40583d97b7601ba6eceb` | | nexuscrm-app-packs | `2fbdac6e3629a4be370c198469beab16a8e1c785` | Public read-only probes at 2026-09-08 03:34 UTC (11:34 Asia/Shanghai): | Endpoint | Staging | Prod | | --- | --- | --- | | POST `/mcp`, no credential | 404, no Bearer challenge | 404, no Bearer challenge | | GET `/.well-known/oauth-protected-resource` | 404 | 404 | | GET `/.well-known/oauth-authorization-server` | 404 | 404 | No response bodies, tokens, object names or CRM records were retained. Authenticated OAuth/read acceptance and actual deployed image/BOM identities remain UNVERIFIED. These results prove public endpoint failure; they do not alone identify a particular running container as the cause. Source and history explain why a redeploy of current code is insufficient: 1. The L2 Nginx template sends all three paths to `__BACKEND_PORT__`, which D3 supplies as the SFCRM AppService HTTP port. The current AppService has no MCP registration. The L0 Runtime process has gRPC and operational HTTP listeners, not a mounted MCP application route. 2. L0 `backend/internal/interfaces/rest/mcp/routes.go` retains `RegisterMCP`, but its current callsites are tests. `newProductionMCPAuthorization` likewise has no production callsite. The L0 graph helped locate these symbols; index SHA was UNVERIFIED, so current source and registration references were checked. 3. [#5448](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/pull/5448) added a Staging check, but [#5493](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/pull/5493) explicitly reverted it and accepted temporary Staging MCP degradation. Its body forbade promotion based on that rollback. Current controllers no longer contain the probe; workflow success alone therefore proves no MCP read. 4. [NexusCRM #4614](https://github.com/pingcap-inc/NexusCRM/pull/4614) is still OPEN at `432c4ab7dad320d73b92a4b0866fc1557eba680d`. Its old host depends on monolith assembly. [NexusCRM #4814](https://github.com/pingcap-inc/NexusCRM/pull/4814) removed `serverruntime/routes.go` at `e013c59433219c8eb26d2bbc84055641b2cab974`. The current upstream AGENTS explicitly forbids restoring that serving process. 5. The latest listed Staging run [34099726451](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/actions/runs/34099726451) and Prod run [34112943515](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/actions/runs/34112943515) both report success (2026-09-07). Those controller results coexist with today's 404s. They are not evidence that the public MCP recovered, nor proof of the currently deployed tuple without release-authority readback. ## Current diagnostic and rollout boundary The executable `scripts/verify_hosted_mcp.py` follows the new L1 native-session bearer contract: protected-resource metadata declares the configured `/mcp` resource and header bearer method; unauthenticated POST receives the Worker's 401 Bearer challenge. It deliberately does not demand an OAuth authorization server, Connected App or resource-profile exchange. It sends no credentials, refuses redirects and ambient proxies, bounds metadata reads and prints no response bodies or proxy credentials. ```sh python3 -m unittest tests.test_verify_hosted_mcp python3 scripts/verify_hosted_mcp.py --origin https://nexuscrm-new-staging.pingcap.com python3 scripts/verify_hosted_mcp.py --origin https://nexuscrm-new-prod.pingcap.com ``` `entryPassed` proves only this entry contract. Both `authenticatedRead` and `authenticatedWrite` remain UNVERIFIED until separate actual-user journeys. A failed probe does not establish the running image identity by itself. This documentation/diagnostic PR can merge independently of rollout. Before changing a deployment, L2 must consume a published Worker, register its distinct workload under existing authority, route `/mcp` to it and validate the actual Runtime/Worker tuple with allowed/denied reads and writes, FLS/RLS, revocation, client login and command reconciliation. Worker metadata/login gaps and release registration are tracked in L1 #869; merging this PR does not close the incident. The obsolete recovery sequence and intermediate PR checklists are preserved in [the prior assessment](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/blob/cc8550b76ec3f1e713ec1424d20d2a8111aad6c1/docs/operations/hosted-mcp-recovery.md). They are superseded, not current gates. Core #4823/#4825 were prerequisite generic changes, and their historic tests are not an independent Worker acceptance. ## Formal independent Worker release and deployment The Worker is independently published by L1, while environment deployment stays inside the existing signed-BOM Staging/Prod lifecycle. The optional, closed `services.mcp_worker` role selects `ghcr.io/pingcap-inc/nexuscrm-mcp-worker` and its own app-packs source commit and explicit immutable image digest; it need not equal the AppService source. Mutable source tags never select MCP releases. The resolved role contains the verified image, provenance and SBOM digests. Its presence adds `;mcp_worker>runtime_api` to the declared execution topology. Old signed BOMs without this role retain their original execution projection. The L1 owner policy binds Sigstore's GitHub OIDC issuer, the exact main `mcp-worker.yml` workflow and source SHA. L2 verifies that certificate identity, transparency-log inclusion, signed source/provenance and linked BuildKit SBOM before signing the resolved BOM. This role does not use or extend the AppService KMS role. The existing KMS verification paths for all other release roles remain unchanged. Changing the producer verifier requires a reviewed producer-closure hash in the existing L2 signing policy. For a selected Worker, the shared D3 controller: 1. Pulls the exact BOM digest and includes it in the sealed process inventory. 2. Provisions a distinct private `mcp-worker` token. Only Runtime's transport identities and `user_api_callers` gain that client; Catalog handler ownership and consumers do not. Current user session/CRUD/FLS/RLS remains authoritative. 3. Derives one loopback HTTP port at Runtime admin port + 1, rejecting role collisions. Staging blue/green use 3503/3603. The Worker receives the public HTTPS origin, installation identity, Runtime endpoint and CA; no database credentials or user token are placed in its environment. 4. Starts the Worker after Runtime, includes readiness before cutover, then routes `/mcp` and `/.well-known/oauth-protected-resource` to it. CRM and OAuth endpoints keep their owners. The public metadata/401 challenge probe is a deployment gate whenever the BOM selects MCP. 5. Includes MCP in exact inventory, retry recovery, reverse-order stop, predecessor retirement, and Runtime TLS rotation. An MCP-only image change keeps other BOM selections but still uses the formal environment lifecycle. Deploy through `deploy-staging-blue-green-environment.yml` using the newly signed resolved-BOM digest. Set `reset_branch=false`; a reset is not an MCP prerequisite. After successful Staging, the existing D3 Tag binds the same BOM and the existing Prod operator consumes it. No independent Prod side deployment or new generic local deployment wrapper is introduced. The public probe proves endpoint ownership and bearer challenge, not login or business authorization. Actual-user initialization, tool discovery, bounded reads, command/permission journeys and fixture cleanup remain separately bound to the deployed image and timestamp. The Worker currently requires a supplied Runtime user bearer token; automated Claude/Codex OAuth onboarding and ordinary user metadata discovery are not added by release integration.