# Statement V2 NGM Proxy

## Background

Clinic currently exposes the Statement V1 routes under `/ngm/api/v1/statements`. Some deployment tiers now use the O11Y Statement V2 datasource and Web Console needs Clinic NGM to expose the matching O11Y diagnosis-query surface.

The existing Statement handlers combine two behaviors:

- Premium/nextgen/BYOC deploy types proxy selected routes to O11Y.
- Dedicated/shared/serverless deploy types query Clinic's existing Statement V1 storage implementation.

The V2 frontend sends `data_source=v2`. The V2 contract has additional fixed routes and its query schema must be passed through without Clinic applying the V1 query binding or validation rules.

## Goal

- Add transparent NGM GET proxies for the O11Y Statement V2-only routes.
- Route the existing list, plans, and plan-detail endpoints to O11Y when `data_source=v2` is present.
- Preserve every existing request without `data_source=v2`, including the current deploy-type routing, V1 query validation, storage lookup, response transformation, and errors.
- Keep Clinic cluster-header validation as the trust boundary before O11Y target resolution.

## Current Behavior

- `pkg/ngm/statements/statements.go` registers `available_fields`, `stmt_types`, `list`, `plans`, and `plan/detail`.
- `pkg/ngm/statements/service.go` proxies Premium/nextgen/BYOC requests to O11Y and serves other supported deploy types from Clinic storage.
- Existing list/plans/detail handlers bind and validate the V1 query structs before selecting the backend.
- `pkg/ngm/utils.ProxyO11yByQueryCluster` resolves the O11Y cluster, copies request headers and the complete query string, and forwards the upstream status, headers, and body.

## Target Contract

Register these fixed GET routes and always proxy them to the same O11Y diagnosis path after validating the existing cluster headers:

| NGM route | O11Y route |
| --- | --- |
| `/ngm/api/v1/statements/datasource` | `/internal/diagnosis/api/v1/statements/datasource` |
| `/ngm/api/v1/statements/field_values` | `/internal/diagnosis/api/v1/statements/field_values` |
| `/ngm/api/v1/statements/summary` | `/internal/diagnosis/api/v1/statements/summary` |
| `/ngm/api/v1/statements/topstats` | `/internal/diagnosis/api/v1/statements/topstats` |

For these existing routes, the exact query parameter `data_source=v2` selects the transparent O11Y path before V1 query binding:

- `/ngm/api/v1/statements/list`
- `/ngm/api/v1/statements/plans`
- `/ngm/api/v1/statements/plan/detail`

All query parameters, including repeated parameters, are forwarded. O11Y response status and response headers are preserved. As with the existing Premium/BYOC detail proxy, a successful V2 plan-detail response safely adds `binary_plan_text` when `binary_plan` is present and decodable; responses without a usable binary plan are returned unchanged. Other V2 response bodies are preserved byte-for-byte.

Requests without `data_source=v2`, including missing, empty, or other values, keep the current V1 behavior exactly.

## Affected Modules and Routes

- Go module: repository root, `github.com/pingcap-inc/clinic`.
- Package: `pkg/ngm/statements`; shared proxy implementation remains in `pkg/ngm/utils`.
- Executable: `cmd/ngm_api/main.go`.
- API prefix: `/ngm/api/v1/statements` through the existing NGM route group.
- Generated API assets: `docs/ngm_api`.

No module ownership, executable entrypoint, image name, workflow, or app-delivery mapping changes, so `docs/clinic-module-overview.md` does not need an ownership update.

## Implementation Steps

1. Add a small helper that recognizes the explicit Statement V2 datasource selector without changing legacy deploy-type decisions.
2. Add a shared Statement O11Y proxy helper that performs existing `ClusterHeader` binding/validation, applies the standard timeout, and delegates to `ProxyO11yByQueryCluster`.
3. Register and document the four new fixed routes with their exact O11Y paths.
4. Short-circuit list/plans/detail to the transparent V2 proxy immediately after header validation and before binding V1 query structs.
5. Keep the current deploy-type switch and storage implementation unchanged for requests without the V2 selector.
6. Add focused handler tests for route registration, header validation, V2 selection, query/header/status passthrough, and V1 regressions.
7. Regenerate NGM Swagger assets and run the repository Swagger consistency check.

## Exclusions

- Do not add `x-deploy-type-v2` or change the supported legacy `x-deploy-type` values.
- Do not infer product eligibility in NGM. The caller limits Statement V2 use to Premium, Premium Reserved, BYOC, and Essential V2 clusters.
- Do not change legacy Essential/shared behavior when `data_source=v2` is absent.
- Do not change Statement V1 response schemas or storage code.
- Do not add non-GET methods or modify unrelated NGM APIs.

## Compatibility Concerns

- The explicit query selector is additive. Existing clients that do not send `data_source=v2` remain on their current path.
- V2 requests bypass V1 query binding intentionally so new or repeated O11Y query parameters remain transparent.
- Header validation remains mandatory for both new and existing routes; optional project ID behavior is unchanged.
- Fixed V2-only routes proxy for every currently supported legacy deploy type. Product eligibility is enforced by the frontend and existing authorization boundary, not by a new untrusted tier header.
- Upstream non-2xx responses must remain non-2xx with their body intact rather than being rewritten as a Clinic 500.

## Test Plan

Focused tests from the repository root:

- Verify all four new exact GET routes are registered and reach their matching O11Y paths.
- Verify required cluster headers are forwarded, project ID remains optional, and invalid/missing required headers return 400 without reaching O11Y.
- Verify arbitrary and repeated query parameters are preserved.
- Verify O11Y success and error status/body/content type are preserved.
- Verify `data_source=v2` sends list/plans/detail to O11Y before legacy query validation.
- Verify a shared/dedicated request without `data_source=v2` still invokes the V1 query implementation and does not contact O11Y.
- Verify Premium/BYOC behavior without the selector remains unchanged.
- Unit-test the V2 selector for missing, empty, V1, exact V2, whitespace, and case variants.

Commands:

```bash
go test ./pkg/ngm/statements -count=1
gofmt -w pkg/ngm/statements/*.go
make generate-ngm-swagger
make check-swagger
go test ./pkg/ngm/... -count=1
```

The exact commands and results will be recorded in the implementation handoff. If private module access blocks a command, report it rather than adding temporary `replace` directives.

## Staging Verification

After review and separate authorization to publish/deploy:

1. Build the feature-branch NGM image through the NGM workflow.
2. Deploy the `ngm-api` component to staging through the normal app-delivery GitOps path.
3. Call each new route with a non-sensitive staging cluster and confirm the O11Y response status/body.
4. Compare one V2 list and detail request with the direct O11Y contract, including repeated query parameters.
5. Call an existing V1 route without `data_source=v2` and confirm unchanged behavior.
6. If read-only EKS access is separately approved, verify the `ng-monitoring` StatefulSet is Ready, has no unexpected restarts, and runs the expected image.

No staging deployment or live API call is authorized by this implementation task.

## Deployment and Release Boundary

- Affected source module: root Go module, `pkg/ngm/statements`.
- Executable and image: `cmd/ngm_api/main.go` -> `ngm_api`.
- Required deployment module: Clinic CD component `ngm-api` via `.github/workflows/build_ngm_api.yml`.
- app-delivery application: `ngm`.
- Kubernetes workload: `ng-monitoring` StatefulSet.
- Target environment for pre-merge validation: staging, only after explicit authorization.
- Production target: `prod-cloud`, only after merge, required review, staging verification, and explicit production-release authorization.
- Desired-state impact is an image tag bump only; no app-delivery configuration change is planned.

A follow-up user request authorizes a local commit on the current feature branch. It does not authorize push, PR creation, staging deployment, merge, or production release.

## Implementation Validation (2026-08-17)

- `go test ./cmd/ngm_api ./pkg/ngm/statements -count=1`: passed; the executable compiles and all focused Statement route/proxy/regression tests pass.
- `make check-swagger`: passed with the Go 1.24.2 toolchain required by the root module. NGM generated assets contain the four new routes, the datasource response schema, and the V2 selector parameters.
- `git diff --check`: passed.
- `go test ./pkg/ngm/... -count=1`: the changed `pkg/ngm/statements` package passed, but the broader existing suite failed in unrelated packages/tests: `pkg/ngm/conprof` (`Test_handleActionTokenByocHostKeepsDeployTypeInToken`), `pkg/ngm/statements/impl` (`TestPlans`), and `pkg/ngm/topsql/vm` (existing query/response expectations). No files in those packages were modified by this work.
- Staging and live O11Y verification were not run because this task did not authorize a deployment or use of staging credentials.