# Athena Query Failure Observability

## Background

Clinic Data Service preserves Athena failure metadata in `engine.AthenaQueryError`, including the Athena query ID, terminal state, state-change reason, error category, error type, and retryable flag. The Data Proxy service currently logs only `query execution failed` and the engine name, while the canonical API intentionally returns a generic `InternalError`. As a result, production logs cannot identify the failed Athena query or explain its failure.

## Goal

Record safe, structured Athena failure metadata at the Data Proxy service boundary so an operator can identify the failed query and determine whether the failure was retryable, without changing the public API response or logging SQL, credentials, storage locations, or arbitrary request payloads.

## Current Behavior

- Athena polling constructs a typed `AthenaQueryError` for `FAILED` and `CANCELLED` executions.
- The service wraps that error but omits it and its structured fields from the final failure log.
- The canonical handler maps the failure to HTTP 500 with `QUERY_FAILED` and the generic message `query execution failed`.
- Only the special `INVALID_GLUE_SCHEMA` path performs a guarded repair and one retry.

## Target Contract

- When the final query error contains `AthenaQueryError`, log:
  - Athena query ID;
  - terminal state;
  - bounded and sanitized state-change reason;
  - Athena error category and type;
  - Athena retryable flag.
- Redact the current cluster ID, S3/HTTP URLs, and AWS ARNs from the logged reason.
- Normalize control characters and limit the reason length.
- Do not log SQL, datasource URLs, output locations, credentials, or request payloads.
- Preserve the current API status, error code/message, timeout behavior, schema-repair behavior, and retry policy.

## Affected Modules and Routes

- Go module: `clinic_data_service` (`github.com/pingcap-inc/clinic/clinic_data_service`).
- Service: `pkg/service.DataProxyService.Query`.
- Engine metadata: `pkg/engine.AthenaQueryError`.
- Routes: SQL query entrypoints that execute through the shared Data Proxy service, including the canonical `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/sql-queries` route.
- Executable/image: `clinic_data_service/cmd/api`, image `clinic/data-service`.
- Deployment ownership: app-delivery `clinic-data-service` application.

The module/deployment overview remains accurate because ownership, entrypoints, images, workflows, and app-delivery mappings do not change.

## Implementation

1. Detect a wrapped `AthenaQueryError` at the final service failure boundary.
2. Build structured Zap fields from the typed error.
3. Sanitize and bound the Athena reason before logging it.
4. Add focused tests covering wrapped errors, all structured fields, redaction, control-character normalization, truncation, and non-Athena errors.

## Exclusions

- No SQL or Athena retry-policy change.
- No timeout change.
- No public error-response change.
- No AWS, Kong, Kubernetes, or app-delivery configuration change.
- No automatic staging or production deployment.

## Compatibility and Security

The change is logging-only. Existing clients continue to receive the same generic errors. The diagnostic reason is treated as potentially sensitive: known location formats and the active cluster ID are redacted, control characters are removed, and the value is bounded before logging.

## Test Plan

From `clinic_data_service`:

```bash
go test ./pkg/service
```

Also run the applicable formatting check and `git diff --check`.

## Staging Verification

After explicit authorization to publish and deploy the branch:

1. Deploy `clinic-data-service` to staging through the dedicated Clinic Data Service CD workflow.
2. Execute an authorized Athena query that produces a controlled failure.
3. Confirm the API contract is unchanged.
4. Confirm logs contain the query ID, state, sanitized reason, category/type, and retryable flag.
5. Confirm logs do not contain SQL, datasource/output URLs, credentials, or request bodies.

## Release Boundary

This plan authorizes local implementation and tests only. Commit, push, PR creation, staging deployment, merge, and production release remain separately authorized actions. Production must follow Clinic CD and app-delivery GitOps after explicit approval.