# ActivityHub Upstream Error Observability

## Background

The canonical Data Proxy activities API returns data through the following chain:

```text
Data Proxy caller
  -> clinic-data-service
  -> Clinic internal activities API
  -> ActivityHub HTTP API
```

Production investigation of repeated activities failures showed that Clinic's internal API received an `activityhub_client.UpstreamError`, but the log only contained the Clinic-classified HTTP status. The original ActivityHub HTTP status, response correlation identifiers, and structured error message were not retained, so the historical failure could only be bounded to the ActivityHub call.

ActivityHub is an independently deployed upstream service configured through `activityhub-endpoint`; it is external to the Clinic request process. This change does not assume that ActivityHub is owned by a third-party vendor.

## Goal

Make future ActivityHub failures diagnosable from Clinic control-plane logs while preserving the public API contract and preventing credentials or arbitrary response payloads from entering logs.

## Current Behavior

- Non-2xx ActivityHub search responses are reduced to `UpstreamError{StatusCode}`.
- A 2xx ActivityHub response with protocol `status=error` becomes a generic error without its structured message.
- The internal activities handler logs only the Clinic-classified response status and Go error type.
- The ActivityHub response body and service authentication headers are intentionally not retained.

## Target Contract

- Preserve the original ActivityHub HTTP status in a typed error.
- Distinguish an HTTP-success response carrying ActivityHub's protocol-level error envelope without logging that envelope.
- Retain safe upstream correlation headers when present, including AWS request ID, API Gateway ID, and a generic upstream request ID.
- Log those fields at the Clinic internal activities boundary with distinct names for the public/classified status and upstream status.
- Propagate the active trace ID from Clinic Data Service to Clinic Internal and from Clinic Internal to ActivityHub.
- Keep the error's `Error()` string limited to the HTTP status so wrapping or incidental logging cannot reveal upstream response content.
- Never retain or log the ActivityHub credential headers, full response body, activity payload, cursor, or request URL/query.
- Do not include the ActivityHub target/cluster identifier in the enriched failure event; use trace and upstream request identifiers for correlation.
- Preserve existing HTTP status mapping, empty-result normalization, and legacy ActivityHub API behavior.

## Affected Modules and Routes

- Go modules: repository root (`github.com/pingcap-inc/clinic`) and nested `clinic_data_service` (`github.com/pingcap-inc/clinic/clinic_data_service`).
- Client: `pkg/activityhub_client.SearchActivities`.
- Handler: `pkg/api/activities.searchAndWriteActivities`.
- Data Service client: `clinic_data_service/pkg/clinic-internal-api`; trace injection applies consistently to activities, metadata, and components calls made through its shared resource method.
- Route: `/clinic/api/v1/internal/data-proxy/v1/orgs/{orgID}/clusters/{clusterID}/activities`, consumed by the public Data Proxy activities facade.
- Executables/images: Clinic API (`cmd/server`, image `clinic`) and Clinic Data Service (`clinic_data_service/cmd/api`, image `clinic/data-service`).
- Deployment ownership: app-delivery `clinic-api` and `clinic-data-service` applications; no desired-state change is required.

The module/deployment overview remains accurate because this change does not alter ownership, entrypoints, image names, workflows, or app-delivery mappings.

## Implementation

1. Extend the typed ActivityHub upstream error with safe diagnostic metadata.
2. Convert 2xx protocol-level error responses to the same typed error representation without retaining response content.
3. Extract allow-listed upstream request identifiers from response headers.
4. Add structured fields to the existing Clinic internal error log.
5. Inject the active OpenTelemetry trace context and explicit `X-Trace-ID` into both outbound hops.
6. Install the shared Clinic propagator and OpenTelemetry Gin middleware in `internal-api` so its inbound `X-Trace-ID` becomes the context used by the ActivityHub client.
7. Add focused client, middleware, and handler tests for identifier retention, bounds, trace propagation, and unchanged HTTP classification.

## Exclusions

- No retry-policy change.
- No Data Proxy timeout or Kong configuration change.
- No response body logging and no public exposure of ActivityHub error details.
- No ActivityHub API or deployment change.
- No changes to legacy ActivityHub routes.
- No automatic staging or production deployment.

## Compatibility and Security

- Public HTTP status codes and response bodies remain unchanged.
- `UpstreamError.Error()` continues to contain only the HTTP status.
- Diagnostic request IDs are bounded and control characters are normalized before structured logging.
- Authentication headers and arbitrary JSON payload fields are neither retained nor logged.

## Test Plan

From the repository root:

```bash
go test ./pkg/activityhub_client ./pkg/api/activities
go test ./pkg/internal_api
```

From `clinic_data_service`:

```bash
go test ./pkg/clinic-internal-api
```

Also run formatting checks applicable to the changed Go files and `git diff --check`.

## Staging Verification

After explicit authorization to publish and deploy the branch:

1. Deploy the `clinic` component to staging through Clinic CD.
2. Trigger or simulate an ActivityHub non-2xx response in an authorized test scenario.
3. Confirm the public activities response remains unchanged.
4. Confirm Clinic logs include the classified status, original upstream status, allow-listed request identifiers, and bounded structured error fields.
5. Confirm logs contain no ActivityHub key/source credential, response payload, cursor, or request URL.

## Release Boundary

This plan authorizes local implementation and tests only. Commit, push, PR creation, staging deployment, merge, and production release remain separate actions. Production deployment requires explicit authorization and the normal Clinic CD/app-delivery GitOps workflow.