# O11Y metrics and logs integration

## Prerequisites

- Connect from an Office IP or Feilian VPN.
- Configure `GITHUB_PAT` with `read:user` and `read:org` for O11Y authentication.
- Installing from the private `tidbcloud/nutshell-skills` repository additionally requires repository read access or an existing `gh`/SSH login that can read the repository.
- Install `curl` and `jq`. The persistent Manual Invoice Python runtime supplies `requests`, `boto3`, and `duckdb`; AWS storage access also needs the AWS CLI.

`$manual-invoice configure` installs these personal Codex Skills under `${CODEX_HOME:-~/.codex}/skills`:

- `clinic-api`
- `o11y-auth`
- `o11y-data`
- `o11y-data-export`
- `o11y-metrics-api`

## Load configuration

Always load the Manual Invoice adapter before sourcing O11Y helpers:

```bash
source "$SKILL_DIR/scripts/o11y_runtime.sh"
export PYTHON_BIN="$MANUAL_INVOICE_PYTHON"
source "$O11Y_SKILLS_ROOT/o11y-data/o11y-data.sh"
o11y_auth aws
```

The adapter sets `ENV_FILE` to the active `.manual-invoice.env` or user config, applies `umask 077`, and maps `GITHUB_PAT` to the GitHub CLI variable names. O11Y's cached bearer tokens and temporary cloud credentials therefore remain in the same private configuration file rather than a project `.env`.

Do not echo authentication responses or capture credentials in run evidence. `o11y_auth` exports credentials without printing their values.

## Select an O11Y Skill

- Use `clinic-api` for exact Clinic-backed cluster metadata when its internal Access Key is already available. The Skill cannot authenticate with GitHub PAT or O11Y tokens.
- Use `o11y-auth` for authentication and bizType mapping.
- Use `o11y-data` as the primary interface for cluster discovery, Prometheus metrics, Loki logs, and S3/GCS/OSS data.
- Use `o11y-data-export` to create and manage historical slow-query, raw-log, or SQL-statement export pipelines.
- Use `o11y-metrics-api` as detailed Prometheus/API reference. Prefer PAT-based authentication from `o11y-data` for Manual Invoice.

Resolve provider, region, and deploy type before querying. Do not ask for them merely because only a cluster ID was supplied. Run the Manual Invoice resolver, which uses nutshell `clinic-api` when its required authentication is already available, then delegates O11Y target discovery to `o11y-data` with bounded GS fallback. Ask the user only if both stages fail.

```bash
"$MANUAL_INVOICE_PYTHON" "$SKILL_DIR/scripts/resolve_o11y_cluster.py" \
  --cluster-id 10658480506726873193 \
  --tenant-id 1372813089209327038 \
  --output runs/example/evidence/cluster.json
```

`--provider`, `--region`, and `--deploy-type` are optional validation hints. The output contains no authentication tokens.

## Billing-source contract

For every O11Y source, save:

- source Skill and installed path;
- cluster/application ID, provider, region, deploy type, and resolved O11Y target ID;
- PromQL or LogQL/export parameters;
- inclusive/exclusive time boundaries and timezone;
- raw response/file hashes;
- normalized schema version and row count;
- retrieval timestamp and any partial-result marker.

Keep credentials out of evidence and lineage.

### Metrics

Save the raw Prometheus JSON response, then normalize each sample to explicit fields such as timestamp, metric labels, and the original string value. Use Decimal in the calculation layer. Record whether the query used the application or infrastructure O11Y target.

Use the persistent metrics runner so cluster discovery and every PromQL request are auditable:

```bash
"$MANUAL_INVOICE_PYTHON" "$SKILL_DIR/scripts/query_o11y_metrics.py" \
  --cluster-id 10658480506726873193 \
  --tenant-id 1372813089209327038 \
  --query-file runs/example/evidence/query.promql \
  --start 2026-06-01T00:00:00Z \
  --end-exclusive 2026-07-01T00:00:00Z \
  --step 60 \
  --cluster-output runs/example/evidence/cluster.json \
  --raw-output runs/example/evidence/metrics-raw.json \
  --output runs/example/normalized/metrics.jsonl \
  --metadata-output runs/example/evidence/metrics-job.json
```

The runner treats the end boundary as exclusive, rejects tenant/provider/region/deploy-type mismatches, and blocks partial Prometheus results. It writes provider, region, deploy type, bizType, target UUID, query hash, boundaries, raw response, and normalized exact string values without persisting bearer tokens.

If the Prometheus endpoint returns HTTP 403, it forces one fresh O11Y authentication, retries the same query once, and records the retry flags in metadata. If the retry is also 403 (or re-authentication cannot complete), stop with a prompt to connect to the Office network/Feilian VPN and try again. Other HTTP statuses do not trigger this refresh loop.

### Logs

Save raw Loki/export responses before parsing. If logs contribute a billable count or amount, implement the parser/aggregation as a deterministic versioned calculation or normalization script. Do not derive invoice totals only from prose or an unversioned grep command.

Use the persistent Loki runner so every query lands on disk:

```bash
"$MANUAL_INVOICE_PYTHON" "$SKILL_DIR/scripts/query_o11y_loki.py" \
  --provider aws \
  --region us-east-1 \
  --query-file runs/example/evidence/query.logql \
  --raw-output runs/example/evidence/loki-raw.json \
  --output runs/example/normalized/loki.jsonl \
  --metadata-output runs/example/evidence/loki-job.json
```

The runner authenticates without persisting bearer tokens in evidence, paginates forward, saves every raw page, and writes normalized timestamp/labels/line JSONL. Treat `possibly_truncated: true` as blocking for billing calculations.

If any Loki page returns HTTP 403, the runner forces one fresh O11Y authentication and retries that page once. A second 403 produces the Office/VPN remediation prompt and stops; metadata records whether the refresh was attempted and succeeded.

### Query manifest example

```yaml
queries:
  - id: cluster_ru
    source: o11y_metrics
    query: queries/cluster_ru.promql
    output: normalized/cluster_ru.jsonl
    schema: schemas/cluster_ru.schema.json
    parameters:
      cluster_id: STRING
      period_start: TIMESTAMP
      period_end: TIMESTAMP
      step: DURATION
```