# Metrics

> Canonical path: `specs/observability/metrics.md`
> Status: living-spec
> Scope: durable rules for naming, labeling, bucketing, and shipping Prometheus metrics from `agentteam-server`. Companion to [`observability.md`](./observability.md).
> Non-goals: dashboard layout; alert thresholds; pre-registering metrics the codebase has not yet emitted.

---

## 1. Naming Rules

- Metric name shape: `agentteam_<area>_<measurement>_<unit>`. The `agentteam_` prefix is the project namespace and never varies.
- `<area>` is a stable code area, not a product-feature word: `http`, `db`, `daemon`, `agent`, `messages`, `archive`, `history`, `rewind`, `realtime`, `trigger`, `webhook`, `attachment`, `runtime`.
- `<measurement>` is the metric noun (`requests`, `duration`, `inflight`, `bytes`).
- `<unit>` follows Prometheus conventions: `_seconds` for durations, `_bytes` for sizes, `_total` always last for counters. Histograms keep their base unit (`_seconds`, `_bytes`) without `_total`.
- Lowercase, underscore-separated.

Label keys are also snake_case. Boolean labels use `outcome=success|failure` form rather than `success=true`.

## 2. Histogram Bucket Families

Histograms must pick a named bucket family from the list below. Custom buckets require an amendment to this section with rationale.

- `BucketsLatencySeconds` — `{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10}`. Use for user-facing request latency.
- `BucketsLongPollSeconds` — `{0.1, 0.5, 1, 5, 15, 30, 45, 60, 90, 120}`. Use for intentional blocking waits. Server-side agent receive long-poll has been removed; do not add new metrics that imply that path exists.
- `BucketsBackgroundSeconds` — `{0.05, 0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600}`. Use for background worker passes whose duration spans several orders of magnitude.
- `BucketsHTTPSizeBytes` — `{128, 512, 2_048, 8_192, 32_768, 131_072, 524_288, 2_097_152, 8_388_608}`. Use for HTTP request and response sizes.

Families live in code under `internal/observability/metrics/buckets.go` as named exports. The identifier in code must match the identifier in this section; the guardrail compares both.

## 3. Cardinality Rules

The full cardinality discipline is in [`observability.md`](./observability.md) §5. The short version:

- A label value must be drawn from a finite, code-controlled enumeration. Closed enums only: HTTP method, `ctx.FullPath()` route template, status code, `outcome` enum, `driver` enum.
- A label value must not be drawn from user input, primary keys, or any unbounded space (`workspace_id`, `agent_id`, `email`, raw URL path).
- New label keys not listed against a metric in this document are rejected by the guardrail.

## 4. Forbidden Names

The following names are reserved as forbidden anti-examples for the reasons in [`observability.md`](./observability.md) §5.4. Registering a metric under any of them is a spec violation:

- `agentteam_scope_messages_rows`
- `agentteam_scope_messages_bytes`
- `agentteam_scope_fanout_outbox_rows`

Each would require a full-table aggregate against a forever-growing table to compute, which violates the database-amplification rule. PromQL derivations from write-path counters provide the same operational insight at `O(1)` per write.

## 5. Storage Volume Is Derived, Not Sampled

Operational dashboards that need the current hot-tail row count or byte size of growing tables (`scope_messages`, `scope_fanout_outbox`) derive them in PromQL at query time from monotonic write-path counters minus removal counters — `sum(written) - sum(archived) - sum(removed)`, with a `_bytes_total` counterpart for byte size.

Those counters live in the domain packages that own the writes and removals (message ingress, archive worker, rewind, targeted-delivery dispatcher), and are added in the same change that introduces each instrumentation. They are not part of the first implementation PR ([§6](#6-initial-required-surface)). Until a counter ships, the corresponding PromQL series is empty and operators rely on the on-demand admin diagnostic at `internal/api/admin` for the exact value; defining that endpoint is the responsibility of [`specs/admin/`](../admin/), not this document.

This separation is the point: a sampled gauge would have been one less follow-up but it would also have made the database the bottleneck of the metrics pipeline (see [`observability.md`](./observability.md) §5.4). Forcing the volume answer to come from the write path is the trade.

## 6. Initial Required Surface

The first implementation PR registers the following collectors. This list is intentionally minimal — it covers the four operator questions in [`observability.md`](./observability.md) §1 without forcing every domain to be instrumented at once.

### 6.1 HTTP server

Owned by `internal/observability/metrics/http.go`, driven by a Gin middleware that runs after route matching (so `ctx.FullPath()` is already resolved before `+1`).

| Metric | Type | Labels | Buckets |
|---|---|---|---|
| `agentteam_http_requests_total` | Counter | `method`, `route`, `status` | — |
| `agentteam_http_request_duration_seconds` | Histogram | `method`, `route`, `status_class` | `BucketsLatencySeconds` |
| `agentteam_http_inflight` | Gauge | — | — |
| `agentteam_http_panics_total` | Counter | `method`, `route` | — |

Label allowlists:

- `method` ∈ {`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`, `"<unknown>"`}; non-standard methods collapse to `"<unknown>"`.
- `route` is `ctx.FullPath()` only; unmatched paths collapse to `"<not_found>"` / `"<method_not_allowed>"`.
- `status` (counter only) is the 3-digit code as a string; codes outside `[100, 599]` collapse to `"<invalid>"`.
- `status_class` (histogram only) ∈ {`1xx`, `2xx`, `3xx`, `4xx`, `5xx`, `<invalid>`}. Histograms multiply series by bucket count, so the class form bounds the cardinality multiplier at 6× per `(method, route)` cell instead of ~30× with raw status codes; debugging an exact response code is still possible through the counter.

Two design choices in this table are tied to specific failure modes; both are intentional and must not be widened without engaging with the trade-off:

- **`agentteam_http_inflight` has no labels.** The gauge needs a balanced `+1` on entry and `-1` on exit; the safest way to guarantee that is to remove the label dimension entirely. A single saturation signal is what alerting needs anyway. If per-route inflight is ever required, it is added in a separate change that reuses the route already captured at middleware entry and reuses the *same* value at exit (otherwise accounting drifts under panic/abort paths).
- **Histogram uses `status_class`, counter uses full `status`.** Rationale above; this also keeps the cardinality budget within the estimate below.

Cardinality budget (current inventory order of magnitude; refresh this when route ownership changes materially):

- Current route inventory is roughly `450` method+route cells (`441` in the implementation inventory; a lightweight non-test Gin method-registration scan currently reports `447` registrations).
- Histogram series ≈ `method+route cells × |status_class| × |buckets+1|` ≈ `450 × 6 × 12` ≈ `32,400`.
- Counter series ≈ `method+route cells × expected status-code variants` ≈ `450 × 12` ≈ `5,400`.
- Panic series is `O(method+route cells)`; `agentteam_http_inflight` remains `O(1)`.
- Tighten this section if method+route cells approach `1,000`, if the histogram estimate approaches `75,000`, or if a profile should stop contributing route-level histograms.

### 6.2 Go runtime and process

Registered through `collectors.NewGoCollector()` and `collectors.NewProcessCollector()` from `prometheus/client_golang`. These expose well-known names (`go_goroutines`, `go_gc_duration_seconds`, `process_cpu_seconds_total`, `process_open_fds`, etc.) and are not renamed.

### 6.3 Database pool

Owned by `internal/observability/metrics/dbpool.go`, driven by a `GaugeFunc` that reads from `internal/coredb/SQLDBPoolController` once per scrape.

| Metric | Type | Notes |
|---|---|---|
| `agentteam_db_pool_open_connections` | Gauge | `sql.DB.Stats().OpenConnections` |
| `agentteam_db_pool_in_use` | Gauge | `sql.DB.Stats().InUse` |
| `agentteam_db_pool_idle` | Gauge | `sql.DB.Stats().Idle` |
| `agentteam_db_pool_wait_count_total` | Counter | `sql.DB.Stats().WaitCount` (monotonic) |
| `agentteam_db_pool_wait_duration_seconds_total` | Counter | `sql.DB.Stats().WaitDuration.Seconds()` (monotonic) |

Pool stats are read from in-memory counters maintained by `database/sql`; no SQL query is issued per scrape, so this is not subject to §4 of this document.

### 6.4 Daemon transport

Owned by `internal/observability/metrics/daemon_transport.go`, driven by the Server-side daemon Remote Endpoint router and Elastic Request Link manager.

| Metric | Type | Labels | Buckets |
|---|---|---|---|
| `agentteam_daemon_transport_requests_total` | Counter | `endpoint`, `link_mode`, `outcome`, `fallback` | — |
| `agentteam_daemon_transport_request_duration_seconds` | Histogram | `endpoint`, `link_mode`, `outcome` | `BucketsLatencySeconds` |
| `agentteam_daemon_transport_route_decisions_total` | Counter | `endpoint`, `link_policy`, `mode`, `reason` | — |
| `agentteam_daemon_transport_elastic_events_total` | Counter | `event`, `reason` | — |
| `agentteam_daemon_transport_warm_events_total` | Counter | `origin`, `profile`, `warm_event`, `reason` | — |
| `agentteam_daemon_transport_elastic_pool` | Gauge | `state` | — |

Label allowlists:

- `endpoint` is the Remote Endpoint descriptor `MetricsName`; it is bounded by `RemoteEndpointRegistry` and collapses invalid values to `"<unknown>"`. It must not contain request IDs, machine IDs, workspace IDs, paths, URLs, or token material.
- `link_mode` ∈ {`session_link`, `elastic_request_link`, `legacy_session_link`, `none`, `"<unknown>"`}.
- `link_policy` ∈ {`session_only`, `session_preferred`, `elastic_preferred`, `elastic_required`, `"<unknown>"`}.
- `mode` ∈ {`session_link`, `elastic_request_link`, `open_elastic_request_link`, `legacy_session_link_fallback`, `stable_error`, `"<unknown>"`}.
- `outcome` ∈ {`success`, `failure`, `timeout`, `canceled`, `busy`, `unavailable`, `"<unknown>"`}.
- `fallback` ∈ {`none`, `route_fallback`, `safe_retry`, `open_failed`, `connect_timeout`, `send_open_failed`, `"<unknown>"`}.
- `event` ∈ {`grant_created`, `grant_cleaned`, `link_accepted`, `link_rejected`, `link_closed`, `link_drained`, `assignment_created`, `assignment_completed`, `assignment_cleaned`, `cleanup_expired_grant`, `cleanup_closed_link`, `cleanup_canceled_assignment`, `"<unknown>"`}.
- `origin` ∈ {`background`, `user_interactive`, `system_internal`, `unknown`}; it is the low-cardinality request-origin class only, never a user/workspace/session identifier.
- `profile` ∈ {`user_interactive_read_review_v1`, `unknown`}; profile names must be registry-owned constants, not request-provided strings.
- `warm_event` ∈ {`lease_refreshed`, `exists`, `grant_created`, `connected`, `cooldown`, `skipped_capacity`, `skipped_background`, `skipped_ineligible`, `drained`, `closed`, `unknown`}.
- `reason` is a bounded reason class from the daemon transport code, such as `missing_transport_capability`, `endpoint_registry_mismatch`, `elastic_request_link_capacity`, `expired`, `replayed`, `node_mismatch`, `deadline_exceeded`, `session_generation_changed`, or `ok`; unexpected values collapse to `"<unknown>"`.
- `state` ∈ {`grants`, `links`, `assignments`, `"<unknown>"`}.

These collectors intentionally do not label by machine, workspace, agent, request ID, link ID, token hash, raw socket error, raw endpoint path, local filesystem path, or URL. Operators can still tune LinkPolicy and Elastic Request Link caps from bounded request, route-decision, lifecycle-event, and pool-size signals.

## 7. Adding New Metrics

When a new metric is introduced:

1. Pick `<area>` from §1, or amend §1 with a new area in the same change.
2. Add a row under §6 (or a new subsection §6.N) including label allowlist and bucket family.
3. Declare the collector in `internal/observability/metrics/` and reference the bucket family symbol from `buckets.go`.

When a metric is removed, delete the row and call out the removal in the commit message; downstream dashboards may break.

Spec amendments that add metrics ship in the same PR as the code that registers them. A row in this document that does not have a registered collector in code (or vice versa) is a spec drift and must be reconciled.