# Observability

> Canonical path: `specs/observability/observability.md`
> Status: living-spec
> Scope: durable contract for `agentteam-server` production observability. Defines the principles, the minimum surface the binary must expose, and the rules that any future metric, trace, or log must follow.
> Related specs: [`specs/observability/logging.md`](./logging.md), [`specs/observability/metrics.md`](./metrics.md), [`specs/storage/storage.md`](../storage/storage.md), [`docs/tech_design.md`](../../docs/tech_design.md) §4.9, [`skills/database-amplification-guardrails/SKILL.md`](../../skills/database-amplification-guardrails/SKILL.md).
> Non-goals: vendor selection; per-dashboard layout; alert thresholds; web/desktop client telemetry; pre-registering every collector the server might ever expose.

---

## 1. Purpose

Operators must be able to answer four questions in production without redeploying:

1. Which request is slow, and which downstream component made it slow.
2. What the current service load is (CPU, memory, goroutines, DB pool saturation).
3. The latency distribution and concurrency of any HTTP route or background worker.
4. What the most recent error wave looks like, with a path from a failing log line to the originating request.

This spec defines the durable contract. The actual collector list and trace propagation rules grow incrementally as code lands; this document anchors the principles those additions must obey.

## 2. Scope

In scope: the `agentteam-server` Go binary and the in-process workers it owns.

Out of scope: `desktop/` daemon, web frontend RUM, vendor-specific SDK choice, alerting policy.

## 3. Pillars

The server exposes three durable surfaces. Each is transport-neutral: code never assumes a specific vendor.

- **Metrics**: Prometheus exposition on `/metrics`, served from a listener separate from the main HTTP listener. Use `github.com/prometheus/client_golang` (already a direct dependency). Collector naming, label allowlists, and bucket families live in [`metrics.md`](./metrics.md).
- **Traces**: OpenTelemetry SDK (`go.opentelemetry.io/otel`) with OTLP/HTTP exporter. Disabled when the OTLP endpoint env var is empty. Propagator is W3C (`traceparent` / `tracestate` / `baggage`); no B3, no vendor-proprietary. Sampling is head-based via `ParentBased(TraceIDRatioBased(ratio))`. The minimum span surface when tracing is enabled is:
    - One server span per inbound HTTP request, named `HTTP <METHOD> <route-template>`.
    - One client span per outbound HTTP call that leaves the process, with `traceparent` injected into the outgoing request headers.
    - One client span per database query/exec. Database spans carry low-cardinality operation metadata such as `db.system` and `db.operation`; they do not record SQL text by default. Operators may explicitly enable caller-supplied SQL statement attributes for controlled diagnostics after auditing query construction risk.
    - Additional span boundaries (daemon WS, Socket.IO, background workers) are added in their own focused changes that amend this spec.
- **Logs**: `log/slog` defaults to JSON on stdout (already wired in `cmd/agentteam-server/main.go`); file/text overrides follow [`logging.md`](./logging.md) §8. Every request-scoped log line must carry `request_id`; once tracing is enabled, the same handler injects `trace_id` and `span_id` from `context.Context`. Cross-component fields, runtime `domain`, stack, privacy, performance, and component-specific output rules are defined by [`logging.md`](./logging.md).

Profiling reuses the existing `internal/observability/debughttp` localhost-only `/debug/pprof/*` surface gated on `AGENTTEAM_ENABLE_PPROF`. The contract is: pprof stays loopback-only forever, and continuous profiling agents are not introduced until on-demand pprof is proven insufficient.

## 4. Deployment Neutrality

The binary must work without any observability sink configured. In that state `/metrics` returns Go runtime metrics and the OTLP exporter is a no-op; nothing fails. Two reference topologies are both supported by the same binary without code changes:

- Self-hosted Grafana stack (Prometheus + Grafana + Loki + Tempo) reading the same surfaces.
- SaaS / Grafana Cloud, with the OTLP endpoint and credentials supplied through environment variables.

Anything that contradicts this is a regression of the contract.

## 5. Cardinality And Cost Discipline

These are the rules that protect both the TSDB and the application database. They apply to every metric, every span, and every log line.

### 5.1 Metric labels

- Label values must be drawn from a finite, code-controlled enumeration. Allowed examples: HTTP method, matched Gin route template (`ctx.FullPath()`), status code, `outcome=success|failure|timeout`, `driver=filesystem|s3`.
- Label values must not come from user input or row identifiers. Forbidden: `workspace_id`, `agent_id`, `channel_id`, `message_id`, `email`, raw URL path, error message string.
- The matched route template is the only acceptable HTTP path label. Unmatched routes collapse to `route="<not_found>"`.
- Histograms must declare a named bucket family from [`metrics.md`](./metrics.md) §2.

### 5.2 Trace attributes

- Tenant identifiers (workspace, member, agent ids) are allowed on span attributes — trace storage is sampled and per-trace, so cardinality cost does not blow up the way it does on metrics.
- Database span attributes must not contain query arguments, DSNs, credentials, or dynamically constructed object identifiers by default. SQL text is disabled by default. When `AGENTTEAM_OBSERVABILITY_DB_STATEMENT_ENABLED=true`, `otelsql` records the SQL string passed to `database/sql` (`db.statement`, or `db.query.text` when stable database semantic conventions are opted in). The tracing boundary does not parse, sanitize, or template that string. Operators must enable this only for deployments where query construction is known to use placeholders and not concatenate user input or secrets into the SQL text. `otelsql` SQL commenter stays disabled.
- Span names are parametric, not value-templated: `HTTP <METHOD> <route-template>`, `db.query`, `archive.pass`.

### 5.3 Log labels at the collector

The deployment-side log collector configuration (Promtail, Vector, container logging driver, etc.) extracts at most a small allowlist as labels: `level`, `service`, `deployment_env`, `instance`, `domain`. `domain` is safe only as the code-controlled runtime-domain enumeration defined by [`logging.md`](./logging.md); arbitrary component names or business identifiers are forbidden. Everything else stays in the JSON body and is queried at read time. This rule lives in the spec because Loki and similar stores degrade silently when high-cardinality labels are added in pipeline configuration.

### 5.4 Scan amplification

Periodic collectors (sampled gauges, `GaugeFunc` registrations, background samplers) must obey [`skills/database-amplification-guardrails/SKILL.md`](../../skills/database-amplification-guardrails/SKILL.md): the cost of a scrape must scale with explicit request intent, job scope, or a product-bounded limit — never with accumulated history.

Forbidden examples (any of these is a spec violation, even if the table is "still small today"):

- `SELECT COUNT(*) FROM scope_messages` on a wall-clock schedule.
- `SELECT SUM(LENGTH(content)) FROM scope_messages` on a wall-clock schedule.
- Any aggregate against `scope_fanout_outbox`, `attachments`, or other forever-growing tables.

The motivation is concrete: by current product configuration (`internal/config/config.go` `MessagesConfig`, `docs/tech_design.md` §4.4.4), `scope_messages` retains all messages indefinitely and automatic archival is disabled. A 1-minute `COUNT(*)` against an unbounded table becomes the most expensive recurring database operation within months.

Allowed shapes:

- Write-path counters incremented in the same transaction that produces or removes a row (`O(1)` per write).
- Index-only point queries bounded by product behavior (e.g. `rewind_operations` rows in `repair_required` status, bounded by operator-driven rewind activity).
- Exact-row diagnostics served by an on-demand admin endpoint under `internal/api/admin`, never by a recurring Prometheus scrape.

Current row counts and byte sums for growing tables are derived in PromQL at query time from monotonic write-path counters minus removal counters; see [`metrics.md`](./metrics.md) §5.

## 6. Ownership

A single package `internal/observability` owns the cross-cutting plumbing: metrics registry, slog handler decoration, OTLP exporter wiring, and Gin/HTTP middleware. Domain packages depend on it; it does not depend on domain packages.

The exact file layout inside `internal/observability/` is an implementation choice. The spec only mandates:

- One process-wide metric registry. Per-package `prometheus.NewRegistry()` instances are forbidden so `/metrics` reflects a single source of truth.
- Collectors are registered explicitly at startup, not in `init()` functions, so tests and helper binaries can opt out.
- `internal/observability` must not import `internal/store`, `internal/api/{domain}`, or `internal/service/<domain>`.

## 7. Configuration Surface

The runtime exposes an `ObservabilityConfig` group through `internal/config`. The supported env vars are:

| Variable | Default | Purpose |
|---|---|---|
| `AGENTTEAM_OBSERVABILITY_METRICS_ADDRESS` | `127.0.0.1:2112` | Listener for `/metrics`. Empty disables the listener. Must not equal `AGENTTEAM_HTTP_ADDRESS`. |
| `AGENTTEAM_OBSERVABILITY_OTLP_ENDPOINT` | empty | OTLP/HTTP base URL. Empty disables trace export. The runtime sends spans to `/v1/traces` under this base URL. |
| `AGENTTEAM_OBSERVABILITY_OTLP_HEADERS` | empty | Optional OTLP/HTTP exporter headers as comma-separated `key=value` pairs. |
| `AGENTTEAM_OBSERVABILITY_TRACE_SAMPLE_RATIO` | `0.05` in `production`, `1.0` otherwise | Head sampling ratio. |
| `AGENTTEAM_OBSERVABILITY_DB_STATEMENT_ENABLED` | `false` | Enables SQL statement attributes on database spans for controlled diagnostics. This records the SQL string passed to `database/sql`; enable only after auditing query construction for placeholders and no concatenated user input or secrets. |
| `AGENTTEAM_ENABLE_PPROF` | unset | Existing; loopback-only pprof endpoints. |

The default loopback bind (`127.0.0.1:2112`) is deliberate. Port `2112` is the canonical `client_golang` example port (Prometheus server itself uses `9090`, so co-locating `agentteam-server` and Prometheus on one host would clash). Loopback is the safe default for single-binary and bare-metal installs; container-and-Compose deployments override to a routable interface (typically the container's default interface, with the host-side port left unpublished).

Additional metrics/tracing env vars, such as a separate service-name override or error-sink DSN, must be introduced with the feature that needs them and must follow the `AGENTTEAM_OBSERVABILITY_*` prefix convention. Logging output, file, and rotation variables use the separate `AGENTTEAM_LOG_*` contract in [`logging.md`](./logging.md) §8.

Validation rules:

- The metrics address must not equal the main HTTP address. Splitting listeners prevents accidental public exposure of `/metrics` and pprof through a misconfigured ingress.
- `AGENTTEAM_OBSERVABILITY_TRACE_SAMPLE_RATIO` must be in `[0, 1]`.
- If `AGENTTEAM_OBSERVABILITY_OTLP_ENDPOINT` is set, it must be a valid `http(s)` URL with a host. It must not include URL user info, a query string, or a fragment. A path prefix is allowed; `/v1/traces` is appended unless the path already ends with `/v1/traces`.
- If `AGENTTEAM_OBSERVABILITY_OTLP_HEADERS` is set, it must contain comma-separated HTTP header `key=value` pairs. Header names must be valid HTTP token names, values must be non-empty, and duplicate keys are rejected.
- `AGENTTEAM_OBSERVABILITY_DB_STATEMENT_ENABLED` accepts only `true` or `false`. When it is enabled with `AGENTTEAM_ENV=production`, the binary must emit a startup `WARN` log reminding operators that tracing records caller-supplied SQL text and that query construction must be audited before enabling it.
- When `AGENTTEAM_ENV=production` and the metrics listener binds all interfaces (empty host, `0.0.0.0`, or `[::]`), the binary must emit a startup `WARN` log advising the operator to bind a loopback/private interface or to enforce ingress-level network isolation. This mirrors `internal/observability/debughttp`'s loopback-only posture for a sibling debug surface.

## 8. Current State

`internal/observability/` owns the process-wide Prometheus registry, the dedicated `/metrics` listener, Go runtime/process collectors, DB pool collectors, HTTP request collectors, the OTLP/HTTP trace exporter wiring, and the request-scoped `slog` handler decorator. The main `agentteam-server` HTTP router injects `X-Request-ID`, records the initial HTTP server metric surface, and keeps `/healthz` and `/readyz` out of request-latency metrics.

Tracing is wired through OpenTelemetry SDK boundaries owned by `internal/observability`. With an empty `AGENTTEAM_OBSERVABILITY_OTLP_ENDPOINT`, the provider uses a no-op tracer and performs no network export. With a configured endpoint, the provider creates a parent-based sampled SDK tracer provider, exports OTLP/HTTP spans with W3C `traceparent`/baggage propagation, and attaches `service.name=agentteam-server`, `service.version`, and `deployment.environment.name` resource attributes. The currently implemented span surface covers inbound HTTP server spans, outbound HTTP client spans, and SQL query spans. SQL statement attributes are off by default and can be enabled with `AGENTTEAM_OBSERVABILITY_DB_STATEMENT_ENABLED=true`; when enabled, they contain the SQL string provided by the caller rather than a tracing-layer sanitized template. The slog decorator injects `trace_id` / `span_id` when a valid span context is present.

To refresh this classification later, run:

```bash
rg -n 'prometheus\.|otel\.|otelhttp\.' internal/ cmd/ | grep -v '_test.go'
```

Every match must either live under `internal/observability/` or import it. Sections that are out of step with the code in a future review must be updated in the same change that changed the code.

## 9. Guardrail

A single guardrail script lives at `guardrails/observability/check-observability.sh`. It scans AgentTeam-owned non-`_test.go` files and intentionally excludes `x/` subtree code that carries imported or separately governed implementations. Test code legitimately constructs `prometheus.NewRegistry()` and inline fixtures, and exempting tests prevents day-one false positives. It fails when:

- A `prometheus.NewHistogramVec` / `NewCounterVec` / `NewGaugeVec` declares a label key that is not present in [`metrics.md`](./metrics.md).
- A metric is registered under one of the forbidden names listed in [`metrics.md`](./metrics.md) §4.
- Any file under `internal/observability/` contains a SQL fragment — `SELECT `, `COUNT(`, `SUM(`, or `AVG(`. This blocks the amplification risk at the entry point: the observability package must source values from in-memory counters or domain-store ports, never inline SQL. Cross-package scan-amplification review (the case where a metric pulls from a store function that does `COUNT(*)` internally) stays a human responsibility through [`skills/database-amplification-guardrails/SKILL.md`](../../skills/database-amplification-guardrails/SKILL.md) — a shell grep cannot follow call graphs and pretending otherwise weakens the guardrail.
- A metric is registered against a `prometheus.NewRegistry()` other than the central registry exposed by `internal/observability`.

Additional guardrails can be added later as concrete violations surface in code review. Splitting prematurely into many scripts adds maintenance burden without proven value. The script is deterministic and repo-root relative.

Project-wide logging discipline belongs to [`logging.md`](./logging.md), not this metrics/tracing guardrail. Its adoption is ratcheted: new or modified operational logging follows the logging contract, while existing runtime diagnostics are migrated in their owning domain instead of being hidden by a broad compatibility wrapper.

## 10. Design Trade-offs

The choices below are durable. Future revisions of this spec must engage with the rationale before changing direction.

- **Prometheus pull, OpenTelemetry SDK, OTLP/HTTP.** Pull-based scraping keeps the binary stateless and lets self-hosted Prometheus, Grafana Cloud agent, or any other consumer share the same surface. OTel decouples the application from a specific vendor. OTLP/HTTP avoids the gRPC dependency surface and works through forward proxies.
- **`slog` stays the canonical logger.** It is in the standard library, already wired, and supports handler composition. Replacing it would re-introduce a third-party logger without measurable benefit at current request volumes.
- **Metrics listener on a separate port.** Sharing the public listener risks accidental exposure of `/metrics` and pprof through a misconfigured ingress.
- **No sampled full-table size gauges, deliberately diverging from `docs/tech_design.md` §4.9.** That section lists `scope_messages` total rows and total bytes as desired observability inputs. This spec rejects them as sampled gauges per §5.4: the table is currently unbounded, so a scheduled aggregate would be the dominant database cost. The same operational insight is preserved through write-path counters minus removal counters, evaluated in PromQL at query time, with exact-row diagnostics delegated to admin-only endpoints. If history retention is later re-enabled and the table is bounded by configuration, this trade-off may be revisited.

## 11. Deferred Scope

The spec deliberately defers the following. Each may be added through a focused follow-up spec amendment when the corresponding code lands:

- Trace propagation across the daemon WebSocket protocol (would change the daemon protocol envelope; out of scope until daemon-side support is also decided).
- Trace propagation across Socket.IO realtime emits (browser side is out of scope for now).
- An optional error-aggregation sink (Sentry or equivalent) as a `slog.Handler` decorator.
- Exporting metrics through the OTel meter API in addition to Prometheus exposition.
- Tail-based sampling configuration on a collector sidecar.

The agentteam-server observability surface consists of Prometheus `/metrics`, Gin HTTP request/duration/inflight collectors, Go runtime + DB pool collectors, the cardinality+scan guardrail script, a `request_id`-bearing slog handler, and the minimum tracing surface: inbound HTTP server span, outbound HTTP client span, DB query span, W3C `traceparent` propagation, and OTLP/HTTP export when configured. Later work should extend the surface only through focused changes tied to one of the deferred items above.