--- title: Shared Observability Requirements document_id: shared-observability-requirements owner: Product Spec Maintainers status: draft created: 2026-03-16 last_updated: 2026-03-23 --- # Shared Observability Requirements Feature specs should reference the applicable observability requirements from this document instead of redefining standard telemetry expectations. If a feature cannot meet one of these requirements, the PRD should state the gap and any compensating operator guidance. ## How to Use - Reference relevant `OBS-###` IDs in PRDs, architecture docs, and rollout plans. - Add feature-local telemetry only for behavior not already covered by shared observability expectations. - Reuse existing product conventions before inventing new metrics, log formats, or dashboards. - In PRDs, state whether RED, USE, both, or neither is the right telemetry model for the feature surface when `OBS-001` applies. ## OBS-001 Health and Progress Metrics Requirement Statement: The feature must expose metrics that indicate health, progress, and backlog or lag where those concepts are relevant to operation. Feature owners must explicitly consider which telemetry model best fits the product surface: request/response or API-serving surfaces should usually define RED-style metrics, while resource-managing, worker, storage, or infrastructure surfaces should usually define USE-style metrics. Some features may require both. If neither model applies cleanly, the PRD must say why and define the substitute telemetry model. Applicability Notes: RED stands for Rate, Errors, and Duration. It is usually the right model for request/response, RPC, API, query-serving, or other user-facing transaction paths where operators need to understand traffic volume, failure rate, and latency from the caller's perspective. USE stands for Utilization, Saturation, and Errors. It is usually the right model for workers, schedulers, queues, storage engines, replication pipelines, controllers, or other resource-managing components where operators need to understand whether a system resource is busy, constrained, or failing internally. Some features span both service and infrastructure behavior and should use both models. For example, an API surface may need RED-style request metrics while its backing queue, worker pool, or exporter path needs USE-style resource metrics. Typical metrics may include request rate, error rate, latency, throughput, retry counts, replication lag, queue depth, concurrency, resource utilization, saturation indicators, and last successful progress timestamp. This requirement is about choosing an appropriate telemetry shape and coverage model, not requiring a fixed universal metric inventory. ## OBS-002 Error Visibility Requirement Statement: User-impacting errors and persistent retries must be visible through supported telemetry surfaces rather than requiring deep manual inspection of internal state. Applicability Notes: Applies to both transient error storms and hard failure states. ## OBS-003 Structured Logs for Debugging Requirement Statement: Operationally significant events should be logged in a structured, queryable form with stable field names where the product already uses structured logging. Applicability Notes: Logs should help correlate component identity, resource scope, and failure reason without leaking sensitive data. ## OBS-004 Alertable Failure Signals Requirement Statement: The feature should produce telemetry that can drive practical alerts for sustained unhealthy states, not just post-hoc forensic analysis. Applicability Notes: Examples include stalled progress, growing lag, error-rate spikes, and repeated failover loops. ## OBS-005 Traceability Across Component Boundaries Requirement Statement: When a workflow spans multiple components or services, the feature should preserve enough correlation context to support end-to-end debugging. Applicability Notes: This may use tracing, correlation IDs, request IDs, or message metadata depending on the product surface. ## OBS-006 Dashboard Compatibility Requirement Statement: New telemetry should fit existing dashboard and operational reporting patterns whenever possible so operators can reason about the feature alongside established system signals. Applicability Notes: Avoid creating isolated telemetry surfaces unless there is a strong product reason. ## OBS-007 Rollout and Migration Visibility Requirement Statement: Features that change traffic patterns, data formats, or operational workflows must provide telemetry that helps operators validate rollout progress and detect migration regressions. Applicability Notes: Applies strongly to upgrades, protocol changes, and replacement of existing pipelines. ## OBS-008 Operator-Debuggable State Requirement Statement: The feature should expose enough state through supported interfaces to let operators determine what it is doing, what it is waiting on, and why it is failing. Applicability Notes: Examples include checkpoint position, active configuration summary, sink status, or last failure reason. ## OBS-009 Convention Consistency Requirement Statement: Metric names, labels, log severity usage, and status surfaces should follow existing product conventions unless a documented exception is needed. Applicability Notes: Consistency reduces cognitive load and simplifies automation. ## OBS-010 Low-Surprise Telemetry Cost Requirement Statement: Observability added by the feature should avoid disproportionate overhead in normal operation and should document any materially expensive debug mode or high-cardinality dimension. Applicability Notes: Relevant to verbose logging, tracing fan-out, and per-resource metric labeling.