# Finding Intake API ## 1. Purpose The Finding Intake API is the single external entry point for SQL discoveries produced by Clinic, AI agents, manual input, and other integrations. The producer is responsible for discovery: collecting data, applying cluster-specific rules or human judgment, deciding that an anomaly exists, and assigning each symptom severity. SQL Catalog does not execute inspection thresholds or re-grade the submitted severity. It validates the common contract, records producer assessment provenance, links the event to a Catalog Item, and starts the post-detection governance lifecycle. A Finding is an immutable external event for exactly one: ```text Organization + Cluster + SQL digest + Detector execution ``` A Finding can contain multiple normalized symptom hits. The API keeps three measures separate: - Finding events: external messages accepted by SQL Catalog. - Symptom hits: normalized symptoms contained by those events. - Affected SQL: distinct Catalog Items or SQL digests. An upstream report containing multiple SQL digests must submit one Finding per digest. Use a Finding Run to group all Findings from the same report or execution. ## 2. Symptom Registry Every `symptoms[].code` must be an active definition returned by: ```http GET /api/v1/symptom-definitions ``` The Registry is a controlled semantic vocabulary. It defines what a symptom means, but contains no cluster thresholds and performs no inspection. Producers keep their own detector and policy logic, then map the result to a registered symptom code. A source-specific inspection item remains visible through `detector_code` and does not need to map one-to-one to a symptom. `unclassified_sql_anomaly` is a manual-only escape hatch. It requires `symptoms[].assessment.summary` and should be replaced by a registered specific definition when the same anomaly becomes recurring. ## 3. Endpoints ```http POST /api/v1/findings POST /api/v1/finding-batches Authorization: Bearer Content-Type: application/json ``` The single endpoint is convenient for manual creation and streaming integrations. The batch endpoint accepts up to 500 Findings and should be preferred for reports and accumulated Agent results. Both use identical validation and event idempotency. The MVP server accepts requests up to 2 MiB. All requests require an Auth0 access token. The current interactive application supplies a PingCAP user token. Automated Clinic and Agent producers will require a dedicated Machine-to-Machine client when service ingestion is enabled; that service credential flow is not part of the current auth slice. Producers must never store or reuse a person's browser token. ### 3.1 Controlled historical replay Normal producers cannot set `ingested_at`, `created_at`, Catalog registration time, relation time, or Timeline time. The checked-in Clinic demo importer uses the internal `X-SQL-Catalog-Replay-At: ` request header on Finding Run, batch Finding, and single Finding requests so a historical report behaves as though SQL Catalog received it when that report was created. The replay time is applied consistently to every system-owned business timestamp created by that request; `reported_at` and `observed_at` remain explicit source facts. This header is enabled by default only when the server binds to a loopback host. A non-loopback deployment rejects it unless `SQL_CATALOG_ALLOW_HISTORICAL_REPLAY=true` is explicitly configured. Future replay timestamps are rejected. This is a controlled import facility, not part of the external producer contract. ### 3.2 Human-entry UI `/findings/new` is intentionally smaller than this external contract. A human selects one Cluster through Clinic search; SQL Catalog derives the Organization from that Cluster. The user then enters the SQL digest, optional normalized SQL, one or more registered Symptoms with severity and a human assessment, observed time, and an optional evidence link. The UI generates `source=manual`, `source_type=manual`, `source_event_id`, `detector_code=manual.user_report`, and actor metadata. It does not ask a human to enter Detector, metric schema, metric JSON, signal codes, metric keys, raw payload, Organization ID, or Cluster ID. Those fields remain available to machine producers through this API. Focus Scope does not constrain the Clinic Cluster search. An exact Cluster scope supplies an editable initial value; Organization and Cluster Group scopes do not filter the available Clusters. ## 4. Finding Request ### 4.1 Top-level fields | Field | Required | Type | Constraints and meaning | |---|---:|---|---| | `source` | Yes | string | Stable producer name, up to 128 characters, for example `clinic_health_check`. | | `source_type` | Yes | enum | `inspection`, `manual`, `ai_agent`, or `integration`. | | `source_event_id` | Yes | string | Producer-owned event ID, up to 512 characters. Must be stable across retries. | | `source_batch_id` | No | string | Transport batch ID, up to 512 characters. It groups retries and supports source-batch deletion; it is not a Finding Run. | | `source_run_id` | No | string | Producer-owned report or execution ID. Links to a previously submitted Finding Run when present. | | `org_id` | Yes | string | Organization identity, up to 128 characters. | | `project_id` | No | string | Project identity, up to 128 characters. | | `cluster_id` | Yes | string | Cluster identity, up to 128 characters. | | `sql_digest` | Yes | string | SQL digest, up to 256 characters. | | `normalized_sql` | No | string | Normalized SQL text, up to 200,000 characters. | | `reported_at` | No | ISO 8601 string | When the producer created or published this Finding event. It drives the default Finding feed and trend. Defaults to the SQL Catalog receive time when omitted. Historical importers should provide it explicitly. | | `observed_at` | Yes | ISO 8601 string | When the producer observed the event. An offset is required in practice; the server normalizes to UTC. | | `time_window_start` | Conditional | ISO 8601 string | Start of the inspected data window. Must be supplied together with `time_window_end`. | | `time_window_end` | Conditional | ISO 8601 string | End of the inspected data window and must not precede the start. | | `detector_code` | Yes | string | Stable detector identity, for example `clinic.slowlog_top_digest`. | | `detector_version` | No | string | Producer or rule version, up to 128 characters. | | `symptoms` | Yes | array | One to 32 structured symptom objects. Duplicate symptom codes are rejected. | | `metric_schema_version` | Conditional | string | Required when `metrics` is not empty. Identifies the exact metric contract. | | `metrics` | No | object | Shared detector output. Defaults to `{}`. | | `dimensions` | No | object | Source-specific dimensions such as ranks, instances, or check names. Defaults to `{}`. | | `evidence_refs` | No | array | Shared event evidence. Each entry must be an object with a non-empty `type`. Maximum 100 entries. | | `raw_payload` | No | object | Source payload retained for audit and replay. Defaults to `{}`. | The audit actor is derived from the authenticated application identity. Producers must not rely on a client-supplied `created_by` value. Unknown top-level fields are rejected. In particular, callers must not submit `severity`, `catalog_item_id`, `is_new_digest`, `created_at`, or other system-owned values. ### 4.2 Symptom fields | Field | Required | Type | Constraints and meaning | |---|---:|---|---| | `code` | Yes | string | Normalized symptom code, unique inside the Finding. | | `severity` | Yes | enum | `critical`, `high`, `medium`, `low`, or `info`. | | `confidence` | No | number | Detector confidence from `0` to `1`. Omit when the detector does not provide confidence. | | `signal_codes` | No | string array | Signals supporting this symptom, for example `slowlog.top_by_total_keys`. | | `metric_keys` | No | string array | Keys in the parent `metrics` object that support this symptom. Unknown keys are rejected. | | `evidence_refs` | No | object array | Evidence specifically supporting this symptom. Each object requires `type`. | | `assessment` | No | object | Provenance explaining how the producer reached this severity. Recommended for every machine source and required for `unclassified_sql_anomaly`. | `assessment` supports: | Field | Required | Type | Meaning | |---|---:|---|---| | `method` | No | enum | `rule`, `model`, `human`, or `external`. | | `summary` | No | string | Human-readable explanation, up to 2,000 characters. | | `rule_code`, `rule_version` | No | string | Producer rule identity and version. | | `policy_id`, `policy_version` | No | string | Producer-owned policy or cluster profile identity and version. | | `policy_url` | No | string | Link to the source policy or inspection configuration. | | `parameters` | No | object | Relevant threshold or decision snapshot needed to explain this result. | The producer owns symptom severity. The Finding event severity is response-only and is the highest submitted symptom severity. SQL Catalog performs this aggregation but does not recalculate any child severity. `confidence` is certainty in the diagnosis; it is not an impact grade. ## 5. Complete Example ```json { "source": "clinic_health_check", "source_type": "inspection", "source_event_id": "clinic:report-20260710:sql_digest:abc123", "source_run_id": "report-20260710", "org_id": "org-example-001", "cluster_id": "cluster-example-001", "sql_digest": "abc123", "normalized_sql": "select * from orders where customer_id = ?", "reported_at": "2026-07-10T07:00:15.636Z", "observed_at": "2026-07-10T07:00:15.636Z", "time_window_start": "2026-07-09T07:00:15.636Z", "time_window_end": "2026-07-10T07:00:15.636Z", "detector_code": "clinic.health_check.sql_digest_report", "detector_version": "clinic-health-check-v1", "metric_schema_version": "clinic.slowlog_top_digest/v1", "metrics": { "slow_count": 2096, "total_latency_s": 3734.286, "max_latency_s": 32.655, "total_keys": 1250000 }, "symptoms": [ { "code": "high_total_latency", "severity": "critical", "signal_codes": ["slowlog.top_by_total_latency"], "metric_keys": ["total_latency_s", "max_latency_s"], "assessment": { "method": "rule", "summary": "Clinic evaluated this digest using the cluster health-check policy.", "rule_code": "clinic.slowlog_top_digest.high_total_latency", "rule_version": "clinic-health-check-v1", "policy_id": "latency-sensitive-oltp", "policy_version": "7", "parameters": { "high_max_latency_s": 10, "critical_max_latency_s": 30 } }, "evidence_refs": [ { "type": "clinic_health_check_result", "dimension": "top_by_total_latency", "rank": 9 } ] }, { "code": "large_key_scan", "severity": "high", "signal_codes": ["slowlog.top_by_total_keys"], "metric_keys": ["total_keys"] } ], "dimensions": { "check_names": ["slow_query_hugeResourceUsage", "slowlog_top_digest"], "top_digest_ranks": { "top_by_total_latency": 9, "top_by_total_keys": 9 } }, "evidence_refs": [ { "type": "clinic_health_check_report", "url": "https://clinic.pingcap.com/portal/#/orgs/..." } ], "raw_payload": { "report_id": "report-20260710" } } ``` ### 5.1 Governance signal role `signal_class` is response-only. It describes how the Catalog governance layer uses the accepted Finding: - `anomaly`: eligible for review admission, current risk, and governance priority; - `ranking_observation`: retained as ranking, metric, and trend evidence but does not independently create human review work. The Clinic Health Check server adapter marks a digest as `ranking_observation` only when it appears solely in `slowlog_top_digest`; any supported anomaly check on the same digest makes the merged Finding an `anomaly`. Existing Clinic records are projected from `dimensions.check_names` with the same rule. Other producers default to `anomaly`. Callers must not rely on a custom Dimensions value to suppress review work; source adapter mappings are owned by SQL Catalog. ## 6. curl Example ```bash curl -sS http://127.0.0.1:8765/api/v1/findings \ -H "authorization: Bearer $SQL_CATALOG_ACCESS_TOKEN" \ -H 'content-type: application/json' \ -d '{ "source": "example_agent", "source_type": "ai_agent", "source_event_id": "agent-run-42:digest-abc", "org_id": "org-1", "cluster_id": "cluster-1", "sql_digest": "digest-abc", "observed_at": "2026-07-10T08:00:00Z", "detector_code": "agent.sql_review", "detector_version": "2026-07-10", "symptoms": [ { "code": "large_key_scan", "severity": "high", "confidence": 0.91, "signal_codes": ["plan.full_table_scan"], "assessment": { "method": "model", "summary": "The agent found a scan amplification pattern and rated its observed impact.", "policy_id": "agent-production-impact-v2" } } ] }' ``` ## 7. Batch Request The batch envelope owns a stable `source_batch_id`; `source` and `source_type` apply to every item. `defaults` are merged into each item before item fields override them: ```json { "source": "clinic_health_check", "source_type": "inspection", "source_batch_id": "report-20260710:part-1", "defaults": { "source_run_id": "report-20260710", "org_id": "org-example-001", "cluster_id": "cluster-example-001", "observed_at": "2026-07-10T07:00:15.636Z", "detector_code": "clinic.slowlog_top_digest" }, "findings": [ { "source_event_id": "report-20260710:digest-a", "sql_digest": "digest-a", "symptoms": [{ "code": "large_key_scan", "severity": "high" }] }, { "source_event_id": "report-20260710:digest-b", "sql_digest": "digest-b", "symptoms": [{ "code": "slow_query_frequency", "severity": "medium" }] } ] } ``` The response contains `created`, `duplicate`, and `rejected` totals plus one result per item. A batch with any rejected item returns `207 Multi-Status`; valid items are still accepted and can be retried safely by `source_event_id`. A batch with only duplicates returns `200`; a fully valid batch that creates at least one Finding returns `201`. ## 8. Response and Idempotency First acceptance returns `201 Created`: ```json { "created": true, "finding": { "id": "fnd_...", "severity": "critical", "signal_class": "anomaly", "symptom_count": 2, "symptoms": [ { "id": "sym_...", "finding_id": "fnd_...", "code": "high_total_latency", "severity": "critical", "confidence": null, "signal_codes": ["slowlog.top_by_total_latency"], "metric_keys": ["total_latency_s", "max_latency_s"], "evidence_refs": [], "assessment": { "method": "rule", "policy_id": "latency-sensitive-oltp", "policy_version": "7" } } ], "catalog_item_id": "cat_...", "is_new_digest": true, "is_first_observation": true, "ingested_at": "2026-07-10T08:00:01Z", "created_at": "2026-07-10T08:00:01Z" }, "catalog_item": { "id": "cat_...", "lifecycle": "active", "current_posture": { "state": "needs_review", "risk": "critical", "attention": "urgent", "reasons": ["critical_without_treatment", "new_findings", "unowned"], "next_action": "review", "review_admission": { "eligible_finding_count": 1, "ranking_observation_count": 0 }, "review_priority": { "tier": "p0", "recommended": true, "reasons": ["current_critical"] } } } } ``` Idempotency is defined by: ```text source + source_event_id ``` Retrying the same event returns `200 OK`, `created: false`, and the original Finding. Producers must not reuse a `source_event_id` for changed content; use a new event ID for a new observation. `source_batch_id` is a transport grouping key. It does not replace per-item idempotency and it does not mean that an inspection execution completed. ## 9. Finding Run Submit a run before its Findings when a report or scheduled check must be tracked, including successful runs with zero Findings: ```http POST /api/v1/finding-runs ``` Required fields are `source`, `source_type`, `source_run_id`, `org_id`, `cluster_id`, `time_window_start`, `time_window_end`, and `status`. Status is one of `running`, `success`, `partial`, or `failed`. Optional coverage fields are `detector_version`, `expected_detector_count`, `completed_detector_count`, `started_at`, `completed_at`, `project_id`, and `metadata`. The run idempotency key is: ```text org_id + cluster_id + source + source_run_id ``` Findings reference it through the external `source_run_id`; the server fills the internal `finding_run_id`. ## 10. Delete and Restore Finding facts are immutable, but erroneous input can be deleted through an audited logical retraction. Default feeds, statistics, Catalog rollups, and new Treatment selection exclude deleted Findings; direct detail remains available with deletion metadata. ```http POST /api/v1/finding-deletions/preview POST /api/v1/finding-deletions POST /api/v1/finding-restorations ``` Selection supports explicit IDs or a server-side Finding filter with exclusions. Query-based selection enables cross-page deletion and source-run/source-batch rollback: ```json { "selection": { "filter": { "source_batch_id": "report-20260710:part-1" }, "exclude_ids": ["fnd_keep_this_one"] } } ``` Preview returns `matched_count`. Deletion repeats the normalized selection and requires that value as `expected_count`, plus a non-empty `reason`. If the count changed after preview, the server returns `409 finding_selection_changed` and does not delete anything. Restoration accepts an `ids` array. Deletion stores `deleted_at`, `deleted_by`, `deletion_reason`, and `deletion_id`, recalculates first observation and Catalog rollups, and hides an empty Catalog Item only when it has no Treatment or Record. Treatments are never cascade-deleted. ## 11. Validation Errors Invalid requests return `400 Bad Request`: ```json { "error": { "code": "invalid_finding", "message": { "key": "api:errors.invalid_finding", "text": "The Finding is invalid." } } } ``` Send `Accept-Language: zh-CN` to receive Chinese product-owned messages. Callers must branch on the stable `error.code`, not localized text; see [Internationalization](../i18n.md). Typical causes are missing required fields, an empty symptom array, duplicate symptom codes, unregistered symptom codes, invalid severity or confidence, an incomplete time window, metrics without a schema version, metric keys absent from `metrics`, and removed or unknown fields. ## 12. Intake Side Effects On first acceptance SQL Catalog atomically: 1. Creates the immutable Finding and its Finding Symptoms. 2. Finds or creates the Catalog Item by `org_id + cluster_id + sql_digest`. 3. Records whether this event created the Catalog Item. 4. Recomputes the first observation when late data arrives. 5. Writes relations and Timeline events. 6. Links the Finding to the single active Treatment for the Catalog Item when one exists.