# TiCDC Cloud Storage WATERMARK Boundary

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "ticdc-cloud-storage-watermark-boundary-2026-08-14",
  "problems": [
    {
      "problemId": "ticdc-watermark-mq-and-cloud-storage-scope-conflated",
      "summary": "The direct-answer support guidance can apply TiCDC MQ WATERMARK message behavior to a Cloud Storage sink and incorrectly claim that TIDB_WATERMARK appears inline in CDC*.json data files.",
      "status": "optimized_verified",
      "applicability": "Direct product-behavior answers about TiCDC Canal-JSON WATERMARK placement or progress semantics where the customer identifies or compares an MQ sink such as Kafka or Pulsar with a Cloud Storage sink such as Amazon S3, GCS, Azure Blob Storage, or NFS.",
      "issueKeys": ["TCID-13011"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/TCID-13011#problem:ticdc-watermark-mq-and-cloud-storage-scope-conflated"
      ],
      "signalTextIncludesAny": [
        "sink does not write tidb_watermark records into the cdc*.json data files",
        "watermark/checkpoint events exist only in the message stream for mq sinks",
        "progress is tracked exclusively by the top-level global metadata object"
      ],
      "observedThrough": "2026-08-11",
      "effectiveAt": "2026-08-14T09:14:26Z",
      "verifiedAt": "2026-08-14T09:31:47.635Z",
      "attemptedAction": "Scope the established periodic WATERMARK, independent-control, and consumer-filtering guidance to MQ message streams, and give Cloud Storage answers the separate no-inline-CDC-file and top-level metadata.checkpoint-ts boundary.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/TCID-13011?focusedCommentId=23454382"
    }
  ]
}
-->

- Optimization ID: `ticdc-cloud-storage-watermark-boundary-2026-08-14`
- Owner: auto-oncall product knowledge consumed by Primary Analysis and the
  Scenario Router
- Source reference: TCID-13011 internal answer and `/ao wrong` review
- Implementation status: deployed to staging in image `8e5847bd85933d4fc5aae445e1236364d3a18763`
- Effective status: `optimized_verified`

## Bounded problem

TCID-13011 explicitly asked whether an Amazon S3 Cloud Storage sink writes
`TIDB_WATERMARK` inline in `CDC*.json` files or uses only the top-level metadata
checkpoint. The persisted direct answer instead applied the existing Canal-JSON
message-stream guidance to S3 and claimed that a WATERMARK could appear between
normal DML records in the same data file.

The existing guidance came from ST-2835, where a customer had already observed
an actual `TIDB_WATERMARK` message and the support discussion used the Kafka
path. That optimization correctly established event meaning, control limits,
parent-extension effects, and consumer filtering for a message stream. It did
not verify physical placement for Cloud Storage sinks.

## Candidate change

- Require the direct-answer Agent to identify MQ versus Cloud Storage before
  applying the established WATERMARK boundary.
- Preserve the verified Kafka/Pulsar behavior: periodic control messages may
  continue without DML, no independent disable or frequency setting exists,
  and consumer-side filtering is the least disruptive default.
- For Amazon S3, GCS, Azure Blob Storage, and NFS, prohibit an inference that a
  generic Canal-JSON WATERMARK description means an inline record exists in
  `CDC*.json`. Use the top-level `{prefix}/metadata` `checkpoint-ts` as the
  global storage completeness barrier.

This changes one independent product-knowledge topic shared by Primary Analysis
and the Scenario Router. It does not put the fact in response-format policy and
does not change Scenario classification, Provider selection, diagnostic
execution, or message ownership. Final Response still receives only the
Router's canonical decision.

## Verification evidence

- Persisted job `120041`, DB run `120043`, and run
  `4e849b9c-6bec-42b9-885d-c0df68198f00` completed with
  `feature_usage_or_capability`, `direct_answer`, and `provider=none`.
  Response generation completed, evidence acquisition recorded no failure, and
  all four web searches succeeded. This rules out a Provider or access failure.
- Jira internal comment `23454382` contains the incorrect inline-file claim.
  The later L1 response and `/ao wrong` comment `23519063` establish that S3
  uses the top-level metadata checkpoint and does not contain an inline
  WATERMARK record; the reviewer said the rest of the answer was correct.
- Service PR `tidbcloud/aisupport#917` and app-delivery PR
  `tidbcloud/app-delivery#77304` deployed image
  `8e5847bd85933d4fc5aae445e1236364d3a18763`; the staging Pod became Ready at
  `2026-08-14T09:14:26Z` with zero restarts and both health endpoints passing.
- Historical read replay `b6d01da4-c1d3-451b-aa75-4675f75b6539` rejected an
  inline `TIDB_WATERMARK` completeness contract for S3, selected top-level
  `metadata.checkpoint-ts`, and kept MQ message-stream behavior separately
  scoped. Its Judge verdict was `ai_better` with no AI gap.
- Write replay `c1518f2a-58d9-49ab-916b-03cd1c18f0f2` persisted DB run `150044`
  and updated Jira internal comment `23454382` in place. Jira readback found
  exactly one Auto-oncall comment, `public=false`, with the expected paired body,
  no-inline S3 boundary, and metadata checkpoint guidance.

## Unresolved gaps

- Public Cloud Storage documentation is ambiguous about WATERMARK emission
  versus physical file placement. Correcting that external documentation is
  outside this repository optimization.

## Next validation action

Treat a later answer that applies MQ WATERMARK placement to Cloud Storage as a
new regression. Keep external documentation clarification separate from this
verified repository behavior.