# Pipeline failure alerts to Feishu

Cloud Monitoring's existing `pingcap-datapipelines job failure` policy watches
failed Cloud Run executions whose job names start with `pingcap-`. A Pub/Sub
notification channel now routes failures through the authenticated
`pingcap-alert-relay` Cloud Run service to a signed Feishu custom bot. Existing
notification channels remain attached. The bridge covers application failures,
startup failures, OOM exits and timeouts reported as failed executions.

Only open incidents are sent. Closing a metric window is not evidence that
source extraction or downstream data recovered. Messages contain a job name,
UTC time, project and Cloud Run/Logging links. Free-form incident summaries,
raw logs and source/business payloads are not forwarded.

The webhook and signing key live only in Secret Manager secrets
`pingcap-alert-feishu-webhook` and `pingcap-alert-feishu-signing-secret`.
The service identity can access those two secrets and its receipt bucket; it
has no pipeline source or warehouse credentials. Only the dedicated Pub/Sub
push identity has service-level `run.invoker`. The webhook is never placed in
Terraform variables/state, GitHub secrets or source control.

Signing follows the [Feishu custom bot protocol](https://open.feishu.cn/document/client-docs/bot-v3/add-custom-bot):
HMAC-SHA256 uses `timestamp + newline + secret` as the key and an empty message,
then Base64 encoding. Delivery requires both HTTP 200 and Feishu code zero.
Rejected/network-failed delivery returns an error so Pub/Sub retries with
60–600 second backoff for up to the one-day retention window. Successful
incident IDs are hashed into seven-day GCS receipts to suppress redelivery.
Delivery is at least once: a crash after Feishu accepts but before the receipt
is saved can still produce a duplicate.

## Deployment

Terraform owns the topic, subscription, service, identities, secret containers,
receipt bucket and notification channel. Populate secret versions out of band,
before deploying the service. Apply a reviewed bounded plan for alert resources
and the existing alert policy; do not apply unrelated pipeline drift. The
GitHub Deploy workflow updates the relay image after it has been provisioned.

For an end-to-end check, execute a temporary `pingcap-alert-smoke-*` Cloud Run
Job that exits nonzero without accessing business data. Monitoring must publish
the incident, authenticated Pub/Sub push must reach the relay, and a stored
`feishu_alert_sent` receipt must show code zero. These jobs are labelled
`【测试告警】` in the message. Delete the temporary job after the receipt is
verified. A direct webhook POST alone is not full-chain acceptance.

Release, test execution and delivery receipts are maintained in the implementing
PR body, which is the authoritative acceptance record for this rollout.