# PingCAP HubSpot Connector Dead-Event Recovery

This companion pack owns the PingCAP CRM adaptation for HubSpot contact-to-Lead
inbound events. A webhook request that returns `202` has already been durably
accepted into `_System_ConnectorEvent`; a later mapping, validation, or flow
failure should be recovered from that saved event instead of re-posting a
synthetic payload.

## What Is Saved

`_System_ConnectorEvent` keeps the accepted event envelope:

- `Payload`: the original connector event payload used by the worker
- `Status`: `pending`, `processing`, `processed`, `failed`, or `dead`
- `RetryCount`, `LastError`, `LastErrorAt`
- `EventKind`, `Namespace`, `IdempotencyKey`
- `ReplayOfId`, `ReplayedAt`
- `SourceObjectKind`, `SourceObjectID`, `SourceLaneKey`

The operator list API intentionally omits raw `Payload` and `LastError`; use DB
access only in a trusted operator shell when those values are needed for RCA.

## Find Dead HubSpot Events

Preferred read-only API:

```bash
curl -sS "$BASE_URL/api/connectors/hubspot-crm-connector/events/dead?limit=100&offset=0" \
  -H "Authorization: Bearer $ACCESS_TOKEN"
```

Trusted DB readback when API metadata is not enough:

```sql
SELECT Id, Namespace, EventKind, IdempotencyKey, Status, RetryCount,
       LastErrorAt, LEFT(LastError, 500) AS LastErrorSample,
       JSON_UNQUOTE(JSON_EXTRACT(Payload, '$.objectKind')) AS ObjectKind,
       JSON_UNQUOTE(JSON_EXTRACT(Payload, '$.sourceID')) AS SourceID,
       ReplayOfId, ReplayedAt, CreatedDate, LastModifiedDate
  FROM _System_ConnectorEvent
 WHERE Namespace = 'hubspot-crm-connector'
   AND Status IN ('failed', 'dead')
 ORDER BY LastErrorAt DESC, CreatedDate DESC
 LIMIT 100;
```

## Known CRM Recovery Classes

### HubSpot lead status typo or unknown value

The contact-to-Lead mapping now translates common HubSpot `hs_lead_status`
values to active PingCAP `Lead.Status` values. The live typo variants
`No Engagement` and `No Enagement` normalize to `OPEN`. Unknown status values
are skipped instead of being written into `Lead.Status`, so a new unknown HubSpot
status should no longer dead-letter the whole event.

If an older event is already `dead` with a status picklist error, apply this
companion pack version, replay the event, and verify `Lead.Status` is active or
unchanged.

### Oversized HubSpot free-text values

High-risk HubSpot free-text fields are truncated in metadata before they write
to bounded CRM text columns:

- `company` -> 255
- `firstname` -> 40
- `lastname` -> 80
- `jobtitle` -> 255
- `first_conversion_event_name` -> 255
- `hs_analytics_source_data_1` -> 255
- `hs_analytics_source_data_2` -> 255

Do not edit `_System_ConnectorEvent.Payload` to trim a saved event. If another
HubSpot field still fails with a string-length error, treat the event as durable
evidence, add the narrow companion-pack mapping transform or field policy, apply
the pack, and then replay.

## Replay

Use the admin replay API after the fixed companion metadata is active:

```bash
curl -sS -X POST "$BASE_URL/api/connectors/events/$EVENT_ID/replay" \
  -H "Authorization: Bearer $ACCESS_TOKEN"
```

The response returns the new replay event id:

```json
{"eventId":"evt_..."}
```

Replay copies the saved payload and binds it to the current active route. For
HubSpot workflow-action recovery, do not build a top-level synthetic payload;
the parser expects the HubSpot-shaped nested envelope (`origin`, `context`,
`object`, and `inputFields`) that was saved from the real accepted request.

## Verify Recovery

Check the replay event and the original event:

```sql
SELECT Id, Status, RetryCount, ReplayOfId, ReplayedAt, ProcessedAt,
       LastErrorAt, LEFT(LastError, 500) AS LastErrorSample
  FROM _System_ConnectorEvent
 WHERE Id IN ('<original-event-id>', '<replay-event-id>');
```

Then verify the CRM side effect by the saved HubSpot source id or email:

```sql
SELECT Id, Email, HubSpot_Id__c, Status, FirstName, LastName, Company,
       First_Conversion__c,
       HubSpot_Original_Source_Drill_Down_1__c,
       HubSpot_HubsOriginal_Source_Drill_Down_1__c
  FROM Lead
 WHERE HubSpot_Id__c = '<hubspot-contact-id>'
    OR Email = '<hubspot-contact-email>'
 LIMIT 5;
```

Recovery is successful only when the replay row reaches `processed` and the
expected Lead or CampaignMember change is visible in the CRM tables. A `202`
from a webhook post or replay call is only enqueue proof.

## When To Escalate To L0

Do not add HubSpot or CRM-specific policy to L0. Escalate only if current
runtime evidence shows a generic substrate gap, such as:

- accepted events do not preserve the original payload
- `ReplayDeadEvent` cannot create a pending row from a dead event
- replay loses the source lane and changes per-record ordering
- the admin replay API cannot bind to the current active route

As of this companion-pack fix, the observed status and string-length failures
are L1 CRM mapping concerns, not L0 substrate gaps.