# JSM Production v2 handoff

Last verified: 2026-07-24 (Asia/Shanghai)

This is the starting point for a new agent working on the JSM Slack/Lark v2
stack. Read this file before changing or deploying anything. Never infer the
deployed state from an old chat transcript: check Git, AWS, and the relevant
SaaS callback configuration again.

## Executive summary

- Working branch: `agent/jsm-production-v2-prep`
- Deployed source commit for the current Slack/Lark code: `38f3186`
- The branch was clean and matched `origin/agent/jsm-production-v2-prep` before
  this handoff document was added.
- The isolated v2 implementation lives entirely under `jsm-production-v2/`.
- **Production has completed the v2 migration. All current Slack, Lark, issue
  creation, comment synchronization, and org snapshot traffic must use the v2
  stack. The original non-v2 implementation is deprecated.**
- Four v2 Lambdas are deployed in AWS account `552185537300`, region
  `us-west-2`, on Node.js 24.
- Slack, Lark, and the issue worker use an isolated v2 SQS queue. The worker's
  event source mapping is enabled.
- The org synchronizer runs every 24 hours and stores production API snapshots
  for 18 configured organizations.
- All locally available v2 tests pass: 77 tests across configuration, Slack,
  Lark, issue creation, and org synchronization.
- ST, TCID, and TCLID now target the formal `Submit a request` request types.
- APID/self-managed issue creation uses the legacy production Bot's atomic
  JSM Customer Request API path.
- The current Slack v2 handler supports a `status_change` webhook for
  `Resolved`. Lark v2 does not yet correctly support that event.

## Safety boundary

### Active versus deprecated stack

The v2 stack is the only active production implementation:

- `slack-bot-v2`
- `lark-support-bot-v2`
- `CreateJsmIssueHandler-v2`
- `jsm-orgs-lambda-v2`
- `JsmBotTicketQueue-v2`

The original non-v2 Lambda functions and source directories are deprecated.
Do not add features to them, deploy them, point callbacks back to them, or use
their configuration as the desired production state. They may be inspected
only as historical reference when explaining old behavior. All fixes and
configuration changes belong under `jsm-production-v2/` and must be deployed
to the matching v2 resources.

The v2 AWS resources are separate from the original production Lambda and
queue resources, but the stack intentionally reuses the production database
and these existing tables:

- `messages`
- `ticket_thread`
- `user_openid_email_mapping`

It adds these snapshot tables:

- `jsm_syncer_org_snapshots`
- `jsm_syncer_cluster_snapshots`
- `jsm_syncer_lake_snapshots`

Therefore, deploying v2 code is isolated at the Lambda/queue level, but it is
not fully isolated at the data level. Bot callbacks, Jira Automation webhooks,
and credentials determine which stack receives real traffic. Treat callback
changes as production changes.

As of the last AWS inspection, the deprecated Lambda functions still retain
some bot credential environment variable names. That does not make them an
active or supported stack; production callbacks are expected to target v2
only. The values were not read or recorded. Do not re-enable the deprecated
callbacks. If the old resources are cleaned up later, first verify that no
Automation rule, callback, event source mapping, or rollback procedure still
references them.

## Source layout

```text
jsm-production-v2/
├── slack-bot/              Lambda: slack-bot-v2
├── lark-bot/               Lambda: lark-support-bot-v2
├── create-issue-handler/   Lambda: CreateJsmIssueHandler-v2
├── org-sync/               Lambda: jsm-orgs-lambda-v2
├── shared/                 shared static channel/group configuration
├── layer/nodejs/           shared dependency Layer source and lock file
├── infrastructure/         resource names, schema, deployment record
├── docs/                   production cutover checklist
└── scripts/                reproducible artifact builder
```

The `shared/` directory is copied into the Slack and Lark deployment archives;
it is not put in the dependency Layer. The Layer contains third-party runtime
dependencies pinned by `layer/nodejs/package-lock.json`.

Useful entry points:

- `shared/channel-config.json`: Slack channel and Lark group routing
- `org-sync/org-sync-config.js`: organizations synchronized into snapshots
- `slack-bot/request-fields.json`: Jira/JSM field snapshot
- `lark-bot/request-fields.json`: must match the Slack snapshot
- `create-issue-handler/snapshot-payload.js`: snapshot payload hydration
- `infrastructure/resources.js`: canonical v2 AWS resource names
- `infrastructure/schema.sql`: snapshot table DDL
- `docs/production-cutover-checklist.html`: detailed cutover checklist

## Deployed AWS resources

AWS profile used for inspection:
`552185537300_DBaaS-DevUser-Role`.

| Resource | Current state |
| --- | --- |
| `slack-bot-v2` | Node.js 24, `slack-bot/app.handler`, 128 MB, 303 s |
| `lark-support-bot-v2` | Node.js 24, `lark-bot/app.handler`, 256 MB, 303 s |
| `CreateJsmIssueHandler-v2` | Node.js 24, `create-issue-handler/index.handler`, 128 MB, 63 s |
| `jsm-orgs-lambda-v2` | Node.js 24, `org-sync/index.handler`, 512 MB, 303 s |
| Layer | `jsm-production-v2-dependencies:3` on all four Lambdas |
| Queue | `JsmBotTicketQueue-v2` |
| DLQ | `JsmBotTicketQueue-v2-dlq`, max receive count 3 |
| Event source mapping | `6ad3bb0e-3a7d-4a5c-a796-371bbc038c04`, enabled, batch size 1 |
| Scheduler | `jsm-orgs-v2-daily-sync`, enabled, `rate(24 hours)`, Asia/Shanghai |

API endpoints:

- Slack:
  `https://knj0kkq1sh.execute-api.us-west-2.amazonaws.com/default/slack-bot-v2`
- Lark:
  `https://abfl58h599.execute-api.us-west-2.amazonaws.com/default/lark-support-bot-v2`

Last observed Lambda code updates (UTC):

- `slack-bot-v2`: 2026-07-24 10:45:01
- `CreateJsmIssueHandler-v2`: 2026-07-24 10:33:05
- `lark-support-bot-v2`: 2026-07-24 10:33:03
- `jsm-orgs-lambda-v2`: 2026-07-23 08:42:58

The Slack, Lark, and issue-handler archives containing the exact legacy APID
request format were deployed and their local SHA-256 values matched AWS
`CodeSha256` exactly. All three functions
were `Active` with `LastUpdateStatus: Successful`; Runtime, Handler, Role, and
Layer remained unchanged.

At 2026-07-24 10:35 UTC, `CreateJsmIssueHandler-v2` was temporarily switched
to the same Jira email and API token as the legacy `CreatJsmIssueHandler` to
isolate APID identity/Assets-permission behavior. Secret values are not stored
in the repository.

No secrets are stored in this document. Runtime configuration currently
expects:

- Common: `DATABASE_URL`, `JSM_SITE`, `JIRA_EMAIL`, `JIRA_API_TOKEN`
- Slack: `SLACK_BOT_TOKEN`, `SLACK_SIGNING_SECRET`, `SQS_QUEUE_URL`
- Lark: `LARK_BOT_APP_ID`, `LARK_BOT_APP_SECRET`,
  `LARK_BOT_UNION_ID`, `SQS_QUEUE_URL`, `LAMBDA_API_URL`
- Worker: Slack/Lark credentials used to post completion results
- Org sync: `JSM_SYNCER_BASE_URL`, `JSM_SYNCER_API_TOKEN`,
  `REQUEST_TIMEOUT_MS`

## Request routing

Product configuration is per channel/group:

- Cloud `premium` creates ST and enables Jira/IM thread comment sync.
- Cloud `enterprise` creates TCID and does not enable thread comment sync.
- Cloud org `self-managed` creates APID.
- Lake creates TCLID. Lake `premium` would enable thread comment sync; Lake
  `enterprise` does not.
- If an entry has only Cloud or only Lake, that form opens directly.
- If it has both, the user chooses Cloud or Lake first.

Formal Jira/JSM request configuration:

| Internal key | Formal project | Portal/service desk | Request type | Org field | Workload field |
| --- | --- | --- | --- | --- | --- |
| ST | ST | 9 | 2164 | `customfield_17125` | Cluster `customfield_17124` |
| TCID | TCID | 6 | 30 | `customfield_17125` | Cluster `customfield_17124` |
| TCLID | TCLID | 4219 | 4398 | `customfield_17125` | Lake `customfield_17123` |

The internal configuration key and created Lake issue project are both
`TCLID`.

ST, TCID, TCLID, and APID use the JSM Customer Request API with their formal
service desk and request type identifiers. APID retains
`customfield_10002` in the creation request so the issue is shared with its
configured JSM organization atomically.

Self-managed routing requires a channel/group-specific Shared Organization ID
in `shared/channel-config.json`.

## APID / self-managed details

APID is represented in Bot configuration by the synthetic Org option
`self-managed`. It is a Cloud request type, not an org snapshot target. Never
add `self-managed` to `org-sync/org-sync-config.js` or any snapshot table.

### Project and field configuration

Both Slack and Lark build APID from the same configuration:

| Setting | Value |
| --- | --- |
| Jira project key | APID |
| Project ID | `10000` |
| Issue type ID | `10003` |
| Request type string | `ticket/trialquestions` |
| Portal/service desk | `1` |
| Request type ID | `3` |
| Customer portal | `https://tidb.support.pingcap.com/servicedesk/customer/portal/1/` |
| Shared Organizations field | `customfield_10002` |

The APID form inherits the current ST field snapshot for Environment,
Severity, and Versions, but uses APID's own production Component snapshot and
forces Affects Versions to be required. ST and TCID expose the same Component
names, but their Jira option IDs are project-specific; the snapshots therefore
retain each project's real IDs. The bot submits Components by name. The
submitted Jira fields are:

- project and issue type
- summary and plain-text description
- Environment: `customfield_10050`
- Severity: `customfield_10090`
- Components
- Affects Versions
- request type string: `customfield_10010`
- Shared Organization: `customfield_10002`

APID does not use the Cloud Org payload field `customfield_17125` or Cluster
payload field `customfield_17124`. It also does not require an org, cluster, or
Lake snapshot.

The Reporter text shown in the Slack modal is not used as the Jira reporter.
The worker reads the submitting Slack user's profile email and resolves the
corresponding Jira account. Lark sends the Reporter email entered in its form,
and the worker resolves that email to a Jira account.

### APID creation path

APID messages are tagged with internal queue `requestType: "op"`. The worker
uses the same atomic JSM Customer Request API path as the legacy production
Bot.

The current sequence is:

1. Slack/Lark converts `self-managed` to the APID project.
2. The Bot requires the channel/group's `selfManagedSharedOrgId`.
3. The Bot puts the Shared Organization ID in `customfield_10002`.
4. `CreateJsmIssueHandler-v2` resolves the reporter email but skips snapshot
   hydration for `requestType: "op"`.
5. The worker creates and shares the request atomically with
   `POST /rest/servicedeskapi/request`, numeric service desk `1`, numeric
   request type `3`, `raiseOnBehalfOf` set to the reporter email, a plain-text
   Description, no `isAdfRequest` flag, and `customfield_10002` retained in
   `requestFieldValues`.
6. It records the APID key in `ticket_thread` and posts the success reply in
   the originating Slack/Lark thread.

An integer Shared Organization ID is required. Account Full Name
(`customfield_10091`) is not sent or configured. Creation and organization
sharing are one request, so the worker cannot report success for an issue that
failed to be shared.

Cluster selection is irrelevant to APID. Slack's Cluster selector is optional
and returns no snapshot options after `self-managed` is selected. Lark can
still display clusters belonging to real orgs configured in the same group,
but any selected `clusterId` is ignored because APID bypasses snapshot
hydration and no Cluster Jira field is constructed.

### Channels and groups exposing self-managed

Slack:

| Channel | Shared Organization |
| --- | ---: |
| `C08D6MG7UUC` | 15 |
| `C08P316CK24` | 15 |
| `C067K20786A` | 62 |
| `C08PE55GQF2` | 62 |

Lark:

| Group | Shared Organization |
| --- | ---: |
| `oc_1591d78068e800aa0d4c0cacb25d9d2e` | 81760 |

There is one known invalid legacy mapping:
`oc_a48537e5b212acb072c3275c7c860244` exposes `self-managed` but has no
`selfManagedSharedOrgId`. Selecting self-managed in that group will fail
validation. Do not describe APID as supported there until the value is
explicitly confirmed and configured.

APID keys are classified as Cloud tickets. Every currently valid self-managed
entry uses Cloud `supportPlan: "premium"`, so APID Jira/IM comment sync is
enabled. Slack also sends its existing Resolved notification; Lark
status-change notification remains unsupported as described later.

## Snapshot synchronization

`jsm-orgs-lambda-v2` calls these production jsm-syncer endpoints for every
configured organization:

- `GET /orgs/{org_id}`
- `GET /orgs/{org_id}/clusters`

`GET /orgs/{org_id}/lakes` is called only for the orgs used by an enabled Lake
channel: `1372813089194331289` and `1372813089209321686`. Skipping Lake sync
does not update or mark existing Lake snapshot rows as deleted.

The cluster endpoint is paginated. None of these v2 calls supplies
`user_email`.

Each organization is processed independently. A failure for one organization
is recorded and later organizations are still attempted. After all targets
are attempted, the Lambda throws a combined error if any target failed, so
the scheduler retry policy can retry the invocation.

Snapshot writes for one organization are transactional. Rows absent from a
successful refresh are marked `is_deleted = 1`; they are not physically
deleted.

Configured sync targets (19):

```text
1372813089187791289
1372813089194331289
1372813089205911288
1372813089209255759
1372813089201471287
1372813089208821887
1372813089209214865
1372813089209284360
1372813089209291473
1311265642040528896
1372813089187041280
1372813089209061633
1372813089209238420
1372813089209289635
1372813089209311529
1372813089209213994
1372813089209292464
1372813089209340225
1372813089209321686
```

The latest read-only check for org `1372813089209292464` found five active
cluster snapshots, zero deleted snapshots, last seen at
2026-07-24 04:03:36 Asia/Shanghai.

## Snapshot use during ticket creation

Bots send IDs to `CreateJsmIssueHandler-v2`; they do not construct Jira Assets
payloads themselves.

The worker:

1. Requires a valid org snapshot for Cloud/Lake requests.
2. Replaces the Org field with the complete stored `org_payload`.
3. If an active Cluster snapshot exists, replaces the Cluster field with the
   complete stored `cluster_payload`.
4. A missing, deleted, invalid, or stale Cluster does not block issue
   creation; the Cluster field is omitted.
5. Lake requests require an active Lake snapshot and use the complete
   `lake_payload`; missing/deleted Lake snapshots do block issue creation.
6. APID/self-managed requests bypass Cloud snapshot hydration.

Slack Cluster/Lake selects are searchable external selects backed by the
snapshot tables. They return at most 100 options, search by ID/name prefix,
cache identical searches for 30 seconds, and use a 2.4-second timeout to stay
inside Slack's acknowledgement window. Searches are restricted to the
organizations configured for the current channel/product.

Lark loads the relevant snapshot choices while building its card.

## Current channel/group configuration

The authoritative and complete list is `shared/channel-config.json`. Every
entry is currently `enabled: true`.

Slack:

| Channel | Products |
| --- | --- |
| `C08S1QZM39V` | Cloud premium `1372813089187041280`; Lake enterprise `1372813089194331289` |
| `C0BJTPN6XBM` | Cloud enterprise `1372813089187041280` |
| `C084CHENLQP` | Cloud premium: 3 configured orgs |
| `C083S02U74H` | Cloud premium `1372813089205911288` |
| `C08D6MG7UUC` | self-managed |
| `C08P316CK24` | self-managed |
| `C067K20786A` | Cloud premium `1372813089209061633` plus self-managed |
| `C08PE55GQF2` | Cloud premium `1372813089209061633` plus self-managed |
| `C08UUDRHT97` | Cloud enterprise `1372813089209255759` |
| `C08D6BSDXFH` | Cloud premium `1372813089209213994` |
| `C08A84SB1PW` | Cloud premium `1372813089209213994` |
| `C03JKD0S3QW` | Cloud enterprise `1372813089201471287` |
| `C056UFRHP0E` | Cloud enterprise `1372813089208821887` |

Lark:

| Group | Products |
| --- | --- |
| `oc_a48537e5b212acb072c3275c7c860244` | Cloud premium `1372813089209291473` plus self-managed |
| `oc_5cda45b3fd965e19cc7b021e49750cb6` | Cloud premium `1372813089209238420` |
| `oc_88e54dae35ec165ad98e97dc02bf35c0` | Cloud enterprise `1372813089209214865` |
| `oc_ce247561d8a2b1c93c67f5d911ab0843` | Cloud enterprise `1372813089209214865` |
| `oc_8b172801a11729dd29b437c70175b592` | Cloud premium: 3 configured orgs |
| `oc_692976a27a7f9bdeffdbc9af755d06a3` | Cloud premium: 3 configured orgs |
| `oc_69dd24b59835a305941b5bdbe0b2d57e` | Cloud enterprise `1372813089209284360` |
| `oc_1fe2745230f9dc7186d7271e04a50b70` | Cloud enterprise `1372813089209284360` |
| `oc_741a848f5e0538f02e4273bbdb1741ef` | Cloud premium `1372813089209340225` |
| `oc_8e43b467448eeb30025414c6468ec091` | Cloud premium `1372813089209340225` |
| `oc_ca98042ab78d27a044d56966df23c3e9` | Cloud enterprise and Lake enterprise `1372813089194331289` |
| `oc_0ffae16096dc33172d7a039ad877c028` | Cloud enterprise `1372813089209291473` |
| `oc_15a7db7552f02e884bfc7b1078bd77a3` | Cloud premium `1372813089209291473` |
| `oc_57b6cf056d5b59732232b0022c6cdf91` | Cloud premium `1372813089209291473` |
| `oc_1591d78068e800aa0d4c0cacb25d9d2e` | self-managed |
| `oc_000902f885d03cfce857e065274e8fd8` | Cloud enterprise `1311265642040528896` |
| `oc_4e6b1403a7e24d1b210395df63f2945c` | Cloud premium `1372813089209292464` |
| `oc_1192a9156d9b2b06627e74cdb039b289` | Lake enterprise `1372813089209321686` |

Do not edit the abbreviated table above as configuration. Edit
`shared/channel-config.json`, run the tests, build, and deploy both Bot
archives because Slack and Lark each receive a frozen copy of `shared/`.

## Comment and attachment synchronization

Implemented improvements include:

- Jira ADF links and inline cards render as clickable Slack/Lark links.
- Jira ADF common block types, tables, mentions, and unknown containers have
  readable fallbacks.
- Jira `media`, `mediaInline`, and `mediaGroup` attachments are collected and
  synchronized without posting empty duplicate Lark messages.
- Slack `<`, `>`, and `&` control entities are decoded before writing ADF to
  Jira, fixing SQL/code text such as `<db-name>`.
- Logs avoid recording the complete API Gateway event and mask reporter email
  data.

Thread synchronization is controlled by the configured product
`supportPlan`, not by a value copied into each thread row:

- premium: enabled
- enterprise: disabled

Changing a channel/group from premium to enterprise therefore also disables
future comment synchronization for existing tickets of that product when the
Bot handles the event and reads the current bundled configuration.

### Status-change webhook behavior

Slack v2 supports:

```json
{
  "eventType": "status_change",
  "issueKey": "{{issue.key}}",
  "status": "Resolved"
}
```

If Jira Automation renders `issueKey` to a real ticket key, the ticket exists
in `ticket_thread`, and thread sync is enabled, Slack posts
`This request was closed.` in the corresponding thread. Other statuses are
ignored. The unsigned JSM webhook path currently has no Jira Automation
User-Agent check.

Lark v2 does **not** currently handle this correctly. Its `status_change`
branch is commented out and there is no active `notifyStatusChange`
implementation. With the expected Jira Automation User-Agent it can fall
through to comment handling and request `comment/undefined`, producing a 500.
Without that User-Agent it returns 200 and ignores the request. Fix this before
adding a Lark status-change Automation rule.

## Known risks and open work

1. Verify the live Slack Event Subscriptions, Interactivity, and Options
   Request URLs. All three matter; updating only one previously caused 500s or
   ignored mentions.
2. Verify the live Lark event callback and card callback URLs. AWS cannot prove
   the SaaS-side callback setting.
3. Implement and test Lark `status_change` if status notifications are needed.
4. Consider authenticating or explicitly validating Jira Automation webhook
   requests. Slack currently routes every unsigned POST to the JSM webhook
   handler.
5. The local request-field snapshots intentionally preserve some Component
   and Version values from earlier snapshots. The user explicitly deferred
   cleaning local extras after the formal ST/TCID forms changed.
6. Recheck the three formal request types after any Jira form migration:
   ST 2164, TCID 30, TCLID 4398.
7. Confirm Slack, Lark, and Jira Automation callbacks still point only to v2.
   The original stack is deprecated and must not be reactivated.
8. Resolve or remove the invalid self-managed option in Lark group
   `oc_a48537e5b212acb072c3275c7c860244`; its required Shared Organization
   mapping is missing.

## Validation and deployment workflow

Run all tests:

```bash
npm test --prefix jsm-production-v2
```

Build reproducible archives:

```bash
npm run build --prefix jsm-production-v2
```

Artifacts are written to ignored `jsm-production-v2/dist/`. Building the Layer
requires npm registry access. Keep `shared/` in the Slack/Lark archives and do
not add it to the Layer.

Before deploying:

1. Confirm the branch and working tree.
2. Review `git diff` for unrelated user changes.
3. Run all tests.
4. Build all artifacts from the exact commit to deploy.
5. Compare artifact hashes or AWS `CodeSha256`.
6. Deploy only the Lambda(s) affected by the change.
7. Wait for `LastUpdateStatus: Successful`.
8. Validate the appropriate API callback, queue, CloudWatch log, and created
   ticket.
9. Never print environment variable values or commit credentials.

Useful read-only checks:

```bash
aws lambda get-function-configuration \
  --function-name slack-bot-v2 \
  --profile 552185537300_DBaaS-DevUser-Role \
  --region us-west-2

aws lambda list-event-source-mappings \
  --function-name CreateJsmIssueHandler-v2 \
  --profile 552185537300_DBaaS-DevUser-Role \
  --region us-west-2

aws scheduler get-schedule \
  --name jsm-orgs-v2-daily-sync \
  --profile 552185537300_DBaaS-DevUser-Role \
  --region us-west-2
```

Filter or query these commands so secret environment values are never copied
into logs, documents, or chat.

## Recent commits that explain the current design

```text
38f3186 refactor(jsm-v2): rename Lake project to TCLID
4565692 feat(jsm-v2): enable self-managed for lark pilot
987f51b fix(jsm-v2): create APID through Jira issue API
c227511 feat(jsm-v2): configure enterprise cloud channel
b5d45ff fix(jsm-v2): include APID account full name
b21e57a feat(jsm-v2): restore APID self-managed routing
22bd1c4 feat(jsm-v2): use formal submit request types
4d3f94c docs(jsm-v2): record enabled org sync schedule
c653f4b refactor(jsm-v2): remove duplicate Lake rollout config
251379e feat(jsm-v2): enable enterprise Lake group
a9ea2de feat(jsm-v2): enable migrated production channels
52edb47 chore(jsm-v2): set Lake pilots to enterprise
993595d feat(jsm-v2): enable Lake for Slack pilot
936e03b feat(jsm-v2): configure TCLID lake pilot
34cd082 fix: decode Slack entities in Jira comments
9aa82be fix: skip empty Lark posts and redact logs
```

When a newer agent starts, the first action should be to compare this dated
snapshot with `git status`, `git log`, AWS resource state, and the Slack/Lark
application callback settings.