# JSM Syncer Service Design

Date: 2026-07-06

Status: proposal.

Service name: `jsm-syncer`.

## 1. Summary

`jsm-syncer` is an independently deployable Go service that keeps TiDB Cloud
account, support-plan, MSP, and Lake membership data synchronized into Jira
Service Management (JSM) and Jira project roles.

The service owns:

- a local materialized view of TiDB Cloud orgs, users, lakes, and membership;
- mapping TiDB Cloud orgs to JSM organizations;
- resolving TiDB Cloud users to Atlassian account IDs by email;
- synchronization of JSM organization/customer membership;
- synchronization of Jira project roles used to gate technical and lake ticket
  creation;
- read APIs for user-related org, lake, and cluster lookup.

The service does not become the source of truth for TiDB Cloud account, billing,
MSP, lake, cluster, or JSM ticket data. Those systems remain authoritative.
`jsm-syncer` consumes events, reconciles periodically, and applies idempotent
side effects to Atlassian.

The first release stores Lake data in the local database and exposes it through
APIs.

## 2. Background

JSM currently has three relevant projects:

| Project | Purpose | Access model |
|---|---|---|
| `TCAB` | Account and billing issues | No special support-plan role requirement in this design. |
| `TCID` | Technical issues | User needs the configured technical-ticket Jira project role. |
| `TCLID` | TiDB Cloud Lake issues | User needs the configured lake-ticket Jira project role. |

JSM-facing data comes from multiple sources:

- org/user events from Account via `github.com/tidbcloud/tiinf/transport/tieventbus`;
- Billing support-plan and tenant-plan events from Billing eventbus;
- MSP events from SQS;
- lake and lake membership events from SQS;
- periodic batch reconciliation for data that may be missed, stale, or emitted
  without enough detail.

The Account sync background document recommends direct MQ publishing with
offline reconciliation as the loss-recovery path. This service follows the same
consumer-side model: event handling is best-effort and fast, while scheduled
reconciliation repairs missed or out-of-order changes.

## 3. Service Boundary

### 3.1 Owns

- Local database schema for JSM sync state.
- Event consumers for Account eventbus and SQS queues.
- Scheduled reconciliation jobs.
- Atlassian/JSM/Jira clients needed for:
  - creating or resolving JSM organizations;
  - resolving Atlassian account IDs by email;
  - adding/removing JSM customers from JSM organizations;
  - adding/removing Jira project role actors.
- Internal APIs backed by the local database plus mgmt-service cluster lookup.
- Observability for event lag, sync failures, retries, drift, and Atlassian API
  rate limits.

### 3.2 Does Not Own

- TiDB Cloud org/user/lake source records.
- Billing support-plan truth.
- MSP relationship truth.
- Cluster truth.
- JSM ticket workflow, SLA, queue, or issue creation.
- Manual operational management of secret values.

### 3.3 Repository Placement

Place the service under:

```text
services/jsm-syncer/
```

Use the existing Go service style in this repository:

- own `go.mod`, `go.sum`, `Makefile`, and `config.example.yaml`;
- `cmd/jsm-syncer/`;
- `internal/app` for HTTP wiring;
- `internal/services` for business orchestration;
- `internal/integrations` for Atlassian, mgmt, eventbus, and SQS clients;
- `models` and `migrate` for database models and migrations;
- shared infrastructure from `packages/sdk-go/infra`.

Do not import code from another deployable service. If logic from
`ticket-manager`'s DevOps client or Jira client is useful, move service-neutral
helpers into `packages/sdk-go/infra` or reimplement a small service-owned client
behind an interface.

## 4. High-Level Architecture

```text
Account eventbus       Billing eventbus      SQS MSP/lake
      |                       |                    |
      v                       v                    v
  event consumers  ------> normalized sync service layer
                              |
                              v
                         local database
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
     Atlassian/JSM       mgmt service          HTTP APIs
     side effects        cluster lookup        user lookup
```

Main runtime components:

| Component | Responsibility |
|---|---|
| API server | Exposes internal read APIs. |
| Eventbus consumers | Receive Account org/user/membership events and Billing plan/support-plan events. |
| SQS consumers | Receive MSP and lake events. |
| Sync service | Normalizes events, updates local state, computes desired JSM/Jira state. |
| Side-effect service | Applies idempotent Atlassian operations with retry and rate limiting. |
| Reconciliation jobs | Pull authoritative source data and repair local/JSM/Jira drift. |
| Mgmt client | Lists clusters for TiDB Cloud orgs through a separate cluster API. |

Use `packages/sdk-go/infra/service` for service manager, event jobs, cron jobs,
metrics, and graceful shutdown where possible.

## 5. Data Model

Assume TiDB/MySQL-compatible schema. Types below are proposed starting points;
string IDs are used because some upstream IDs may exceed signed integer range or
may already be string identifiers.

### 5.1 Core Tables

#### `tidbcloud_orgs`

| Column | Type | Notes |
|---|---:|---|
| `id` | `varchar(64)` | TiDB Cloud org ID. Primary key. |
| `name` | `varchar(255)` | TiDB Cloud org name. |
| `status` | `varchar(64)` | Source org status. |
| `support_plan` | `varchar(64)` | Normalized support plan. |
| `plan` | `varchar(64)` | Billing tenant plan, for example `new_user`, `poc`, or `on_demand`. |
| `jsm_org_id` | `varchar(64)` | JSM organization ID. Nullable until resolved. |
| `msp_org_id` | `varchar(64)` | MSP provider org ID when this org is managed by an MSP. Nullable. |
| `msp_notification_enabled` | `tinyint(1)` | Whether MSP notification is enabled. |
| `biz_region` | `varchar(64)` | Business region from DevOps tenant data. |
| `time_zone` | `varchar(64)` | Org time zone. |
| `created_at` | `datetime(3)` | Local row created time. |
| `updated_at` | `datetime(3)` | Local row updated time. |

Indexes:

- primary key on `id`;
- index on `jsm_org_id`;
- index on `msp_org_id`;
- index on `support_plan`;
- index on `plan`;
- index on `biz_region`.

#### `tidbcloud_users`

| Column | Type | Notes |
|---|---:|---|
| `id` | `varchar(64)` | TiDB Cloud user ID. Primary key. |
| `email` | `varchar(320)` | Normalized user email. |
| `first_name` | `varchar(128)` | User first name. Nullable. |
| `last_name` | `varchar(128)` | User last name. Nullable. |
| `atlassian_account_id` | `varchar(128)` | Atlassian Cloud `accountId`. Nullable until resolved. |
| `created_at` | `datetime(3)` | Local row created time. |
| `updated_at` | `datetime(3)` | Local row updated time. |

Indexes:

- primary key on `id`;
- unique index on normalized `email`;
- index on `atlassian_account_id`.

Identity conflict handling:

- Account is the source of truth when an incoming user ID and a locally stored
  user ID claim the same normalized email;
- resolve the current canonical user through Account before changing local
  state, and do not fall back to last-writer-wins when Account is unavailable;
- before reassigning the local user ID, mark every active org and lake
  membership for the previous ID as `removed`;
- create `removed` membership shadows for the canonical ID so the normal user
  access sync can apply the corresponding JSM and Jira removals after the
  transaction commits;
- preserve `atlassian_account_id` when the normalized email is unchanged, and
  clear it when Account reports a different current email.

Terminology note:

- Use the column name `atlassian_account_id`.
- Jira Cloud and JSM Cloud both identify Atlassian users with `accountId`.
- Account events are not expected to carry Atlassian `accountId`; this service
  resolves it asynchronously by email.

#### `tidbcloud_lakes`

| Column | Type | Notes |
|---|---:|---|
| `id` | `varchar(64)` | TiDB Cloud Lake ID. Primary key. |
| `name` | `varchar(255)` | Lake display name. |
| `tier` | `varchar(64)` | Lake tier: `Personal`, `Business`, `Dedicated`, etc. |
| `support_plan` | `varchar(64)` | Normalized support plan derived from `tier`. |
| `org_id` | `varchar(64)` | Owning TiDB Cloud org ID. |
| `status` | `varchar(64)` | Source lake status, for example `Active` or `Deleted`. |
| `created_at` | `datetime(3)` | Local row created time. |
| `updated_at` | `datetime(3)` | Local row updated time. |

Indexes:

- primary key on `id`;
- index on `org_id`;
- index on `support_plan`;
- index on `status`.

#### `tidbcloud_org_memberships`

| Column | Type | Notes |
|---|---:|---|
| `org_id` | `varchar(64)` | TiDB Cloud org ID. |
| `user_id` | `varchar(64)` | TiDB Cloud user ID. |
| `status` | `varchar(32)` | `active` or `removed`. Prefer soft state over immediate hard delete for auditability. |
| `created_at` | `datetime(3)` | Local row created time. |
| `updated_at` | `datetime(3)` | Local row updated time. |

Indexes:

- unique key on `(org_id, user_id)`;
- index on `user_id`;
- index on `status`.

#### `tidbcloud_lake_memberships`

| Column | Type | Notes |
|---|---:|---|
| `lake_id` | `varchar(64)` | TiDB Cloud Lake ID. |
| `user_id` | `varchar(64)` | TiDB Cloud user ID. |
| `status` | `varchar(32)` | `active` or `removed`. |
| `created_at` | `datetime(3)` | Local row created time. |
| `updated_at` | `datetime(3)` | Local row updated time. |

Indexes:

- unique key on `(lake_id, user_id)`;
- index on `user_id`;
- index on `status`.

## 6. Event Inputs

All event handlers must be idempotent. They should upsert by source business
IDs and tolerate duplicate delivery. Current Account, Billing, and MSP events
do not carry a stable row-level source update timestamp, so MVP does not keep
`source_created_at` or `source_updated_at` columns. Use business-key upserts,
desired-state recomputation, and scheduled reconciliation rather than
timestamp-based stale event checks.

### 6.1 Account Eventbus

Use `github.com/tidbcloud/tiinf/transport/tieventbus` for org/user related
events. Add a thin adapter in `internal/integrations/account/events` if upstream
event names differ; keep the service layer working on the normalized names
below.

Expected envelope:

```json
{
  "event_id": "evt-10001",
  "event_type": "user.changed",
  "version": "1.0",
  "timestamp": "2026-07-06T10:00:00Z",
  "data": {}
}
```

Current Account event payload fields:

- `data.user`: `user_id`, `email`, `first_name`, `last_name`;
- `data.org`: `org_id`, `org_name`, `time_zone`, `status`;
- `data.role_bindings[]`: `scope_type`, `scope_id`, `role`.

`user.changed` example:

```json
{
  "event_id": "evt-user-1008611-20260706100000",
  "event_type": "user.changed",
  "version": "1.0",
  "timestamp": "2026-07-06T10:00:00Z",
  "data": {
    "user": {
      "user_id": 1008611,
      "email": "user@example.com",
      "first_name": "Ada",
      "last_name": "Lovelace"
    }
  }
}
```

Handling:

- upsert `tidbcloud_users`;
- do not create a local task table row;
- let the scheduled Atlassian account resolver find `atlassian_account_id` by
  email.

`org.changed` example:

```json
{
  "event_id": "evt-org-1372813089209282513-20260706100000",
  "event_type": "org.changed",
  "version": "1.0",
  "timestamp": "2026-07-06T10:00:00Z",
  "data": {
    "org": {
      "org_id": 1372813089209282513,
      "org_name": "Acme",
      "time_zone": "America/Los_Angeles",
      "status": "active"
    }
  }
}
```

Handling:

- upsert `tidbcloud_orgs`;
- ensure JSM organization exists using display name `ORGNAME (ORGID)`;
- persist the returned JSM organization ID in `tidbcloud_orgs.jsm_org_id`.

`user.role_binding.changed` example:

```json
{
  "event_id": "evt-org-user-1372813089209282513-1008611-20260706100000",
  "event_type": "user.role_binding.changed",
  "version": "1.0",
  "timestamp": "2026-07-06T10:00:00Z",
  "data": {
    "user": {
      "user_id": 1008611,
      "email": "user@example.com",
      "first_name": "Ada",
      "last_name": "Lovelace"
    },
    "org": {
      "org_id": 1372813089209282513,
      "org_name": "Acme",
      "time_zone": "America/Los_Angeles",
      "status": "active"
    },
    "role_bindings": [
      {
        "scope_type": "ORG",
        "scope_id": 1372813089209282513,
        "role": "org:owner"
      }
    ]
  }
}
```

Handling:

- ensure user and org exist, creating missing records from `data.user` and
  `data.org` without overwriting existing records;
- ensure the JSM organization when the local org is created by this event;
- mark `tidbcloud_org_memberships` active whenever the event contains both `data.user` and
  `data.org`;
- when the event user identity conflicts with local state, resolve the
  canonical user from Account and query the current Account org membership;
  write that current membership status instead of trusting a potentially
  delayed event;
- ignore `role_bindings` values for local storage and JSM/Jira role computation;
- sync JSM org-customer membership if both `jsm_org_id` and
  `atlassian_account_id` are known;
- recompute the user's desired technical-ticket role from all active orgs.

`user.removed_from_org` example:

```json
{
  "event_id": "evt-org-user-removed-1372813089209282513-1008611-20260706100000",
  "event_type": "user.removed_from_org",
  "version": "1.0",
  "timestamp": "2026-07-06T10:00:00Z",
  "data": {
    "user": {
      "user_id": 1008611,
      "email": "user@example.com",
      "first_name": "Ada",
      "last_name": "Lovelace"
    },
    "org": {
      "org_id": 1372813089209282513,
      "org_name": "Acme",
      "time_zone": "America/Los_Angeles",
      "status": "active"
    }
  }
}
```

Handling:

- ensure user and org exist, creating missing records from `data.user` and
  `data.org` without overwriting existing records;
- ensure the JSM organization when the local org is created by this event;
- mark `tidbcloud_org_memberships` as `removed`;
- when the event user identity conflicts with local state, resolve the
  canonical user and current Account org membership exactly as for
  `user.role_binding.changed`; the latest Account membership may therefore
  keep the local relation active;
- remove the user from the JSM organization only if no active local relation
  still requires that membership;
- recompute the user's desired technical-ticket role.

### 6.2 Billing Eventbus

Billing publishes two relevant eventbus event types.

`binded-support-plan-update` updates the active support plan:

```json
{
  "tenant_id": 123,
  "level": "Basic/Developer/Enterprise/Premium"
}
```

`tenant_plan` updates the tenant plan:

```json
{
  "tenant_id": 123,
  "changed_plan": "on_demand",
  "changed_status": "active",
  "effective_plan": "on_demand"
}
```

Handling:

1. Treat `tenant_id` as the TiDB Cloud org ID.
2. For `binded-support-plan-update`, store `level` in
   `tidbcloud_orgs.support_plan`.
3. For `tenant_plan`, store `effective_plan` in `tidbcloud_orgs.plan`.
4. Recompute technical-ticket role for all active users in the org when
   `support_plan` changes.

Scheduled reconciliation still calls Billing's active support-plan API to repair
missed `binded-support-plan-update` events.

Technical-ticket eligible plans:

```text
Developer, Enterprise, Premium
```

`Basic` does not grant the technical-ticket role.

### 6.3 MSP SQS

Input example:

```json
{
  "type": "add_msp_relation",
  "msp_info": {
    "msp_org_id": 123,
    "user_org_id": 456
  }
}
```

Notification input example:

```json
{
  "type": "enable_msp_notification",
  "msp_info": {
    "msp_org_id": 123
  }
}
```

Handling:

| Type | Handling |
|---|---|
| `add_msp_relation` | Set `tidbcloud_orgs.msp_org_id` on `user_org_id`, then reconcile the customer's JSM organization membership. |
| `remove_msp_relation` | Clear `tidbcloud_orgs.msp_org_id` when it matches the payload `msp_org_id`, then reconcile the customer's JSM organization membership. |
| `enable_msp_notification` | Set `tidbcloud_orgs.msp_notification_enabled = true` on the MSP provider org row. |
| `disable_msp_notification` | Set `tidbcloud_orgs.msp_notification_enabled = false` on the MSP provider org row. |

MSP notification behavior should be modeled as a separate field
`msp_notification_enabled` and updated by explicit notification events. Do not
infer it from relation add/remove.

### 6.4 Lake SQS

Expected event shape:

```json
{
  "type": "create_lake",
  "lake_info": {}
}
```

`create_lake` / `update_lake` example:

```json
{
  "type": "create_lake",
  "lake_info": {
    "lake_id": "lake-123",
    "org_id": "1372813089209282513",
    "display_name": "prod-lake",
    "tier": "Business"
  }
}
```

`delete_lake` example:

```json
{
  "type": "delete_lake",
  "lake_info": {
    "lake_id": "lake-123"
  }
}
```

`invite_user_to_lake` / `update_user_lake_role` example:

```json
{
  "type": "invite_user_to_lake",
  "rolebinding": {
    "lake_id": "lake-123",
    "user_id": "1008611",
    "role": "account_admin"
  }
}
```

`remove_user_from_lake` example:

```json
{
  "type": "remove_user_from_lake",
  "rolebinding": {
    "lake_id": "lake-123",
    "user_id": "1008611"
  }
}
```

Handling:

| Type | Handling |
|---|---|
| `create_lake` | Derive support plan from tier and upsert `tidbcloud_lakes`. |
| `update_lake` | Derive support plan from tier, update lake fields, and recompute lake-ticket roles for active lake users if support plan changed. |
| `delete_lake` | Mark lake deleted; recompute lake-ticket roles for active lake users. |
| `invite_user_to_lake` | Find the existing `tidbcloud_users` row by `rolebinding.user_id`, upsert `tidbcloud_lake_memberships`, and recompute the lake-ticket role for the user. |
| `update_user_lake_role` | No local relation change is needed unless the event also changes membership status; source role is ignored. |
| `remove_user_from_lake` | Mark relation removed; recompute lake-ticket role for user. |

Lake tier to support-plan mapping:

| Tier | Derived support plan |
|---|---|
| `Personal` | `Basic` |
| `Business` | `Enterprise` |
| `Dedicated` | `Premium` |

## 7. JSM and Jira Synchronization

### 7.1 Atlassian User Identity

Store `atlassian_account_id` as the canonical Atlassian user identifier.

Resolution strategy:

1. Account events provide TiDB Cloud user ID and email only; they do not provide
   Atlassian `accountId`.
2. A scheduled background job scans users where `atlassian_account_id` is null.
3. For each user, search Jira/JSM by normalized email.
4. If a matching Atlassian account is found, store `atlassian_account_id`.
5. If no account is found, do not create or invite a customer. Leave the column
   null and try again on the next scheduled run.
6. If an Account event changes a user's email, clear the existing
   `atlassian_account_id`; the scheduled job will resolve the new email.

Missing `atlassian_account_id` should not block local org/lake state updates.
JSM membership and Jira role writes for that user wait until the ID is resolved.

### 7.2 JSM Organization

Keep TiDB Cloud org ID and JSM organization ID separate:

- `tidbcloud_orgs.id` is the TiDB Cloud org ID.
- `tidbcloud_orgs.jsm_org_id` is the JSM organization ID returned by Atlassian.
- API responses should expose both fields clearly.

Desired behavior:

- When a TiDB Cloud org is created or first seen, ensure one JSM organization
  exists and persist `jsm_org_id`.
- Use `jsm_org_id` as the primary lookup after it is known.
- Ensure the JSM organization is associated with all three JSM service desk
  projects: account/billing, technical, and lake.
- If `jsm_org_id` is missing, call the JSM organization search endpoint with
  the TiDB Cloud org ID and match its exact suffix in the production display
  name:

```text
ORGNAME (ORGID)
```

- The `ORGNAME` prefix may be stale after a TiDB Cloud org rename and must not
  be used for identity matching.
- If multiple JSM organizations have the same exact org ID suffix, select the
  organization with the smallest numeric JSM organization ID.
- If not found, create the JSM organization with the current display name.

Because TiDB Cloud org names may collide, the org ID suffix is required in the
display name.

### 7.3 JSM Organization-Customer Membership

Desired behavior:

- For every active `tidbcloud_org_memberships` relation:
  - ensure JSM organization exists;
  - ensure the user has `atlassian_account_id`;
  - add the user/customer to the JSM organization.
- When an org references an MSP provider through `msp_org_id`, its desired JSM
  organization membership is the union of active members from the customer org
  and the MSP provider org.
- For removed `tidbcloud_org_memberships` relation:
  - remove the user from the JSM organization only if no active local relation
    still requires that membership, including an active MSP provider relation.
- Reconcile desired membership against the users currently in the JSM
  organization so clearing or replacing an MSP relation removes stale provider
  members.

If the Atlassian account cannot be found yet, skip the write. The next account
resolution or reconciliation run will retry.

### 7.4 Technical Ticket Role in `TCID`

Role config:

```yaml
atlassian:
  projects:
    technical:
      key: TCID
      roleID: "<technical ticket user role id>"
```

Desired state:

A user should have the technical-ticket role when the user belongs to at least
one active org whose support plan is in:

```text
Developer, Enterprise, Premium
```

Removal rule:

- Do not remove the role just because one org downgrades or one membership is
  removed.
- Remove only when the user has zero active eligible orgs.

Implementation:

- Compute desired state from the local database.
- Query Jira project role actors by `roleID` when applying or reconciling.
- Add/remove the user's `atlassian_account_id` only when the actual Jira role
  state differs from the desired state.

### 7.5 Lake Ticket Role in `TCLID`

Role config:

```yaml
atlassian:
  projects:
    lake:
      key: TCLID
      roleID: "<lake ticket user role id>"
```

Desired state:

A user should have the lake-ticket role when the user belongs to at least one
active lake whose support plan is in:

```text
Developer, Enterprise, Premium
```

If lake support plan is derived from tier:

- `Personal` does not grant the role because it maps to `Basic`.
- `Business` grants the role because it maps to `Enterprise`.
- `Dedicated` grants the role because it maps to `Premium`.

Removal rule:

- Remove only when the user has zero active eligible lakes.

Implementation matches technical-ticket role sync: compute desired state from
the local DB and query Jira role actors by `roleID` before writing.

## 8. API Design

Base path:

```text
/api/v1/jsm-syncer
```

Only the Forge query surface is exposed through the public Kong gateway:

```text
/api/v1/jsm-syncer/forge/orgs
/api/v1/jsm-syncer/forge/lakes
/api/v1/jsm-syncer/forge/orgs/{tidbcloud_org_id}/lakes
/api/v1/jsm-syncer/forge/orgs/{tidbcloud_org_id}/clusters
```

Kong validates the Forge Invocation Token before forwarding these requests.
The FIT identity claims authenticate the Forge invocation but are not used to
resolve the TiDB Cloud user. Forge endpoints resolve users only from the
required `user_email` query parameter. Org-scoped Lake and Cluster queries also
require an active membership between the email-resolved user and the requested
org.

The non-Forge API paths remain available only on the internal Kubernetes
service and are not routed through the public Kong gateway. They also include
direct resource lookup endpoints for trusted internal callers:

```text
GET /api/v1/jsm-syncer/orgs/{tidbcloud_org_id}
GET /api/v1/jsm-syncer/orgs/{tidbcloud_org_id}/lakes
GET /api/v1/jsm-syncer/clusters/{cluster_id}
```

### 8.1 Get User Orgs

```text
GET /api/v1/jsm-syncer/orgs?user_email=user@example.com
```

Behavior:

1. Normalize `user_email`.
2. Find `tidbcloud_users`.
3. Join active `tidbcloud_org_memberships`.
4. Return associated `tidbcloud_orgs`.
5. If an org has `msp_org_id`, include `msp_org` by default:
   - `msp_org.tidbcloud_org_id` and `msp_org.name` from the MSP provider org;
   - `msp_org.msp_notification_enabled` from the MSP provider org row;
   - `msp_org.msp_members` from active users in the MSP provider org.

Response example:

```json
{
  "user": {
    "tidbcloud_user_id": "1008611",
    "email": "user@example.com",
    "atlassian_account_id": "5b10a2844c20165700ede21g"
  },
  "orgs": [
    {
      "tidbcloud_org_id": "1372813089209282513",
      "name": "Acme",
      "status": "active",
      "support_plan": "Premium",
      "jsm_org_id": "123",
      "msp_org": {
        "tidbcloud_org_id": "1372813089209282999",
        "name": "Acme MSP",
        "msp_notification_enabled": true,
        "msp_members": [
          "msp-admin@example.com",
          "msp-operator@example.com"
        ]
      }
    }
  ]
}
```

### 8.2 Get Org Clusters

Cluster lookup is a separate API and is not folded into the org list response.

```text
GET /api/v1/jsm-syncer/orgs/{tidbcloud_org_id}/clusters?page_size=50&page_token=&cluster_id=123
```

Behavior:

1. Validate that the org exists locally. This trusted internal endpoint does not accept or require `user_email`.
2. Accept pagination parameters:
   - `page_size`: maximum number of clusters to return, with a service-defined cap;
   - `page_token`: opaque token returned by the previous page;
   - `cluster_id`: optional exact TiDB Cloud cluster ID filter.
3. Call internal mgmt service to list clusters by TiDB Cloud org ID.
4. Return mgmt-service results without storing clusters locally.

The public Forge route keeps its required `user_email` query parameter:

```text
GET /api/v1/jsm-syncer/forge/orgs/{tidbcloud_org_id}/clusters?user_email=user@example.com&page_size=50&page_token=&cluster_id=123
```

Before listing clusters, this route resolves `user_email` and requires an active
membership in the requested org. It returns `403 Forbidden` when no active
membership exists, including when the org ID is unknown, so callers cannot use
the endpoint to enumerate organizations.

Mgmt client interface:

```go
type ListClustersOptions struct {
    PageSize  int
    PageToken string
    ClusterID string
}

type ClusterPage struct {
    Clusters      []Cluster
    NextPageToken string
}

type MgmtClient interface {
    ListClustersByOrgID(
        ctx context.Context,
        orgID string,
        opts ListClustersOptions,
    ) (ClusterPage, error)
}
```

Follow the DevOps internal-service pattern: call mgmt gRPC
`ClusterService.ListClusters` with `ClusterFilter.tenant_ids`, `page`, and
`page_size`, and call `ClusterService.CountClusters` only if needed to decide
whether to return a next page token. Do not use the BFF cluster-management API
for service-to-service traffic.

The upstream internal mgmt API is page-based. Keep the public API cursor-like by
translating inside the mgmt client:

1. For the first page, use `page=1`, `page_size`, and `tenant_ids = [org_id]`.
2. For the next page, decode `page_token` into an internal payload such as:

```json
{
  "org_id": "1372813089209282513",
  "page": 2,
  "page_size": 50,
  "cluster_id": "123"
}
```

3. Validate that the request org and filters match the token payload. If
   `org_id`, `cluster_id`, or `page_size` changes, reject the token and require
   the caller to start a new query without `page_token`.
4. Call upstream with `ClusterFilter.tenant_ids`, optional
   `ClusterFilter.cluster_ids`, `page`, and `page_size`, and exclude deleted
   clusters.
5. Use `CountClusters` internally if needed. If another page exists, return a
   new opaque token with `page + 1`; otherwise return an empty
   `next_page_token`.

The token should be treated as opaque by callers. It can be base64url-encoded
JSON because it is not a secret, but it must not become a long-term stable API
contract.

Response example:

```json
{
  "tidbcloud_org_id": "1372813089209282513",
  "clusters": [
    {
      "id": "cluster-1",
      "name": "prod",
      "status": "available",
      "service_plan": "Premium"
    }
  ],
  "next_page_token": "opaque-token"
}
```

`service_plan` should use mgmt's ServicePlan enum names:

```text
SERVICE_PLAN_UNSPECIFIED, Starter, Essential, Premium, BYOC, Dedicated,
Premium_Reserved, Essential_V2
```

### 8.3 Get User Lakes

```text
GET /api/v1/jsm-syncer/lakes?user_email=user@example.com
```

Behavior:

1. Normalize `user_email`.
2. Find `tidbcloud_users`.
3. Join active `tidbcloud_lake_memberships`.
4. Return active or recently deleted lakes according to query options.

Response example:

```json
{
  "user": {
    "tidbcloud_user_id": "1008611",
    "email": "user@example.com",
    "atlassian_account_id": "5b10a2844c20165700ede21g"
  },
  "lakes": [
    {
      "id": "lake-123",
      "name": "prod-lake",
      "tier": "Business",
      "support_plan": "Enterprise",
      "tidbcloud_org_id": "1372813089209282513",
      "status": "Active"
    }
  ]
}
```

### 8.4 Get Lakes By Org ID

This endpoint is available only to trusted internal callers and does not
require a user identity:

```text
GET /api/v1/jsm-syncer/orgs/{tidbcloud_org_id}/lakes
```

It validates that the organization exists in the local sync database, then
returns all locally synced lakes for that organization ordered by lake name.
An unknown organization returns `404 Not Found`.

The public Forge route requires `user_email` and returns the same organization
Lake list only after verifying an active organization membership:

```text
GET /api/v1/jsm-syncer/forge/orgs/{tidbcloud_org_id}/lakes?user_email=user@example.com
```

It returns `403 Forbidden` when the email-resolved user is not an active member
of the requested organization.

Response example:

```json
{
  "tidbcloud_org_id": "1372813089209282513",
  "lakes": [
    {
      "id": "lake-123",
      "name": "prod-lake",
      "tier": "Business",
      "support_plan": "Enterprise",
      "tidbcloud_org_id": "1372813089209282513",
      "status": "Active"
    }
  ]
}
```

### 8.5 Get Users By Org ID

This endpoint is available only to trusted internal callers; there is no Forge
equivalent:

```text
GET /api/v1/jsm-syncer/orgs/{tidbcloud_org_id}/users
```

It validates that the organization exists, then returns its active members
ordered by email and user ID. `jsm_account_id` is the locally resolved
Atlassian account ID and is `null` while resolution is pending.

Response example:

```json
{
  "tidbcloud_org_id": "1372813089209282513",
  "users": [
    {
      "id": "1008611",
      "email": "ada@example.com",
      "jsm_account_id": "5b10a2844c20165700ede21g"
    }
  ]
}
```

### 8.6 Operational APIs

Optional admin-only endpoints:

| Endpoint | Purpose |
|---|---|
| `POST /api/v1/jsm-syncer/admin/reconcile/orgs/{tidbcloud_org_id}` | Force org/user/JSM reconciliation for one org. |
| `POST /api/v1/jsm-syncer/admin/reconcile/users/{tidbcloud_user_id}` | Force account-ID resolution and role recomputation for one user. |
| `POST /api/v1/jsm-syncer/admin/reconcile/lakes/{lake_id}` | Force lake and lake-role reconciliation for one lake. |

These should be protected by internal auth and can be postponed until after MVP
if operational tooling is not required on day one.

## 9. Source API Findings

These source integration points are available from existing services. The new
service should call them through service-owned clients and must not import code
from those deployable services.

### 9.1 Account

Account already publishes data-change events through eventbus with these event
types:

```text
user.changed
org.changed
user.role_binding.changed
user.removed_from_org
```

The event envelope includes `event_id`, `event_type`, `version`, `timestamp`,
and `data`. Current Account payload fields are the same as Section 6.1:
`data.user`, `data.org`, and `data.role_bindings[]`. For
`user.role_binding.changed`, `jsm-syncer` should mark membership active whenever
the event contains both `data.user` and `data.org`; it should not depend on the
contents of `role_bindings[]` because local JSM sync only needs the user-org
association, not the source role.

Available Account APIs for backfill and reconciliation:

- provider gRPC org listing: `ListOrgsAdvance`, with `GetOrgByID` for targeted
  repair;
- provider gRPC user listing: `ListUsersByFilter`, with `GetUserByEmail` and
  `GetUserByID` for targeted repair and user identity-conflict resolution;
- provider gRPC MSP customer lookup: `ListPartnerCustomers`, using
  `partner_customer_ids` as customer org IDs, `status = active`, and
  `partner_type = MSP` to repair `msp_org_id`.

No Account OpenAPI endpoint is required for MVP backfill. For `msp_members`,
list active users in the MSP provider org. MSP relation is repaired by
`ListPartnerCustomers`; MSP notification state remains event-driven from MSP SQS
enable/disable events and is not overwritten by full reconciliation.

### 9.2 Billing

Billing publishes eventbus events that `jsm-syncer` can consume directly:

- `binded-support-plan-update` with `tenant_id` and `level`;
- `tenant_plan` with `tenant_id`, `changed_plan`, `changed_status`, and
  `effective_plan`.

Use Billing eventbus for near-real-time updates instead of the legacy JSM SQS
support-plan queue.

Billing also exposes an active support-plan lookup API for reconciliation:

- `GET /api/v1/billing/tenants/{tenant_id}/binded_support_plan` returns the
  active support plan for one tenant and defaults to `Basic` when no active
  binding exists.

On scheduled reconciliation, query this endpoint and use the returned level
directly for tenants without an MSP relation. For MSP customers, query the
customer tenant and MSP provider tenant concurrently, then store the higher
level according to `Basic < Developer < Enterprise < Premium`. If either lookup
fails or returns an unknown level, keep the previously stored support plan and
report a reconciliation error instead of risking an incorrect downgrade.

### 9.3 DevOps Tenant API

Use DevOps tenant APIs for full org metadata reconciliation, because DevOps
tenant data already combines Account tenant fields with Billing tenant plan and
business-region enrichment:

- `TenantServiceAPI.ListTenants` for full org metadata backfill and scheduled
  reconciliation;
- `TenantServiceAPI.GetTenant` for targeted repair by TiDB Cloud org ID;
- Account-derived fields used by this service: org ID, name, status, and time
  zone;
- Billing-derived field: `plan`;
- enrichment fields: `country_region` and `sales_force_region`, with
  `GetTenantRegion()` selecting the business region used by DevOps.

### 9.4 Mgmt

Use internal mgmt gRPC for service-to-service cluster lookup, following the
existing DevOps pattern:

- call `ClusterService.ListClusters` with `ClusterFilter.tenant_ids`,
  `page_size`, and `page`;
- call `ClusterService.CountClusters` with the same filter only if needed to
  compute whether another page exists;
- exclude deleted clusters in the filter;
- do not use BFF cluster-management APIs for this internal service.

The current internal `ClusterFilter` supports tenant/project/cluster IDs,
regions, states, excluded states, view, cluster type, deleted flag, and service
plan. Use `cluster_ids` for exact cluster ID filtering.

### 9.5 Lake

Lake API is not confirmed yet. Define a service-owned `LakeClient` interface and
use a mock provider for MVP backfill/reconciliation. Keep real Lake
reconciliation disabled until the Lake contract is available.

## 10. Scheduled Reconciliation

Scheduled jobs are required because event publishing and delivery are
best-effort.

| Job | Suggested frequency | Source | Action |
|---|---:|---|---|
| Incremental user/membership reconciliation | Every 5 minutes if source cursor exists | Account provider gRPC | Upsert changed users/memberships and apply JSM/Jira side effects where possible. |
| Full org sync | Daily | Account provider gRPC, DevOps tenant API, Billing API | Page through Account orgs, merge DevOps tenant plan/business region, query Billing active support plan, repair MSP relation, repair org memberships, and apply JSM/Jira side effects. |
| Atlassian account ID resolution | Every 5-15 minutes | Jira/JSM user search by email | Fill missing `atlassian_account_id`; retry unresolved users later. |
| Lake reconciliation | Every 5-15 minutes | Lake mock provider until API exists | Upsert lakes/memberships, derive support plan from tier, and recompute lake roles. |
| JSM org membership drift check | Daily or sampled hourly | JSM APIs | Compare expected vs actual and repair. |
| Jira role drift check | Daily or sampled hourly | Jira project role APIs | Compare expected vs actual role actors and repair. |

If a source system does not yet expose reconciliation APIs, keep the scheduled
job interface but mark it disabled in config until the source API exists.

## 11. Idempotency and Ordering

All handlers must be safe under duplicate, out-of-order, and partially failed
delivery.

Rules:

- Use upstream business IDs as primary keys.
- Upsert source records rather than insert-only writes.
- Do not add an `event_receipts` table for MVP; duplicate delivery is handled by
  business-key upserts and scheduled reconciliation.
- Compute role desired state from the full local database, not from a single
  event.
- Query Jira role actors before adding/removing role membership.
- Apply Atlassian side effects as "ensure desired state" operations.
- Treat Atlassian 404/409/already-exists responses as state signals, not always
  as terminal failures.
- Rely on source redelivery plus scheduled reconciliation for retries; do not
  require a local side-effect task table for MVP.

Examples:

- Billing event downgrades one org from `Premium` to `Basic`; user still belongs
  to another `Enterprise` org. Desired technical-ticket role remains `true`.
- User removed from one lake; user still belongs to another `Business` lake.
  Desired lake-ticket role remains `true`.
- `create_lake` is delivered twice. `tidbcloud_lakes` upsert produces the same
  final state.
- JSM org creation is retried after a timeout. The service searches by TiDB
  Cloud org ID, verifies the exact `(ORGID)` suffix, and stores the existing JSM
  org ID instead of creating a duplicate, even if the `ORGNAME` prefix changed.

## 12. Configuration

Example shape:

```yaml
apiServer:
  address: ":8080"

environment: "staging" # staging or prod

secret:
  rootConfigSecretName: "" # e.g. "jsm-syncer/root-config"
  awsASM:
    region: "us-west-2"

database:
  host: ""
  port: 4000
  user: ""
  password: ""
  database: "jsm_syncer"

eventbus:
  account:
    enabled: true
    topic: "account.events"
    consumerGroup: "jsm-syncer"
  billing:
    enabled: true
    supportPlanTopic: "binded-support-plan-update"
    tenantPlanTopic: "tenant_plan"
    consumerGroup: "jsm-syncer"

sqs:
  msp:
    enabled: true
    queueURL: ""
  lake:
    enabled: true
    queueURL: ""

atlassian:
  baseURL: "https://tidb.atlassian.net"
  username: ""
  apiToken: ""
  projects:
    accountBilling:
      key: "TCAB"
      roleID: ""
    technical:
      key: "TCID"
      roleID: ""
    lake:
      key: "TCLID"
      roleID: ""

account:
  grpcAddress: ""
  timeout: "10s"
  pageSize: 500

billing:
  baseURL: ""
  timeout: "10s"

devops:
  baseURL: ""
  apiToken: ""
  timeout: "10s"
  pageSize: 100

mgmt:
  grpcAddress: ""
  timeout: "30s"

reconciliation:
  accountIDResolution:
    enabled: true
    recentSpec: "*/10 * * * *"
    fullSpec: "0 2 * * *"
  orgFullSync:
    enabled: true
    spec: "0 2 * * *"
  orgMembershipSync:
    enabled: true
    spec: "0 4 * * *"
  lakeSnapshot:
    enabled: true
    spec: "*/10 * * * *"
```

Secrets should be loaded from AWS Secrets Manager by reusing the repo's shared
config path:

- add a service `SecretConfig` matching `ticket-manager`: embed
  `packages/sdk-go/infra/config.Secret` and add `rootConfigSecretName`;
- load config with `packages/sdk-go/infra/config/loader` in this order:
  defaults, YAML file, then
  `loader.NewConfigurationSourceSecret(...)`;
- use `packages/sdk-go/infra/secret.NewSecretSource`, which currently supports
  AWS Secrets Manager through `awsASM`;
- store secret values such as DB password, Atlassian API token, SQS queue URLs
  if sensitive, and mgmt API token in the ASM root config YAML;
- rely on the pod's AWS identity/default AWS config for ASM access. Do not
  commit secret values.

Staging and prod should use separate config files and AWS Secrets Manager root
config secrets. Environment-specific values such as source service base URLs,
SQS queue URLs, Atlassian project role IDs, and secret names must come from
configuration, not code constants.

## 13. Observability

Metrics:

- event consumed count by source/type/result;
- event lag by source;
- Atlassian account ID resolution count by result;
- Atlassian API count, latency, status code, and rate-limit responses;
- mgmt cluster lookup count and latency;
- reconciliation drift found/repaired count;
- role desired/applied mismatch count.

Logs:

- include `event_id`, `event_type`, `org_id`, `user_id`, `lake_id`,
  `atlassian_account_id`, and `jsm_org_id` when available;
- do not log API tokens or secret-bearing headers;
- avoid logging full user payloads when email alone is enough.

Alerts:

- consumer stopped or queue age above threshold;
- Atlassian 401/403 spike;
- Atlassian 429 sustained rate limit;
- sustained Atlassian account ID resolution failure;
- reconciliation drift above threshold.

## 14. Testing Strategy

Follow repository Go testing guidance:

- use standard `testing` plus `testify`;
- use `go.uber.org/mock` generated mocks for clients;
- write parallel-safe unit tests where possible.

Recommended coverage:

| Area | Tests |
|---|---|
| Event normalization | One test per event type and malformed payload. |
| DB upsert | Idempotent create/update, stale event ignored, deletion/removed state. |
| Atlassian account resolution | Found, not found, already resolved skipped, API error. |
| Role computation | Multi-org and multi-lake eligibility, downgrade/removal edge cases. |
| JSM org/customer sync | Existing org lookup, missing account ID skipped, retryable API error handled by reconciliation. |
| Jira project role sync | Add, remove, no-op when desired already applied. |
| API handlers | Email lookup, MSP expansion, cluster API success/error. |
| Reconciliation | Source snapshot repairs local and Atlassian drift. |

Provide dummy Atlassian clients for local tests. Real Atlassian integration
tests should be opt-in and gated by explicit config.

## 15. Rollout Plan

### Phase 0: Contract Confirmation

- Confirm Account eventbus schemas.
- Confirm Billing eventbus topics and schemas for `binded-support-plan-update`
  and `tenant_plan`.
- Confirm SQS queue URLs and message schemas for MSP and lake.
- Confirm DevOps tenant API contract and auth for full org metadata
  reconciliation.
- Confirm Jira project role IDs for `TCID` and `TCLID`.
- Confirm mgmt cluster list API contract and auth.
- Confirm the final Lake API contract when Lake is ready.

### Phase 1: Read Model and APIs

- Add `services/jsm-syncer` Go skeleton.
- Add database migrations for core tables.
- Implement repository layer and read APIs.
- Implement Account provider gRPC, Billing, DevOps tenant, and mgmt clients.
- Implement mgmt client interface and separate cluster lookup API.
- No Atlassian write side effects yet.

### Phase 2: Event Consumers

- Add eventbus consumer for Account org/user/membership events.
- Add eventbus consumer for Billing support-plan and tenant-plan events.
- Add SQS consumers for MSP and lake.
- Update local DB idempotently.
- Add metrics and logs.

### Phase 3: Atlassian Side Effects

- Implement Atlassian clients behind interfaces.
- Implement scheduled `atlassian_account_id` resolution by email.
- Implement `ensure_jsm_org`.
- Implement JSM org-customer sync.
- Implement Jira role sync for `TCID` and `TCLID` using role IDs.

### Phase 4: Reconciliation

- Add full org sync that combines Account org/user data, DevOps tenant metadata,
  Billing support plan, MSP relation backfill, and JSM/Jira side effects.
- Keep Atlassian account-ID resolution and Lake snapshot reconciliation as
  separate jobs.
- Add JSM/Jira drift checks.
- Add operational admin APIs if needed.
- Run in dry-run mode first: compute desired changes and log without writing.

### Phase 5: Production Cutover

- Enable writes in staging.
- Backfill local DB from source systems.
- Run full dry-run drift report.
- Enable JSM org/customer writes.
- Enable project role writes.
- Monitor unresolved Atlassian accounts, role mismatches, and Atlassian rate
  limits.

## 16. Open Questions

1. What are the exact Jira project role IDs for `TCID` and `TCLID`?
2. What is the final Lake API contract after the MVP mock phase?

## 17. References

- Account eventbus data-change schema.
- Billing support-plan and tenant-plan eventbus schemas.
- Billing active support-plan API.
- DevOps tenant API.
- Internal mgmt cluster gRPC APIs.
- Atlassian JSM organization REST API:
  <https://developer.atlassian.com/cloud/jira/service-desk/rest/api-group-organization/>
- Atlassian Jira user REST API:
  <https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-users/>
- Atlassian Jira project role actors REST API:
  <https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-project-role-actors/>