# Cloud lifecycle — operations runbook

## 云角色目录:「默认下拉?」关掉的角色在新子账号上会复活(2026-09-07,迁移 623)

- **现象**:管理员在 `/admin/cloud/role-catalog` 把某角色的「默认下拉?」关成「否」,员工的
  **申请云访问** 主角色下拉框里它还在。生产实测:10 个 `DBaaS-Prod-*` CloudSSO 角色
  2026-06-11 被 demote,而 6-13 至 8-24 期间新接入的 8 个阿里云海外子账号上它们全是开着的
  (`BaiLian` / `rds-secret` 同型,新的 4 个上还开着)。
- **原因**:`is_promoted` 是 `(integration, scope, role_key)` 三元组上的列,promote-by-key
  只能改点下去那一刻已存在的行。之后 `cmdb.roles_sync` 或单 scope 预热给新子账号插的是**新行**,
  取驱动默认值 —— 所有云驱动对自定义 / CloudSSO 角色一律 `is_promoted=True`。
  目录页按**跨 scope 一致值**上报,8/29 显示成「否」,和「全部关掉」长得一模一样,
  管理员既看不见也点不掉(开关已在「否」位,再点是开)。
- **修复后**:promote / demote 会记 `cmdb_cloud_roles.promotion_overridden_by_admin`,
  roles_sync 用它给新行播种、并把跨 scope 漂移的行重新盖成同一个值(每次跑最多两条 UPDATE,
  日志行 `roles_sync re-stamped N role row(s)`)。「重置为默认」清掉这个标记 —— 它的语义就是
  「忘掉我的编辑」。
- **2026-09-07 已执行(不用再点)**:12 个阿里云键在发布后手动跑了一次
  `cmdb.roles.alicloud-intl-primary`(`manual:post-deploy-migration-623`,8.5 秒,
  `upserted=476 deleted=17 errors=[]`)→ 各 0 promoted / 28 scope;31 个 AWS 权限集
  经 `promote-by-key` 统一为「否」(4123 行,审计 `cloud.role_catalog.demote`)——
  方向不是猜的:那 31 个键上 `is_promoted=0` 的 2110 行 `first_seen_at` 在
  2026-04-29…06-15,`is_promoted=1` 的 990 行在 06-15…09-03,一条干净的时间分界线,
  与阿里云那 12 个同一机制。**现在全平台混合态 0 组**(修复前 43 组)。
- **Operator 该做什么**:① 自定义角色(含 CloudSSO 权限集)**无需动作** —— 迁移 623 回填了
  `is_custom=1 AND is_promoted=0` 这批(这个组合只可能是人关的),下一次 roles_sync 会把
  残留的 **108 行**拉回「否」(10 个 DBaaS 键各 9 行 = 8 个活跃子账号 + 1 个已软删的,`BaiLian` / `rds-secret` 各 5 行;发布前对着生产只读算的,可拿来对账 —— 日志行 `roles_sync re-stamped N role row(s)`);② **厂商内置角色要点一次**:目录页新增「各 scope 不一致」筛选
  和一个「统一为否」按钮(单行的黄色标签 + 顶部横幅的批量),生产上等着的是 31 个 AWS
  permission set(32/100 混合态)。迁移**故意不回填**这批 —— 内置角色「部分 scope 关着」在
  数据上和驱动默认值分不开,方向得由管理员定,不能猜;③ 判断某角色现在到底会不会出现在
  某个子账号的下拉里,看 `GET /cmdb/scopes/{id}/roles`(默认 `promoted_only=true`),
  不要只看目录页那个按一致值显示的开关。


## 「谁负责这个 scope」七处谓词收敛(2026-09-07)

**Operator 该做什么:无需任何动作。**(初版这里写着「一次性清掉 2 条解不掉的离职 blocker」—— **那是错的,已更正,见下面第 1 条**。)

起因是一条用户报告:同一个 Owner 筛选,管理员台账 7 行、员工「我的云订阅」12 行。
顺着这一类查下去,「这个人负责哪些 scope」在 5 个模块里写了 7 遍、给出 3 种答案。
现已收敛为 `cmdb/models.py` 的 `owner_governed_scope_criteria()` /
`employee_directory_scope_criteria()`,七处全部调用。

**对运维的实际影响**(生产实测,不是假设):

1. **离职 blocker 扫描不再对「不是账号」的 scope 建 blocker。** 此前它只过滤
   `deleted_at IS NULL`,因此对 `azure_resource_group` / `legacy_billing_unit` 也建了
   `cloud_scope_owner` blocker。

   🔴 **更正(2026-09-07,发布后核对生产数据得出)**:本条初版写的是「已经给真实离职者
   写下 2 条**谁也解不掉**的 blocker,会**卡住离职流程**,需要 operator 去手工跳过」——
   **这三点都不成立**,是我从扫描器的行为推断出来的,没有查 blocker 自己的状态。实测:
   全库 `cloud_scope_owner` blocker 共 **16 条,全部已 resolved**(11 `transferred` /
   5 `revoked_manual`),**全库任何类型的未解决 blocker 数为 0**,相关离职流程全部
   `completed`。我点名的那两条都是被 T+0 `force_transfer` **自动**解决的(一条 15 分钟,
   一条 3 天)。**没有任何需要 operator 处理的存量,当时也没有。**

   而且其中一条(`APAC-Technical_Service`)建于 2026-08-05,**当时该 scope 还是
   `standalone_iam`**(合法的 tenant-level 类型)—— 是迁移 466 在 8-12 把它重定型成
   `legacy_billing_unit` 之后,这条 blocker 才回头看起来像垃圾。**它在创建那一刻是对的。**

   真实(小得多的)代价:force-transfer 把一个成本归集行 / 一个已下线资源组的「负责人」
   转给了接手人 —— 无意义的归属churn 和噪声,不是卡流程。修复消除的是未来的同类噪声:
   库里还有 **22 条** closed-but-owned 的 scope,修复前每个持有它们的未来离职者都会再来一次。
2. **离职残留对账(`/admin/reconciliation` 离职残留 tab)的 cloud_scope_owner 行会变少。**
   它此前已排除 closed/decommissioned,现在同时排除 `legacy_billing_unit`
   —— 生产有 1 条 `left` 的人持有的 legacy 行会从该列表消失。**这不是漏报**:那是成本归集行,
   迁移 466 起「no owner governance applies」,管理员台账默认也不显示它。
3. **`/admin/reconciliation` 「归属完整性」透镜的 cloud_scope 行会从 8 条变成 0 条 —— 这 8 条全是误报。**
   (初版写 7 条,是漏算了「owner 存在但已离职」的那一类 —— 该透镜同时报 NULL owner 和非在职 owner。
   按透镜真实谓词复算是 8 条,依然 100% 是 `legacy_billing_unit`。)
   它们全部是 `legacy_billing_unit`:迁移 466 给这些行定型,正是为了让它们**不需要 owner**,
   所以管理员点开也永远清不掉,和该查询里既有的「GCP billing account 排除」防的是同一种永久假阳。
   **如果你之前把这几条当成待办记在别处,可以直接划掉。**
4. **经理「我的团队 → 团队云资产」的计数徽章与列表现在必然一致**(此前是两条独立查询)。
5. **员工「我的云订阅」少 30 行**(28 `legacy_billing_unit` + 2 `azure_resource_group`),
   这些行此前还带着一颗点下去必被 `CLOUD_SCOPE_NOT_GRANTABLE` 拒绝的「快速申请」按钮。

**管理员台账本身不收窄** —— 它仍显示 `azure_resource_group` / `standalone_iam`,
`legacy_billing_unit` 仍可用 `include_legacy_billing=true` 调出。契约是
**员工 ⊆ 管理员**,不是相等。

0 migration / 0 cron / 0 RBAC / 0 MCP 工具数改动。

## DAX Owner-only 审批上线（2026-09-07）

- 仓库个人与 SA 定义都只保留 `data_owner_review`；当前生产个人模板仍可能是经理→Owner，
  SA 旧模板可能还残留 workload-manager/technical-owner 自动跳过节点。不要把 main 的 artifact
  或 Helix mapping 当作已发布的飞书定义。
- 使用既有 `Deploy DAX Feishu Approval` 分别选择 `data_access_personal_grant` 与
  `data_access_service_account_grant`，发布后独立读取 definition，确认实际审批节点只有 Owner；
  再核对各自 `integration_approval_flows` 的 external code、sequence 和 approver node id 一致。
  不通过发布模板顺便开启任何 Provider apply 开关或修改 Group 历史 mapping。
- 旧实例通常保留创建时的流程，不能假设改模板就会解锁经理待办。先只读核对在途实例；如需
  迁移，走经确认的取消/重新申请流程，不代 Owner 批准、不伪造回调、不直接修改 approved 状态。
- Owner-only 流程仍须实际 Owner 通过；接着检查 request/binding/effect 的 applied 状态和精确
  BigQuery IAM，不能把审批通过等同于 Provider 授权成功。


## DAX Service Account 已批未执行：principal 白名单退役（2026-09-07）

- **新版行为**：SA 授权不再需要 `service_account_apply_principal_allowlist`。旧字段即使存在或
  格式异常也被忽略，下一次 apply-mode config merge 清理；无需迁移。不要再为新申请手工登记 UUID。
- **上线影响**：只有 integration apply 与 SA apply 都已开启的集成受影响；原有符合条件的
  pending/retryable SA effect 将由 `data_access.bigquery_apply_reconcile` 自动落地。发布前只读核对
  积压申请、审批、member/role 和 exact dataset/table locator；不自动打开开关、不扩大 Terraform IAM。
- **验收**：不要把 scheduler 的 `last_status=ok` 当成授权成功。核对 blocked/applied/error 汇总、
  `data_access_effect_runs`、request/binding/effect rollup，再独立读取对应 BigQuery dataset/table IAM。
  资源失效或 Owner/locator 不匹配仍会拦截，Provider 错误仍按原机制保留 retryable 证据。
- **本次定点修复（已获用户授权）**：Heidi 的两笔已批 SA 请求共 14 项，通过正常 reconciler
  按 effect 逐项执行；只在每项事务内临时允许该 principal，同事务恢复 integration 原配置后提交，
  其他 worker 从未看到持久化白名单扩大。修复前 1 项已存在、本次补齐 13 项；修复后独立 IAM
  回查 14/14 存在、两笔 request 和全部 effect 均 `applied`。审计 trace：
  `d72c03f5-f061-4c15-a4b4-78dc29f671d0`（`data_access.approved_sa.ops_repair` + reconciler evidence）。
  首项 table IAM 写入后曾因运维脚本遗漏 audit `status` 参数而回滚本地事务；先独立读回确认已存在，
  再用幂等 reconciler 补证据，恢复经过写入上述审计，不把这一项误记为原先就有的权限。
- **回滚**：旧二进制会重新执行 principal gate，尚未完成的非名单 SA 会再次 pending；已落地 IAM
  不会因代码回滚自动删除。需要回收时走正常 revoke，禁止盲删整个 dataset policy。


> **统一云资源目录索引（2026-08-24，迁移 529）**：`cmdb.resource_directory_refresh` 每 15 分钟
> 只读并投影现有 CMDB 专用表；`cmdb.resource_inventory_refresh` 每小时调用已由云管理员显式开启的
> 官方只读 inventory API（AWS Resource Explorer、Azure Resources、GCP Cloud Asset）。两者都不
> 创建角色、不修改云资源。前者显示 `integrations/seen/created/changed/disappeared/restored/
> history_pruned`，后者显示同组资源计数以及 `inventory_skipped/inventory_partial/inventory_errors`。
> `inventory_skipped` 表示未开启服务或能力，
> `inventory_partial` 表示部分账号/订阅/项目不可读且本轮已禁止消失扫描，`inventory_errors` 才是失败。
> 首次上线 `created` 较大是全量建索引的预期；稳定后通常只有 `seen`。投影按主键游标分批读取
> 宽 JSON 行并逐批提交（默认 `config.batch_size=100`，边界 1–500），避免首次回填触发 TiDB
> `tidb_mem_quota_query`。若仍出现 8176，先将任务配置的 `batch_size` 降到 25 后重跑并检查具体
> 源表；不要提高 TiDB 全局内存额度，也不要清空目录表。任务失败时再检查迁移 529 是否已应用及
> 具体源表/FK；刷新幂等且每批可恢复，修复后直接重跑即可。
> `disappeared` 表示源表已软删除/下线，目录保留历史；不要手工物理删除变化账本。
> 开通广覆盖发现前由云管理员在 Provider 侧配置只读能力：AWS 建 Resource Explorer 聚合索引与
> default view，并给每个组织成员只读角色开放 Search；Azure 在预期订阅授予 Reader；GCP 为组织内
> 预期项目授予 `cloudasset.assets.searchAllResources`。随后只在集成配置打开
> `resource_inventory_enabled`；Helix 不代建角色。AWS 还应填写聚合索引区域。开通后首次运行先核对
> `inventory_partial=0` 再把空结果视作权威；不要用清表方式解决权限或分页问题。

> **治理评估与趋势（迁移 530）**：`cmdb.resource_governance_evaluate` 每日 01:43（Asia/Shanghai）
> 只读统一目录并更新 Helix 本地发现项和长期聚合。正常结果应包含 `rules/resources/opened/updated/
> resolved/rollups_written`；首次运行 `opened` 较大是建立基线，稳定后重点关注持续新增与未关闭项。
> 失败时先检查 530、规则 evaluator 是否在应用白名单、最近目录刷新是否成功；不要手改 finding
> fingerprint 或删除历史聚合。抑制最长 366 天且到期自动重开，治理例外过期后也会重新命中复核。
> 此任务不调用云 API、不执行云变更；重跑是幂等的。同一天的聚合按维度覆盖更新，不产生重复点。

> **目录报表投递（迁移 534）**：`cmdb.resource_report_dispatch` 每小时第 17 分钟扫描到期的
> 周报/月报订阅。正常结果包含 `due/generated/queued/skipped/failed`；`queued` 只表示已写通知
> outbox，不等同渠道送达。投递前会重新加载用户在职状态、真实角色和目录关系授权，离职或失去
> 目录权限会关闭订阅，绝不复用订阅创建时的可见范围。单条失败推迟 6 小时并写 failed run，
> 不阻断同批其它订阅。排障顺序：确认迁移 534、任务启用、`next_run_at`、时区与通知 outbox；
> 不要手工改 report snapshot 或 permission fingerprint。AI 对所有目录读者开放，但输入只有脱敏且按
> 收件人实时权限裁剪的聚合事实；普通员工只能解释本人/下属/有效授权范围。同一收件人同一事实
> 24 小时复用，不能通过重复点击绕开成本控制。

> **覆盖状态与责任继承（迁移 540）**：排查“无监控”时只处理 `monitoring_status=missing`；
> `not_applicable/not_supported/not_expected` 都不是采集故障。成本同理，`cost_source=scope/integration`
> 表示有订阅级/共享账单证据但不能冒充资源精确金额，只有 `cost_status=not_available` 才检查账单连接。
> 第二个同厂商连接发现同一原生账号 ID 时可继承唯一既有 Owner；若历史上存在冲突 Owner，会保留
> 治理缺口而不是自动选人。修复连接凭据或账号同步后依次重跑 `cmdb.scopes_sync`、资源同步和
> `cmdb.resource_directory_refresh`，不要直接批量 UPDATE 目录行（目录是可重建读模型）。

> **目录读性能（迁移 538）**：员工列表/摘要慢时先确认迁移已应用，索引
> `ix_cmdb_cloud_resources_integration_scope_deleted` 存在；再分别测 `/summary` 与当前 Tab 接口。
> 客户端只应请求当前 URL Tab，不能为“切换更快”预挂载五个 Tab。普通员工关系权限在请求开始时
> 解析一次，后续 SQL 不应出现逐资源相关的 `cmdb_cloud_access_grants` 子查询。若仍超过 30 秒，保留
> 用户角色、接口、筛选条件和查询计划；不要调大浏览器超时来掩盖慢查询。

> **订阅工作台与全资源 AI（2026-08-25）**：订阅详情“资源”Tab 的第一块是统一目录清单，下面仍是
> 账单分摊；两者口径不同，不要用账单资源行数判断扫描是否完整。AI 的 `resources.total` 是目录全部
> 类型，`server_total/classified/actionable` 才是机器子集。排查 AI 数字时先用相同订阅的目录 summary
> 对账；`without_monitoring` 只计 `monitoring_status=missing`，`without_cost_data` 只计真正没有任何账单
> 证据的资源。证据链接在展示时会重新校验 integration + scope，失去授权后旧快照引用会隐藏。
> 事实构建按 UUID 游标每 500 条处理并只保留 20 条下钻证据；若分析任务变慢，应检查单批覆盖查询
> 和数据库计划，不要把实现改回一次性加载整订阅资源，也不要提高 TiDB 内存上限掩盖问题。

> **(2026-08-20, Owner 仍显示但成员为 0 的修复路径)**：Owner 字段不是云侧 IAM
> 的证据；若历史到期/撤销留下 `owner_person_id` 但没有 active grant，成员页会显示 0。
> 管理员现在可以在「添加成员」选择该 Owner 并添加所需 Owner-equivalent role：仅
> `global_admin` / `cloud_admin` 允许这条受审计的 break-glass 路径，普通 scope Owner
> 仍被拒绝自授予。成功修复的 owner grant 为永久有效，不会再被到期扫描回收。处理前先
> 查该 Owner 的 grant 历史和云侧角色，**不要**把没有 owner 的 legacy scope 随意分配给
> 当前操作者；没有明确历史归属时应先确认责任人再转移 Owner。2026-08-20 已恢复
> `gcp-dm-dev` 与 `gcp-qa-dp-testing-dev` 的 `bear.c@pingcap.com` `roles/owner`。

## Shared AI subscription analysis

Subscription detail now exposes a top-level **AI 分析** tab to administrators, the subscription Owner, and effective members. Opening the tab only reads the current shared result and deterministic five-dimension health baseline; it never invokes a model. Operators should first check the data-confidence message: when confidence is low, repair scope synchronization or monitoring before requesting an AI interpretation. **开始 AI 分析** is the explicit cost-bearing action; the result is shared for the subscription, respects the seven-day unchanged-facts cache and existing cooldown, and keeps evidence links filtered to the viewer’s ordinary scope permissions. Use the resulting risks and priority actions to reach the existing underlying object/workflow, then return to the tab and verify the deterministic baseline rather than treating an AI narrative as an automated remediation.

> **云订阅 AI 普通成员 H5 详情（2026-08-16；功能开关仍关闭）**：有效成员/Owner 的
> 普通订阅深链进入 `/m/cloud/scopes/:scopeId` 并读取同一共享分析；没有员工菜单。
> 账号联系邮箱、预算、成员清单和管理动作不会出现在该页。403/404 继续按既有非枚举
> 权限闸处理；带 `renew=<grant_id>` 的到期提醒仍进入授权页，不应改到分析页。

> **云订阅 AI H5 长时生成（2026-08-16；功能开关仍关闭）**：自动 GET 最多持续 10 分钟
> （200 次），随后暂停；用户点继续检查只恢复 GET，不会重发 POST。仍为 generating 时
> 检查共享 lease、百炼请求与生成审计，禁止通过重复生成规避卡住状态。

> **云订阅 AI H5 生成轮询（2026-08-16；功能开关仍关闭）**：生成中每三秒 GET，只有
> 返回仍为 generating 才继续，终态或离开页面即停止。持续 generating 应查共享 lease/
> 百炼调用，不要重复 POST；瞬态读取失败使用页面重试。

> **云订阅 AI 飞书 H5 卡（2026-08-16；功能开关仍关闭）**：管理员订阅详情已有原生
> 移动分析卡，首屏不读历史、不调用模型；历史 Sheet 只读，生成与强制重算遵守同一
> 缓存/冷却/额度/确认边界。证据链接先走桌面到移动的安全映射，无法映射时不导航。
> H5 出现 generating 会每三秒 GET 状态，不应反复点击或手工重放 POST。

> **云订阅 AI 请求隔离（2026-08-16；功能开关仍关闭）**：latest/history/generate 的
> 移动与桌面调用必须保持独立请求序列，不能用读取轮询取消历史或生成状态。release
> gate 尚未被成功响应证明启用时，首次 GET 的 5xx/403/404 都隐藏；成功启用后的 5xx
> 才给重试。排障时不要反复点击生成或把无权访问误判为服务故障。

> **云订阅 AI 桌面历史（2026-08-16；功能开关仍关闭）**：分析历史只有用户点击后才
> 读取，最多 50 条摘要且绝不调用模型。历史出现 `integrity_status != valid` 时保留原始
> 记录用于审计，界面标记完整性失败；不要改快照 Hash 或手工补关联。

> **云订阅 AI HTTP/MCP 入口（2026-08-16；功能开关仍关闭）**：启用后，订阅详情
> 首屏只调用 `GET .../ai-analysis`，不会生成快照或产生模型费用；历史同样只读。
> 只有用户点击“开始 AI 分析”才调用 `POST .../generate`，MCP 对应工具
> `cloud_scope_generate_ai_analysis` 也是明确的成本 mutation。Owner 和 active 未到期
> 成员共享同一结果；无权限/pending/到期统一返回 404，这是防枚举设计，不要改成
> “更友好”的订阅存在提示。普通成员不能强制重算；global/cloud admin 使用
> `force=true` 时必须同时提交 `confirm_force=true`。未变化事实七天内直接复用并标记
> `cache_reused`；30 分钟冷却或每日额度触发时按标准 `Retry-After`/错误码退避，不要
> 连续重试。最新生成失败时保留上一成功分析和当前确定性健康分，先查生成审计与百炼
> 连接器，不要删除历史。三个 MCP 工具总数提示更新为 466。
> 若订阅账单尚未同步、Azure RG 缺少安全归属键或对应成本事实已过留存期，机器/服务
> 证据会 fail closed 为不可见；这是防止同租户跨订阅泄漏的安全降级。先恢复账单归属
> 数据并重新分析，不要绕过归属校验或把其他订阅对象手工写入 reference map。

> **云订阅 AI 证据投影（2026-08-16）**：响应中的 AI 匿名引用会在 Helix 内按当前
> 权限映射，不代表模型见过机器名、角色名或资源名。普通成员看不到他人的授权角色；
> Owner/管理员仍只能沿既有订阅/机器详情入口查看。若返回
> `integrity_status != valid`，表示分析与不可变快照关联不一致：健康分仍可信可看，
> AI 内容会被隔离；先保留记录并查审计/数据库一致性，禁止手工改 reference map。

> **云订阅 AI 按需生成边界（2026-08-16）**：当前服务层已完成但入口仍关闭，无需
> operator 配置或回填。启用后的普通读取不会刷新事实或调用模型；只有用户点击生成
> 才先写确定性快照并进入统一缓存。普通 Owner/成员不能绕过 7 天同事实复用，只有
> global/cloud admin 在界面二次确认后可强制重算；关闭订阅禁止生成。若最新尝试失败，
> 页面应继续展示上一成功版并标记失败，不能删除旧结果或反复点击消耗额度。普通成员
> 必须持有 active 且未到期授权；pending/到期授权不算分析权限。

> **云订阅 AI 快照留存（2026-08-16）**：订阅快照使用现有 Intelligence 长期账本，
> 不复制领域原始行；同事实 Hash 不新增版本，并发只允许一份成功。Hash 包含本地
> 证据映射的摘要语义，因此“数字没变但授权/机器换人”仍会生成新历史。当前尚无
> 用户入口或定时任务，无需执行回填；后续显式生成会在调用模型前先写确定性快照。
>
> **云订阅 AI 分析事实口径（2026-08-16）**：共享事实包只读业务折前 usage 成本；
> 只有 `budget_measure=list` 的预算才参与预测与 burn rate。看到
> `budget_basis_mismatch=true` 表示该订阅仍是旧实付预算，应由财务按既有预算转换
> 流程处理，禁止为“让 AI 有数字”而直接改口径。租户级订阅按 `scope_id` 归属成本，
> Azure 资源组按 integration + resource_group；无安全归属键会显示数据不足，不会
> 回退到整个租户。模型包仅有匿名引用，真实服务/机器/授权对象只在 Helix 本地映射。
>
> **云订阅 AI 分析评分内核（2026-08-16）**：`cloud-scope-health-v1` 已固定五维
> 权重和 18 项确定性指标，但功能入口仍关闭；此版本不增加同步任务、不调用模型，
> operator 无需执行回填或配置。后续事实接入必须继续使用业务折前成本，不能把实付
> 金额放进共享订阅快照；资源节省只读取既有 efficiency 结果，不重复计算。

> **AI 大脑读取运维提示（2026-08-16）**：云与成本维度的确定性快照读取使用独立
> `intelligence.reads_per_minute` 预算（默认 120，合法范围 30–600），不会消耗敏感
> 云数据接口额度。计数使用兼容旧版 Redis 的原子 Lua INCR+TTL，不会留下永久 key，
> 且会自动修复上线前遗留的无 TTL 计数；
> 超限客户端按 `Retry-After` 退避；Redis 故障时读取 fail-open，
> 应排查日志 `Intelligence read limiter unavailable`，无需停用云同步或快照任务。

> **Audience**: IT admin running multi-cloud onboarding, sub-account creation, access lifecycle, cost watch, decommission.
> **Architectural reference**: [docs/architecture/PLATFORM.md §13](../architecture/PLATFORM.md#13-cloud-lifecycle-cmdb).
> **Last refreshed**: 2026-08-17 (CAM 凭证信投递真值 + 死信留档). **Operationally relevant cloud-lifecycle changes**:
>
> - **(2026-08-17, Tencent CAM 首登凭证信:飞书指引的措辞现在可信 + 没投出去时有可查的死信)**. **两件事**。① 之前「凭证是否送达」是在**写完通知台账之后**才判定的,于是台账那一步失败(FK / 事务中毒 / 连接超时)会让一封**已经投到公司邮箱的信**被判成没送达 —— 飞书登录指引随即告诉新同事「初始登录凭证未能送达你的公司邮箱,请联系 Helix IT 管理员重新发送」。**运维影响**:如果你在 2026-08-17 之前收到过这类求助并据此重置了密码,那次重置很可能是多余的,而且**作废了对方邮箱里那封信上的密码**。现在投递真值只取自邮件驱动的返回值,指引措辞与实际投递状态一致。② 一封都没投出去时(模板渲染为空、主邮件集成缺失或类型不对、发送调用自身抛错),现在会在**通知台账**里留一条 `template_key=tencent_cam_initial_password`、`status=dead`、`failed_channels=["email"]` 的行,标题以「[未送达] 腾讯云 CAM 初始登录凭证」开头,正文带主账号 ID / 子用户名 / **临时密码**。**Operator 该做什么**:看到 outcome 里的 `Tencent CAM account created but credential email failed`,先去通知台账按上述 `template_key` 查这条死信行 —— **优先线下把这串密码交给本人(仍是首登必改),而不是直接重置密码**;确认交付后再排查邮件集成 / 模板本身。密码照旧**不要**贴进飞书或任何聊天。**0 migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-08-14, 已审批的 DNS 申请不会再永久卡在「处理中」—— 云写搬出飞书回调 + `executing` 回收 cron)**. **现象(用户报告)**:申请人在飞书里三级审批全过,Helix「我的申请」却一直显示「处理中」,一个多小时没有任何变化。**根因(两个 bug 叠加)**:① 表单 TTL 填了 300,而 Helix 驱动的这个 DNSPod 租户**最低 600**(实测:10 个 zone / 900 条解析,没有一条低于 600),云侧以 `LimitExceeded.RecordTtlLimit` 拒绝——而这个必然失败的值,是走完三级人工审批之后才被发现的;② 本该把失败落到行上的兜底代码**自己也炸了**:它先 `db.rollback()`(会 expire 掉所有 ORM 属性),紧接着读 `row.id` 触发懒加载 → `MissingGreenlet`,handler 当场死掉,于是行既不是 `failed` 也没有审计、没有飞书评论。而 `executing` 这个状态**当时没有任何东西在扫**(审批 reaper 只看 `pending` 的审批行,「重跑」只认写锁失败),所以它会一直停在那里。**改了什么**:① 兜底路径在云写之前就把 id 存成普通值,任何 provider 报错都能落到 `failed` + 审计 + 飞书评论;② **提交时**就按 provider 的 TTL 下限拒绝(`min_dns_ttl`,腾讯云 600;付费套餐可在集成的 `config_public.min_dns_ttl` 单独放宽),**只拒绝不静默改写**——否则执行出来的记录会和审批人看到的不一致;前端 TTL 输入框下限也从 60 改成 600(它的提示文案本来就写着 600);③ 云写从飞书回调里搬走——回调只落 `approved` 立刻返回,由**每分钟**的 `cmdb.dns_request_executor` 在 worker 里执行(飞书 ~3 秒收不到响应就重投,那条实例 26 秒内被投了 4 次,而第一次还卡在 DNSPod 调用里);④ 新增**每 10 分钟**的 `cmdb.dns_request_execution_reaper` 回收 `executing`:`create` 先对账云端(记录已存在=写入其实成功,补台账置 `executed`,**绝不重复创建**),其余动作幂等直接回退 `approved` 重跑,超过 3 次尝试则置 `failed` 并写入**可「重跑」**的错误信息。**Operator 该做什么**:① **常态无需动作**;② **注意执行延迟从「秒级」变成「最多 1 分钟」** —— 审批通过后申请状态先短暂停在「已批准」再变「已完成」,这是预期;③ 看到申请详情里出现「执行被中断」类失败:审批仍然有效,直接点「重跑」,**不要让申请人重新发起**;④ 有人报 TTL 填不进去/提交被拒:让他填 600 或更大(错误信息会直接报出该 provider 的下限)。**migration 469 / +2 cron / 0 RBAC / 0 新 MCP 工具**。
> - **(2026-08-13, Tencent CAM 首次登录推送)**. 新建 Tencent CAM 用户除了普通“云访问已生效”通知，还会收到一张 Feishu-only 首次登录卡，包含当前可用入口 `https://cloud.tencent.com/login`、选择“子账号登录”的提示、主账号 ID 和子用户名。初始密码**不会**出现在飞书或 Helix inbox，只发到公司的工作邮箱；邮件同样使用该有效入口。多角色授权只会有一张开户指引，不会每个 role 重复推送。若用户报告旧 `/login/sub-account` 404，发送/补发这张登录指引，不要把密码贴进聊天。
> - **(2026-08-13, Tencent CAM API 密钥身份验证)**. CAM 子用户创建时会从 HRIS 的 `mobile_phone` 和手机号所属国家码携带手机号，严格格式化后写为 CAM `PhoneNum` / `CountryCode`，因此用户可完成创建 API 密钥所需的二次身份验证。手机号为空、国家码未知或号码与国家不一致时会跳过（不猜测或写错手机号）；应先在 HRIS 修正后再新建账号。**已存在的 CAM 用户不会因再次授权而覆盖个人资料**：若其控制台提示“信息缺失暂时无法完成身份验证”，由拥有 CAM 用户资料编辑权限的管理员通过 `UpdateUser` 对该用户名补齐手机和区号；不重置密码、不在飞书发送手机号或密码。该资料修复只解决身份验证，若控制台仍显示 `cam:QueryCollApiKey` 未授权，则应走最小权限的 API-key 自助策略审批，禁止授予 `QcloudCamFullAccess`。
> - **(2026-08-13, Tencent CAM grants sync 同页重复策略)**. Tencent `ListAttachedUserAllPolicies(AttachType=0)` 的 `PolicyList` 可能为直接和组继承两种路径各返回一条相同策略，且 `TotalNum` 计原始行。同步会在**同一页**按 `(PolicyId, PolicyName)` 合并为一个有效授权，同时用原始行数完成 `TotalNum` 校验；跨页重复、总数漂移、提前空页和短页仍拒绝快照。若成员与权限页显示旧的 “overlapping/repeated page at page=1”，发布后手动执行一次该腾讯集成的 grants sync 即可刷新；不需要手工改库，失败期间也不会触发该 scope 的 drift revoke。
> - **(2026-08-12, 离职人员云 IAM 权限残留对账)**. 新增每日 cron `cmdb.leaver_iam_residue_sweep`（02:10 Asia/Shanghai，排在 `cmdb.leaver_grant_sweep` 01:30 之后），用云端 IAM 镜像 `cmdb_iam_users` 对账 HR 离职名单，把「Helix 记录已回收、云上其实还在」的权限登记为治理待办并通知 IT 管理员。**这个 cron 只读，不做任何云端写操作**，因此不受 prod 写锁影响；清理动作在 IAM 用户页的「离职残留」tab 人工确认后执行，复用既有 `POST /cmdb/iam-users/{id}/cleanup`（不可撤销，`global_admin` 限定，受 prod 写锁约束）。**Operator 该做什么**: 上线首夜后打开 `/admin/cloud/iam-users?tab=residue` 处理队列 —— 当前生产已知两条真实残留（腾讯云 `chujie` 4 条策略、`tangpeng` 1 条），都是 2026-06-02 组继承回收修复之前留下的。镜像超过 13 小时未同步的行会标「镜像过期」并禁用清理按钮：**先修该租户的 IAM 同步，不要照着过期快照删账号**。确属正常的（账号转团队共享等）用「忽略」并填原因，会记录操作人；残留真的消失后该行自动关闭。单次检出超过 200 条时 cron 直接失败而不是发告警 —— 那通常意味着 HRIS 把一批人误刷成 `left`，先核 `identity_persons.employment_status`。**migration 463 / +1 cron / 0 RBAC / +2 MCP**（`list_leaver_cloud_iam_residue`、`dismiss_leaver_cloud_iam_residue`）。
> - **(2026-08-09, 云成员角色显示名称回填)**. `cmdb.grants_sync` 现在在写入 discovered grant 前，优先按当前 scope、其次按租户级 `cmdb_cloud_roles` 目录解析 `role_key` 的展示名。AWS Identity Center 的 permission-set assignment 只给 ARN，因此已有目录名（例如 `SRE-Admin`）会在下一次 grants sync 自动回填到成员/授权页；其他云的匿名或遗漏展示名也走同一条路径。**Operator 该做什么**: 发布后对受影响云租户执行一次 roles sync（确保目录最新）再执行 grants sync；无需手工改库、不会新增或撤销任何云侧权限。目录找不到名称时，仍显示厂商返回值作为回退。**0 migration / 0 cron / 0 RBAC / 0 MCP**.
> - **(2026-08-03, production Terraform adoption Phase 3 — #2515)**. The desired baseline now has no known live drift: DNSPod and SecureString values are externally owned, all observed SG rules are explicit, nonexistent SSH/ECR lifecycle resources are disabled, and ALB health/TLS settings preserve live behavior. The manifest has 63 importable addresses plus two still-blocked provider-only state-record addresses. This is not plan proof: keep the gate blocked until production inputs are reviewed and plan JSON reports zero create/update/delete/replace after the two isolated state records are separately approved. **0 migration / 0 cron / 0 RBAC / 0 MCP change**.
> - **(2026-08-03, production Terraform adoption Phase 2 — #2515)**. The configured production state key is absent while live AWS resources remain. A reviewed 76-address manifest and adoption contract now live under `infra/aws/environments/prod/adoption/`; desired ForceNew identities have been coordinated to live. This is evidence and configuration only: `execution_gate` remains `blocked`. Operators must not render or run imports, state commands, plans, applies, or AWS mutations until external DNS ownership, non-importable resources, unmanaged/live-absent children, and behavioral in-place drift have approved strategies and fresh live metadata is reverified. Never retrieve SecureString values for this adoption workflow. **0 migration / 0 cron / 0 RBAC / 0 MCP change**.
> - **(2026-08-03, integration/provider diagnostic credential redaction — #2469)**. Cloud provider exceptions and `/api/v1/integrations/{slug}/test` now share the same credential scrubber. Tencent `SecretId`/`SecretKey`, Basic/Bearer headers, signed URL parameters and OSS STS tokens are removed before an error reaches the test response, `last_test_message`, audit detail, sync history or logs. Operators still see bounded provider error codes/request context; if a connector test now shows `<redacted>`, inspect the provider-side request ID and error code instead of expecting credential fragments. No deployment-time action or data backfill is required; historical stored diagnostics are not rewritten. **0 migration / 0 cron / 0 RBAC / 0 MCP change**.
> - **(2026-08-02, GCP organization / billing tenant boundary + bounded lifecycle transport — #2446 / #2448)**. **安全变化**:GCP 项目发现不再使用凭证全局可见的 `projects.search`;现在从配置的 `organizations/{id}` 开始递归调用 `projects.list(parent=...)` + `folders.list(parent=...)`,并严格校验每条返回值的直接 parent、分页 token、重复节点与遍历上限。账单账号也不再 `billingAccounts.list()` 全扫;每条 integration 必须显式配置 `billing_scope_mode=projects_only`(账单 partition 为「权威空」)或 `allowlist` + `billing_account_allowlist`(逐个 `get` 并沿 billing parent 链验证到配置 organization;`create_scope_billing_account` 自动并入白名单)。任何权限不足、parent 不符、配置缺失或响应畸形都会把 partition 标为 incomplete,在写库、orphan sweep 和离职 IAM mutation 前 fail closed。生命周期 Google SDK 同时统一为 30s socket timeout;幂等 GET/list 最多重试 3 次,`setIamPolicy` 永不盲重放,未知提交结果靠重新读取 policy 判定 applied / not-applied / unknown。**上线前迁移（必须先做,无 DB schema migration）**:① 给 SA 在 organization 及全部后代 folder 上补齐 `resourcemanager.projects.list` 与 `resourcemanager.folders.list`（保留既有 projects.get / getIamPolicy / setIamPolicy）;② 在集成编辑页选择账单模式;若要开放账单 IAM,只填经财务确认属于该 organization 的账号;③ 点「测试连接」做只读 dry-run——必须为绿;响应 `detail.discovered_scopes` 是本次严格边界下的完整 identity 报告,`detail.scope_partitions` 是每个 partition 的权威性;红色时不要 run scopes-sync / 离职回收;④ 导出当前 active GCP scopes，与该 dry-run 报告核对跨组织/历史脏行及其 active grants，先人工审计；确认后 run `cmdb.scopes_sync`，让不再权威的旧 scope 走正常 soft-close，再复核审计记录。**兼容策略**:不保留 credential-visible legacy 开关（它无法提供租户隔离证明）;已有 `create_scope_billing_account` 且未填 mode 的配置仅向后推断为该单账号 allowlist，其他存量配置缺 mode 一律失败并提示迁移。**0 DB migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-08-02, AWS / Aliyun / Tencent / UCloud IAM 用户名派生拒绝非 ASCII 字母数字 —— #2390)**. **现象**:邮箱 local-part 含中文等 Unicode 字符时，Python `str.isalnum()` 会把它们当作字母数字保留下来；原实现因此生成云厂商不接受的 IAM 用户名，导致授权创建失败，后续按用户名回收也可能找不到账号。**修复**:四个云 driver 现在只保留各厂商允许的 **ASCII** 字母数字与标点；若过滤后为空，继续使用原有确定性 fallback `helix-<sha1[:12]>`，同一邮箱的授权与回收仍稳定命中同一个名字。**Operator 该做什么**:① 发布后新请求自动生效，无需配置；② 此前卡在 `PENDING` 的相关授权可直接重试；③ 若曾人工建立过不符合该派生规则的云侧账号，清理前先按 Helix 标签/人员归属核对，避免误删同名账号。**0 migration / 0 cron / 0 RBAC / 0 MCP 改动**。
> - **(2026-08-02 follow-up, raw baseline + tenant-wide role prewarm 收口)**. 历史 job 的 raw `upserted` 若被重复页放大，sweep baseline 现以实际活跃行数为上限，force audit 会标记 `previous_sync_capped_by_existing`。tenant-wide `scope_id=NULL` role 的全量 cron 与 post-provision prewarm 现共用 canonical upsert，护栏按 `(scope_id, role_key)` 逻辑计数，可在普通 sweep 中清理历史 NULL 重复行。**Operator 该做什么**：无需迁移或手工清理；首次 roles sync 可能显示软删了旧重复目录行，这是逻辑去重，不是云权限被撤销。**0 migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-08-02, sweep 小基线精确阈值 + 唯一观测计数 + Aliyun RD/管理员同步链 fail-closed)**. resource/scope/role 三条 sweep 现在按精确比例判断 drop，不再把最小保留数向下取整；默认 20% 时 2→1 与 6→4 会拒绝，5→4 恰好边界仍可继续。护栏分子也改为本 job 实际落库的唯一活跃行，不再用可被重复页/租户级 role fan-out 放大的 raw `upserted`。出现 `refusing ... sweep` 的处理方式、平台 tolerance 临时调整与 force-sweep 流程均不变；force audit 会同时保留 raw/unique 计数便于核对。Aliyun scope sync 只有收到明确 `ResourceDirectoryNotInUse` 才按单账号工作；STS master 身份解析、member AccountId 至少 10 位纯数字、目录 master ID 必须与 STS 一致、`Accounts`/`TotalCount` 必填结构、`TotalCount` 必须为非负整数、每页剩余行数与最终唯一总数全部属于权威快照契约，403/429/5xx/timeout、重复/畸形 AccountId、双 master 身份、提前空/短/超长页、TotalCount 变化或类型错误、非 JSON 200 和超过 1000 页均让 job 失败并保留原 scopes/grants。管理员「立即同步」的 scopes→roles→grants 链在真实前序 failed/partial_failure 时跳过后续步骤；审计的 `failed_stages` / `skipped_stages` 与 failure/partial/skipped 状态明确指出实际执行范围，嵌套错误文本写入前递归脱敏云凭据。**Operator 该做什么**：常态无需动作；若发布后小租户或重复数据源首次暴露真实大幅删减，先在云控制台核对，再按下文 force-sweep 流程显式确认，不要通过反复重跑绕过；Aliyun STS、权限、限流或分页一致性故障应修复连接器/等待目录稳定后重试，不能当作 no-RD；立即同步审计为 failure/partial 时先处理第一个 failed stage，不要把 skipped 后续阶段当成独立故障。**0 migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-08-02, Tencent CAM role/grant/revoke 权威分页 + OwnerUin 租户身份 fail-closed)**. `ListPolicies` 与逐用户 `ListAttachedUserAllPolicies` 任意页授权/限流/服务故障、提前空/短页、分页总数变化/非法、重复页、空 PolicyName、最终唯一行数不等于 `TotalNum` 或超过安全上限都会让同步失败并保留现有角色/授权，不再把前缀交给 destructive sweep。单 grant 的 direct/group 撤权与离职账号 policy 清扫也走完整分页：第 2 页策略、查询失败或页数上限不会再误报已撤销，离职清扫在拿不到权威清单时会停止并告警。Tencent scope 现在用主租户 `OwnerUin`，当前 CAM 服务子账号 `Uin` 只留作诊断；换子账号 AK/SK 不会另建幽灵租户，OwnerUin 缺失则不写 scope。所有相关 SDK 错误统一脱敏。**Operator 该做什么**：查看失败信息中的 operation/page，修复 CAM 权限、用户缺失 Uin/email/PolicyName 等数据，或等待限流/目录变化稳定后重跑；不要对 provider 一致性故障使用 force-sweep。若历史上曾轮换子账号凭据，发布后检查是否存在以旧子账号 Uin 为 external_id 的重复 Tencent scope，确认无关联活跃 grant/cost 后再走正常下线/回收流程，禁止直接删库。**0 migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-07-30, 月度账单超预算的飞书**群** webhook 不再带绝对金额 —— 双口径保密红线第二批收口)**. **背景**:云成本有折前(list)/实付(invoice)两个口径,实付只有保密圈(`can_view_invoice_measure` = finance / cloud_admin / global_admin)可读,红线保护的是两者的**比值**(公司折扣率)。这条告警的收件人是**飞书群成员** —— 群成员资格代码无从校验,结构上不可能是保密圈的子集;而它的金额取自 `cmdb_billing_monthly.total_cost_usd`,**该表没有 list 列**,所以那个数恒为实付。群里任何非圈成员拿它去除自己在 `/cmdb/cost/*` 上合法可读的折前月合计,即得折扣率。**改了什么**:卡片正文改为发**超出预算的百分比**(无量纲,正是「为什么这条告警会发」要的那个数)+ 对账 deep link + 一句「金额为实付口径,仅财务 / 云管理员可见,请在对账明细页查看」;`账单金额` / `预算阈值` / `超出金额` 三个绝对数不再出现在群里。**Operator 该做什么**:① 无需任何配置动作,发布即生效;② **预期会被问「卡片怎么没数字了」** —— 答案是去云资产总览「账单」tab(deep link 已在卡里)看,圈内账号能看到完整金额;③ 取证/对账**没有损失**:`audit_logs` 的 `cmdb.billing_alert_sent` 行仍记录完整的 `total_usd` / `budget_usd`,只是读取时按读者是否在圈内脱敏(`app.core.masking`);④ 告警的**触发条件、去重窗口、收件人**一律未变,只有文案变了。无 migration / cron / RBAC / MCP 改动。
> - **(2026-07-24, 离职时自动清扫离职人残留的 AWS Identity Center 账号授权 —— 根治控制台「姓名为空」的孤儿记录)**. **现象(用户报告)**:AWS IAM Identity Center 里某账号(如 `qa-dp-testing-dev` / `448141240262`)的「Assigned users and groups」有一堆**用户名为空**但仍挂着权限集(`DBaaS-DevLeader-Role` 等)的记录。**根因**:Identity Center 的一条授权 = `(PrincipalId, PermissionSet, Account)` 三元组;控制台显示的「用户名」是实时拿 PrincipalId 去 Identity Store 反查出来的。离职时 SCIM 反配**删掉了目录用户**,但 AWS **不级联删除**这条 account assignment → 它继续指向一个已不存在的 PrincipalId,反查不到名字 = 空姓名 = **孤儿授权**。凡是当年直接在 AWS 控制台/别的工具授、Helix 从没建过 grant 行的授权,离职后都会变成这种空姓名孤儿。**改了什么**:离职转 `left` 后、在**SCIM 删用户之前**,自动解析该离职人还活着的 Identity Center principal,把 TA 名下的 account assignment 直接删掉(写 `status='revoked'` 历史行,带真实 person_id + `revoke_reason='leaver_principal_swept'` + 审计),从此不再产生新的空姓名孤儿。`run_tick` 自动离职 + leaver「立即推进」两条路径都走 `sweep_then_offboard_sync`(先清扫后反配)。**护栏**:**只清非生产账号**——有 Helix scope 记录且非 prod 的才删(prod 走默认锁死的生产写闸,跳过);**没有 Helix scope 记录的账号只上报不删**(无法判定是不是生产就不碰);已被标准离职 grant 回收覆盖的授权让给它,不重复删。幂等。**Operator 该做什么**:① **常态无需动作**——以后离职者的 AWS 残留授权在离职当天自动清掉;② **清理存量空姓名孤儿(principal 已删的历史记录,不归自动流程)**:用既有 `scan_cloud_orphan_assignments`(只读,**务必带 `scope_id` 单账号扫**)预览 → 复核 → `cleanup_cloud_orphan_assignments` 逐个删,按**非生产 scope** 逐个来;或用 MCP `scan_leaver_cloud_assignments`(某即将离职者的删除预览)/ `sweep_leaver_cloud_assignments`(执行);③ **生产账号的空姓名孤儿**:默认锁死不动;确需清理,先在平台设置把 `cmdb.prod_cloud_writes_enabled` break-glass 翻 `true` → 清 → 立即翻回 `false`;④ **观测**:sweep best-effort,失败只告警不阻断离职目录反配;真删了会在审计 `cloud.grant.leaver_principal_swept` 与 scope 详情「已结束」tab 可见。**0 migration / 0 cron / +2 MCP(`scan_leaver_cloud_assignments` / `sweep_leaver_cloud_assignments`,hint 433→435)/ 0 RBAC**。
> - **(2026-07-24 follow-up, 离职 SSO sweep 历史写幂等修复 —— 消除「重加后再离职」假 CRITICAL)**. 上条 sweep 写 `status='revoked'` 历史行时是**盲插**,会撞两个唯一键:(a) 同 `(scope,person,role)` 此前已有 REVOKED 行(即某人先被 Helix 授过并回收、之后又在 AWS 控制台重加、再离职——正是本能力要清的核心场景),(b) `orphan_cleanup` 早先写的 `person_id=NULL` 同 principal 行。撞键→IntegrityError→被 catch 成 `logger.critical("cloud-side delete SUCCEEDED but Helix history write FAILED — manual reconciliation required")` + 该 assignment 记为 error,并破坏重跑幂等。**改了什么**:改为按两个唯一键 upsert——命中即原地 UPDATE(把 NULL person_id 升级为真实可归属的),不再盲插。**Operator 影响**:云侧删除行为不变(访问照样切断);此前会看到的「假 CRITICAL / 需人工对账」告警在这个场景下不再触发,历史归属行也不再丢。**0 迁移**。
> - **(2026-07-21, AWS Organizations 跨账号扇出 —— EC2 库存 + 利用率覆盖全组成员账号,配置开关默认关)**. **背景**:云成本洞察上线后 AWS 机器数=0。真相(生产凭证实测):`aws-primary` 接入的凭证落在**组织管理/付款账号** `317766989874`,该账号自身 0 台 EC2——所有机器在它下面的 **90 个成员账号**里;管理账号能 `organizations:ListAccounts` 列出全部,但**没有跨账号 `sts:AssumeRole` 权限进不去**(AccessDenied)。`aws-sg`(`841162690310`)是独立账号、自身也 0 台。**能力**:`org_fanout` 开启后 `list_servers`/`list_server_metrics` 枚举每个 active 成员账号、STS-assume `org_member_role`(默认 `OrganizationAccountAccessRole`)进入并发拉取(并发 8、逐账号故障隔离),机器 `raw` 打 `_aws_account_id` 戳,利用率同步据此重新 assume 到正确账号查 CloudWatch。**Operator 该做什么**:① **上线前提(须云团队一次性授权,Helix 侧从不改 IAM)**:给管理账号 `PingCAP` 用户加 `sts:AssumeRole` —— **方案A**(最省事,秒生效):附一条 `{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:iam::*:role/OrganizationAccountAccessRole"}`(该角色 org-created 成员账号自带、新账号自动有;管理员级角色,Helix 只读用);**方案B**(最小权限,推荐):CloudFormation StackSet(service-managed + 自动部署新账号)全组建只读角色 `HelixCloudReadOnly`(仅 `AmazonEC2ReadOnlyAccess`+`CloudWatchReadOnlyAccess`,信任 `arn:aws:iam::317766989874:user/PingCAP`),再给 `PingCAP` 加 `sts:AssumeRole` 指向 `arn:aws:iam::*:role/HelixCloudReadOnly`。② **权限到位后**:在 `aws-primary` 集成配置置 `org_fanout=true`(方案B 再把 `org_member_role=HelixCloudReadOnly`),开启 AWS 库存/利用率 cron,`sync-now` 验证——90 账号机器+利用率自动进,新账号自动覆盖。③ **未开启前一字节行为不变**(默认关,AWS 走原单账号路径)。④ **观测**:成员账号 assume 被 `AccessDenied` 拒(受邀账号没有那个角色)→归入 `skipped_legs`(`reason=member_role_unavailable`,管理员可见、能力保持绿、**不告警**——结构性稳定状态,同 OptInRequired);瞬态传输错误才落 `partial_failures`(可重试);`ListAccounts` 整体失败才 `ProviderAuthError`。**实测(2026-07-22 权限到位后)**:90 成员里 64 个 org-created 可进、26 个受邀账号被静默跳过;只读扇出已读到 **5585 台 EC2**(44 账号有机器)。给受邀账号补角色/铺 StackSet 后下次同步自动点亮。**0 迁移 / 0 cron 行数变化 / 0 RBAC / 0 新 MCP 工具**。
> - **(2026-07-20, 云服务器自助转让拒绝「从未被实时同步确认」的僵尸行 —— Oncall_bot 悬空转让事故 follow-up)**. **现象**:员工收到「X 把 1 台云服务器转给你了」的邮件,但在「My Resources → Servers」查不到那台机器。**根因**:被转让的主机(如 `Oncall_bot` / `uhost-dyvjjq4e`)其实**云侧早已释放**,只因连接器一度漏扫其 project 而作为陈旧行(`last_seen_sync_job_id IS NULL`、实时同步从未确认)滞留台账;它在被 `cleanup_stale_migration` 软删**之前**被转让出去,收件人于是拿到一封指向不存在资源的「你已成为 Owner」邮件(连接器漏扫本身已在 2026-07-14 #1932 修复,见对应条)。**改了什么**:`transfer_owned_servers`(自助转让 + MCP 同源)现拒绝转让 `last_seen_sync_job_id IS NULL` 的行——对**所有人含管理员**;整批被拦回 400「资源待同步对账、可能已不存在,请刷新」,混合批转已确认行、跳过僵尸行并在 `summary.unconfirmed` 报告。**Operator 该做什么**:① **常态无需动作**——存活库全部是同步确认行,守卫永不误伤;② **有人报「转让报错说资源可能不存在」**:说明那台机器云侧已不在了、只是台账陈旧——去云控制台核实,确已释放则该行会被下一轮同步/`cleanup_stale_migration` 软删清掉,**无需强行转让**;③ **确需给某台真实存在但显示未确认的机器换 owner**:先对其云账号跑一次 sync-now 让实时同步确认它(`last_seen_sync_job_id` 落值),再转让;④ 收件人名下若已有一条这类软删僵尸(历史遗留),完全惰性——半年 Owner 确认 cron 本就 `deleted_at IS NULL` 过滤,不会追问、不显示于任何面,无需清理。**0 migration / 0 cron / 0 RBAC / 0 MCP**。
> - **(2026-07-19, `cmdb.resource_sync` 默认节奏分档 —— 每小时 → 热 4h / 冷 12h,migration 418)**. **背景**:生产台账审计发现 27 个启用的 resource_sync(云账号 × 能力)全部**每小时**跑,3 天窗口 ~1600 次 ok-run **100% 零变更**(每次 `upserted=0 deleted=0`)——500 人公司的云资产一周变不了几次,纯空转;这批 cron 也正是 2026-07-19 连接池打满事故的聚堆源头。**改了什么**:默认节奏按能力分档——**热(每 4h)**:`servers` / `iam` / `applications`;**冷(每 12h)**:`domains` / `dns_records` / `certificates` / `cdn` / `object_storage` / `load_balancers`(证书还有每日 `cmdb.cert_probe_sweep` 直接探测,12h 发现延迟对到期告警不可见)。小时相位 + 分钟都按 job 名哈希铺开,不会在 0 点/12 点重新聚堆。**migration 418** 只改写仍等于旧代码默认(`哈希分钟 * * * *`)的存量行——**管理员在 /admin/scheduled-jobs 手改过 cadence 的行不动**。**Operator 该做什么**:① 常态无需动作——变更检测降到 4h/12h 一次,数据面(billing/cost/grants 每日)不受影响;② **需要即时新鲜度**(刚开了新机器/改了 DNS 想立刻看到):对应 job 点「立即执行」,或云账号详情页 sync-now;③ **想要某个能力回到更密节奏**:直接在 /admin/scheduled-jobs 改该行 cron——reconcile 和未来迁移都不会碰你改过的行;④ 同步历史里看到 run 间隔变大是预期,不是任务坏了(staleness 口径都是天粒度,不受影响)。**1 migration(418,纯数据)/ 0 cron 行数变化 / 0 RBAC / 0 MCP**。
> - **(2026-07-16, API Key AK/SK 自助轮换 —— 云侧凭证双活轮换 + 自动退役)**. **背景**:员工需要「换一把新的 AK/SK」而不停机。轮换在云侧的语义按家不同,operator 需知道每家实际动了什么:**Bedrock** = 在同一 IAM 用户上多加一把 access key(AWS 每用户硬顶 2 把),旧的到期 `delete_access_key`;IAM 用户/策略/组/标签/成本分摊规则**全不动**。**Bailian** = 在同一工作区新建一个 `sk-`(工作区可多把并存),旧的到期 `DeleteApiKey`;RAM 子账号不动。**Azure OpenAI** = 重生成账号的**另一个** key 槽(key1/key2 原生双槽)拿去发给员工,旧槽到期再重生成使其失效;账号/RG/project 不动。**退役时机**:员工发起时自选宽限期(立即 / 3 / 7 / 14 天),`apikey.rotation_retire` cron(每小时)到点退役旧凭证;24h 内到期发提醒。**Operator 该做什么**:① **常态无需动作** —— 员工在「我的 API Keys」自助轮换,cron 自动退役旧的;② **看轮换态**:`api_key_assignments.rotation_retire_at` 非空 = 轮换中,`detail.<provider>.rotation` 有完整 handle;③ **退役失败观测**:cron 对云侧幂等(凭证已被吊销/离职清理删掉的直接 no-op),某行退役异常落 `summary.errors` 但**仍清空轮换态**(旧凭证可能还活着 → 失败通知器告警,需人工到云控制台确认删除);④ **Bedrock 2-key 限制**:若某 IAM 用户已有 2 把 key(上一次轮换没退役,或有人在控制台手工建了 key),再轮换会被 `ROTATION_KEY_LIMIT` 拒——等上一轮退役或手工清掉多余 key;⑤ **吊销优先**:`revoke_key` 会把整个云身份连同两把凭证一起拆掉并清空轮换态,轮换中的行被吊销后 cron 自动跳过。**migration 401 / +1 cron(`apikey.rotation_retire`)/ 0 RBAC / +3 MCP(`rotate_api_key`/`complete_api_key_rotation`/`cancel_api_key_rotation`)**。
> - **(2026-07-14, 金山云(历史归档租户)退出半年盘点 + 541 台服务器 + 13 LB 软删)**. **背景**:半年盘点(云服务器 owner 确认)一直把金山云机器推给 owner 确认,但金山云早已是**只读历史归档租户**(DECISIONS 2026-04-23,PingCAP 不再在金山云跑任何东西,驱动无凭证/无同步/无 capabilities),确认这些机器毫无意义。**改了什么**:① 代码护栏——`KSYunDriver.archival=True` + `registry.archival_providers()`,`ServerOwnerConfirmationService.list_due_owners` 排除归档租户下的服务器(将来任何归档云自动排除,非硬编码 `ksyun`);② 数据清理——生产**软删**(`deleted_at=NOW()`)金山云全部 541 台服务器 + 13 个 LB,它们从盘点/CMDB 服务器列表/所有管理面消失,整合租户行(`金山云 (历史)`)与历史成本数据保留。**Operator 该做什么**:① **常态无需动作**——金山云机器不再进盘点,CMDB 里也看不到;② **需恢复某台/全部**(误软删补救):金山云无在线同步不会自愈,须手工把对应行 `cmdb_servers` / `cmdb_load_balancers` 的 `deleted_at` 置回 `NULL`(按 `integration.provider='ksyun'` 定位);③ **彻底物理删**(如将来确认永不需要):需再次显式确认后跑 SQL——默认不做。**0 migration / 0 cron / 0 RBAC / 0 MCP 工具数改动**(纯代码护栏 + 一次性数据软删)。**⚠️ 更新(同日 follow-up,用户指示"历史的也清理、咱们不管这个了")**:金山云数据已在生产**物理硬删**——`cmdb_servers` 541 + `cmdb_load_balancers` 13 + `cmdb_network_owner_history` 150 + `integrations`(`金山云 (历史)`)1,**永久不可逆,上面第②条的 `deleted_at` 恢复路径已失效**(数据不复存在)。盘查确认金山云**无成本数据**故硬删无成本损失。`KSYunDriver` + `archival` 护栏**保留**(通用归档租户排除,defense-in-depth)。若将来要新接金山云,须在管理台重新添加租户。
>
> - **(2026-07-14, UCloud cost_facts 补齐 region + 资源名 —— 按区域/资源看「小账」不再空白)**. **现象**:明细成本 `cmdb_cost_facts` 的金额/项目/服务/计费方式/资源 ID 全对且与月度对账一致(2026-06 = 398 行/13,152 CNY,按项目 IT_Infra/CICD/IT_Safe/CEG 拆分吻合),但 **`region` 100% 空、`resource_name` 回落成资源 ID**。**根因**:UCloud `ListUBillDetail` 行没有 `Region` 字段(恒 null),区域在 `AzGroupCName`(中文名「华北(北京2)」「香港」),资源名在 `ResourceExtendInfo` 的 `{KeyId:"name"}`。**改了什么**:`list_cost_facts` 的 region 回落到 `AzGroupCName`(真 `Region` 仍优先),resource_name 从 `ResourceExtendInfo` 取 `name`(无则回落 ID);仅 cost_facts(daily 无区域维度)。**Operator 该做什么**:① 常态无需动作——每日 `cmdb.cost_facts_sync` 起新行即带 region/名字;② **历史回填(已随本次运维做)**:部署后对 `cmdb.cost_facts_monthclose`/`cost_facts_sync` 重跑 2026-03~07,补历史行的 region/资源名。**0 migration / 0 cron / 0 RBAC / 0 MCP 工具数改动**。
> - **(2026-07-14, UCloud 月度账单从「订单」改「消费」口径 —— 修 `cmdb_billing_monthly` 长期全 0 + 账单强制账户级)**. **现象**:`cmdb_billing_monthly` 每月 0.00(2026-03~06),而 daily/cost_facts 有真实数。**实测(现拉 API,主账号 50602349)**:2026-06 真实消费(`ListUBillDetail`)= 13,167 CNY,Helix 存的日汇总 13,173 CNY,差 0.04% —— **明细账单本来就对**;错的只有月度表:它用 `DescribeOrderDetailInfo`(订单 API),①解析读错响应键(读 `OrderDetails`,实际 `OrderInfos`)恒 0,②订单≠消费(6 月订单 47.6 CNY vs 消费 13,167 CNY,~276× 低报)。**改了什么**:月度 `get_monthly_billing` 改用当月 `ListUBillDetail` 聚合(与 daily/cost_facts 同源、含 per-service);并给 driver `_call` 加 account-wide 覆盖,三条账单读全部显式账户级(`project_id=None`)—— UCloud 账单是账户级,不带 ProjectId 即返回全项目消费,堵住「填了旧单项目字段会把账单过滤到单项目少报」的隐患。**Operator 该做什么**:① **常态无需动作**——日账单/成本(财务视图真正用的)一直是对的;下个月 2 号 `cmdb.billing_sync` 会自动写出正确月度值;② **历史回填(已随本次运维做)**:部署后对 `cmdb.billing_sync` run-now 传 `config={"month":"YYYY-MM-01"}` 逐月重跑 2026-03~07,把历史 0 行改写为真实消费;③ **别填 UCloud 集成的单个 `project_id` 字段**——它只影响资源发现的单项目过滤,且账单已强制忽略它;多项目用 `project_ids`。**0 migration / 0 cron / 0 RBAC / 0 MCP 工具数改动**。
> - **(2026-07-14, UCloud `project_ids` 补进管理台可配字段 + 主账号权威盘点清理 89 条迁移遗留漂移行)**. **现象**:云服务器管理里 `DLP-test`(`uhost-c1pa0wrhq5u`,项目 `org-eneelp`)显示「运行中」、同步时间停在 2026-05-18,实际早已释放。**排查(prod 只读)**:Helix 用的正是 UCloud 主账号 key(`sre-bot@pingcap.com` / `UserType=0` / CompanyId `50602349` / 北京星云深图,`GetUserInfo` 证实),全 7 项目全区域全量分页盘点**只有 23 台**(22 台 `org-z2uz3o`/IT_Infra + 1 台 `org-d50rns`/CEG-DevCon),`org-eneelp` 权威计数为 0 —— DLP-test 确已销毁。而 Helix 台账有 **111** 条 UCloud 行,其中 **89 条**(`last_seen_sync_job_id IS NULL`、名字带 2022 日期)是当年旧 CMDB 迁移灌入、从未被实时同步确认的陈旧行。**根因**:UCloud 列表接口按单个 `ProjectId` 过滤,`UCloudProvider` 自 PR#558 起读 `project_ids`(CSV)支持多项目,但该字段**没进 `UCloudDriver.config_schema`** → 管理台无从填 → `ucloud-primary` 只扫默认项目(`org-z2uz3o`),其余 6 项目资源不可见、老行既不刷新也 sweep 不掉。**改了什么(代码)**:`project_ids` 加进 `UCloudDriver.config_schema`(带 hint;单个 `project_id` 保留为旧字段;纯 driver 声明,无 migration/前端改动)。**已落地的 prod 运维**:① 给 `ucloud-primary` 写入全 7 项目的 `project_ids`(直写 `integrations.config_public`,后续 UI 编辑因 merge 语义不会丢);② 跑全量同步 upsert 23 台(含之前不可见的 `uhost-gk4jmzb5eoo`);③ 显式软删 89 条陈旧行(可逆,`deleted_at`)+ 审计 `cmdb.server.cleanup_stale_migration`(detail 含全部 instance_id + 原因)。**Operator 该做什么**:① **常态无需动作** —— 同步现覆盖全 7 项目,稳态 23 台;② **误删补救**:89 条是软删,`/admin/cloud/recycle-bin` 或把 `cmdb_servers.deleted_at` 置 NULL 即恢复,审计里有完整清单;③ ⚠️ **清理迁移遗留行必须显式软删,`force-sweep` 也不管用** —— `_safe_sweep` 只处理 `last_seen_sync_job_id IS NOT NULL` 的行(2026-05-28 收窄以永不误删迁移/手工行),迁移导入行(NULL)对 sweep 免疫;④ **确认单账号后再清理**:主账号 key 只能看自己账号,判「云侧缺失=已销毁」前须确认无第二个 UCloud 账号(本次已向 owner 确认只有 50602349 一个),否则那些行可能活在别的账号里、需各自的连接器,**切勿删**。**0 migration / 0 cron / 0 RBAC / 0 MCP 工具数改动**。
> - **(2026-07-08, 离职 Google 账号挂起 2 天后自动硬删除 —— 收口「永久挂起」终态,不做交接)**. **背景**:2026-07-01 翻案后离职者 Google 账号在末日次日被自动**挂起**,但此前永久停在挂起态、无删除动作。现收口为终态:挂起 **2 天后**硬删除。**改了什么**:新增每日 03:00 UTC 定时任务 `hr.offboarding_google_account_delete`——对已 `completed` 的离职流程,在 `leaver_date + delete_lead_days`(平台 config,默认 **2**,按**员工本地时区**)到达后,对**已确认挂起**(`session_enforcement.google_account=='suspended'`)且**非保留号**的账号调 `users().delete` 硬删。**安全网**:Google 删除后保留 **~20 天可 `undelete` 恢复**,加上挂起→删除的 1 天间隔,足以覆盖误离职发现窗口——所以自动、不做数据交接(资源在 blocker 级联早已转交/回收)。**护栏**:① 删除永不跑在它依赖的挂起之前(挂起失败/未确认的**不删**,等 reenforce 补挂后下轮再删);② 删除时**再查一次**保留分类(`external_account_registry` 服务/共享/外部号永不删);③ 返聘(`employment_status='active'`)跳过;④ 幂等键 `session_enforcement.google_account_deletion`(`deleted`/`already_absent`/`retained` 为终态,永不重删)。**Operator 该做什么**:① **常态无需动作**——离职号挂起 2 天后自动删,Google 侧保留 ~20 天可恢复;② **误删/误离职补救**:到 **Google Admin 控制台**在 ~20 天恢复窗口内 `undelete` 该账号(Helix 侧无 undelete 路径);③ **改删除时机**:平台设置调 `hr.offboarding_google_account_delete` 的 `config.delete_lead_days`(如设更长给更宽发现窗口);④ **删除失败观测**:真·厂商 `users.delete` 失败(如 SA 丢 `admin.directory.user` 域授权 → 403)会让该 run 落 `failed` 并触发失败通知器,账号停在「已挂起未删除」,修好 SA 授权后下一轮自动重删(保留跳过/未到期不算失败);⑤ **想保留某离职号**:把其邮箱在 `external_account_registry` 标为服务/共享/外部类,则既不挂也不删。**migration 395 / +1 cron / 0 RBAC / 0 MCP 工具数改动**。
> - **(2026-07-06, 自助 DNS 写操作从生产云写冻结中解耦 —— DNS 专属开关 `cmdb.dns_writes_enabled` 默认允许,被锁失败行可「重跑」而非重新发起)**. **背景**:DNS 变更(改一条解析记录)之前和云 IAM 写操作共用 `cmdb.prod_cloud_writes_enabled` 这把 kill-switch —— 于是一次为保护生产 IAM 而下的冻结,把每一条**已审批**的 DNS 申请也失败掉(见下方 2026-07-04 条),而 DNS 记录改动其实低风险、易回滚,不该被 IAM 锁连带。**改了什么**:① DNS 写操作现由**独立开关 `cmdb.dns_writes_enabled`** 管控,**默认 `true` = 允许**(与 IAM 锁的默认锁死相反;缺行也算允许,migration 386 seed `true` 仅为在平台设置里可见可调),`prod_cloud_writes_enabled` 冻不冻结**不再影响** DNS;② 被 DNS 开关锁住而失败的行**可恢复**——审批仍有效,IT 把开关翻回 `true` 后,在申请详情点「重跑」即按原审批直接下发变更(不重开审批、不 bypass），成功回原飞书实例发成功评论;入口另有 `POST /cmdb/dns-record-requests/{id}/resume` 与 MCP `resume_blocked_dns_request`(均需 `cloud_admin`/`global_admin`)。「重跑」只认「仅因写锁失败」的行(同时匹配新开关与旧 `prod_cloud_writes_enabled` 两种错误,故**历史上被生产冻结失败的 DNS 申请解锁后也能直接重跑**);真·云厂商写失败的行不可重跑(避免重复下发),仍开着锁时点重跑会被拒(先解锁)。**Operator 该做什么**:① 常态把 `cmdb.dns_writes_enabled` 保持 `true`(默认即是),自助 DNS 正常执行;② 冻结生产云 IAM(`cmdb.prod_cloud_writes_enabled=false`)**不再**连带停掉 DNS —— 若确需单独冻 DNS,单独把 `cmdb.dns_writes_enabled` 翻 `false`;③ 处理历史/冻结期被拒的 DNS 申请:确认 `cmdb.dns_writes_enabled=true`,到申请详情点「重跑」即可,**无需让申请人重新发起**。**migration 386 / 0 cron / 0 RBAC / +1 MCP(`resume_blocked_dns_request`)**。
> - **(2026-07-05, `cmdb.cost_facts_monthclose` 月结重拉从「单进程串行全账号」改为「按账号扇出 + 拆窗」调度器 —— 修「从未跑完过」)**. **现象**:月结重拉定时任务失败,`timed out after 1800s`。**只读 prod 实证**:该 job 一次都没成功过——2026-06-04 跑 >6h 被 6h stale-running 清扫器判死、2026-07-04 命中 1800s 超时;`config={}`(窗口=上一整月,正常)。**根因**:旧实现把 8 个启用云账号 × ~205 tenant scope × 整月 line-item 重拉串行塞进**一个** 1800s 超时;实测每账号仅 3 天窗口的日跑就已 Aliyun 海外 max 1040s、Azure max 1649s,整月 ≈ 10× → 单账号整月即可能数千秒,聚合远超 7200s 上限,**调超时修不好**。**改了什么**:`run_cost_facts_monthclose` 重构为**月度调度器**——对每个启用云账号把上月切成 `chunk_days`(默认 7)天子窗,逐个经该账号自己的 `cmdb.cost_facts.<slug>` 任务(`execute_job` + 显式过去窗口)重拉;同账号子窗串行(共享 `sched_job:{id}`/`cloud_sync:cost_facts:{id}` 锁,防撞每日 tick + 防重复计费),不同账号有限并发(`max_parallel` 默认 3)后台 drain;调度器秒级返回,慢/失败账号按账号隔离、可独立重试。**关键陷阱(已修)**:`_resolve_recurring_billing_window` 会丢弃 `end_date<昨天` 的窗口 pin(Azure stale-pin 冻结事故护栏),月结的过去窗口若直接派发会被静默丢弃回退成 3 天——加**仅经临时 override 注入、不落库**的 `allow_past_window` 旁路(每日 cron 仍照旧自愈)。**Operator 该做什么**:① **正常无需动作**——每月 5 号自动按账号×周窗扇出对账;② 想立刻补:对该 job run-now 传 `config={"start_date","end_date"}`(可选 `chunk_days`/`chunk_timeout_secs`/`max_parallel`);③ **失败观测面变了**:月结硬失败现落在**各账号 `cmdb.cost_facts.<slug>` 的 run 行 + 该账号失败告警**(按账号隔离),`cmdb.cost_facts_monthclose` 自己的 run 行只记「dispatched N 账号 × M 子窗」派发计划、不再因某账号失败整体标红;④ 某子窗异常大月仍超时 → 该子窗标 failed+告警、不阻断该账号其余子窗,把 `chunk_days` 调小(如 3,实测稳 <1800s)重试。无 migration / cron / RBAC / MCP 改动。
> - **(2026-07-04, 生产写冻结期间的自助 DNS 申请改为「失败并通知申请人」,不再无声搁死)**. **背景**:自助 DNS→飞书审批→执行链路命中全局生产写冻结(`cmdb.prod_cloud_writes_enabled=false`)时,旧行为把申请行留在 `approved` 并声称「解冻后自动重放」—— 但**没有任何 reconciler 会重驱动 approved 行**,飞书回调也已返回 200 不再重投,申请就此无声搁死,除非申请人自己发现。**修复后**:命中冻结即把行置 `failed`(error_message 说明「生产云写被全局冻结,Helix 未执行」)并给申请人发飞书评论「解冻后请重新发起」,与同链路的自审批 / 离职申请人拦截路径、以及平台 DNS 失败恢复模型(重新提交)一致。**Operator 该做什么**:① 冻结期间被拒的自助 DNS 申请**不会自动补跑** —— 解冻后让申请人重新发起即可;② 注意该 DNS 冻结门用的是**全局**开关(不区分该 DNS 是否指向 prod scope,因 DNS 记录无 prod-scope 标签,这是有意让 DNS 也随全局冻结);若你把 `cmdb.prod_cloud_writes_enabled` 长期保持 `false`(默认锁死姿态),则**所有**自助 DNS 申请都会走「失败并通知」路径 —— 需要放行自助 DNS 时,与其它 prod 写操作同姿态临时开锁。无 migration / cron / RBAC / MCP 改动。
> - **(2026-07-03, auto-Owner-grant backfill 候选谓词收紧 —— 只修自助开通的 scope,owner 已持 owner 等价角色(任意来源)即出集)**. **现象(prod)**:`cmdb.auto_owner_grant_backfill` 每 tick `scanned=20 / fixed=0`,零审计行。98 个 CMDB 迁移导入的存量 scope 匹配旧谓词(owner 无 `source='helix'` ACTIVE grant),但 owner 早已在云侧持有 IAM(grants_sync 镜像成 `source='discovered'` ACTIVE 行),finalize 撞 ConflictError 静默 no-op;`ORDER BY created_at ASC` 让最老 20 个永久占满 batch,真候选被饿死。**修复后**:候选必须能溯源到自助开通请求(`cmdb_cloud_scope_requests.created_scope_id`),且 owner 持有 owner 等价角色的 ACTIVE grant(不限 source)即出集(AWS 按 `cloud.aws.scope_request_default_owner_role_name` 权限集名匹配:grant 行 `role_display_name` 或 `cmdb_cloud_roles` 目录表按 ARN 翻译 —— discovered 行 display name 为 NULL,靠后者);ConflictError 分支补 `status=skipped` 审计。**Operator 该做什么**:正常稳态应为 `scanned=0`;存量导入 scope 的 owner 若确需补 owner 等价 IAM,走 scope 详情页 owner-add / `assign_scope_owner` 人工路径 —— cron 不会也不应自动给存量 owner 提权(候选里曾包含 `pingcap-security`、AWS 管理账号等敏感 scope)。无 migration / cron / RBAC / MCP 改动。
> - **(2026-07-03, auto-Owner-grant 遇到「principal 在云厂商 IdP 不存在」(如已删除的 Google 账号) 不再每 10 分钟空转 —— 复用 2026-06-02 的 unresolvable 退避标记)**. **现象(prod)**:`cmdb.auto_owner_grant_backfill` 对 `gcp-storage-datarepl-dev`(owner weiwan@ 的 Google 账号已被删除)每 tick 重试一次 `setIamPolicy`,31+ 条 `cloud.scope_request.auto_owner_grant status=failure` 审计、无退避。**根因两层**:① GCP cloudresourcemanager v3 对已删除账号返回 400 「Request contains an invalid argument.」+ `details` 里的 `ProjectIamPolicyError`(点名 member),驱动的子串分类(`does not exist` / `not a valid`)不认识这个形态 → 当成 generic 失败;② 即使驱动抛了 `CloudPrincipalNotFoundError`,`_dispatch_grant` 的兜底 `except Exception` 也会把它拍平成 generic PENDING(全云驱动同病,AWS/Azure/UCloud 的同名 raise 一样被吞)。**修复后**:驱动识别 ProjectIamPolicyError-点名本次-member 形态;`_dispatch_grant` 把 principal-not-found 落成带因由的失败审计(`detail.reason=principal_not_found`)+ grant 行 `detail.principal_not_found=true`(行保持 PENDING,不是终态)+ 抛类型化 409 `CLOUD_PRINCIPAL_NOT_FOUND`;backfill/finalize 路径捕获后对该 scope 盖 `auto_owner_grant_unresolvable_at` 退避标记(audit `reason=principal_not_found`)—— 与 AWS perm-set unresolvable 同一套 6h 窗口 + 失败通知器每窗口至多一次。**Operator 该做什么**:收到失败通知器卡片(消息含 `owner principal not found at the provider IdP`)时,三选一:恢复/重建该 owner 的云侧账号(Google Workspace 账号等)、`PATCH owner_person_id` 换一个有效 owner(立即清标记重试)、或确认该 scope 该下线就走下线。修完**无需手动重跑** —— 过 6h 窗口 cron 自动重试,成功即清标记 + 清 grant 行上的 principal_not_found 痕迹。**人工路径同步受益**:owner-add / 审批 / 续期撞到同样情况现在直接给 409「账号在云厂商侧不存在」而非「pending + 看不懂的 invalid argument」。**顺带**:`gcp-primary` integration 未配 `admin_impersonation_email`,Admin SDK 预校验(能在 IAM round-trip 之前拦住这类主体)整体失效 —— 属配置项非代码;驱动的报错信息现在会提示这一点,建议有条件时补配。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-19, AWS CUR-in-BigQuery 成本同步 ~38× 扫描量削减 —— 逐账号 fan-out 改为「单趟全账号扫描 + 内存扇出」)**. **背景**:这正是 2026-06-18 那条月结修复里记下的「遗留(已知、独立、上游归属)」——「每个 scope 整月扫全分区天然重复(60 scope 各扫 ~70 GiB),根治需上游 CUR 表按账号聚簇**或单趟全账号查询后内存扇出**」。本 PR 落地后一个方案,不依赖上游改表。**根因实证(只读 prod BQ dry-run)**:CUR 2.0 月表(`gcp-digital-bi.aws_billing_no_payer`)只按 `line_item_usage_end_date` 按日分区、**未按账号聚簇**,`WHERE line_item_usage_account_id = ?` **不减少扫描字节**(加过滤 0.209 GB vs 不加 0.209 GB,逐字节相同),所以 `cmdb.cost_facts_sync` 逐账号发 query(prod 38 个)= 同一批分区被重扫 38 遍。这些扫描费按运行 SA 的项目记入 `gcp-digital-infra`(「Digital Infra」云 scope 的 BigQuery 底噪 ~$6/天 ≈ ~$2.3k/年),并被偶发回填尖峰放大(2026-05-20 全量回填单日 **$536 / 71 TiB**),还反触发了 Helix 自己对该 scope 的「日环比异常」告警。**改了什么**:`AwsCloudDriver` 在 `per_month`+`per_product` 粒度下,首个 scope 的 BQ 调用去掉账号过滤、按 `account_id` GROUP BY **扫一次覆盖全账号**,结果按账号分桶进按 `(start,end)` 键的实例缓存(cost_facts 同步本就复用单 driver 实例),其余 scope 命中缓存零额外扫描。fact 仍逐账号携带真实 scope,`compute_fact_key`/逐 scope DELETE+UPSERT/幂等性逐字节不变。护栏:窗口跨度 ≤ 70 天且 `per_product` 才走批量,否则回退原逐账号流式路径(防超大单次回填把全账号 fact 一次性驻留内存)。**Operator 该做什么**:① **正常无需动作** —— 发布后下一次 `cmdb.cost_facts_sync` 日跑即按单趟全账号扫描,稳态 ~$2.3k/年 → ~$60/年;② **回填请按 ≤2 个月分批**:`/cmdb/cost/backfill` / MCP `cost_backfill` / `cmdb.cost_facts_monthclose` run-now 传 ≤70 天窗口可走便宜的批量路径;传更宽的单次窗口会**有意回退**到逐账号(内存安全),仍能跑完但贵——大回填手工切成多段月窗即可全程便宜;③ **「Digital Infra」BigQuery 底噪不会归零**:Helix 跑 AWS-CUR 查询的扫描费按运行 SA 计入 `gcp-digital-infra` 是结构性的(共享 SA),本 PR 把它降 ~38× 但不消除,该 scope 仍会有少量自查 BQ 成本——属预期,不是漏修。④ **同源姊妹(同 PR 一并修)**:`cmdb.billing_daily_sync`(写权威表 `cmdb_cloud_billing_daily`、驱动日环比告警)对同一批 CUR 月表也逐账号 fan-out,且每账号 ×3 query(totals/service/owner),同样改为单趟全账号扫描 + 内存按账号扇出(driver 仍逐 scope 返回该账号的行,caller 零改动)。⑤ **经核实非 BQ 浪费**:Bedrock per-profile(`list_bedrock_profile_facts`)走 Cost Explorer(不碰 BigQuery)且对 `billing_source=bigquery` 主账号整段 skip——无 BQ 扫描费,无需改。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-18, 帐单获取·告警·增长异常 上线前深度复审 —— 阿里云 CNY ~7× 低报 / 日账单整跑中止 / 异常误判 三处修复)**. **改了什么**:① **阿里云月度账单 ~7× 低报 + 假超预算告警**:阿里云驱动此前把 `total_cost_usd` 预乘硬编码陈旧 `0.14` 却仍标 `currency='CNY'`,在 FX 解析失败回退时被总览读取端再除一次 ~7.2(双重换算);并把 native CNY 值拿去跟 USD 预算阈值比 → 假「超预算」告警。修复:阿里云驱动改为返回 native 值(与腾讯/UCloud 同契约,由 handler 用**动态**汇率换算);`run_billing_sync` 仅在拿到**可信 USD 数**(`effective_currency=='USD'`)时才发预算告警,FX 失败月 skip(记 `budget_alert_skipped`)、FX 恢复后下月重评。② **日账单同步可被一个 CN scope 整跑拖垮**:新注册 CN 云 scope(<7 天基线 + 默认告警规则)在 FX 陈旧时,用户规则分支的换算无护栏会抛异常逃逸整个 per-scope 循环,**丢本 scope 及后续所有 scope 的数据 + 跳过未注册账号对账**。修复:该分支加与 spike 块同形的 FX 护栏(失败 → 记错误、提交已落日行、跳过本 scope),整跑降级为 partial_failure 而非崩。③ **成本异常检测跨集成 resource_id 碰撞**:`detect_anomalies` 此前按 `resource_id` 单键分组,两个云租户的同名资源(如都叫 `i-bp1...`)被求和成一条序列、误判/掩盖 spike;改按 `(integration_id, resource_id)`(对齐 `top_resources`)。**Operator 该做什么**:① **正常无需动作** —— 修复随发布生效,FX 正常时账单/告警口径不变;② **阿里云历史月修正(建议)**:`cmdb_billing_monthly` 是月度 upsert、不会自动重算历史月,如需让此前 FX-失败窗口写错的阿里云月行归正,对相关阿里云 integration 的 `cmdb.billing_sync` run-now 传对应 `month` 重抓(FX 现已恢复即写出正确 USD/currency);③ FX 周期 cron(`ai_spending.cny_rate_refresh` / frankfurter.dev)若长期失败,CN 云的预算告警会被**有意跳过**(而非发错值)——看到失败通知器报 FX 陈旧就排查,别等被错告警误导。④ **遗留(独立、已建 follow-up)**:`alicloud_intl` 驱动同型预乘,但海外租户实际按 USD 计费(无真触发),且本次告警口径门控已挡其假告警——单独 PR 对齐。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-18, AWS `cmdb.cost_facts_monthclose` 月结重拉超 BigQuery bytes-billed 上限而失败 —— CUR 2.0 按自然月分段 + 上限 50→128 GiB)**. **现象**:`cmdb.cost_facts_monthclose`(每月 5 号整月重拉对账)对 **AWS 主账号**的每个 scope 都失败,失败通知器卡片含 `bytesBilledLimitExceeded` / `Query exceeded limit for bytes billed`,AWS 上月对账永远跑不完(其它云正常)。**根因(只读 prod BQ dry-run 实证)**:AWS 主账号走 CUR 2.0 数据集 `gcp-digital-bi.aws_billing_no_payer`——60+ linked account 合在一张按 `line_item_usage_end_date` 按日分区的月表,per-scope 的账号过滤**不是分区/聚簇键**,故每个 scope 的整月查询都扫该月**全部账号**分区,30 天单查询 **73.57 GB(68.5 GiB)> 原 50 GiB 上限**。**修了什么**:① `AwsCloudDriver._list_cost_facts_bq_sync` 把窗口**按自然月**切片逐月查询(单月窗=月结场景=1 查询,与改造前一致;跨月回填=每月 1 查询);② BQ `_BQ_MAX_BYTES_BILLED` 50 GiB→**128 GiB**(稳过单月 ~70 GiB + 留增长余量,仍拦真 runaway)。按自然月而非任意 ≤15 天切点,是因为 fact 按 `start_date` 定日、表按 `end_date` 分区,月中切点会静默丢「月初起、跨月结束」的 SP/RI recurring fee 行;月界切点等价于独立月结、复用既有 +2d pad,零丢行、对账口径不变。**Operator 该做什么**:① **正常无需动作** —— 月结自动按月分段跑过;② 想立刻补:对 `cmdb.cost_facts_monthclose` run-now 传整月或跨月窗口(现已自动按月分段,不必再手工切 ≤15 天)、或 `/cmdb/cost/backfill` / MCP `cost_backfill`;③ **遗留(已知、独立、上游归属)**:每个 scope 整月扫全分区天然重复(60 scope 各扫 ~70 GiB),根治需上游 CUR 表按账号聚簇或单趟全账号查询后内存扇出——该表由 pchub Data Exports 产出、Helix 不拥有,留待上游优化。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-17, cost_facts 全云虚高 —— fact_key 碰撞静默丢负向折扣/抵扣行;ingest 层逐 fact_key 求和 + Aliyun 空 OwnerID 只归 master)**. **现象**:`cmdb_cost_facts`(多维下钻/资源排行/分摊/异常/预测/部门归因)比权威日表 `cmdb_cloud_billing_daily` 与云控制台**虚高**。实证(只读 prod BQ CUR + TiDB,2026-05-18..06-16):AWS cost_facts $3.17M vs billing_daily $2.80M vs CUR 真值 $2.91M(+9% / +$370K);alicloud_intl $119K vs $53K。**根因(承接 2026-06-03 腾讯同型,但这次根治在 ingest 层而非逐驱动)**:`compute_fact_key` 自然键不含 product_code/service,驱动产出的多条同键 CostFact(AWS CUR EdpDiscount/PrivateRateDiscount 行 NULL resource_id 合成成单个折扣桶 id 却横跨 67/8 个 product_code;Aliyun 信用日一个 resource_id 多条调账行)被 `ON DUPLICATE KEY UPDATE` **只留一条、丢其余**——丢的多是**负向**行,故总额虚高(billing_daily 在内存按天求和、不受影响)。**改了什么**:① `cost_service.ingest_cost_facts` 在分摊+分块 upsert 前用 `_preaggregate_facts_by_fact_key` 把同 fact_key 的 facts **求和**(provider-agnostic、幂等、跨分块、无碰撞时 no-op);② Aliyun 空 OwnerID(master-allocated)BSS 行此前对每个 scope 都保留→在 cost_facts 与 billing_daily 双双按 scope 扇出重复计(~$16.8K/30d),改为只归 RD master scope(`_billing_row_belongs_to_scope` + 缓存 `_own_account_id`,三条 BSS 路径共用),`list_scopes` 注册 RD master(ListAccounts 不返回管理账号)。**Operator 该做什么**:① 正常无需动作;② **本次发布后一次性重对账**(必须):对 AWS + alicloud_intl(及其它云)的 `cmdb.cost_facts_sync` 在 cloud_sync 锁下全量重拉(`/cmdb/cost/backfill` 端点 / MCP `cost_backfill` / 对 `cmdb.cost_facts_monthclose` run-now 传整月窗口;`on_duplicate_key_update` 覆盖回填,billing_daily 无需动),cost_facts 月度即对齐 billing_daily/控制台;③ **会多出一个 scope**:Aliyun RD master(显示名 `… (RD master)`)首次同步后出现在云资产里,承载此前被扇出的 master-allocated 花费——预期、正确(它本就有自有花费);④ 重对账并跑 per-provider billing_daily-vs-cost_facts 对账(1-2% 内)+ AWS vs BQ CUR 校验通过后,可移除云资产总览下钻页的 `CostBasisNote` 口径说明(#1196 加的临时提示)。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-16, 云证书终于接入回收站 —— 补全 2026-06-15「复活改由回收站 Restore」承诺的缺口)**. **背景**:上一条（2026-06-15）让证书同步不再自动复活退役证书,并告诉 operator「确实仍在用就走 `/admin/cloud/recycle-bin` 回收站 Restore」——但 `certificate` 当时**根本没接进回收站**(`_RECYCLE_BIN_TYPES` 只有 server/domain/dns_record/load_balancer/cdn_domain),于是被 `_safe_sweep` 扫掉或管理员 `DELETE` 的云证书(带 owner/environment/remark 注解)**永久不可恢复**,只能改 DB。**改了什么**:把 `certificate` 接入回收站全链路——`CmdbService.list_deleted_certificates`(按 common_name/serial_number/fingerprint + `JSON_SEARCH(sans)` 搜)、`GET/POST /cmdb/recycle-bin/certificate[/{id}/restore]`、MCP `list_cmdb_recycle_bin`/`restore_cmdb_resource` 支持 `certificate`、前端 `CmdbRecycleBinPage` 新增「证书」tab(三语)。**Operator 该做什么**:① 退役证书被云 sweep 误删但仍在用 → `/admin/cloud/recycle-bin` → 证书 tab → 恢复(或 MCP `restore_cmdb_resource resource_type=certificate`);② 审计 `cmdb.resource.restore`(resource_type=`cmdb_certificate`)= 一次证书恢复。无 migration / cron / RBAC / 新 MCP 工具改动。
> - **(2026-07-08, 离职 Google 账号级挂起链路 4 修 —— defer 补跑 / 返聘防挂 / reopen 清证据 / cancel 反向解挂)**. **背景**:2026-07-01 ADR 翻案后 `_push_force_logout` 会 SUSPEND 离职者 Google 账号(不止切会话),对抗性 review 抓到 4 处缺陷(#1739/#1748/#1749/#1750,详见 CHANGELOG)。**Operator 该知道**:① **未来日期离职者**:HR 提前录 endDate 时,会话切断 + Google 挂起会 **defer 到 leaver_date 当天由 T+0 advance 自动补跑**(此前永久跳过);若历史上有「已完成离职但 Google 会话从未切断」的残留,现在会出现在**数据质量巡检 `offboarding_session_not_cut`**(新增「已完成但 `session_enforcement` 为空」类,原因 `cutoff_never_attempted`),点该发现走「重新执行会话切断」(reenforce)即可,幂等安全。② **返聘员工**:对返聘(`employment_status='active'`)员工的旧离职进程点「重新通知」现在会被 **409 `PERSON_ACTIVE`** 挡下,不再误挂活跃账号。③ **误标离职回滚(cancel)**:若离职流程已把 Google 账号挂起,取消流程时会**要求 confirm_irreversible**(账号挂起属不可逆云侧变更)并**自动反向解挂**(`unsuspend_google_account`,只 `_set_suspended(false)`、不重置密码/不重发欢迎信);自动解挂失败(无 Google 集成 / 厂商 5xx)时,取消响应的 `skipped_restore_reasons` + 审计 `google_account.restore_manual_required` 会提示**去 Google Admin 控制台手动启用该账号**。无 migration / cron / RBAC / MCP 工具数改动。
> - **(2026-06-15, `alicloud_intl` CloudSSO 租户的申请下拉残留旧 RAM 角色 → 批准时报「role catalog is stale」)**. **现象**:同事申请阿里云海外子账号访问时选了 `System:ReadOnlyAccess`,scope owner 点批准弹 `Cannot grant RAM-shape role 'System:ReadOnlyAccess' on a CloudSSO-managed AliCloud tenant … catalog appears to be stale`。**根因**(查 prod 确认):`cmdb.roles.alicloud-intl-primary` cron 自 CloudSSO 迁移后**每天 failed**——`provider returned 368 roles vs 10754 active baseline (>20% drop) — refusing soft-delete sweep`。新 CloudSSO 权限集每天正常 upsert(`last_synced_at`=昨天),但 10386 条旧 RAM 形态行(108 条 promoted、`last_synced_at` 冻在 2026-05-28)因 20% 跌幅保护永远 sweep 不掉,继续出现在申请下拉里;grant 路径的形态护栏 fail-closed 拒绝它们(护栏本身对的),漏的是下拉还在给。**与「PROD 显示」无关**。**生产侧已做**:按 `roles_sync force_sweep` 同语义软删了 10386 条孤儿(`integration_id=alicloud_intl AND last_seen_sync_job_id NOT NULL AND <> 最近成功 job AND deleted_at IS NULL`;0 误删 bare CloudSSO 名;158 条 CloudSSO promoted 保留;写 `cmdb.roles_sync.force_sweep_override` 审计行)→ cron 下次转绿、下拉只剩 CloudSSO 权限集。**Operator 该做什么**:① 通常无需动作;② 若其它 CloudSSO 租户再现「下拉里有 `Type:Name` 旧角色 / cron 卡 >20% drop」,跑 `POST /cmdb/integrations/{id}/roles-sync/force-sweep`(或同语义 SQL 软删冻结 `last_synced_at` 的孤儿行);③ **已卡在 pending 的旧角色申请批不动** —— 让申请人驳回/撤回后改选 CloudSSO 权限集重发(代码侧现已在提交时拦,见下)。**代码侧防复发**:新增驱动钩子 `validate_request_role`,`submit_access_request`/`owner_add_member` 落库前预检 role_key 形态,CloudSSO 租户上的 RAM 形态 key **当场 422 拦回申请人**(fail-open:目录探测瞬时失败则放行,grant 路径仍是最终闸)。无 migration / cron / RBAC / 新 MCP 改动。
> - **(2026-06-15, 云证书同步不再复活管理员退役的证书 —— 对齐 server PR #559 / DNS 已修同型 bug)**. **现象**:管理员经 `DELETE /cmdb/certificates/{id}` 软删一张云证书后,它会在下次 `cmdb.resource_sync` 后**静默重现**在活跃列表里。**根因**:`_sync_certificates` 的 `on_duplicate_key_update` 里写了 `deleted_at=None`——云厂商最终一致性窗口内仍返回刚退役的证书,upsert 把 `deleted_at` 清回 NULL 复活该行(servers 在 PR #559、DNS 早已去掉这个写法,唯独证书漏了);证书路径还**漏掉** `_audit_resurrections` 取证调用。**改了什么**:从证书 upsert 的 update set 去掉 `deleted_at`(行保持软删,**复活改由 `/admin/cloud/recycle-bin` 回收站 Restore**,与 server/DNS 一致)+ 补 `_audit_resurrections` 取证(软删行被同步命中时写一条 `cmdb.sync.resurrected` 审计)。**Operator 该做什么**:① **正常无需动作** —— 退役的云证书不再被同步复活;② 若某张证书是被云 sweep 误删后**确实仍在用**,走回收站 Restore(而非等同步自动救);③ 审计里出现 `cmdb.sync.resurrected`(resource_type=`cmdb_certificate`)= 一张软删证书仍被云端返回,供取证判断是否该 Restore。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-13, 云同步 per-capability 可观测 + 单能力重跑 —— 角色对称 PR4)**. **背景**:同步引擎一直把每个能力的明细(状态/原因/计数/分腿失败/skipped/超时/强扫)写进 `CmdbSyncJob.capability_results`,但管理员排障时没有「这个云账号每个能力最新结果是什么、哪个在失败」的聚合视图,只能读原始 JSON。**改了什么**:新增诊断端点 `GET /cmdb/integrations/{id}/sync-capability-status`([sync_status.py](../../server/app/domains/cmdb/sync_status.py)`build_capability_status`)——对该账号**配置的每个能力**取最近一条 job 的结果,含**从未同步**的能力(`status='never'`,让静默缺口可见);automation-health 新增 `cloud_sync` lane(窗口内 failed/partial 同步 job + top 失败的 集成×能力 带原因)。前端集成页加「同步能力状态」抽屉,每个能力一行(状态/计数/原因/上次运行 + forced-sweep/timed-out 徽标),失败行可点**「重新同步」单独重跑该能力**(复用既有 `POST /cmdb/sync/{id}/trigger` 传单个 resource_type,无新端点)。**Operator 该做什么**:① 某云账号同步看着不对 → 集成页点「同步能力状态」看哪个能力 failed/partial + 原因;② 单能力失败(如 IAM AccessDenied)修好权限后**点该能力行的「重新同步」单独重跑**,不必全量同步;③ `status='never'` = 该能力配了但从没成功同步过(查权限/驱动支持);④ 运管驾驶舱「自动化健康」cloud-sync 卡红数字 = 窗口内有同步失败,点 top_failing 深链直达对应账号能力状态。纯读诊断 + 复用既有触发,无 migration / cron / 新 RBAC;MCP +1 `get_cloud_sync_status`(hint 293)。
> - **(2026-06-12, 生产写锁第二豁免口 —— 管理员可在锁定状态下转让订阅 owner;owner 本人自转让仍锁挡)**. **背景**:owner 转让(`assign_scope_owner`)是纯 Helix 元数据操作(DB rebind + 审计 + 治理时间线,**不触达云厂商 IAM**),与「改环境」豁免同类;此前管理员要为生产订阅交接 owner(典型:owner 离职)只能 break-glass 开全局锁,窗口内所有 prod 写全放开,风险反而更大。**改了什么**:`PATCH /cmdb/scopes/{id}` 中,调用者为 `can_manage_cloud`(cloud_admin / global_admin)且 owner 是唯一可锁字段时豁免写锁;MCP `transfer_cloud_scope_owner` 同步加同一管理员旁路。**owner 本人(非管理员)自转让仍被 403 PROD_CLOUD_LOCKED 拦截**;捆绑 budget/notes/status/ttl/auto-suspend 任一字段照旧触锁。**Operator 该做什么**:**转 owner 无需 break-glass 开锁** —— 云管理员直接在子账号详情页点「转让 Owner」即可(锁定 prod scope 上该按钮现在对云管理员可用);改预算/加人/关停等仍需照旧临时开锁。员工/owner 看到的锁定 banner 文案已标注此例外。无 migration / cron / RBAC / MCP 工具数改动。
> - **(2026-06-09, 云厂商 TLS 证书同步 —— AWS ACM / 阿里云 CAS / 腾讯云 SSL,默认按 integration opt-in)**. **背景**:证书台账(`cmdb_certificates`)新增第三条进料「云证书管家同步」,补主动探测覆盖不到的托管证书。新增 `ProviderCapability.CERTIFICATES`,AWS/阿里云/腾讯三家驱动实现 `list_certificates()`,`cmdb.resource_sync` 经 `_sync_certificates` upsert(`source='cloud'`)。**Operator 该做什么**:① **默认不动**——CERTIFICATES 受每个 integration 的 `capabilities` 配置门控,**除非显式给该云账号加 `certificates`(或用 `*` 通配),否则不会同步证书**;② 想开启:集成配置里给目标云账号 capabilities 加 `certificates`,下次 `resource_sync` 即开始拉;③ **前置权限**:AWS 需 `acm:ListCertificates`+`acm:DescribeCertificate`(各 region),阿里云 CAS 只读(`DescribeUserCertificateList`),腾讯 SSL 只读(`DescribeCertificates`)——缺权限清洁报 partial_failure,不影响其它能力;④ **sweep 安全**:证书 sweep 只清本 integration 的 `source='cloud'` 行,探测/手动行(integration_id NULL)绝不受影响;⑤ 云证书与探测证书暂以不同 `dedup_key` 并存,跨源去重在 PR3。无 migration(列写进 266)/ cron / RBAC / MCP 改动。
> - **(2026-06-09, DNS 自助申请现在支持 NS 记录 —— 子域委派不再只能走管理员直写)**. **背景**:自助 `POST /dns-record-requests`(+ MCP `submit_dns_request`)此前白名单只有 A / AAAA / CNAME / TXT,NS 被刻意挡在管理员直写路径(`DNS_REQ_SELF_SERVICE_TYPES`);但把子域委派给外部权威 nameserver(`sub.example.com NS ns1.vendor.com`)是常规需求,没理由每次都开工单找管理员。**改了什么**:① `DNS_REQ_SELF_SERVICE_TYPES` 加入 `NS`;`DnsRequestRecordType` literal 同步;② NS 的「解析目标值」按 FQDN 校验(与 CNAME 同形),前后端双校验对齐;③ 申请仍走同一条 Feishu 审批 + 管理员/自动下发到云端 DNS 的路径,审批与下发逻辑零改动。**仍是管理员专属**:MX / SRV / CAA(改邮件投递 / 服务发现 / CA 签发授权,误改 blast radius 大)。**Operator 该做什么**:**通常无需动作** —— 员工现在能在「DNS 管理 → 申请新增解析」选 NS 自助提交,审批人照常在飞书审批卡看到 `name.zone NS ns1.target.com`。**注意**:zone 顶点(`@`)的权威 NS 是云厂商自管的,厂商的记录 API 一般会拒绝覆写,所以自助 NS 实际只对子域委派生效;若审批通过后下发报厂商错误,即为该原因,引导申请人改用子域 host 名。无 migration / cron / RBAC / 新 MCP 工具改动(既有 `submit_dns_request` 工具自动接受 NS,仅描述串补了 NS)。
> - **(2026-06-06, GCP 账单账号成为可授权 scope —— 员工要看 GCP 控制台账单走这里)**. **背景**:`roles/billing.viewer` 授在 project 上必 400(账单角色是账单账号级)。现在 GCP 账单账号(`billingAccounts/XXXX-XXXX-XXXX`)作为一类 scope 出现在云资产里,授权经 Cloud Billing API。**Operator 该做什么**:① **开放申请**:账单账号 scope 同步后**默认无 owner、不可申请**(不触发未指定 owner 合规告警);要让员工能申请,云管理员用「转移/指派 owner」给它指派一个账单/财务负责人,该人即成为审批人。② **SA 权限前提**:GCP 集成的 service account 需有 `billing.accounts.setIamPolicy`(列举账单账号只需 `cloud-billing.readonly`,已有);缺权限时授权会清洁报错,去 GCP 给 SA 补 `roles/billing.admin` 或等价自定义角色。③ **blast radius**:账单账号上的 `billing.viewer` = 能看**该账单账号下所有项目**的成本,比单项目访问大 —— 指派 owner 即等于对自助开放,owner 要清楚这点;只 promote 只读的 `billing.viewer`,`billing.admin` 不进自助。④ 若员工只是想看**某个项目**的成本,Helix 自己的「账单」tab 对 scope member/owner 已直接显示,通常无需 GCP 控制台账单角色。
> - **(2026-06-06, 高危角色审批权收紧:elevated/restricted 角色只能云管理员审批,scope owner 批不了)**. **背景**:框架 Bucket B。**Operator 该知道**:① 一个 `elevated`/`restricted` 角色的访问申请,**普通员工 scope owner 点「批准」会拿到 403**(`CLOUD_APPROVAL_NEEDS_ADMIN`,提示转交云管理员)—— 这是有意的,防止 owner 一键放行提权。云管理员(`global_admin`/`cloud_admin`)审批正常。② **存量影响**:本次发布后,历史上还 pending 的 elevated/restricted 申请,其 owner 审批人会突然批不了 —— 需云管理员接手审批。发布前可查 `cmdb_cloud_access_grants WHERE status='pending'` 交叉角色档位,提前知会。③ **拒绝不受限**:owner 仍可随时驳回任何申请。④ **判错纠偏**:若某角色档位判错,云管理员可调 `POST /cmdb/role-catalog/governance-tier`(role_ids + governance_tier)覆盖,置覆盖位后同步不回写。
> - **(2026-06-06, 云角色治理框架:角色现在分四档 normal/elevated/restricted/ungrantable,影响下拉显示与申请/审批)**. **背景**:承接 billing.viewer 400 的系统性治理。每个可申请云角色按危险度分四档(分类器 [`app/integrations/cloud_role_governance.py`](../../server/app/integrations/cloud_role_governance.py),驱动声明各 provider 的角色集)。**Operator 该知道**:① **申请下拉**:员工看不到 `ungrantable`(该 scope 授不了的,如 GCP billing.* 在 project 上)和 `restricted`(owner 等价/全 IAM admin,如 roles/owner、Administrator);`elevated`(Contributor/PowerUser/editor/IAM-write)员工能看能申请,但(PR3 起)**只有云管理员能审批**。② **谁能改档**:`cmdb_cloud_roles.governance_tier` 可由管理员在角色目录覆盖(`governance_overridden_by_admin=1` 后同步不回写);默认值由驱动分类器算,每次角色同步刷新。③ **若某角色档位判错**(误把正常角色判成 restricted、或漏判高危),在角色目录 UI 改 tier 即可(PR3 提供写接口),分类器默认集也可在 [`cloud_role_governance.py`](../../server/app/integrations/cloud_role_governance.py) / 各驱动里加白/加黑。④ **申请报错**:选了 ungrantable 角色提交直接 422 + 中文原因(不再撞 provider 原始 400);非管理员申请 restricted 角色 403。这些拦截在 service 层,HTTP 与 MCP 一致。
> - **(2026-06-06, GCP 项目访问审批选了 `roles/billing.viewer` 必失败 400「not supported for this resource」—— 账单/组织级角色不能授在项目上, migration 258)**. **现象**:审批人在某 GCP 项目批准 `roles/billing.viewer` 访问申请时前端弹 `HttpError 400 ... :setIamPolicy ... Role roles/billing.viewer is not supported for this resource`,审批失败、权限没开通。**根因**:`roles/billing.viewer`(及 billing.user/admin/creator/costsManager)是 GCP **账单账号/组织级**角色,GCP 不允许把它们绑到 project 的 IAM policy —— 项目 `setIamPolicy` 直接 400。它被早期 seed(migration 054)错误放进了项目授权目录的下拉里。**修复**:① 驱动 fail-fast 拦截这族角色(调 API 前返回可操作提示,grant 保持 PENDING);② 下拉过滤掉;③ migration 258 清掉存量库 `platform_settings` 里的该族角色。**Operator 该做什么**:**正常无需动作** —— 发布后申请下拉不再出现 Billing Viewer;**历史卡住的 billing.viewer 申请请驳回**,让申请人改选项目级角色(如 roles/viewer)。GCP 的账单可见性是**账单账号级**授权,不在 Helix 项目授权范围内 —— 需要在 GCP 账单账号上单独授(`roles/billing.projectManager` 才是项目级可授的账单角色,管的是项目↔账单账号关联,不是看账单数据)。无 cron / RBAC / MCP 改动。
> - **(2026-06-04, 「改环境」是生产写锁的唯一豁免口 —— 误标为 prod 的子账号现在可降级到 staging/dev)**. **背景**:生产写锁(`cmdb.prod_cloud_writes_enabled`,默认锁死)挡掉 prod scope 的**所有**写,包括改 `environment` 本身 —— 这造成死结:一个被误标成 `prod` 的子账号永远没法降级,因为降级用的那次 `PATCH` 自己就被锁挡。**修复**:`PATCH /cmdb/scopes/{id}` 里,**只改 `environment` 一个字段**的请求豁免生产写锁(改环境标签是纯 Helix 元数据、**不触达云厂商 IAM**),仍受 `can_manage_cloud` 管控(仅云管理员)。请求里**只要还带其它字段**(owner/budget/status/notes/告警…),照旧被锁挡。前端配置页:锁开启时整张配置表单仍禁用,但「环境」下拉 + 新增的「保存环境」按钮对云管理员可用。**Operator 该做什么**:**改环境无需 break-glass 开锁** —— 想把一个 prod scope 降级到 staging/dev,云管理员直接在子账号详情「配置」tab 改「环境」→ 点「保存环境」即可;改预算/加人/转 owner 等仍需照旧临时开锁。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-04, orphan-cleanup 现在也受生产写锁约束 —— 之前是唯一绕过 kill-switch 的人工云侧 IAM revoke 路径)**. **背景**:上线前审计发现 `POST /cloud-grants/orphan-cleanup`(+ MCP `cleanup_cloud_orphan_assignments`)对 prod scope 会发**真实的** provider IAM revoke(`delete_orphan_assignment`),却没过 `ensure_prod_scope_writable` —— 正是生产写锁要防的「在生产账号上误撤角色」那一类。**修复**:守卫加在 `cleanup_orphan_assignments` 内部 per-target(一处覆盖 HTTP + MCP 两个入口）。锁开启时一个 prod 目标返回 `ok=False` + 锁提示、**绝不调用云端 revoke**;同批次的非 prod 目标照常清理。**Operator 该做什么**:① 日常无需动作 —— 想清 prod 孤儿权限时,先去平台设置把 `cmdb.prod_cloud_writes_enabled` break-glass 翻 `true` → 跑 orphan-cleanup → 立即翻回 `false`(与其它 prod 写操作同一姿态）;② 锁开启时跑 cleanup,prod 目标会在结果里显示「被锁挡」而非静默跳过,非 prod 目标不受影响。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-04, 自助申请域内阿里云(`aliyun`)子账号时 owner 永远绑不上云 IAM —— owner role_key 发错形态)**. **现象/根因**:#696 给 `_DEFAULT_OWNER_ROLE_BY_PROVIDER` 补了 `aliyun` / `alicloud_intl` 两个 owner 条目,但都发 `("admin","admin")`。海外 `alicloud_intl` 驱动 override 了 `grant_access`、把裸 `admin` 映射成 CloudSSO AccessConfiguration —— OK;但**域内 `aliyun` 驱动没 override**,走 RAM 基类 `grant_access`,它拒绝任何不含冒号的 role_key(`admin` 无冒号 → 永远 ok=False），自助子账号 owner 落成「Helix 侧 owner 但云侧零 IAM」—— 正是 #696 想消除的 bug 的回归。**修复**:按 provider 区分 —— 域内 `aliyun` 发 `System:AdministratorAccess`(RAM 系统策略,匹配域内驱动 `owner_role_keys`），海外 `alicloud_intl` 保留 `admin`(CloudSSO）。**Operator 该做什么**:**无需动作** —— 本次发布后域内阿里云自助子账号的 owner 授权自动绑上 RAM AdministratorAccess。历史上若有「域内 aliyun 子账号 owner 云侧无权限」的个案,人工换一次 owner(`PATCH owner_person_id`)即触发重绑。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-03, 腾讯成本明细 cost_facts 漏算 ~99%(COS 真实 ¥3,344 落库 ¥30.51)—— 驱动按 fact-key 粒度聚合求和)**. **现象**:#712 让账单概览对齐控制台后,成本明细(多维下钻/资源排行)月度总额明显偏低(2026-05 cost_facts ¥9,242 vs 概览 ¥12,782)。**根因**:cost_facts 按 `(integration,date,scope,resource_id,usage_type)` 唯一键 UPSERT(REPLACE 幂等),但腾讯 `DescribeBillDetail` 对同一 (天×资源×ActionType) 返回多条子行(不同 ItemCode/子周期)撞同键,逐行 emit 后写覆盖前写。生产实证:4037 行→523 grain,434 个 grain 多行。**修复**:腾讯驱动 emit 前先按该粒度聚合求和(不动共享 key/ingest,不影响其它云、不影响 billing_daily)。**Operator 该做什么**:① 正常无需动作;② **本次发布后一次性重对账**:删腾讯 integration 旧 `cmdb_cost_facts` 再全量重拉(billing_daily 本就正确、无需动),cost_facts 月度即对齐 billing_daily/控制台。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-03, 腾讯云账单/明细数值跟控制台对不上 —— 按「计费周期」归集,改用 `Month` + `FeeBeginTime`,不再用 `PayTime`)**. **现象**:#703 修好金额后对账,Helix 跟腾讯控制台「费用趋势/按计费周期」逐月差 ~¥1k,**6 月差 35×**(控制台 ¥161.50,Helix ¥5,626)。**根因**:驱动用 `DescribeBillDetail` 的 `PayTime`(扣费时间)归日,而腾讯把 CDN/COS 月结在**次月 1 号扣费** —— 5 月那笔 ¥5,390 CDN 月结(`FeeBegin=5/1`)被算进了 6/1。生产实证:`Month=2026-05` 拉取合计 **¥12,782.47 = 控制台 5 月一分不差**。**修复**:驱动改成逐计费周期(`req.Month="YYYY-MM"`)拉 + 用 `FeeBeginTime` 归日(clamp 进所查周期),删掉 `_split_by_calendar_month`。**Operator 该做什么**:① 正常无需动作 —— 发布后每日 cron 自动按计费周期归集,月度对齐控制台;② **本次发布后做了一次性重对账**:删除腾讯两个 integration 的旧 `cmdb_cloud_billing_daily` + `cmdb_cost_facts`(PayTime 错配数据)再全量回填 —— 改腾讯成本归集口径时**必须先删后填**(否则旧口径有数据、新口径没数据的日期会残留旧值,upsert 只覆盖新返回的日期)。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-03, 生产云账号写操作全局锁 —— 默认锁死,任何人都不能写 prod 云账号, migration 246)**. **背景**:有人在生产云账号上 self-release 了一个角色(重大误操作)。现在 `environment=prod` 的云账号在锁开启时**任何人都不能做写操作**(加人 / 释放角色 / 续期 / 转移 owner / 改配置 / 设告警 / snooze / 立即同步 / 下线),**与角色无关 —— 连 IT 管理员 / global_admin 都被挡**,返回 403 `PROD_CLOUD_LOCKED`。**开关**:平台设置 `cmdb.prod_cloud_writes_enabled`(JSON bool),**默认 `false` = 锁死**(缺行也算锁死)。**Operator 该做什么**:① **日常什么都不用做** —— 生产云账号保持只读是预期姿态;员工 / owner 看到按钮置灰 + tooltip + 页面顶部黄色 banner 说明已锁。② **确需改某个 prod 子账号**(例:加人、撤一条 grant、改预算、下线账号):去「设置 → 平台设置」把 `cmdb.prod_cloud_writes_enabled` 临时翻 `true` → 完成你的那一步操作 → **立即翻回 `false`**。这是 break-glass,不是常开。③ **离职 / TTL 过期自动回收不受此锁影响** —— 离职员工的 prod 云权限照常被系统自动撤掉(自动化走的是 service 层、不经此守卫),无需为离职手动开锁。④ 锁状态下若有人硬调 API / MCP,会拿到 403,Helix 侧不会发生任何云侧变更(守卫在调用云驱动之前)。**不锁的**:新建 prod 子账号的 scope-request 审批(那是独立上线流程,锁开才能批);integration 级 force-drift/roles-sweep(是 reconciliation,不是 per-scope 人为写)。**无新 cron / 无新 RBAC 角色 / 无新 MCP 工具**(只给既有写工具加守卫)。
> - **(2026-06-03, 云子账号访问审批 `POST /cmdb/grants/{id}/approve` 返回 500「can't subtract offset-naive and offset-aware datetimes」—— 但权限其实已开通)**. **现象**:审批人(prod: Alex.Quan,trace `fe188c39…`)在 Helix 点「批准」云订阅访问申请,前端弹错,但被授权人**确实拿到了访问** —— 看起来像「审批失败」其实是审批成功后**响应序列化崩了**。**根因**:approve / renew 的 dispatch 把 tz-**aware** 的 `datetime.now(UTC)+timedelta` 写进 `grant.expires_at`;session 配 `expire_on_commit=False`,`commit()` 后 router 的 `db.get(grant)` 拿回**同一内存对象**(仍 aware),`_serialize_grants` 里 `(expires_at - now).days` 减 naive `now` → `TypeError`。列表页重新 fetch 时行从 DB 读回(tz 被剥成 naive)→ 不复现,所以只在「刚审批完那一下」炸、像「跟时间有关」。**修复**:源头改写 naive UTC(对齐全库约定)+ 序列化边界统一归一 naive 再做减法/比较(防御所有 caller);回归测试 `test_serialize_grants_aware_expires.py`。**Operator 该做什么**:**无需补救** —— 历史上凡是「审批报这个错」的申请,访问权限都已真实生效(检查 `cmdb_cloud_access_grants.status='active'` 即可确认);不要因为看到过 500 就重复审批。本次同时把审批人飞书/邮件通知的跳转链接从裸 `/cloud/scopes/{id}` 改成 `?tab=pending`,审批人点开直接落「待审批」tab。无 migration / cron / RBAC / MCP 改动。
> - **(2026-06-03, 腾讯云 账单/概览/成本明细 金额恒 $0 —— `DescribeBillDetail` 读了不存在的 `RealTotalCost`,真金额在 `ComponentSet[].RealCost`)**. **现象**:`云资产总览 → 账单 → 概览`(腾讯云租户)期内总支出 / 日均 / 各云厂商占比 / by-account 排行 全是 0,`cmdb_cost_facts` 腾讯 0 行 —— 但连接器 `last_test_status=ok`、`cmdb_cloud_billing_daily` 有 136 天行且服务名(COS/边缘安全加速/月度计费精度差异)解析正确。**即连接器、权限、汇率都没问题,纯金额恒 0**。**根因**:`tencent_cloud._list_billing_daily_sync` + `_list_cost_facts_sync` 按 `getattr(row,"RealTotalCost") or getattr(row,"TotalCost") or "0"` 取金额,但 `DescribeBillDetail` 的 `BillDetail` 行**没有**这俩字段(它们属于另一个 API `DescribeBillSummaryByProduct` 的 `SummaryOverview`,月度 provider 用的是那个、正确);`BillDetail` 金额在 `ComponentSet[].RealCost`。`getattr` 全 `None` → 回落 "0" → billing_daily 写满 0、cost_facts 因 `if native==0` 丢光每行。**修复**:新增 `_bill_detail_real_cost_cny(row)` 对 `ComponentSet` 求和 `RealCost`(空回退 `Cost`),两处共用;全量驱动 SDK 审计确认仅腾讯有此病。**Operator 该做什么**:部署后**重跑腾讯两个 integration 的 `cmdb.billing_daily_sync` + `cmdb.cost_facts_sync`**(`/admin/cron/scheduled-jobs` 或 admin「立即同步」;`on_duplicate_key_update` 覆盖历史窗口回填),概览金额随即恢复。无 migration / cron schedule / RBAC / MCP 改动。
> - **(2026-06-03, 云成本明细对账缺口 + 新增「月结重拉」月度作业 `cmdb.cost_facts_monthclose`, migration 247)**. **现象**:财务把 Helix 账单页某云账号某月支出与该云控制台(AWS Cost Explorer 等)对账,发现差一点点(实测 AWS SG 2026-05 差 $3.84,全在「无 Owner 标签」桶,打了标签的逐项分毫不差)。**根因**:每日 `cmdb.cost_facts.<slug>` 只滚动重拉「昨天起往前 3 天」窗口,一个使用日过了 3 天就再也不重拉、冻结在厂商出账后 ~2-3 天的值;而 Cost Explorer 在**月末 finalize** 时还会继续核价(日累积 S3/ECR/Cost Explorer、晚到的未打标签用量),这些 >3 天后的修订永远进不来,造成系统性小额漂移。**会复发**,每月都有。**修复**:新增**单例**月度作业 `cmdb.cost_facts_monthclose`(每月 5 号 07:00 SH,等厂商 finalize 上月后),遍历所有 enabled 云账号把**整个上月**整月重灌一次追平。复用 `sync_cost_facts_for_integration` backfill 路径 + `cloud_sync:cost_facts:{id}` 锁,绝不与每日 tick 并发。**与 `cmdb.cost.nightly_recompute_allocation` 正交**:那个重「归属」(重跑 allocation rules),这个重「拉取」(重灌厂商金额),互不替代。**Operator 该做什么**:**正常无需动作** —— 每月 5 号自动整月对账。① 想立刻补任意区间:对该 job `run-now` 传 `config={"start_date":"YYYY-MM-DD","end_date":"YYYY-MM-DD"}`,或对单个 integration 用 `/cmdb/cost/backfill` 端点 / MCP `cost_backfill`。② 失败通知器卡片(`cmdb.cost_facts_monthclose ... raised`)= 某云账号月结重拉硬失败(凭证过期 / 厂商 5xx),卡片消息含 `N/M integration(s) failed` + 失败账号名,去查该账号连接器凭证后等下月或手动 run-now 重试;not_supported(stub driver)/ skipped_locked(撞每日 tick)不算失败。③ **账单四件套**(每云账号):月汇总 `cmdb.billing`(月→cmdb_billing_monthly)、日趋势+告警 `cmdb.billing_daily`(日→cmdb_cloud_billing_daily)、明细 `cmdb.cost_facts`(日滚动3天→cmdb_cost_facts,归属唯一源)、月结重拉 `cmdb.cost_facts_monthclose`(月→对账)。四者写不同表/不同粒度/不同厂商 API,**刻意不合并**(互为交叉校验)。每个任务的 description 已写清角色定位,admin 在 `/admin/cron/scheduled-jobs` 直接可读。无 schema / RBAC / MCP 改动。
> - **(2026-06-03, 腾讯云每日账单抓取 `cmdb.billing_daily_sync` 每月 2/3 号必失败 —— `DescribeBillDetail` 不支持跨月查询)**. **现象**:`cmdb.billing_daily_sync` 报 `RuntimeError: Tencent DescribeBillDetail offset=0 failed: [TencentCloudSDKException] code:InvalidParameterValue message:call billcenter3 ...`(offset=0 即第一页就炸)。**根因**:腾讯 `DescribeBillDetail` 官方约束「BeginTime 和 EndTime 必须为相同月份,**不支持跨月查询**,查询结果是整月数据」;而每日 cron 默认窗口是 `今天-3 .. 今天-1`,在每月 **2 号 / 3 号**会跨越上月→本月边界(例:today=06-03 → 窗口 2026-05-31..06-02),腾讯直接拒。其它云(AWS/Azure/GCP/Aliyun)接受任意窗口,只有腾讯受影响。**修复**:driver 新增 `_split_by_calendar_month` 把窗口按自然月切段,`_list_billing_daily_sync` + `_list_cost_facts_sync` 改成「按月分段、段内分页、累加合并」;腾讯按月返回整月数据,既有 per-day 过滤裁回窗口,总额精确。**Operator 该做什么**:**通常无需动作** —— 每月 4 号起窗口回到月内自然恢复;此 fix 部署后每月 2/3 号不再失败、告警噪音消除。若想立刻补回 6/1–6/2 缺口,可手动重跑该 integration 的 billing sync(`/admin/cron/scheduled-jobs` 或 admin「立即同步」按钮),现已支持跨月窗口。无 migration / cron schedule / RBAC / MCP 改动。
> - **(2026-06-03, 已 `left` 的人离职流程卡死时,其 cloud_server / DNS / cloud_iam_user / 邮件组 owner 不再永久挂在离职者名下 —— migration 245, cron `hr.recover_orphaned_left`)**. **现象**:lifu.wu@ / chujie@ 两人 `employment_status` 已 `left`,但 offboarding_process 永远 `blocked`,他们名下的金山云服务器 / finops·cloudcost DNS / IAM 用户 owner 一直指向离职者,T+0 force-transfer 从未对其触发。**根因**:唯一写 `left` 的 `LeaverBlockGuard.flip_to_left` 锁了 process 行但用**快照读**读 pending blocker(MySQL/TiDB RR 下 locking read 不推进后续普通 SELECT 快照),flip 事务快照早于并发 `start_process` 扫描提交 blocker 时,读到零 pending → 放行 flip;此后人已 `left`,`hr.process_advance`(只选 `leaver_pending`)永不再碰,`reconcile_processes` 又只翻已完成 task 的 blocker、不 force-transfer owner 绑定基础设施 → owner 永久孤儿。**修复**:① guard + listener 的 pending-blocker 读改 `.with_for_update()` current read 堵竞态;② 新增 cron `hr.recover_orphaned_left`(每 15 分钟,**默认启用**)扫"`left`+`blocked`+有 pending"的孤儿,跑 `_advance_single(bypass_grace=True)` 把 owner 转给交接人/manager 兜底并完成流程(写 `cmdb_network_owner_history` / 同 T+0 force-transfer 路径)。**Operator 该做什么**:正常无需动作——部署后该 cron 自动把残留 owner 转给 `default_handover_person_id`(吴→左武辉、褚→孟凡生)并完成流程;失败通知器卡片(`hr.recover_orphaned_left ... raised`)说明某孤儿连 manager 兜底都解析不出,去给该流程设有效交接人或 manager 链后等下一 tick 自愈。瞬时无 fallback 的孤儿归 `still_blocked` 并已由内部记 CRITICAL compliance incident。
> - **(2026-06-02, auto-Owner-grant 遇到不可解析的 AWS permission-set 时不再每 10 分钟空转 —— migration 244)**. `cmdb.auto_owner_grant_backfill` cron 对 AWS scope 在运行时按名字(platform setting `cloud.aws.scope_request_default_owner_role_name`,默认 `DBaaS-DevLeader-Role`)去 Identity Center 解析 owner permission-set。**名字配错 / perm-set 不存在**时解析确定性失败、不写任何 grant 行,导致该 scope 永远是候选 —— 每 tick 重跑一次**付费的** `list_scope_roles` + 写一条 `cloud.scope_request.auto_owner_grant status=skipped` 审计。**修复后**:`cmdb_cloud_scopes.auto_owner_grant_unresolvable_at` 退避标记盖戳后,cron 在 `unresolvable_retry_hours`(默认 6h,可在 job config 覆盖)内跳过该 scope;过窗自动重试自愈。**Operator 该做什么**:收到失败通知器卡片(标题含 `unresolvable=N` + scope_id 列表)时,去 Identity Center 确认该 permission-set 名是否存在 / 拼写;要么在 Identity Center 建/改名,要么改 platform setting `cloud.aws.scope_request_default_owner_role_name` 对齐实际名字。改完**不需要手动重跑** —— 过 6h 窗口 cron 自动重试并在成功后清标记;想立即重试就对该 scope `PATCH owner_person_id`(人工换 owner 即清标记)。失败通知器对同一类不可解析按稳定签名**只通知一次**(scope_id UUID 被签名函数归一),admin ack 前不刷屏。瞬时 driver 崩溃 / `LockBusy` **不**盖标记(照常下个 tick 重试)。
>
> - **(2026-06-02, Tencent CAM 离职清云权限改 group-aware revoke —— 修正 PR #663 走错方向的 "DetachUin" 假修复)**. lifu.wu@/chujie@ 的 13 条 grant **全部是 group-inherited**(管理员 / 账单计费 / EE 三个 CAM group),而不是 direct attach。PR #663 加 `DetachUin` 让 API 接受请求,但 `DetachUserPolicy` 只对 direct attach 有效,对 group-inherited 是 **silent no-op**(返 RequestId 但策略仍活)。**Helix 侧反映 "revoked" 但云上 IAM 策略附加关系仍存活,离职后该账号保留完整云权限,高严重度 prod 数据残留。**
>
>   **三层套娃 root cause**:
>   1. **SDK 3.1.79 字段 bug** — `DetachUserPolicyRequest` / `AttachUserPolicyRequest` 只接受 `PolicyId` + `DetachUin`/`AttachUin`,**完全没有** `DetachUserName` / `AttachUserName` 字段。旧 driver `de.DetachUserName=user_name` 是写到 ad-hoc 属性,SDK 序列化 drop 首字母 D 变 `EtachUserName`(同 `_Page`/`_Month` 已知 bug 模式)→ API 拒绝 `UnknownParameter`
>   2. **`DetachUserPolicy` 只对 direct attach 有效** — group-inherited policy 必须 `RemoveUserFromGroup(Uid, GroupId)` 才能真撤
>   3. **`Uid` ≠ `Uin`** — `ListGroupsForUser.Uid` / `RemoveUserFromGroup.Info[].Uid` 是 CAM 内部 id,跟 sub-account UIN 是不同数字;`GetUser` 同时返两个,旧代码把 Uin 当 Uid 传 → API 报 `ResourceNotFound.UserNotExist`(撒谎)
>
>   **Driver 修复** [`tencent_cloud.py`](../../server/app/integrations/drivers/tencent_cloud.py):
>   - 新 `_resolve_user_ids(client, user_name)` helper:`GetUser` 同时返 `(Uid, Uin)`,UserNotExist → `None`,caller 视作 `already_absent`
>   - `_revoke_sync` 重写为 **group-aware**:
>     1. resolve `(Uid, Uin)`
>     2. `ListAttachedUserAllPolicies(TargetUin=uin, AttachType=2)` 找这条 policy 的 source group → 对每个 group `RemoveUserFromGroup(Uid, GroupId)`
>     3. 找不到 group-inherited → `AttachType=1`(direct attach)→ `DetachUserPolicy(DetachUin=uin, PolicyId)`
>     4. 都找不到 → `already_absent`
>   - `_revoke_account_sync`:同步用 `Uid` 调 group APIs;detach loop 去掉 ad-hoc `DetachUserName` 写
>   - `_grant_sync` `AttachUserPolicy` 只传 `AttachUin` + `PolicyId`(去掉 `AttachUserName`)
>
>   **副作用警告(运维必读)**:撤一条 group-inherited grant 会 **把 user 整个移出该 group**,同 group 给 user 的其他 policy **一起被撤**。这是 Tencent CAM 的固有语义(group 是 atomic 撤权单位)。**离职场景天然 correct(全撤);admin 单撤场景** UI 必须在 confirm modal 显示 `result.detail.groups_removed` 和 `side_effect_warning`,让操作员看到这点。
>
>   **lifu.wu@ / chujie@ 13 条 prod 残留处理记录(2026-06-02 03:20 SSM 手动清完)**:
>   ```bash
>   # 1. SSM 到 prod worker 容器
>   aws ssm start-session --target i-0c15d98cba3ebba9c --region ap-southeast-1
>   sudo docker exec -it compose-worker-1 bash
>
>   # 2. 用 /app/.venv/bin/python + PYTHONPATH=/app 跑 inline RemoveUserFromGroup
>   #    (脚本走 GetUser(Name) → 拿 Uid → ListGroupsForUser(Uid) →
>   #     RemoveUserFromGroup(Uid, GroupId) 全部 group)
>
>   # 3. 验证云侧:ListAttachedUserAllPolicies(TargetUin=...) 必须返 0 条
>   ```
>   实际清理结果:lifu.wu@ Uid=22001053,从 EE/账单计费/管理员 3 个 group 移除,after 0 policies;chujie@ Uid=22782094,从 管理员 1 个 group 移除,after 0 policies。
>
>   **同类未来事故的自然恢复**:`sweep_pending_manual_cloud_revokes` 每小时跑,对 grants 上 `detail.requires_manual_cloud_revoke=True` 的会自动调新 group-aware `_revoke_sync`,自然清。**不再需要 ad-hoc backfill 脚本** —— PR #663 留下的 `scripts/backfill_tencent_detach_uin_skipped_2026_06_01.py` 在本 PR 中改为直接调 `driver.revoke_access`(绕过 `service.revoke_grant` 看到 status='revoked' 就短路的 bug),但保留作为"已 skipped + last_error 含 DetachUin"这类特殊残留的紧急 ops 工具。
>
>   **回归测试** [`tests/test_integrations/test_tencent_driver_cam.py`](../../server/tests/test_integrations/test_tencent_driver_cam.py) 7 用例锁住:group-inherited path / multi-source-group / direct-attach / not-attached(idempotent)/ user-missing / Uid-vs-Uin / AttachUin-without-AttachUserName。
>
> ---
>
> - **(2026-06-01, role-change review flow + 30-day sweep — closes [#623](https://github.com/pingcap-inc/Helix/issues/623))**. When an employee's dept / title / manager changes (HRIS sync detects via `_compute_org_change_flags`), the new `role_change_coordinator` flips EVERY active or pending grant they hold to `review_status='pending_review'` and records the moment in `review_status_changed_at`. Access is **NOT auto-revoked** — the legal team's standing policy is human-in-the-loop on org moves (prior incidents where auto-revoke mid-quarter hit the wrong person). Admin path: `/admin/role-changes` lists role-change events + a per-person 「N / total 待决断」 chip; clicking opens a Drawer with every grant + `保留 / 撤销` buttons. **「撤销」 fully revokes**: `status='revoked'`, `revoked_at=NOW()`, `revoke_reason='role_change_confirmed_revoke'` — the cloud-side cleanup cron picks it up next tick (same code path as manual admin revoke). **「保留」 just stamps `review_status='confirmed_keep'`**, no cloud-side change.
>
>   Stalled rows (`pending_review` AND `review_status_changed_at < now - 30d`) get a Feishu card from the new daily cron `role_change.scope_review_sweep` (10:00 Asia/Shanghai) to **scope owner + IT admin pool**. Dedup is via `NotificationOutbox.(template_key='role_change_review_30d', related_resource_id=grant.id)` — second tick does not re-card. **`related_resource_id` is CHAR(36) without GUID adapter; the WHERE bind must use `str(grant.id)`** or asyncmy returns 0 rows silently and the cron re-cards every day (memory `reference_asyncmy_uuid_bind`). The cron commits **per row** after each grant's notification dispatches — a partial-loop failure does not roll back the first N cards already written.
>
>   **(2026-08-29, migration 565 — the flow above was reaching almost nobody, and its trigger has been narrowed.)** Production held **one** `role_change` event in fourteen months and all **5273** `cmdb_cloud_access_grants.review_status` values were NULL: three independent defects in the HRIS staging path meant an org move on an ordinary employee never reached this coordinator at all. Two changes matter for cloud operators:
>
>   1. **The flow now actually fires.** Expect `pending_review` rows to appear where there were none — that is the feature working for the first time, not a new fault. `role_change.dispatch_reconcile` (daily 03:20) additionally replays any staged org move whose inline coordinator call failed silently, dated to when the move was **observed** rather than to today.
>   2. **A title-only move no longer flips grants.** Access review is triggered by a **department or manager** change only. Cloud access follows who a person reports to and which org they sit in, not their title string — and a bulk SF position-catalogue rename would otherwise put every grant in the company into `pending_review` in one sync, which is a review storm that teaches scope owners to click through without reading. The lifecycle event is still written for a title change (the forensic timeline wants it); `summary["title_only"]` and an INFO log record that grants were deliberately left alone.
>
>   🔴 **(2026-09-05, migrations 610 / 611 — this sweep is now OFF, and the card it used to send was unactionable for half its recipients.)** Three things a cloud operator needs to know:
>
>   1. **`role_change.scope_review_sweep` is disabled and policy-gated.** `enabled=0` (migration 610), and the handler re-reads `platform_settings['role_change.review'].stale_grant_push_enabled` **at send time** — turning the `scheduled_jobs` row back on sends nothing on its own. It was stopped because the review packet behind it was reporting a mover's whole **inventory** as though the move had caused it (`provision_policies` is empty in production, so no baseline could say otherwise). At the time it was switched off there were **35 grants** flagged on 2026-09-04 that would have produced ~175 cards on 2026-10-04.
>   2. 🔴 **The scope owner may now resolve a grant on their own scope.** `PUT /api/v1/role-changes/grants/{id}/review` was `require_admin()` while the card went to **scope owner + IT** — so the primary recipient got a 403 at the endpoint and a bounce at the page, which is a large part of why this sweep never produced decisions. The gate is now `global_admin` **or the owner of the grant's scope**, checked per scope. This matches what the cloud domain already believed: `_revoke_grant_locked` has admitted `is_cloud_scope_owner_of` all along; only this one endpoint disagreed with its own domain.
>   3. **The card sends each audience a different landing page.** Owner → `/my/todos?tab=cloudGrantReview` (a new inbox tab listing exactly the flagged grants on scopes they own). Admin → `/admin/role-changes`. The template body no longer names a page, because naming one is wrong for the other half. ⚠️ Deliberately **not** the scope detail page: its own `?tab=pending` is pending *requests awaiting the owner's approval*, a different kind of pending, so it would land them on a page that does not contain what the card asked about.
>
>   ⚠️ **Unchanged and worth restating:** 「回收」 from that tab is a **real** revoke — `CloudLifecycleService.revoke_grant` with `suppress_driver_error=True`, exactly as the admin path always did. Helix marks the grant revoked even if the cloud-side teardown failed; the confirm dialog says so in the owner's own words. Widening the actor set widens exposure to that pre-existing gap, which the separate 访问对象闭环 epic is addressing.
>
>   **(2026-08-29, migration 568 — the review is no longer cloud-only, and it is no longer yours to answer alone.)** Cloud grants turned out to be ONE of ~24 kinds of thing a mover holds; a senior engineer's real holdings measured on production are 238 items across ten categories. The new **review packet** covers all of them, and it is routed to the **new manager**, not to IT — an admin looking at a grant list cannot know whether the person still needs it. Three consequences for cloud operators:
>
>   1. **`/admin/role-changes` now opens on the packet**, with the cloud-grant table kept as a second tab. The grant review flow itself (`pending_review` → `confirmed_keep` / `confirmed_revoke`, and the 30-day sweep) is unchanged.
>   2. **A packet decision does NOT touch a grant.** Recording 「不再需要」 on a cloud-access item in the packet writes to `hr_role_change_review_items.decision` and nothing else — the grant stays `active` (there is a test pinning exactly this). Actually revoking still happens on the grant surface, with its approval and audit. Issue #623's no-auto-revoke-on-a-move policy is why.
>   3. **Cloud scope / IAM-user / DNS / domain / LB / CDN ownership now appear in a mover review**, where before only grants did. Expect ownership questions to reach scope owners' managers that previously reached nobody at all.
>
>   Auditor's view: every role_change event lands on the forensic timeline (issue #622's per-person case file at `/admin/lifecycle/forensic/<id>`) with the full `{before, after}` blob; the page's "in-flight" KPI lights up when there's a `role_change` event in the last 60 days.
>
> ---
>
> - **(2026-05-29, `cmdb.grants_sync` 50% drift-sweep 闸触发 — 修复 driver 分页 + scope-close 级联,migration 217)**. 生产事故 2026-05-28 21:19:UCloud 反推 cron 报 `drift_sweep refused: matched 11 of 29 existing discovered grants (38%) is below the 50% safety floor`。**安全闸正确拒绝了一次会误吊销 18 条合法权限的 mass-revoke**。诊断后落地三项修复:
>   - **修 1:UCloud driver `ListPoliciesForUser` 加分页**([ucloud_cloud.py:825-862](../../server/app/integrations/drivers/ucloud_cloud.py:825))。UCloud 该 API 是 `Offset/Limit/TotalCount` 分页,但 driver 之前单次调用就 return,用户策略超过默认页大小时尾部静默丢失 → drift sweep 把丢失尾部当云侧撤销 → mass-revoke 候选。同文件 `ListUsers` / `ListPolicies` / `_resolve_user_id` 早就分页,补齐 `ListPoliciesForUser` 让所有 List 调用语义一致。中段失败 raise(不写 partial 快照)。
>   - **修 2:AWS IAM `list_attached_user_policies` 用 boto3 paginator**([aws_cloud.py:2101-2125](../../server/app/integrations/drivers/aws_cloud.py:2101))。同类隐患,目前 AWS IAM 10-policy/user 配额一页能放下,但配额可申请到 20+,预防性切换到 `iam.get_paginator("list_attached_user_policies").paginate(...)`(跟同文件 `list_users` 用同一套 paginator 家族)。Aliyun RAM `ListPoliciesForUser` 是有意单次调用 — 2015-05-01 API 无分页字段 + RAM 硬配额 10/user;加了一段注释固化结论。
>   - **修 3:`scope_sync._safe_sweep` 关 scope 必须级联关 grants**([lifecycle_service.py:6029-6151](../../server/app/domains/cmdb/lifecycle_service.py:6029) + [migration 217](../../server/alembic/versions/217_cascade_close_grants_on_closed_scopes.py))。**关键发现**:`scope_sync` 把云侧不再返回的 scope 一键 `status='closed'`,但**没级联关掉它上面的 grants**;`grants_sync` 的 drift sweep 只跑 `synced_scope_ids`,关掉的 scope 永远进不去 → `status='active'` 僵尸 grant 无限累积。生产数据:UCloud closed scope `1ccf3d0a-…` 在 2026-05-05 关闭,留下 **29 条 active discovered grants 滞留 `last_seen=2026-05-04` 至今 24 天**。
>     - **修复:同事务级联 disable**。把 `status='active' OR 'pending'` 的 grant 全置 `status='disabled'` + `revoke_reason='scope_closed'`(新枚举 `GRANT_REVOKE_REASON_SCOPE_CLOSED`)+ `revoked_at=NOW()` + 清 `last_notified_stage`(防 inactivity sweep 在已关行上再发提醒)。
>     - **审计行**:`cloud.scope.sweep_close` 永远写(凡是 sweep 关了 scope 就写,canonical "哪些 scope 关了" 表面);`cloud.scope.sweep_close_cascade` 只在真有 grant 级联时才写(详情里的 `grants_disabled` 永远非零)。
>     - **手工 decommission 同步对齐**:[`CloudLifecycleService.decommission_scope`](../../server/app/domains/cmdb/lifecycle_service.py)(admin "下线" 按钮)的 grant cascade 也在同 PR 补上 `revoke_reason` + `revoked_at` + 清 `last_notified_stage`,两条 close 路径语义对齐。
>     - **Migration 217 回填**:`UPDATE … JOIN cmdb_cloud_scopes ON … WHERE s.status='closed' AND g.status IN ('active','pending')`。downgrade 双重收紧 — 只复原 `revoke_reason='scope_closed'` **且** scope 已重新 `status='active'` 的行,防止运维误操作把已关闭账户的权限再放出来。
> - **运维处理这次 UCloud incident**(不是上面 PR 的修复范围):事故剩下的 18 条 missing 是 **3 个 active UCloud IAM user 整体从 `ListUsers` 响应里消失**(`name-8pto9smh` / `shengbo.ma` / `suhaihuang`,对应在职员工 chujie@pingcap.com / shengbo.ma@pingcap.com / suhaihuang@pingcap.com)。整 user 蒸发 + 当事人都在职是 50% 闸要拦的典型 — **不要直接按 force-drift-sweep**。先去 UCloud 控制台用 driver 同把 AK/SK 跑 `ListUsers` 看这 3 个 user 是否还在:
>   - 还在 → 一次 API 抖动 / 分页响应不完整,下一轮 cron(05:00 daily)自然恢复,什么都别做;
>   - 被刻意删除 → 调 `POST /api/v1/cmdb/integrations/{id}/grants-sync/force-drift-sweep`(`can_manage_cloud` 权限,会写 `cloud.grant.drift_sweep_force_override` 审计行)。
>   - UCloud 返回但被 driver 过滤(没 Email 字段且 UserName 不含 `@`)→ driver 过滤逻辑要修,不是数据问题。
>
> ---
>
> **Earlier**:
>
> - **(2026-05-28, server lifecycle action menu + detail page — PR #559)**. 用户报 `/admin/cloud/servers` 上"以前 CMDB 截图里有开机/关机/重启/删除/重装/下工单,现在云服务器管理页什么都没有,搜索点不动也没有详情"。落地范围:**4 个 lifecycle 操作 + 详情页**,覆盖 UCloud / Aliyun / AWS 三家;其余 5 家 provider(AliCloud Intl / Azure / GCP / Tencent / KSyun)抛 `CapabilityNotSupported` → 后端返 501 + 前端按 provider 灰菜单项。AliCloud Intl 暂不支持是因为它走多账号 AssumeRole 路径,需要先 resolve 实例所属 sub-account 再代理调用 ECS,这部分尚未对接 → naive 继承 Aliyun 会用 master AK 打 `ecs.aliyuncs.com`(domestic endpoint)操作错账号的实例,故显式 override 抛 501。"重装"老 CMDB 也是 stub 没实现,"下工单"老系统只 KSY 裸金属有 —— 两个都不补。**操作上变化**:
>   - `/admin/cloud/servers` 列表新增"详情"列 + 行点击跳详情 + Edit 旁的"更多 ↓"下拉(开机/关机/重启/删除)。`/admin/cloud/servers/:id` 复刻老 CMDB 服务器详情布局(主机名/状态/类型/Owner/云厂商/项目组/应用/备注/地域/可用区/VPC/子网/实例 ID/内网 IP/公网 IP/CPU/内存/磁盘/镜像)。
>   - 4 个新 HTTP 路由 `POST /cmdb/servers/{id}/{power-on|power-off|restart|destroy}`,RBAC `can_manage_cloud`(it_admin / cloud_admin 才看得到按钮)。成功后 DB `status` 乐观置过渡态(`starting` / `stopping` / `rebooting` / `terminated`),destroy 还 soft-delete 落到 [`/admin/cloud/recycle-bin`](../../web/src/pages/cloud/CmdbRecycleBinPage.tsx) 可恢复。
>   - **幂等性**:UCloud RetCode 173/174(already in target state)被吞为成功;Aliyun `IncorrectInstanceStatus.*` + `InvalidInstanceId.NotFound` 同理;AWS EC2 本身就幂等;destroy 在 AWS 上额外把 `InvalidInstanceID.NotFound`(实例已经被云外释放掉)也算成功。双击 / 重复点击不会 5xx。
>   - **状态过渡的时效边界**:`starting` / `stopping` / `rebooting` 是 Helix-side 乐观值,不代表云端已经完成。真正的状态由下一次 `cmdb.servers` sync cron(5 分钟)回填。如果你下完指令立刻看到一台 `starting`,5 分钟后才变 `running`,这是预期行为。**不要**因为状态没变就再点一次开机——指令已经下了,只是云端 still propagating。
>   - **错误码语义**(admin 弹窗里看到的):501 `CLOUD_LIFECYCLE_NOT_SUPPORTED` = provider 不支持(Azure/GCP/Tencent/KSyun 上这 4 个按钮本来就该灰);502 `CLOUD_LIFECYCLE_AUTH_ERROR` = integration 的凭证被云端拒绝(去 `/admin/integrations` 检查 AK 是否已轮换 / 是否仍有 ECS 操作权限);422 = 服务器行的 region 字段为空(理论上不该发生,如果发生说明 sync 流水线某天漏抓 region — 联系开发);502 `CLOUD_LIFECYCLE_FAILED` = 其它 provider 拒绝,消息已脱敏。所有错误路径都写 `audit_logs.action='cmdb.server.{action}', status='failure'`,可在 `/admin/audit` 按操作人 + 时间过滤回放。
>   - **删除的不可逆性**:点删除按钮 = UCloud `TerminateUHostInstance` 带 `ReleaseEIP=true + ReleaseUDisk=true` / Aliyun `DeleteInstance` 带 `Force=true` / AWS `terminate_instances`,云端实例 + EIP + 数据盘**直接物理释放**。Helix-side cmdb 行可以从回收站恢复(`/admin/cloud/recycle-bin` → 服务器 tab),但**只能让 cmdb 行回来,云端 VM 不会回来**。前端 Modal.confirm 上已加红色警告文案;destroy MCP tool 描述也明确告知 AI client "AI clients SHOULD confirm with the human before invoking"。
>   - **MCP 镜像**(`_REGISTERED_TOOLS_COUNT_HINT` 188 → 192):`server_power_on / server_power_off / server_restart / server_destroy`,1:1 HTTP 路由镜像,`mutates=True`。同 PR 落,服从 MCP-sync 规则。
>   - **审计样例**:`SELECT action, status, JSON_EXTRACT(detail, '$.instance_id'), JSON_EXTRACT(detail, '$.new_status') FROM audit_logs WHERE action LIKE 'cmdb.server.%' ORDER BY created_at DESC LIMIT 20;` 可以一眼看到最近 20 条操作。
>
> - **(2026-05-27, provisioning `_resolve_execution_mode` was downgrading every `software_id=NULL` revoke to `manual_notify`)**. Root cause behind a prod report of repeated "请手工回收 (未绑定软件)" pushes: `ProvisioningService._resolve_execution_mode` keyed ONLY off `software_id`, so the two task shapes that carry `software_id=NULL` by design — `target_type='account'` (Jira full-access revoke, #537) and `target_type='api_key'` (cloud-credential teardown, the 2026-05-19 PR3 entry below) — were both force-resolved to `manual_notify` and never reached the auto driver path. **Jira impact (the reported symptom)**: every leaver's `revoke_all_access` task sat `notified` instead of auto-running; prod showed all 6 such tasks ever created were `manual_notify`, 0 ran auto — i.e. offboarded staff kept Jira access until a human actioned the card. **api_key impact (latent)**: the 2026-05-19 PR3 entry below says api_key auto-revoke "really tears down cloud now", but PR3 only fixed the connector short-circuit + post-success hook — the task still had to be `auto_api` to get there, which this same bug prevented. No api_key offboarding auto-revoke had ever actually fired in prod (0 such tasks). Fix ([provisioning/service.py](../../server/app/domains/provisioning/service.py)): `_resolve_execution_mode` is now target-aware — `account` → `auto_api` iff the integration driver `supports_full_access_revoke` (else `manual_notify`), `api_key` → `auto_api` (teardown runs in `_post_success_side_effects` via `ApiKeyService.revoke_key(actor_role=SYSTEM_OFFBOARDING)`). **Operationally**: from this deploy, offboarding a Jira user auto-removes them from ALL groups without IT clicking anything; clicking "auto-revoke" on an api_key blocker now actually deletes the AWS IAM user / Aliyun RAM user / Azure OpenAI account instead of just notifying. The 5 Jira leaver tasks currently stuck `notified` in prod (qiang.wu / chenyiwen / cheeyan.ho / qiang.cao / oliver.tajiki, created 2026-05-27 10:58) are pre-fix rows: reset each to `pending` + `execution_mode='auto_api'` so the `provisioning.dispatch` cron runs `revoke_all_access`, OR re-run their offboarding scan after deploy.
> - **(2026-05-27, `cmdb.resource_sync` + cost/IAM-read UCloud body-level transient `RetCode` retry)**. Follow-up to the 2026-05-24 transport-layer retry entry below. That fix covered httpx transport flakes (`ReadTimeout` / `ConnectError`); this one covers the OTHER UCloud failure shape: UCloud returns **HTTP 200 + a non-zero `RetCode`** for transient server faults too, so `RetCode=8498` "Call third api service timeout, please retry" arrives as a successful HTTP response and bypassed the transport/5xx retry budget entirely — it was classified + raised immediately. Prod alert: `cmdb.resource_sync <int>/servers: UCloud DescribeUHostInstance RetCode=8498 ... · 1 leg(s): transient × 1 [region:jpn-tky]` failed the whole `servers` capability and paged, even though the next call would have recovered. Fix ([providers/ucloud.py::_call](../../server/app/domains/cmdb/providers/ucloud.py)): `_is_retryable_retcode(ret, msg)` now classifies a transient body-level RetCode (8498 + message hints `timeout`/`please retry`/`try again`/`time out`; EXCLUDES region-skip 150/230 and the auth bucket) and the call retries it in-place within the same 3-attempt budget (exp backoff `2**attempt`), raising only after exhaustion. The lifecycle/billing driver ([drivers/ucloud_cloud.py::_signed_call](../../server/app/integrations/drivers/ucloud_cloud.py)) — which previously had NO retry budget at all — gained a `retryable` flag, enabled ONLY on idempotent reads (`GetRegion`/`List*`/`ListUBillDetail`); non-idempotent IAM mutations (`CreateUser`/`DeleteUser`/`Attach`/`DetachPolicies`) stay single-shot so a timed-out write never silently re-fires (offboarding/provisioning idempotency is handled at the job level via next-tick re-run). **Operationally**: from this deploy, a single UCloud gateway timeout (RetCode 8498) on any region's UHost/ULB/DNS/CDN/UFile sweep — or on the UCloud cost_facts / IAM-read crons — self-heals silently instead of paging. The per-region `list_servers`/`list_load_balancers` re-raise behavior is UNCHANGED: a transient that survives all 3 retries still marks the capability failed (NOT a partial sweep) because `_safe_sweep` does not consult `partial_failures` and a partial sweep could soft-delete a transiently-failed small region within drop-tolerance. Cross-checked aws/azure/gcp/aliyun/alicloud_intl/tencent — no same-class gap (each has SDK-level or hand-rolled body-transient retry). If you see fewer `cmdb.resource_sync` UCloud RetCode=8498 Feishu cards after deploy, that is expected and healthy; sustained UCloud outages still page one tick later.
>
> - **(2026-05-27, long-transaction 1205 锁等待超时 bug class — billing_daily / roles_sync / grants_sync 全部改 per-scope commit)**. GCP 每日账单抓取生产失败,run record 报 `PendingRollbackError ... Original exception was: (asyncmy.errors.OperationalError) (1205, 'Lock wait timeout exceeded') ... [SQL: INSERT INTO notification_outbox ...]`。**真因**(查 prod `scheduled_job_runs` 确认,2026-05-26 22:18 run,scope `pingcap-gardener`):该 cron 把**整轮多 scope 抓取跑在一个事务里**,只在最后(+reconciliation)commit 一次;大账号一轮实测 122s–894s,长事务横跨多次外部 cloud API 调用,长时间持有目标表行锁,与并发写入者争锁触发 1205;而 1205 被 best-effort dispatch handler 的 `except: log` **吞掉但没 rollback**,session 进入 rolled-back-pending,下一条语句即抛 `PendingRollbackError` 整轮失败、丢弃所有已抓取行。**同一个 1205 class 在 prod 还命中过 `cmdb.roles_sync`(2×,`INSERT INTO cmdb_cloud_roles`)、`cmdb.scopes_sync`、`cmdb.cost_facts_sync`** —— 不是单点,是结构性的「长事务跨外部 I/O」反模式。**修复后运维变化**:(1) `billing_daily` / `roles_sync` / `grants_sync` 现在**每 scope commit 一次**——锁随每个 scope 及时释放,1205 几乎不再发生,单个 scope 失败不再丢弃其他 scope 已抓取的数据(对齐 `cost_facts` per-batch commit)。**`grants_sync` 的 drift-sweep mass-revoke 仍是单一原子块 + 50% 安全下限不变**,per-scope commit 只提交发现到的 grant upsert,不会扩大撤销半径;(2) `NotificationService.dispatch` 的 outbox INSERT 包进 savepoint,告警卡写入的瞬时 DB 错误只回滚该卡、不再毒化整个 cron 的 session;(3) **billing reconciliation(unregistered-account scope_id=NULL 行)不再硬编码 `currency="USD"`**——CN 云(阿里云/腾讯/UCloud)写的是 native CNY,之前硬编码 USD 会让 admin billing-overview 总额虚高 ~7×;现在保留 driver 的 currency label。`scopes_sync`(已 upfront list + 50 批量 commit)、`cost_facts_sync`(已 per-batch commit)结构本就安全,未改动。**你会看到**:`/admin/cron/scheduled-jobs/{billing_daily,roles_sync,grants_sync}` 历史里瞬时锁争用不再表现为整轮失败;若 1205 仍偶发,真实错误码会进 worker 日志而非被 `PendingRollbackError` 掩盖。无 migration / cron schedule / RBAC / MCP 改动。
>
> - **(2026-05-26, Round-52 decommission 错误友好化 + 跨 driver 对称性)**. 用户在尝试 Azure subscription 下线时碰到 Azure 返回 `400 ResourcesExistingOnSubscription`,Helix 把 ARM 的原始 JSON 直接吐到 admin 弹窗,完全没有可操作的提示。本轮把所有 cloud driver 的 `close_scope_on_provider` 4xx/5xx 错误路径补齐成"友好 + 可操作"形态。**操作上变化**:
>   - **Azure 取消订阅**(`POST /cmdb/scopes/{id}/decommission` on Azure scope):
>     - `ResourcesExistingOnSubscription` → 提示先 `az group list --subscription <id> -o tsv --query '[].name' | xargs -I {} az group delete --subscription <id> --name {} --yes --no-wait` 删完所有资源组再重试。`error_code='resources_existing'`。
>     - `SubscriptionNotInRegisteredState` → 视为 `already_closed=True`,Helix-side 自动同步到 closed(订阅其实已在 Azure 侧出局了)。
>     - `PendingCharges` / `BillingAccountInactive` / `OutstandingCharges` / `BillingProfileNotActive` → 提示先到 Azure Cost Management 把欠费结清。`error_code='pending_charges'`。
>     - `SubscriptionMustHaveOwner` → 提示 Azure 要求 cancel 时必须有 Owner role(且不能是 SP-only),先给个人 admin assign Owner 再重试。`error_code='missing_owner'`。
>     - `401` → `auth_failed` + 提示 SP `client_secret` 是不是过期/轮换了。
>     - `429` → `throttled` + retry-after 写进 detail。
>     - `5xx` → `transient_failure` + 提示可安全重试。
>     - 未识别的 400 ARM code → `cancel_precondition_failed` + 把 ARM code 原样带进 message 方便后续 grep。
>   - **Aliyun ResourceDirectory DeleteAccount** 同样补齐:
>     - 包含 `HasResource` / `HasUndeleted` / `PrePaidResource` → `resources_existing` + 提示先释放 ECS/RDS/OSS/SLB/云盘 再重试。
>     - 包含 `NotPassRealNameAuth` / `RealNameAuth` → `not_real_name_authed` + 提示去 Aliyun 控制台完成实名认证。
>     - 包含 `Bill` / `Charge` / `Arrear` / `PaymentMethod` → `pending_charges` + 提示去 Aliyun 账单结清。
>   - **GCP `projects.delete`** 400/412 提示包含 3 个最常见原因(unresolved lien / 仍 attach billing / Shared VPC host),并对应附上具体 `gcloud` 命令。从 `details[].reason` 取 `LIEN` → `liens_exist`;message 含 `billing` → `billing_attached`;否则 `precondition_failed`。新增 429 → `throttled`、5xx → `transient_failure` 分类。
>   - **AWS** 已经覆盖 `ConstraintViolationException` 等具体错误(2026-05-24 上线),本轮不动。
>   **回滚 / 调试**:`cmdb_audit_log` 现在每一次失败的 `close_scope_on_provider` 都带上 `error_code` + provider-specific 原始 code(Azure: `arm_code`,Aliyun: `provider_message`,GCP: `gcp_reason`),grep audit 就能聚合"这一周有几个 scope 因为 `resources_existing` 失败"。
>
> - **(2026-05-26, Round-51 上线前账单/告警/异常综合扫 — 16 个真 bug)**. 10 轮独立扫描汇总,操作上有 5 个关键 invariants 改变了运维行为:
>   1. **launch-day 不再爆发 Feishu 卡片**. semiannual 和 zero-spend cron 现在尊重 platform-level `cmdb.owner_confirmation.first_observed_at` 时间戳 —— 第一次跑时 stamp `now`,30 天 cooling-off 期内只发 ever-confirmed-but-180d-overdue 的卡。如果你在 launch day 想*立即*让 cron 触发某些已知 scope,手动 `UPDATE platform_settings SET value = JSON_OBJECT('at', '2026-01-01T00:00:00+00:00') WHERE key = 'cmdb.owner_confirmation.first_observed_at'` 把 stamp 改到 30 天前。回滚:删除 key,下次 cron 会重新 stamp。
>   2. **zero-spend 现在需要 ingest-presence 证据**. `list_scopes_with_zero_spend_last_month` 加了 "至少 [month-60d, month) 内有 1 条 cost_fact 行" 的预过滤。换言之,一个 cloud account 接进来但 cost_facts driver 还没跑过,zero_spend cron 不会误报。运维含义:cost_facts 同步必须比 zero_spend cron 早至少 1 个月跑成功,否则真零花费的 scope 不会被告警。
>   3. **spike alert 最小 7 天 baseline + multiplier 范围校验**. `cloud.daily_spike_multiplier` 和 `cloud.daily_spike_min_usd` 现在被 plausibility band 钳制:multiplier `[1.0, 100.0]`、min_usd `>= 0`。出界配置 fall-back 到默认 1.5 / 10 + warning log,不会再 silently 让告警风暴或永远静音。Spike 在 `past < 7 天` 时不触发(新 scope 第 1-7 天豁免)。
>   4. **FX rate staleness gate**. `cmdb` cron 和 cost service 现在拒绝 `>14 天` 陈旧的 CNY/USD rate(与 `ai_spending.cny_rate_refresh` cron 的 hard-fail 阈值对齐)。如果 frankfurter.dev 长期挂,这两个 cron 会一起 raise → failure_notifier page IT admin。运维含义:别等 14 天后被告警风暴打脸,看到 cny_rate_refresh 第一次失败就要排查。
>   5. **billing 告警决策有完整 audit trail**. 每次 spike / threshold alert dispatch 现在都写 `cloud.scope_alert.fired` audit_log 一行,含 baseline/threshold/multiplier/FX rate/recipients_count/dispatched 快照。"为什么 3am 收到这张卡" 现在能 forensic 回溯。
>
> 同 PR 还修了 leaver retry 重发 / `cmdb.daily_spike` 卡片单位混报 / AllocationTab 越权显示 / DeptCost 路由暴露 / Feishu / ECharts XSS / `UnsafeURLError` 泄漏 webhook bot token 到日志 等 11 项 —— 详见 [ROADMAP](../../ROADMAP.md) 顶部 Round-51 entry。
>
> - **(2026-05-25, resource-creator sweep first-run shook out 2 PR #466 bugs)**. After PR #500 seeded the cron and the user manually triggered the first sweep, the run raised `RuntimeError: 116/200 resolves failed (≥50% threshold)` and `cmdb_resource_creators` was empty (zero rows landed). Two architectural bugs that PR #466's tests didn't cover:
>   1. **handler-wide rollback**: `_upsert_creator_row` only `flush()`s and the outer `_execute_job_locked` runs the whole handler in one `async with session` that only commits on clean return. The threshold-raise at the end of the sweep handler exited the `async with` WITHOUT commit → every row (the 84 successes and the 116 errors) rolled back. Operator-visible symptom: empty `cmdb_resource_creators` + generic threshold message + no per-resource `last_error` rows to diagnose from. Fix: per-iteration `await db.commit()` inside the handler loop so each row is durable the instant it's written; the threshold-raise still fires for cron-failure-notifier signaling but doesn't destroy the evidence.
>   2. **Aliyun ActionTrail wrong endpoint**: PR #466 hardcoded `host="actiontrail.aliyuncs.com"` (no region) but `actiontrail` is region-pinned on Aliyun — that global host doesn't resolve in DNS. Verified 2026-05-25 from prod EC2 (`compose-backend-1` in ap-southeast-1): `getent hosts actiontrail.aliyuncs.com` → empty, `getent hosts actiontrail.cn-shanghai.aliyuncs.com` → 3 IPs, `getent hosts actiontrail.ap-southeast-1.aliyuncs.com` → 2 IPs. All 116 failures were `httpcore.ConnectError: [Errno -2] Name or service not known` against the bogus host. Fix: new `_ACTIONTRAIL_REGIONS: tuple[str, ...]` class attribute; default `("cn-shanghai",)` on `AliyunCloudDriver`, override to `("ap-southeast-1", "us-east-1", "eu-central-1")` on `AlicloudIntlCloudDriver` (matches the intl integration's `config_public.regions`). `get_resource_creator` walks the regions, transport-error on one region falls through to the next, every-region transport-fail re-raises so the service layer marks `status='error'` (NOT terminal `not_found` — that would bury transport-broken resources permanently after a one-shot network blip).
>
>   **Per-cloud first-tick status from the broken run** (preserved here because the bug ate the row evidence): GCP `entries.list` returned 200 OK (proves the `roles/logging.privateLogViewer` org-level grant works end-to-end). Azure `microsoft.insights/eventtypes/management/values` returned 200 OK (the SP's existing `Reader` role on tracked subscriptions includes the activity-log read). AWS: no observable CloudTrail traffic — the candidate pool's AWS resources were all `aws-account:%` / `aws-principal:%` synthetic ids that the driver short-circuits (CUR-derived cost_facts don't carry resource ARNs for those product lines), so AWS validation will happen on the first sweep that picks up a non-synthetic AWS resource id. **Aliyun**: blocked end-to-end by Bug 2 above; revalidate after the fix lands by watching `cmdb_resource_creators.status` distribution per `alicloud_intl` integration on the next manual tick.
>
> - **(2026-05-25, resource-creator sweep is now actually running)**. `cmdb.resource_creator_sweep` was shipped 2026-05-22 (PR #466) but the migration to INSERT the matching `scheduled_jobs` row was forgotten — the in-process decorator registered the handler but APScheduler had no row to attach to, so the sweep had never fired in prod. Migration 201 seeds it: `trigger_type='cron'`, `cron_expr='0 2,14 * * *'` UTC = 10:00 + 22:00 Asia/Shanghai, `config={"limit":200}` = 400 attribution calls/day, throttled to 1 req/s per resource. Paired with a 90-day candidate-freshness filter (`list_unresolved_resources` only considers resources with a `cmdb_cost_facts.date >= NOW() - 90d` row). Without the filter the prod pool of 276,709 distinct rids would take 690 days at 400/day; with it the candidate pool caps at ~38k initial + ~150/day steady-state, ~95 days to drain the back-history. The 90d cutoff matches AWS CloudTrail / Aliyun ActionTrail / Azure Activity Log default audit retention — older resources would just burn rate budget for a guaranteed `window_exceeded` terminal. GCP audits 400d but the uniform 90d cap keeps cross-cloud sizing predictable; a one-shot wider sweep via `run-now {"candidate_window_days": 365}` is supported when an operator wants the GCP back-history. **IAM matrix needed for the sweep** (verify before you expect green status rows):
>   * **GCP** — SA needs `roles/logging.privateLogViewer` (covers `logging.privateLogEntries.list`, required for `entries.list` against `cloudaudit.googleapis.com/activity`). Granted 2026-05-25 at org level (478164292948) to `pch-resource-manager@gcp-digital-infra`, covers all 68 tracked projects + future ones automatically. Verify: `gcloud organizations get-iam-policy 478164292948 --filter="bindings.members:pch-resource-manager*" --format=json | jq '.bindings[] | select(.role=="roles/logging.privateLogViewer")'`.
>   * **AWS** — IAM user/role needs `cloudtrail:LookupEvents` (included in `ReadOnlyAccess` and `SecurityAudit` managed policies). The integration's credentials are inside `integrations.secrets_encrypted` so the lazy verification path is: watch the first sweep tick's `cmdb_resource_creators.status` rows for `error` + `last_error` containing `AccessDenied` against this integration's id; if present, add the policy via AWS console / CLI.
>   * **Azure** — service principal `83c4593d-66b5-4592-9e5f-a78fc58b6ab3` (tenant `6c865c34-...`) needs `Microsoft.Insights/eventtypes/values/read`, included in `Reader` role on every tracked subscription. Lazy-verify path: `cmdb_resource_creators.status='error' + last_error LIKE 'AuthorizationFailed%'` → add `Reader` to the SP on the affected subscription via Azure portal `Access control (IAM)` blade.
>   * **Aliyun (alicloud_intl)** — AK/SK's RAM user needs `actiontrail:LookupEvents`, included in `AliyunActionTrailReadOnlyAccess` system policy. Lazy-verify: `status='error' + last_error LIKE '%NoPermission%'` → attach the policy in Aliyun RAM console.
>
>   **Reading the first 24h of sweep results**: open `/admin/cron/scheduled-jobs/cmdb.resource_creator_sweep/runs` after the first tick (10:00 Asia/Shanghai post-deploy). Expected per-tick result_summary distribution: `resolved` (resource ≤ 90d old, audit log contains create event, creator extracted) + `not_found` (resource > 90d old but a few create events for unrelated resources matched the name; rare) + `window_exceeded` (resource older than provider audit retention; expected for the back-history) + `unsupported` (UCloud / Tencent / KSyun — drivers don't implement `get_resource_creator`, terminal once attempted) + `error` (transient). If one cloud's `error` count is >50% of its share and stable across ≥2 ticks, that's the IAM gap signal — the handler raises `RuntimeError` and the failure-notifier fires.
>
> - **(2026-05-24: `cmdb.resource_sync` Azure / Aliyun OSS / UCloud transport-layer retry)**. Prod alert `cmdb.resource_sync 6310dc15-... /object_storage: partial_failure · 1 leg(s): ReadTimeout × 1 [subscription:d3a296ee-...]` on 2026-05-24 traced to: `domains/cmdb/providers/azure.py` had NO transport-layer retry on `_token` / `_arm_get` / `_arm_list_all` nextLink hop / `get_monthly_billing` — any `httpx.ReadTimeout` / `ConnectError` / `PoolTimeout` raised before a status code existed, bypassed every status-code branch, and landed in `partial_failures`. fc08c02e had shipped the same fix to the cost driver (`integrations/drivers/azure_cloud.py`) three days earlier but resource-sync was a separate file and missed the sweep. Same-shape sibling bugs found and fixed in two more drivers: `providers/aliyun.py::list_object_storage` (OSS REST path used a separate `httpx.AsyncClient` with no try/except, while the same driver's RPC `_call` path had retry) + `providers/ucloud.py::_call` (one-shot wrap with no retry, while same-position Aliyun `_call` is a 3-attempt loop). Retry semantics: 5 attempts (Azure / Aliyun OSS) / 3 attempts (UCloud, matching Aliyun `_call`), exponential backoff `min(2 ** attempt, 60)`s, catches 5 httpx transport exception classes (`ReadTimeout` / `ConnectError` / `PoolTimeout` / `RemoteProtocolError` / `WriteError` and their timeout variants). Re-raises ORIGINAL httpx exception on budget exhaustion so failure-notifier surfaces the real type (PR #465 observability). **Operationally**: from this deploy onwards, a single transient socket flake against any of these three clouds' API endpoints will NOT mark a CMDB capability `partial_failure` — only a sustained outage (≥5 attempts over ~30s) will. If you see fewer Feishu cards about `cmdb.resource_sync` `ReadTimeout` / `ConnectError` after deploy, that is expected and healthy. Real outages still page (just one tick later). Status-code retry (429 / 5xx) semantics on the affected drivers are UNCHANGED — only transport-layer behaviour shifted. Audit: 9 cloud providers reviewed (AWS / GCP / Tencent use SDKs with built-in retry; Alicloud-Intl wraps `aliyun._call` and inherits the OSS fix; Ksyun is a stub) — Azure resource-sync + Aliyun OSS + UCloud were the only three drivers using raw httpx without transport retry.
>
> - **(2026-05-24: `cmdb.resource_sync` Azure / Aliyun OSS / UCloud transport-layer retry)**. Prod alert `cmdb.resource_sync 6310dc15-... /object_storage: partial_failure · 1 leg(s): ReadTimeout × 1 [subscription:d3a296ee-...]` on 2026-05-24 traced to: `domains/cmdb/providers/azure.py` had NO transport-layer retry on `_token` / `_arm_get` / `_arm_list_all` nextLink hop / `get_monthly_billing` — any `httpx.ReadTimeout` / `ConnectError` / `PoolTimeout` raised before a status code existed, bypassed every status-code branch, and landed in `partial_failures`. fc08c02e had shipped the same fix to the cost driver (`integrations/drivers/azure_cloud.py`) three days earlier but resource-sync was a separate file and missed the sweep. Same-shape sibling bugs found and fixed in two more drivers: `providers/aliyun.py::list_object_storage` (OSS REST path used a separate `httpx.AsyncClient` with no try/except, while the same driver's RPC `_call` path had retry) + `providers/ucloud.py::_call` (one-shot wrap with no retry, while same-position Aliyun `_call` is a 3-attempt loop). Retry semantics: 5 attempts (Azure / Aliyun OSS) / 3 attempts (UCloud, matching Aliyun `_call`), exponential backoff `min(2 ** attempt, 60)`s, catches 5 httpx transport exception classes (`ReadTimeout` / `ConnectError` / `PoolTimeout` / `RemoteProtocolError` / `WriteError` and their timeout variants). Re-raises ORIGINAL httpx exception on budget exhaustion so failure-notifier surfaces the real type (PR #465 observability). **Operationally**: from this deploy onwards, a single transient socket flake against any of these three clouds' API endpoints will NOT mark a CMDB capability `partial_failure` — only a sustained outage (≥5 attempts over ~30s) will. If you see fewer Feishu cards about `cmdb.resource_sync` `ReadTimeout` / `ConnectError` after deploy, that is expected and healthy. Real outages still page (just one tick later). Status-code retry (429 / 5xx) semantics on the affected drivers are UNCHANGED — only transport-layer behaviour shifted. Audit: 9 cloud providers reviewed (AWS / GCP / Tencent use SDKs with built-in retry; Alicloud-Intl wraps `aliyun._call` and inherits the OSS fix; Ksyun is a stub) — Azure resource-sync + Aliyun OSS + UCloud were the only three drivers using raw httpx without transport retry.
>
> - **(2026-05-23, Azure cost_facts_sync transport-layer retry)**. `azure_cloud.py` now retries every outbound POST on `httpx.ConnectError` / `ReadTimeout` / `RemoteProtocolError` / `PoolTimeout` / `WriteError` / `ConnectTimeout` / `ReadError` (the umbrella `_TRANSPORT_RETRY_EXCEPTIONS` tuple), in the same loop that already handled 429/5xx. 5-attempt budget, exp backoff `min(2^attempt, 60)` for transport (no Retry-After yet), Retry-After-honouring with a new 90s ceiling for status retry — when server states `Retry-After > 90s`, the helper short-circuits to surface the throttle immediately instead of burning all 5 attempts on guaranteed-throttled retries. Triggered by 2 consecutive nights of `cmdb.cost_facts_sync` partial_failure on Azure subscription `7bc8ab17-...` (PingCAP Azure OpenAI) — 2026-05-21 (empty error pre-PR #465 obs fix) + 2026-05-22 (`ConnectError: All connection attempts failed` post-fix). Same sub's `resource_sync` succeeded both nights → transport-only, not auth/quota. Applied to all 4 Azure POST sites: `_token` (AAD — every cost call's prerequisite, transport flake here cascades 38/38 instead of 1/38), `list_cost_facts`, `list_billing_daily`, `list_billing_by_dimension`. **Operationally**: from this deploy onwards, single-night transient transport failures on the Azure cost cron will self-heal silently. Admin UI's `/admin/cron/scheduled-jobs/cmdb.cost_facts_sync` history will only show failures that exhausted 5 retry attempts (real outages, not flake). Other cloud drivers were audited and confirmed not affected — AWS / GCP / Tencent / Aliyun / UCloud delegate to vendor SDKs (boto3 / google-cloud-bigquery / TencentCloud / Aliyun / UCloud) which have built-in transport retry. Azure was the only driver using raw httpx for cost. Also fixed silent `driver.close()` exception swallow in `cost_service.py` — pool / socket close failures are now `logger.exception`'d.
>
> - **(2026-05-23, Round 46: 4-eyes DNS bypass + IAM owner rebind require browser session, not PAT)**. `POST /cmdb/dns-record-requests/{id}/bypass-execute` and `PATCH /cmdb/iam-users/{id}/owner` are the two most-destructive admin overrides on the cloud surface (the first writes prod DNS without a second approver; the second rebinds IAM principal ownership, which feeds offboarding's leaver-FK invariants). Both previously accepted any cloud_admin / global_admin PAT; both now require an interactive browser session via the same `Depends(require_browser_session)` gate that the scheduled-jobs mutations and the matching `cleanup_iam_user` endpoint use. **Operationally**: scripts / CI / cron callers cannot drive these two endpoints with a PAT any longer. The MCP equivalents (`bypass_4_eyes_dns_request`, no MCP twin for owner-rebind today) bypass the HTTP gate and remain available for AI-assisted recovery — that path still requires `mutates=True` + the right MCP-side admin role gate. For a stolen-PAT incident response: revoke the PAT and you are immediately covered on both endpoints.
>
> - **(2026-05-23, Round 46: cleanup_iam_user cascade also revokes PENDING helix-source grants)**. Round-45 added `GRANT_STATUS_PENDING` to the cascade only on the discovered-grants branch (driver-listed grants). The helix-source branch (Helix-internal grants like the access bundle) was still only matching `GRANT_STATUS_ACTIVE`. Approving a stale helix-source pending grant pointing at a now-deleted IAM principal would 502 with a confusing audit trail. The helix branch now matches `(ACTIVE, PENDING)` — Round-45 closed half of this; Round-46 closes the other half.
>
> - **(2026-05-23, Round 46: PATCH /cmdb/servers/{id} + batch-annotate write per-server owner-history rows)**. `RESOURCE_TYPE_CLOUD_SERVER` was added to `cmdb_network_owner_history` in Round-45 but only the offboarding T+0 force-transfer path was wired into the new history surface. Admin-direct re-binds via PATCH or the batch-annotate API were a bigger surface and were silently leaving only generic `audit_logs` rows (with `body.model_dump()` and no before-owner). Both paths now `SELECT FOR UPDATE`, snapshot the previous owner, and call `record_owner_change(...)` per-server when the owner is touched. **Operationally**: the standard "who has owned cmdb_server X over time" report at `/admin/cloud/network-owner-history` now covers all cloud_server owner mutations — admin PATCH, bulk annotate, AND offboarding T+0.
>
> - **(2026-05-23, Round 46: DNS request cancel admits admin on-behalf)**. `POST /cmdb/dns-record-requests/{id}/cancel` previously refused with 403 for anyone other than the original requester. Combined with the new leaver-flip cascade sweep that includes `CmdbDnsRecordRequest`, automatic handling covers the bulk of stale rows; but a manual admin path was still missing for the case where an admin spots the stale row first or wants to terminate a pending request without going through bypass-execute. The endpoint now admits cloud_admin / global_admin callers and the audit row distinguishes `cancelled_by_admin`.
>
> - **(2026-05-23, Round 46: Azure driver scrubs raw error bodies)**. 21 sites in `integrations/drivers/azure_cloud.py` that interpolated `resp.text[:N]` into `RuntimeError` / `AppException` messages or audit detail now go through the new `_safe_body(resp, max_len=...)` helper which wraps `redact_provider_error`. Same family as the Aliyun fix Round-45 already shipped — Azure AAD token endpoint + ARM/Graph error responses occasionally echo request metadata (client_id, signature fragments on retries) and correlation tokens. **Operationally**: every Azure-driver exception surfaced via failure-notifier or admin audit views is now credential-safe.
>
> - **(2026-05-23, Round 46: `scheduled_job_runs.error_message` is scrubbed at persist time)**. Round-44 added `_scrub_message` at notification dispatch but the DB persistence at `service.py::_execute_job_locked` was raw. Any admin viewing `/admin/cron/scheduled-jobs` could read raw vendor errors including credential fragments. Scrub now happens before the row is written; only the redacted form lives in the DB. **Operationally**: failure messages in the admin UI may show `[REDACTED]` markers where credential-shaped substrings were found — that's expected; the audit chain still names the handler / job / signature for triage.
>
> - **(2026-05-23, Round 46: audit `ip_address` now reads `get_real_ip(request)` everywhere)**. 113 audit / mutation sites across asset / cmdb / apikey / hr / inventory / scheduled_jobs / approval routers were calling `request.client.host` directly — which behind nginx is always the loopback proxy peer, defeating forensic IP traceability. Codemod replaced with `get_real_ip(request)` (which Round-45 already fixed to pick the rightmost `X-Forwarded-For` element, i.e. what nginx appended). **Operationally**: from this deploy onwards, mutation audit rows record the real client IP. Historical rows still show the loopback peer — keep that in mind when correlating across the cut-over date.
>
> - **(2026-05-22, Round 45: shadow-IAM revival is now loud + alarms IT admins)**. `cmdb.sync` (`_sync_iam_users`) previously cleared `revoked_at` on a previously-revoked IAM user that reappeared in the cloud listing (after a 5-minute grace) with only a `logger.warning` — the shadow-IAM threat pattern (admin manually re-creates an offboarded IAM user in the cloud console) was invisible in `audit_logs` and no human got paged. The sync now writes one `cmdb.iam_user.revival_detected` audit row per revived row (resource_type=`cmdb_iam_user`, status=`warning`) AND fans out a Feishu + email push titled `[Security] Shadow-IAM revival on <integration>` to every active global_admin / it_admin. **Operationally**: every such push should be investigated within 24h — pull the cloud-side IAM CreateUser audit (CloudTrail / RAM audit / Aliyun Operation log) to identify who recreated the principal out-of-band; either re-revoke (preferred — that admin's action was unsafe) or document an exception. Failure of the IT-admin push is best-effort (`logger.warning`), but the audit row IS durable — `SELECT * FROM audit_logs WHERE action='cmdb.iam_user.revival_detected' AND status='warning' ORDER BY created_at DESC LIMIT 50` is the canonical query.
>
> - **(2026-05-22, Round 45: DNS 4-eyes self-bypass is refused)**. The bypass-4-eyes admin recovery endpoint `POST /cmdb/dns-record-requests/{id}/bypass-execute` previously accepted bypass-by-the-requester-themselves if the requester also happened to be a cloud_admin / global_admin — defeating exactly the principle 4-eyes exists to enforce. The endpoint now refuses with `DNS_REQ_BYPASS_SELF_FORBIDDEN` 403 when `actor.id == row.requester_person_id`. **Operationally**: even in a single-admin shop, a 4-eyes bypass MUST be performed by a peer (any cloud_admin / global_admin other than the requester). If you have exactly one admin and one DNS request from them gets stuck, the workaround is to add a temporary cloud_admin role grant to a second person (manager / CTO / vendor PM) so they can review and execute the bypass, then revoke the temporary role.
>
> - **(2026-05-22, Round 45: IAM owner re-bind / cleanup leaver-target parity)**. `PATCH /cmdb/iam-users/{id}/owner` and `cleanup_iam_user` (admin disown) both previously rejected target persons with `employment_status='left'` but NOT `'leaver_pending'`. The scope-owner assign path always refused both. The two are now consistent — re-binding / transferring an IAM principal to a `leaver_pending` employee is refused with `IAM_OWNER_TARGET_INACTIVE` 409.
>
> - **(2026-05-22, Round 45: cleanup_iam_user cascade also revokes PENDING grants)**. When an admin disowns an IAM user via `cleanup_iam_user`, the cascade-revoke previously only swept `GRANT_STATUS_ACTIVE` rows — pending grants tied to the now-deleted IAM principal lingered. Approving a stale pending grant later would try to provision against a missing cloud-side user and 502. The cascade now includes `GRANT_STATUS_PENDING` so disowned-user grants are killed in both states.
>
> - **(2026-05-22, Round 45: cloud_server force-transfer writes a per-resource history row)**. The offboarding T+0 force-transfer of a `cmdb_server` previously only wrote an `audit_logs` row — the highest-cost cmdb resource had the weakest per-resource forensic chronology of the 5 cmdb resource types. `cmdb/owner_history.py` now exposes `RESOURCE_TYPE_CLOUD_SERVER` and the advance_service T+0 cloud_server branch writes a `cmdb_network_owner_history` row alongside the audit log (with prior/new owner + label snapshots + offboarding context). **Operationally**: the standard "who has owned cmdb_server X over time" report under `/admin/cloud/network-owner-history` now covers cloud_server too.
>
> - **(2026-05-22, Round 45: software-seat auto-revoke cron is now atomic enqueue-first)**. The monthly `cron_auto_revoke_pending_cancels` previously flipped `alloc.status='revoked'` BEFORE enqueuing the vendor-side revoke task; on enqueue failure the row was committed `revoked` in Helix while the user kept their seat in Zoom/Slack/Atlassian. The order is now enqueue → success-only status flip → audit; failures preserve `status='active'` for next month's retry AND the handler raises a summary `RuntimeError` so the failure-notifier pages ops. **Operationally**: if you see a failure-notifier page for `cron_auto_revoke_pending_cancels` mid-month, the affected `cancel_*` metadata is intact and the seat is still active in both Helix and the vendor — investigate the connector / DB blip, then trigger the cron manually from `/admin/cron/scheduled-jobs` to retry.
>
>
> - **(2026-05-20, API Key revoke teardown — silent skip fixes)**. Two paths in the cloud-side revoke chain were silently skipping cloud teardown on every revoke — discovered pre-launch:
>   - **Azure OpenAI**: `_revoke_cloud_resources` was reading `aoai.get("account_name")` but the autograny path writes `foundry_account_name`. The `if` always evaluated False → DELETE Cognitive Services account was **never called**. AOAI accounts stayed live + KEY1 still worked + per-resource billing meter kept running after revoke. Fixed at [service.py:3165](../../server/app/domains/apikey/service.py) — reads `foundry_account_name` with one-tick fallback to `account_name`. **Operationally**: any AOAI assignment revoked between PR #408 ship date (2026-05-19) and today's deploy must be manually verified — the Cognitive Services account is likely still live. Verify with `az cognitiveservices account list --resource-group <rg-name>` and DELETE manually if Helix shows the assignment `status='revoked'` but the account is present.
>   - **Bailian RAM**: `_dispatch_cloud_revoke` was called with `expected_provider="aliyun"` but Bailian autogrant only ever uses `alicloud_intl` provider slug. The strict `integration.provider != expected_provider` mismatch raised → caught as `CLOUD_REVOKE_FAILED` 502. The sk- DashScope key DELETE (line ~3104) succeeded (inference auth killed), but the companion RAM sub-account was **never deleted** → orphan RAM users accumulate forever in the Aliyun-intl tenant. Fixed at [service.py:3148](../../server/app/domains/apikey/service.py) — `expected_provider=("alicloud_intl", "aliyun")` tuple. **Operationally**: post-deploy, audit Aliyun RAM users with `aliyun ram ListUsers --region ap-southeast-1` — orphan users named `helix-bailian-<id>` whose corresponding Helix assignment is already `revoked` should be hand-deleted via `aliyun ram DeleteUser`. From now on, every revoke cleanly tears down both halves.
>
> - **(2026-05-20, API Key transfer + extend leaver gates)**. Two cloud-adjacent paths now refuse to act when the assignee/recipient is `leaver_pending` or `left`:
>   - `apply_transfer` (called when a recipient accepts a transfer) — refuses with `RECIPIENT_NOT_ACTIVE` 409 if the recipient is offboarding. Pre-fix, a stale Feishu/email card sitting in a leaver's inbox could be accepted post-departure and the AK would flip to the leaver's `assigned_to`, the CmdbCostAllocationRule's `assign_person_id` would re-point to them, and AWS IAM `Owner` tag would push their email — all of which the runtime leaver-gate exists to prevent for new grants.
>   - `extend_validity` (admin extending an expired key) — refuses with `ASSIGNEE_NOT_ACTIVE` 409 if the original assignee is now offboarding. Admin must transfer the key to a current employee or revoke instead.
>
> - **(2026-05-20, `apikey.budget_check` cron commit fix)**. The daily budget-alert cron was missing `await session.commit()` before its `async with async_session_factory()` block exited → every `ApiKeyBudgetAlertsDedup` SAVEPOINT row was silently rolled back on session close → next morning the same (assignment, billing_month, threshold_pct) re-evaluated and re-fired the 80% / 100% alert. **Operationally**: budget-alert recipients (assignee + manager + IT admin) should expect to see per-(key, threshold) duplicates from before today's deploy — those are not new threshold breaches, just the cron re-firing the same window. Audit log under `apikey.budget_alert_sent` has the true history (audit writes via a separate independent session and weren't rolled back).
>
> ----
>
> - **(2026-05-19, API-Key lifecycle PR3) Offboarding admin "auto-revoke API Key" really tears down cloud now + admin IAM tag retry endpoint**. Pre-PR3, clicking "auto-revoke" on an api_key offboarding blocker queued a ProvisioningTask with software_id=None, which the provisioning dispatcher rejected with ``ValueError`` — task landed FAILED, blocker stayed pending, AWS IAM user / Aliyun RAM user / Azure OpenAI account kept working. PR3 wires the api_key short-circuit so the task succeeds at the connector layer and the post-success hook dispatches into ``ApiKeyService.revoke_key(actor_role=SYSTEM_OFFBOARDING)`` for the real teardown (`_revoke_cloud_resources` deletes the cloud credential). **Operationally**: post-deploy, any historical offboarding processes with stuck "auto-revoke api_key" provisioning tasks can be retried via the admin console — they should now resolve cleanly. New admin retry endpoint `POST /apikeys/assignments/{id}/retry-iam-retag` rescues transfers whose best-effort cloud-tag push (PR2) failed; the red "IAM 标签失败" chip in `ApiKeyManagementPage` is the surface, clicking it re-issues `iam.tag_user` for the current owner.
>
> ----
>
> - **(2026-05-19, API-Key lifecycle PR2) AWS IAM user tag rewrite on API-Key ownership transfer**. When an employee transfers an API Key to a colleague (or when offboarding cascades a key to the leaver's manager), Helix now pushes the new owner's email into the AWS IAM user's ``Owner`` + ``helix-owner-email`` tags via the new [`AwsCloudDriver.tag_iam_user`](../../server/app/integrations/drivers/aws_cloud.py) method. This is required for AWS Cost Explorer to attribute future Bedrock spend to the new owner — without it, AWS keeps grouping by the old email tag and chargeback runs on stale data.
>   - **Cloud side**: a single `iam.tag_user` upsert (no IAM permission change required — `iam:TagUser` is already covered by the `IAMFullAccess` policy that Helix's master integration uses). Idempotent on the same payload.
>   - **Failure mode**: best-effort. A tag-update failure does NOT roll back the transfer — the local Helix row + cost-allocation rule are already authoritative for Helix's own cost attribution. Failures land in the `apikey.transfer.complete` audit row's `cloud_tag_summary` field with `status='failed'` + the boto3 error. To retry manually post-failure: `aws iam tag-user --user-name helix-bedrock-<id> --tags Key=Owner,Value=<new_email> Key=helix-owner-email,Value=<new_email>` (run with the master integration's AK/SK, ap-southeast-1 region irrelevant since IAM is global).
>   - **Aliyun + Azure**: no equivalent operation needed. Bailian's auto-grant doesn't emit a cost-allocation rule (per-key BSS cost isn't available); Azure OpenAI's rule keys on the per-employee resource-group name, which doesn't change on ownership transfer.
>
> ----
>
> - **(2026-05-19) GCP scope sync — billing-account gate**. PR #389 caught ~3 named families of GCP personal sandbox projects via displayName / projectId patterns; a follow-up 2026-05-19 audit found ~111 org-root personal projects with no regular naming pattern that the filter missed. The durable signal added: **billing-account membership**. When the integration has `config_public.create_scope_billing_account` set, org-root projects (parent = `organizations/*`) must be linked to that billing account; folder-hosted projects (`parent = folders/*`) pass without billing check.
>   - **New IAM dependency**: the integration's service account must have `roles/billing.viewer` (or any role with `billing.resourceAssociations.list`) **on the configured billing account**. For PingCAP prod: SA `pch-resource-manager@gcp-digital-infra.iam.gserviceaccount.com` on `billingAccounts/01D088-8F9CF2-8AF1C6` (granted 2026-05-19). Without this grant, scope sync will fail loud with `ProviderAuthError` — there is no silent fallback (the whole point of the gate is to prevent the silent re-flooding that PR #389 narrowly missed).
>   - **Verification post-grant**: `gcloud billing accounts get-iam-policy 01D088-8F9CF2-8AF1C6 --filter="bindings.members:pch-resource-manager"` should show the binding. From the Helix side, trigger a manual scope sync (`POST /cmdb/sync?integration_id=c1a4f301-adc2-46de-8525-fb8212268c46&capability=scopes`) and inspect the resulting `cmdb_sync_jobs.capability_results.scopes` — successful run = gate is reading the billing account correctly.
>   - **Prod cleanup that landed alongside this PR** (2026-05-19 03:50 UTC, SQL on TiDB): 134 active org-root GCP scopes soft-deleted; cron `cmdb.scopes.gcp-primary` (`scheduled_jobs.id='f408dce8-2b04-45e0-858c-f471fbd7c52c'`) temporarily disabled to prevent the legacy code from re-upserting the sandboxes before this PR ships. After Production Deploy succeeds, re-enable the cron: `UPDATE scheduled_jobs SET enabled=1, updated_at=NOW() WHERE id='f408dce8-2b04-45e0-858c-f471fbd7c52c';`. The next scope sync tick uses the new filter — folders/* 67 + billing-linked org-root ~90 expected to re-surface as active; un-billed remain soft-deleted.
>   - **Operator impact on other clouds**: zero. AWS / Azure / Aliyun / Tencent / UCloud scope sync unchanged. Only `provider='gcp'` integrations pick up the new filter, and only when `create_scope_billing_account` is configured on the integration (orgs that haven't centralized billing yet retain legacy behavior).
>   - **(2026-05-19 follow-up) OAuth-scope fix for Cloud Billing API**. Hours after PR #397 shipped, every `cmdb.resource_sync` / `cmdb.scopes_sync` tick failed with `auth error: GCP cloudbilling auth failed ... Request had insufficient authentication scopes` — **not** an IAM gap (SA already had `roles/billing.viewer`). Root cause: [providers/gcp.py](../../server/app/domains/cmdb/providers/gcp.py)'s `_READ_SCOPES` was narrowed to `cloud-platform.read-only` in PR #334 (pre-launch audit) and that scope does NOT cover `cloudbilling.googleapis.com` — Cloud Billing API's accepted scopes are `cloud-platform` / `cloud-billing` / `cloud-billing.readonly` only. Fix added `https://www.googleapis.com/auth/cloud-billing.readonly` to `_READ_SCOPES`; matches what [`gcp_cloud.py::_BASE_SCOPES`](../../server/app/integrations/drivers/gcp_cloud.py) had all along. Same bug class as the existing `bigquery` (vs `bigquery.readonly`) scope comment in the same file: new API surface = must extend OAuth-scope list explicitly, no compile-time check.
>
> ----
>
> - **(2026-05-19) `POST /cmdb/sync/{id}/force-sweep` 改异步 + nginx /api/ proxy_read_timeout 300s 兜底**. GCP integration `c1a4f301-adc2-46de-8525-fb8212268c46` 触发 `cmdb.resource_sync` 5%-drop 阈值后,admin 点「强制以最新同步为准」收到通用"后端返回错误，但没有详细消息"文案。Root cause:endpoint 同步执行 `run_sync` 跨过 nginx 默认 60s `proxy_read_timeout` → 504 → 响应体是 nginx HTML(非标准信封)→ 前端 `apiMsg` undefined → fallback 通用文案。
>   - **Endpoint** ([cmdb/router.py:529](../../server/app/domains/cmdb/router.py) + 新增 [cmdb/sync.py::run_sync_with_lock](../../server/app/domains/cmdb/sync.py)):200 → 202;响应即返回 `status='queued'` sync_job 行;真活通过 FastAPI `BackgroundTask` 在 `run_sync_with_lock` wrapper 内拿 `cloud_sync:<id>` 锁后跑 `run_sync`。锁竞争(原 409 `CLOUD_SYNC_ALREADY_RUNNING`)现在表现为 job `status='failed'` + 显式 retry 文案,前端轮询读到展示。
>   - **前端** ([ScheduledJobRunsPage.tsx](../../web/src/pages/admin/ScheduledJobRunsPage.tsx) + [cloud.ts:getSyncJob](../../web/src/api/cloud.ts)):`handleForceResourceSweep` 收到 202 后 2s 节奏轮询 `GET /cmdb/sync-jobs/{id}`,5min deadline;terminal `failed` 渲染 `job.error_message`(覆盖 LockBusy);deadline 命中走 "BG task 可能仍在跑,稍后查 sync-jobs 页" warning。
>   - **Nginx defense-in-depth** ([web/nginx.conf:29](../../web/nginx.conf)):`/api/` 加 `proxy_read_timeout 300s; proxy_send_timeout 300s;` 兜底任何 60s+ 同步端点(billing 大表 export、CSV 导出、grants force-drift-sweep)。
>   - **Operator impact**:admin 「强制以最新同步为准」流程不变;UI 上需要 2s+ 的进度条等待(以前是 nginx 60s 后才报错,现在 5min 内任何时刻完成都能正确反馈);LockBusy 现在能看到清晰文案而不是 409;5min 超时后告诉你"任务还在跑",到 `/admin/scheduled-jobs` 查 sync_job 最终状态即可。
>   - **`force-drift-sweep` 保持同步** ([lifecycle_router.py:1233](../../server/app/domains/cmdb/lifecycle_router.py)):grant fan-out 通常 < 60s,nginx 300s 兜底足够;如果未来观察到 drift-sweep 同样超时,再走同套异步重构。
>   - **`force_cloud_resource_sweep` 仍刻意不暴露 MCP** ([mcp/tools.py:6376-6385](../../server/app/mcp/tools.py)) — 与 `force_cloud_grants_drift_sweep` 同一 defense-in-depth 决策。
>
> ----
>
> - **(2026-05-18) Round-29 — platform-wide pre-launch audit (cloud-lifecycle touch points)**.
>   - **Monthly billing CNY rebase** ([scheduled_jobs/handlers.py::run_billing_sync](../../server/app/scheduled_jobs/handlers.py)). For CNY-native cloud accounts (国内阿里云, 部分 AliCloud Intl tenants 计费人民币), `cmdb.billing_sync` cron now recomputes `total_cost_usd` via `app.core.fx_rate.resolve_cny_per_usd` after the provider returns its native amount. Pre-fix the Aliyun providers used a hardcoded `_FX_TO_USD["CNY"]=0.14` (the ECB rate moves 3-5% per quarter; the daily `cmdb.billing_daily_sync` already used the dynamic resolver). **Operator impact**: monthly rollup USD numbers now match daily cost_facts dashboards. Per-integration override `config_public.cny_per_usd` still wins; otherwise reads the weekly-refreshed `ai_spending.cny_to_usd_rate` platform setting. If both are missing, falls back to the provider's stale-but-bounded value and logs `warning` (the `ai_spending.cny_rate_refresh` failure-notifier should already have paged).
>   - **Scheduled-job per-handler timeout** ([scheduled_jobs/service.py::_execute_job_locked](../../server/app/scheduled_jobs/service.py)). Every cron handler now wrapped in `asyncio.wait_for(timeout=timeout_secs)`. Default 1800s (30min); per-job override via `scheduled_jobs.config['timeout_secs']`, clamped `[60, 7200]`s. Wedged cloud-sync handlers (slow vendor API with no client timeout, stuck DB deadlock) are now force-cancelled and the run row lands `status='failed'` — the failure-notifier pages instead of the wedge being invisible until the worker restarts. Cloud-sync handlers commonly run 60-300s; the 30-min default is generous headroom.
>   - **Audit `user_agent` capped to 200 chars** — administrators reading `audit_logs` for cloud-lifecycle actions (sync, force-sweep, scope grants/revokes) will see only the first 200 chars of the calling User-Agent. The truncation removes device-fingerprint suffixes (hardware model / build SHA) but keeps the browser/OS family needed for "was this Chrome or curl" forensic attribution.
>   - **Scope unchanged**: no provider-driver behaviour change for AWS / GCP / Azure / UCloud / KSYun. Resource-sync sweeping, IAM lifecycle, scope sync, role catalog sync, grants sync, cost facts ingestion behaviour preserved exactly.
>
> ----
>
> - **(2026-05-18) Round-28 — network module full-lifecycle pre-launch audit (cloud-lifecycle touch points)**. Network module (域名 / DNS / CDN / LB) ships to production. 25+ hardening fixes; cloud-lifecycle-relevant deltas:
>   - **Sync sweep race fix** ([_safe_sweep](../../server/app/domains/cmdb/sync.py)). Pre-fix: sync starts at T0, fetches cloud rows at T0+5s; `_execute_create` from an approved Feishu DNS self-service request inserts a new row at T0+15s with `last_seen_sync_job_id=NULL`. Sweep at T0+30s matched the `IS NULL OR != job_id` filter and **soft-deleted the just-created record**. Round-28 adds `model.first_seen_at < sync_job.started_at` to sweep WHERE. Applies to every network-resource model with `first_seen_at`. Mid-sync HTTP creates no longer disappear.
>   - **DNS self-service input hardening** ([dns_request_service.py](../../server/app/domains/cmdb/dns_request_service.py)). `POST /dns-record-requests` now rejects: (a) any role other than `employee` + `global_admin`; (b) wildcard `*` in `name`; (c) reserved underscore labels in any position (`_dmarc` / `_acme-challenge` / `_spf` / `_dkim` / `_domainkey` / `_smtp._tls` / `_mta-sts` / `_tls` / `_tlsa`); (d) TXT values matching email-auth or domain-ownership prefixes. Admin direct-write (`can_manage_cloud`) unchanged. **Operator impact**: ACME-DNS-01 cert issuance / DMARC reporting / SPF changes now require the IT-admin direct-write path; file a ticket. Self-service still works for the A / AAAA / CNAME / non-policy TXT long tail.
>   - **Owner_history snapshots person labels** ([owner_history.py](../../server/app/domains/cmdb/owner_history.py)). Every history row from Round-28 onward carries `context.previous_label` / `new_label` / `actor_label` denormalized JSON. Leaver IdentityPerson hard-delete (per `reference_prod_person_cleanup`) flips FK to NULL via `ondelete=SET NULL` but the label string survives — forensic "who owned dns.example.com in 2026 Q1" still answers. Pre-Round-28 rows rely on the FK; deletion loses the trail for those.
>   - **Network alert cron at kill-switch flip** ([handlers.py](../../server/app/scheduled_jobs/handlers.py)). When operator flips `network.user_notifications_enabled` false → true (migration 174 default false), `cmdb.domain_expiry_alert` first tick SUPPRESSES backlog: domains with `prev=None` get state stamped, no DM. Owners get DMs on the NEXT bucket transition. **Exception**: freshly-discovered (synced within 24h) in short bucket (`<=1d` / `<=7d` / `expired`) fires immediately — no "next transition" before expiry. Counters: `suppressed_first_observation` + `first_observation_emitted_short_bucket`.
>   - **Feishu unconfigured = raise** — `cmdb.dns_owner_recertify` + `cmdb.dns_pre_leaver_handover` raise `RuntimeError` instead of returning `status="skipped"` when Feishu creds are missing. Operator impact: a misconfigured Feishu that previously left these green will now page via failure-notifier. Use kill switch for intentional silence, not Feishu deletion.
>   - **Pre-leaver handover covers LB + CDN** — DM body includes `Load balancers (N)` + `CDN domains (N)` sections with "...and N more" overflow. Leavers with only LB or CDN now get T-7/T-3/T-1 reminders. Filter expanded to `employment_status IN ('active', 'leaver_pending')`. Recertify cron correspondingly excludes leavers within the 7-day handover window.
>   - **TXT secret redaction in audit_logs** — DNS audit rows hash-prefix TXT values (`[TXT redacted; sha256:<16char> length=<n>]`) in `audit_logs.detail.value` / `before.value` / `after.value` / `patch.value`. Forensic compare-by-equality still works (same input → same hash); original plaintext stays in `cmdb_dns_records.value`. Non-TXT plaintext preserved (operationally valuable).
>   - **DNS recertify cron index** — [migration 178](../../server/alembic/versions/178_cmdb_network_audit_hardening.py) adds composite `ix_dns_recertify_due (deleted_at, last_recertify_nag_at, last_owner_review_at)` covering the "due records" query that was full-table-scanning on every tick.
>   - **DNS create / delete atomic with owner_history** — helpers no longer commit internally; caller commits AFTER writing owner_history seed/terminal. Pre-fix split-tx left orphan history.
>
> ----
>
> - **(2026-05-22) Empty-`str(exc)` 观测性 bug-class 全平台 sweep + canonicalise on `format_exception_for_log`** — 2026-05-21 Azure `cmdb.cost_facts_sync` partial_failure notifier 显示 `cmdb.cost_facts_sync 6310dc15-... partial_failure: 7bc8ab17-...: `(scope_id 后裸冒号,零异常类型信息)是入口,但根因更广:任何 `f"...: {exc}"` 落到 `str(exc)` 上,对 httpx 传输层异常(ReadTimeout / RemoteProtocolError / 某些 ConnectError)以及所有 AppException 派生异常都返回空字符串(AppException 故意不调 `super().__init__(message)`,所以 `str()` 是空、`.code`+`.message` 是真载体)。Pre-fix 整个 executor `scheduled_jobs/service.py:806` 也用 `str(exc)[:4096]` 写 `scheduled_job_runs.error_message` —— 任何 handler 顶层 raise 的 httpx 传输错或 AppException 都会让 `error_message` 落空、Feishu 卡片正文落空、dedup signature 把所有空消息失败合并到同一个 bucket(第二次起被静默)。Fix:`str(exc)` / `f"...: {exc}"` 全替换为 `format_exception_for_log(exc)`(已在 `provisioning` / `offboarding` / `hr_lifecycle` / `leaver_blocked` 生产了)—— 共 ~30 站点,横跨 `scheduled_jobs/{service,handlers,ai_spending_handlers,bailian_handlers,jira_handlers,slack_handlers,zoom_handlers}.py` + `worker.py` + `domains/cmdb/{cost_service,lifecycle_service}.py`(scope-iter loop + driver-load + FX + BQ-config + reconciliation + notify + revoke + disable + grant + billing-daily-precheck + billing_by_dimension)。**`cmdb.cost_facts_sync` raise message 同时加 `scopes ok=X failed=Y total=Z` 后缀**(handler raise → result_summary 落 null,计数挂在 message 上是失败时唯一可观测的 blast radius);早返回路径(driver load failed / FX unavailable / BQ misconfigured)`per_scope=[]` 时后缀自动省略。**What you'll see**:下次任何 cloud cost sync / billing-by-dimension / scope-grants-sync / API-key 撤销 cron / 任何 scheduled job 单 scope / 单整合报错,failure-notifier 飞书卡会显式带异常类 + 消息或 repr 兜底(`HttpReadTimeout: <repr>`、`AzureCostQueryError: status=429 body=...`、`AppException: PROVIDER_BAD_CREDS: ...`),不再是裸冒号 + 空字符串。Dedup signature 现在按异常类区分,operator 会收到「每种失败一张卡」而不是「所有空消息失败合并一张卡」。**Operator action for the 2026-05-21 Azure incident**: 已通过 prod 手动 trigger 修复 — 在 worker 容器 `compose-worker-1` detached 执行 `execute_job('8b839f6e-...')`,run id `73d5a85e-be55-4bfe-bc96-eb78ebd2d56b` 6 min 37 s 完成,38/38 scope ok,119,011 facts upserted,失败的 7bc8ab17 PingCAP Azure OpenAI scope 恢复正常(37 facts)。No migration / cron / RBAC / MCP-tool 改动。
>
> - **(2026-05-17) Round-23 follow-up — sync error-path `partial_failures` drain**. PR #340 (Round-23) fixed the GCP `aggregatedList` zone-warning driver bug but its visibility goal didn't fully land in production: the driver populated `provider.partial_failures` with each bad zone, but `_safe_sweep`'s >5%-drop guard raises `ProviderTransientError` BEFORE the success-path drain in `run_sync` runs — so legs sat in memory and never reached `cmdb_sync_jobs.capability_results.servers.partial_failures`. Admin sync-history UI kept showing "refusing to sweep" with no leg pointing at the bad zone. Fix: `_run_sync_impl`'s `except Exception` branch now drains `provider.consume_partial_failures()` symmetrically with the success path (same `redact_leg_entry` wrap; second-level try/except so leg-collection failure doesn't mask the original exception). Pre-existing gap for every fan-out driver but PR #340's GCP fix made it operationally visible. **What you'll see**: starting on the next `cmdb.sync.gcp-primary.servers :55` tick after deploy, the sync_jobs row's `capability_results.servers.partial_failures` will list each problem zone as `{leg: "project:X:zone:Y", reason: "aggregatedList.zone_warning_<CODE>"}` — finally letting ops correlate the 60-70-instance drop to specific GCP zones.
>
> - **(2026-05-17) Round-23 — `cmdb.resource_sync` 双 driver bug 修复**. Two production cron failures on `alicloud-intl-primary` and `gcp-primary` traced to driver bugs (NOT cloud-side IAM gaps as initially suspected):
>   - **AliCloud OSS `list_object_storage` missing STS SecurityToken** — `auth_error × 18/18 sub_accounts` every tick. The OSS list-buckets path was the only place in the AliCloud driver that bypassed `self._security_token`; OSS uses REST-style header auth (`x-oss-security-token`) and the token must also appear in the V1 signing string's `CanonicalizedOSSHeaders` block. Fix lives in [`providers/aliyun.py::list_object_storage`](../../server/app/domains/cmdb/providers/aliyun.py) — master single-account behaviour identical to before; only the assumed-role sub-account path changed.
>   - **GCP `compute.instances.aggregatedList` ignored zone warnings** — `gcp-primary` `servers` cron blocked by the >5%-drop sweep guard for 8+ hours (provider returned ~870 vs baseline ~938) with `partial_failures = NULL`, no leg pointing at the bad zone. With `returnPartialSuccess=True`, GCP replaces failed-zone `instances` with a `warning` block; pre-fix the driver only iterated `instances` and dropped the rest silently. Fix in [`providers/gcp.py::_collect_zone_warnings`](../../server/app/domains/cmdb/providers/gcp.py) emits `partial_failure` legs with `reason=aggregatedList.zone_warning_<CODE>` and `leg=project:<id>:zone:<name>` (matches sibling GCP DNS leg shape so failure-notifier dedup keys stay consistent across capabilities). Per-call dedup set collapses cross-pagination duplicates. `NO_RESULTS_ON_PAGE` / `NO_RESULTS_FOR_LIST` remain silent (every empty zone hits these). **What you'll see**: GCP servers cron starts succeeding again once the underlying zone recovers; persistently bad zones surface in `cmdb_sync_jobs.capability_results.servers.partial_failures` and route through the failure-notifier dedup keyed by warning code (one alert per problem zone, not hourly).
>   - **Operator impact**: no DB migration, no config change required. After deploy, the alicloud-intl `:58` OSS cron and GCP `:55` servers cron should self-recover on the next tick. Watch `scheduled_jobs.last_status` for `cmdb.sync.alicloud-intl-primary.object_storage` and `cmdb.sync.gcp-primary.servers` — both should flip from failed → ok within 1 hour of deploy.
>
> - **(2026-05-14) Cloud-revoke driver fix — P0092 round 3**: huohao@pingcap.com offboarding cycled 7 cloud-grant blockers 55–58× each. Two distinct driver bugs took 3 PRs to land correctly:
>   - **UCloud Attach/Detach Scope value** (6 blockers, PR #295 → #296 → #297). Legacy code passed `Scope="Account"` which is not a legal UCloud value; current API expects `"All"` (tenant-wide, Helix's mode) or `"Specified"` (project-scoped, requires `ProjectID`). Round 1 (#295) added `Scope=Account` (wrong value) → RetCode=230. Round 2 (#296) removed it → RetCode=220 again. Round 3 (#297) confirmed `Scope="All"` against UCloud docs and applied to Attach, Detach, and the revoke_account chain. The same PR set also fixed an `external_ref` URN parse bug (URN itself contains `::`, parser was truncating it). See §8 step 4 callout.
>   - **AliCloud Intl CloudSSO `DetachPolicyFromUser` — `InvalidParameter.PolicyName.InvalidChars`** (1 blocker). CloudSSO refs (`sso::…`) reached the inherited RAM-based revoke. New override in `AlicloudIntlCloudDriver.revoke_access` routes `sso::` direct-user refs to `DeleteAccessAssignment` and `sso::…::group::*` synthetic-member refs to `RemoveUserFromGroup`. See §8 step 4 callout.
>   - **What you'll see**: the offboarding admin "立即推进" button will now succeed on these and similar blockers. The 55–58× error counters in `offboarding_blockers.last_error_message` reset to 0 on the next successful revoke. No DB migration shipped — pure driver fix.
> - **(2026-05-13) Driver fail-closed fixes**: Aliyun `ListPoliciesForUser` transient errors now raise instead of silently continuing with a partial policy list (could have triggered drift-sweep revocation on partial data). AliCloud Intl SSO pagination errors raise instead of `break`ing mid-page (partial assignment/member data was passing as authoritative). GCP `_list_cost_facts_sync` row guard added: rows missing `day` or `project_id` are skipped instead of crashing mid-batch.
> - **(2026-05-26) Scope detail page redesign (frontend-only)**: both [`CloudScopeDetailPage`](../../web/src/pages/cloud/CloudScopeDetailPage.tsx) and [`MyCloudScopeDetailPage`](../../web/src/pages/cloud/MyCloudScopeDetailPage.tsx) now render a Hero card + 6-tile KPI ribbon + 9 tabs (Summary / Billing / Members / Pending / Access requests / History / Governance / Alerts / Config). The legacy admin `SetAlertsModal` popup is replaced by the inline "Alerts" tab (same backend endpoint, same authz, unified UI). `ScopeBillingPanel` is lazy-mounted inside the Billing tab — `/billing-detail` is no longer fired on default page load (saved a round-trip; tab click triggers it). All previously-available actions remain (Console / Sync now / Snooze 30 days / Transfer Owner / Decommission for admin; Console / Snooze for owner). KPI tiles + Summary tab derive from already-loaded scope + grants + daily — no new endpoints. Indigo-600 + Slate palette aligned with [`App.tsx`](../../web/src/App.tsx) `customTheme`. New shared components under [`web/src/components/cloud/scope-detail/`](../../web/src/components/cloud/scope-detail/).
> - **(2026-05-13) Skipped-unresolved UI**: cloud-side principals whose email doesn't match any Helix identity are counted per-scope and stored in the sync audit row. The scope detail page's members tab now shows an Alert banner when principals were skipped (count from `GET /cmdb/scopes/{id}/last-sync-summary`). This is expected for GCP users added directly in the console without a matching Helix account — the fix is to ensure the principal's email matches a Helix identity.
> - **(2026-05-13) Inactive owner notification guard**: `sweep_expiring_grants` and `sweep_inactive_grants` no longer send notifications to scope owners whose `employment_status` is `left` or `leaver_pending`. Pre-fix, a leaver who was a scope owner would continue to receive prod-overdue and inactivity alerts indefinitely.
> - **(2026-05-13) Multi-role approval partial failure audit**: when a multi-role access request has some roles fail and others succeed, a group-level `status=partial_failure` audit entry is now written with `request_group_id`, total count, failed count, and per-role errors — in addition to the existing per-grant audit entries.
>
> - **GCP grants_sync mass-revoke safety**: `_list_scope_grants_sync` no longer returns `[]` on 403. If the GCP service account loses `resourcemanager.projects.getIamPolicy` (most common cause: an admin yanked the SA's role binding by mistake; second most common: a freshly-created project hasn't propagated the SA's binding yet), the next `cmdb.grants_sync` run will REPORT THE 403 against that scope and skip it (the scope drops out of `synced_scope_ids` so the drift sweep won't fire on it). What you'll see: a run summary with `errors: ["projects/<id>: HttpError 403 ..."]` and ZERO grants revoked on that project. Triage: re-grant the SA at minimum `roles/resourcemanager.projectViewer` (read) + `roles/resourcemanager.projectIamAdmin` (read+write IAM bindings) on the affected project (or at the org for blanket fix). Same posture as AWS / Aliyun / Tencent on auth-class errors.
> - **AWS / Tencent driver fail-closed on missing creds**: a partial `integrations` row (decrypt skipped, key rotation race, hand-typed DB edit clearing `secrets_encrypted`) used to silently fall through to ambient creds — boto3's chain → env vars → EC2 instance role on the prod host. The driver would then operate against the WRONG AWS account (instance role's account, not the integration's), silently corrupting `cmdb_cloud_billing_daily` rows or running IAM mutations against the wrong tenant. Now the driver hard-fails at construction with a clear "missing access_key_id / secret_access_key" message. Operator fix: re-save the integration via `/admin/integrations/<id>` (web console) with valid credentials. Aliyun + Azure already had this guard.
> - **Feishu billing-alert delivery check**: a transient Feishu webhook failure (`{"code": 9499, ...}`) was being treated as success — Helix wrote a 24h dedup row and silenced the alert. Now ANY non-zero indicator (HTTP status, top-level `code`, top-level `StatusCode`) is treated as failure; the dedup row only writes after a confirmed-200 send. **What you'll see**: `cmdb.billing_alerts` cron summaries on a misconfigured Feishu webhook will report status `exceeded-post-failed` instead of `exceeded-alerted`, and the alert will retry on the next eligible budget evaluation.
> - **Scope-request `failure_reason` cleared on re-approval**: previously a scope-creation request that failed once and was re-approved kept the prior failure message in the row, confusing the audit reviewer ("approved but says it failed?"). Now the field is cleared on each fresh approve.
> - **`revoke_grant` divergence flag**: when offboarding sweep swallows a real cloud failure (5xx / IAM throttle, NOT `already_absent`) and locally marks the grant `revoked`, `cmdb_cloud_access_grants.detail.requires_manual_cloud_revoke = true` flags the row for a future reconciliation pass to retry the revoke. To find affected rows: `SELECT id, scope_id, person_id, detail FROM cmdb_cloud_access_grants WHERE JSON_EXTRACT(detail, '$.requires_manual_cloud_revoke') = TRUE;` (TiDB MySQL JSON path).
> - **lifecycle_router UUID parse safety**: `/billing/daily?scope_id=foo`, `/billing/by-dimension?integration_id=foo`, `/role-catalog/promote-by-key`, `/role-catalog/reset-promotion`, `/billing/dimensions`, and `/admin-overview/...?integration_id=foo` no longer 500 with stack trace on a malformed UUID; they return 404 instead. Operator-visible: less log noise from frontend bugs / fat-fingered cURL probes.
>
> **Earlier (2026-05-09, Tencent SDK throttle — `cmdb.grants_sync` `RequestLimitExceeded` fix)**. **Operationally relevant**: every Tencent SDK call (CAM, billing, CVM, CLB, CDN, DNSPod, 域名注册) now goes through the per-`(secret_id, service)` token bucket in [`server/app/integrations/drivers/_tencent_throttle.py`](../../server/app/integrations/drivers/_tencent_throttle.py) (4.5 RPS default — under Tencent's documented 5 RPS CAM ceiling) plus a bounded retry-on-`RequestLimitExceeded` loop (5-attempt cap with exponential backoff + jitter, then re-raise). Effect on `cmdb.grants_sync` runs: a Tencent integration with >5 active CAM users used to abort mid-snapshot with `[TencentCloudSDKException] code:RequestLimitExceeded message:您当前每秒请求 6 次，超过了每秒频率上限 5` (and the partial snapshot would be a mass-revoke hazard via the mark-and-sweep at the end of the job). Now the per-user `ListAttachedUserAllPolicies` loop self-paces; concurrent admin "Sync now" clicks during the cron run cost a few backoff retries instead of failing the job. Operator impact: a Tencent grants_sync that previously took ~5s on a 30-user tenant now takes ~7s (30 users × 0.22s = 6.6s — the price of staying under the cap); throughput is bounded by the throttle, not by network/SDK. If you ever need to override the rate (Tencent docs publish a new ceiling, or you have a special-case quota uplift), edit `SERVICE_RATES` in `_tencent_throttle.py`. **Earlier (2026-05-08, 11th pre-launch audit — JML hardening, see ROADMAP.md)**. **Operationally relevant cloud-driver changes**: Aliyun (`list_iam_users`), Tencent (`_list_scope_grants_sync`, both outer ListUsers and per-user ListAttachedUserAllPolicies), and AWS SSO (`list_scope_grants_sso_sync`, both outer and per-permission-set) NO LONGER swallow exceptions on transient errors. Effect on grants_sync runs: a flaky 5xx / throttle / regional outage now SURFACES as an error on that scope (caller at `lifecycle_service.py:3760-3777` already adds the scope to `errors` and excludes it from the stale-discovered-grant sweep), so a transient failure becomes a no-op for that scope on that tick — instead of producing an empty-grant snapshot that the drift detector treats as "all grants gone, revoke everything Helix-tracked." Only `ResourceNotFoundException` (AWS, "permission set never provisioned to this account") still degrades to `[]`. Operator impact: expect grants_sync run summaries to occasionally show `errors: ["account_id: <SDK exception>"]` per scope when there's a transient cloud-side blip; the synced_scope_ids set excludes those, so no revocation will fire for that scope until the next clean tick.
>
> **Earlier (2026-05-08, 10th pre-launch audit)**. Operations-relevant changes: (1) `_dispatch_grant` durability — every grant issuance path (`approve_grant` / `owner_add_member` / batch group-approve) now commits the PENDING row before the cloud-side `driver.grant_access` call AND commits the success/failure outcome before any post-grant side effect (member notification). Operationally this means: a route handler crash mid-dispatch no longer rolls back the audit trail or leaves a "ghost grant" where the cloud-side assignment is live but Helix shows nothing. The PENDING-stays-PENDING-on-driver-failure semantics are unchanged — admin retry still works as before. (2) MCP tool `force_cloud_grants_drift_sweep` was REMOVED from the PAT-reachable surface (matched the launch-blocker pattern denylist). The HTTP route `POST /cmdb/integrations/{id}/grants-sync/force-drift-sweep` remains for admin web access — IT operators MUST use the web console / curl, not an LLM/MCP client. The audit row attribution (`cloud.grant.drift_sweep_force_override`) is unchanged. Earlier this date: extended §11b with `scope_id` requirement on the scan endpoint (whole-integration scan can take 90s+ on a real prod-shaped AWS Org and was the first-cut bug; per-scope scan is the supported interactive flow), per-target commit isolation in the cleanup service (so a mid-batch DB error doesn't roll back already-deleted-on-cloud history rows), idempotency belt-and-braces via a second unique key on `(scope_id, role_key, external_principal_id)`, PII redaction on `external_principal_*` fields for non-privileged readers, and the paired `LeaverBlockedGuard.flip_to_left → strip_person_from_alert_cc` hook so future leavers don't accumulate as `未知用户` in alert configs. Earlier (2026-05-07): added §11b (cloud-side orphan-principal cleanup): admin-confirmed sweep of AWS Identity Center "User not found" assignments. Migration head = 130. Pairs with the new "打开云 Console" pivot button on scope detail pages (resolved per-integration via `Integration.config_public.console_url` → `platform_settings.cloud.console_url.<provider>` → hardcoded baseline). Earlier (2026-05-05, migration head = 116, post 5th-audit hardening — Redis-backed advisory locks replaced 6 long-held FOR-UPDATE patterns; AWS create_account `ClientRequestToken` for retry idempotency; Aliyun bare-except → ProviderAuthError/Transient so mark-and-sweep stops nuking on transient blips; currency normalization across Aliyun/Tencent/UCloud/AliCloud-Intl with unknown-currency fail-safe; OSS pagination; Tencent/UCloud `list_scopes` ghost-scope canonical-id resolution).

This document is for the day-to-day. If you need to know how a piece works under the hood, read PLATFORM.md §13. If you need to know what to TYPE when something is on fire, this is the right file.

---

## 0. Glossary (so the runbooks below read clean)

| Term | Meaning |
|------|---------|
| **Tenant credential** | The org-level credential Helix holds in `integrations(kind='cloud')` for one provider — e.g. AWS Organizations master, Azure tenant app, GCP Org SA, Aliyun RD master. |
| **Scope** | A billable unit. AWS account / Azure subscription / GCP project / Aliyun RAM member / standalone IAM bucket. Row in `cmdb_cloud_scopes`. |
| **Grant** | (Person × Scope × Role) tuple. Row in `cmdb_cloud_access_grants`. |
| **Owner** | Single Helix person on `cmdb_cloud_scopes.owner_person_id`. Approves grants, gets cost alerts. |
| **Curated role** | Role catalog entry with `is_promoted=true` — shows in the request modal dropdown. |
| **Discovered grant** | Grant Helix didn't issue but found cloud-side via `cmdb.grants_sync`. `source='discovered'`. |
| **Helix grant** | Grant Helix issued via approval flow or owner direct-add. `source='helix'`. |
| **standalone_iam scope** | Single-tenant clouds (AWS SG, Tencent, UCloud) — one degenerate scope row per integration; no sub-accounts to discover. Grant = IAM user. |
| **Org-root creds** vs **sub-account creds** | Org-root sees + manages all units. Sub-account sees only its own. Helix supports both shapes. |
| **prod scope** | `environment='prod'` on `cmdb_cloud_scopes`. Opted out of auto-revoke at expiry. |

---

### Who can see what on the cloud pages (2026-09-04)

| | 读 | 写 |
|---|---|---|
| `global_admin` / `cloud_admin` | 全部 | 全部(不可逆下线仍限 `global_admin`) |
| `finance` | 云服务器 / 域名 / 证书 / DNS / CDN / 负载均衡 / 对象存储 / IAM 用户(含离职残留)/ 子账号·订阅(含待审批、已下线)/ 网络总览 / 专线 | **无** |
| `auditor` / `management` | 同 `can_read_cloud` 名单 | 无 |

**Three cloud pages are NOT open to `finance`** — 回收站、角色目录、阿里云百炼(以及 API Key 管理)。判据是它们的 **READ 端点本身**就是 `can_manage_cloud`(API Key 是 global_admin + asset_admin),给了入口只会得到首屏 403。前端路由和侧栏按同一条线切分。

**Answering "为什么财务看不到某个云页面"**: 先看那个端点的读闸是哪一个谓词 —— `can_read_cloud` / `can_view_cloud_resources` 都含 `finance`,`can_manage_cloud` 不含。如果读闸含 finance 而页面仍进不去,那是前端 `App.tsx` 的 `RoleRoute` 没带上 `'finance'`(2026-09-04 之前整块云路由都是这个状态:**403 在路由上,不在 API 上**,所以后端日志里一条 403 都不会有 —— 别从后端日志开始查)。回归闸在 [`test_finance_cloud_read_parity_2026_09_04.py`](../../server/tests/test_cmdb/test_finance_cloud_read_parity_2026_09_04.py)。

---

## 1. Provider capability matrix

What each cloud driver can do today. Empty cells are NOT bugs — they're "not implemented" (and most aren't planned).

| Provider | Scopes | Grant | Revoke | Roles sync | Grants sync | Servers | DNS | LB | CDN | OSS | IAM users | Billing | Account close |
|----------|--------|-------|--------|-----------|-------------|---------|-----|-----|-----|-----|-----------|---------|---------------|
| AWS (Organizations + Identity Center) | ✓ multi-account | ✓ | ✓ | ✓ permission sets | ✓ assignments | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ daily + monthly | `organizations.close_account` |
| Azure | ✓ subscriptions | ✓ | ✓ | ✓ role definitions | ✓ role assignments | ✓ | ✓ | ✓ | ✓ | ✓ | (later) | ✓ Cost Management API | subscription cancel |
| GCP | ✓ projects under Org | ✓ | ✓ | ✓ predefined + custom | ✓ IAM policy | ✓ | ✓ | ✓ | ✓ | ✓ | (later) | ✓ BigQuery export | `projects.delete` |
| Aliyun (RD) | ✓ RAM members | ✓ | ✓ | ✓ system + custom | ✓ per-user policies | ✓ | ✓ | ✓ | — | ✓ | (later) | ✓ BSS monthly | `RemoveCloudAccount` |
| Aliyun Intl | ✓ RAM members | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | partial (since 067) | ✓ | Delete |
| Tencent | standalone_iam (1 row) | ✓ CAM user create + policy attach | ✓ | ✓ CAM policies | ✓ | ✓ CVM | — | — | — | ✓ COS | — | ✓ monthly | no-op |
| UCloud | standalone_iam | ✓ IAM user + policy | ✓ | ✓ | (discover) | ✓ uhost | — | ✓ ULB | ✓ ucdn | ✓ ufile | — | ✓ | no-op |
| KSyun | — | — | — | — | — | ✓ | — | — | — | ✓ | — | ✓ | — |

When you read "(later)" or "—", that capability is genuinely not wired. If a user asks for it, it's a real backlog item, not an oversight you can paper over.

---

## 2. Onboard a new cloud tenant

Use this when adding e.g. a new AWS Organization, a new Azure tenant, a new Aliyun RD master.

1. **Mint cloud-side credentials** with the right scope:
   - **AWS**: Master account → IAM user (or assumed-role) with `organizations:*` + `sso-admin:*` (Identity Center) + `iam:ListUsers` + `ce:GetCostAndUsage` + service-list permissions for every capability you'll enable.
   - **Azure**: Tenant-level app registration with `Microsoft.Resources/subscriptions/read`, `Microsoft.Authorization/roleAssignments/*`, `Microsoft.Billing/*`, `Microsoft.CostManagement/*`, plus reader on every subscription you'll discover.
   - **GCP**: Org-level service account with `roles/resourcemanager.organizationViewer`, `roles/resourcemanager.projectIamAdmin`, `roles/billing.viewer`, `roles/iam.roleViewer`.
   - **Aliyun (RD master)**: RAM user under master account with `AliyunResourceDirectoryFullAccess`, `AliyunBSSReadOnlyAccess`, plus per-account assume-role chain to RAM members.
   - **Tencent / UCloud / KSyun**: single-tenant; one IAM user per integration.

2. **Create the integration row**:
   ```
   POST /api/v1/integrations
   {
     "kind": "cloud",
     "provider": "aws",                // aws | azure | gcp | aliyun | alicloud_intl | tencent | ucloud | ksyun
     "slug": "aws-prod-us",            // stable, used in scheduled_jobs naming
     "display_name": "AWS Production (us-east-1)",
     "config_public": { "default_region": "us-east-1" },
     "secrets": { "access_key_id": "...", "secret_access_key": "..." },
     "capabilities": ["servers", "scopes", "roles", "grants", "billing", "user_revoke", "sso_admin"]
   }
   ```
   Backend encrypts `secrets` with `CREDENTIAL_ENCRYPTION_KEY` (AES-256-GCM) before persisting. If the env var isn't set you'll get a 500 — fix infra before retrying.

3. **Test connection**: `POST /api/v1/integrations/{slug}/test`. Should enumerate ≥1 server. If it returns `503 ENCRYPTION_KEY_MISMATCH`, your `CREDENTIAL_ENCRYPTION_KEY` doesn't match the one used at insert — rotate or re-insert.

4. **Verify scheduled jobs auto-created**: `IntegrationService.create()` calls `_reconcile_cloud_scheduled_jobs()` to insert per-capability rows. Check `/admin/scheduled-jobs` — you should see e.g. `cmdb.resource_sync` rows for each capability you enabled. They start **disabled**.

5. **Enable the jobs**: flip them on in `/admin/scheduled-jobs`. Default schedules are sensible; tweak per-tenant if needed (e.g. servers every 6h, dns every 12h).

6. **Run scope discovery once now**: `POST /api/v1/cmdb/sync?integration_id=...&capability=scopes`. Don't wait for the 04:00 cron. New `cmdb_cloud_scopes` rows appear with `owner_person_id=NULL` if no owner-equivalent role holder is detected.

7. **Run roles + grants sync** (one each), so the request modal has a role catalog and you can see what existing access there is:
   ```
   POST /api/v1/cmdb/sync?integration_id=...&capability=roles
   POST /api/v1/cmdb/sync?integration_id=...&capability=grants
   ```

8. **Set scope owners** for any unowned rows — `PATCH /api/v1/cmdb/cloud-scopes/{id}` with `owner_person_id`. Without an owner: grants are blocked + a `cloud_scope_unowned` compliance incident is raised.

9. **Verify billing flows**: `POST /api/v1/cmdb/sync?integration_id=...&capability=billing` — new row in `cmdb_billing_monthly`. Threshold lookup: `platform_settings.billing.budget_usd.{slug}` (per-tenant) or `billing.budget_usd.default`. Alerts fire to `platform_settings.feishu.alert_webhook_url`.

---

## 3. Provision a new sub-account (employee self-service)

This is the path employees take to "give me a new AWS account / Azure subscription / GCP project for project X".

1. **Employee submits**: `/my/cloud/scopes` ▸ "申请新子账号" → fills name, description, monthly budget, **account type** (free-text — production / test / POC / personal). Backend writes `cmdb_cloud_scope_requests(status='pending', metadata_json={cloud_account_type, department})`.

2. **Approval routing** (Feishu, since 2026-05-02):
   - Backend resolves the `cloud_scope_request` flow under the primary `chat_approval` integration, fetches the live Feishu definition, populates the form (cloud provider / owner / name / department auto-pulled from HRIS / monthly budget / type / unique ID = the Helix request UUID), and creates a Feishu instance.
   - The parallel `ApprovalRequest` row carries the link via `source_payload.scope_request_id` so the Feishu callback dispatcher can route the decision back into `CloudScopeRequestService.handle_feishu_callback()`.
   - On Feishu approve → callback runs `service.approve()` (existing path; same audit + notification trail an admin-clicked approval would have hit) **then awaits `_post_provision_finalize(scope_id, trace_id=feishu_callback:...)` inline** (PR #671, 2026-06-02) — the service-level approve doesn't go through the router, so without this explicit invocation the Feishu-driven happy path would still silently skip the auto-Owner finalize. Same backstop (`cmdb.auto_owner_grant_backfill` every 10 min) covers any inline failure.
   - **Account-ID backfill onto the Feishu instance** (added 2026-06-13): after provisioning succeeds, `_backfill_account_id_to_feishu` posts the freshly minted cloud account/project `external_id` (+ display name) as an **instance comment** on the still-live Feishu approval, replacing the submit-time "(待开通后填写 / TBD)" placeholder so approvers/CC see the real ID in the approval-card timeline. It is a COMMENT, not a form-field write: Feishu forbids editing the form of a submitted approval, and the instance is already terminal (`APPROVED`) by the time the ID exists — the "Cloud account (project) ID" widget being editable at the IT 开通 node does not enable a post-approval API write (`approve_task`'s form path needs a *pending* task; there is none here). Best-effort — the durable `provisioned` state is already committed, so a Feishu hiccup only logs a warning. Fires on both the auto path (`handle_feishu_callback`) and the manual path (`manual_provision`, where the admin-supplied `external_id` is what gets posted).
   - On Feishu reject → callback runs `service.reject()`. Requester can resubmit a new request.
   - **Withdraw**: requester calls `POST /api/v1/cmdb/scope-requests/{id}/withdraw` while pending — local row flips to `withdrawn` AND the Feishu instance is cancelled in the same call.

3. **On admin approval**:
   - For **AWS** (Organizations supports `create_account`): backend calls `driver.create_scope()` → `status='provisioning'` → on success `status='provisioned'`, new row in `cmdb_cloud_scopes` with requester as `owner_person_id`. AWS path includes **owner-alias auto-derive** (added 2026-05-25, prefix rule simplified same-day): the request name is slugged verbatim into `<slug>@pingcap.com` — NO forced `aws-` prefix. Names that already start with `aws-` stay 1:1 (`aws-pingcap-datalake` → `aws-pingcap-datalake@pingcap.com`); names without it stay un-prefixed (`pingcap-foo` → `pingcap-foo@pingcap.com`). Empty / over-64-char slugs raise so the admin renames the scope instead of silently colliding with a `unnamed@pingcap.com` placeholder. the Google Workspace driver `add_alias` is called against the requester's mailbox (409 ALREADY_EXISTS treated as success), the alias is persisted to `cmdb_cloud_scope_requests.metadata_json.owner_alias_email`, and then used as the `owner_email` for AWS Organizations CreateAccount. AWS sees a globally-unique email (avoids `EMAIL_ALREADY_EXISTS`), the requester still receives the AWS root verification mail. **Idempotency**: AWS Organizations CreateAccount accepts no `ClientRequestToken` (corrected 2026-05-25 — the parameter does not exist on this API); the driver sweeps `list_create_account_status(States=[IN_PROGRESS, SUCCEEDED])` for an entry matching AccountName before issuing `create_account`, resuming polling its RequestId on match.
   - For **Aliyun / AliCloud 海外** (Resource Directory `CreateResourceAccount`): backend calls `driver.create_scope()` against the RD **master** credentials. **`rd_parent_folder_id` is OPTIONAL** (fixed 2026-06-03): when set, the new sub-account is filed under that 资源夹; when blank — the prod default, neither `aliyun-primary` nor `alicloud-intl-primary` ever had it configured — the driver resolves the RD **Root** folder id via `GetResourceDirectory` and files the member there (same default as AWS Organizations; resolved explicitly rather than relying on Aliyun's undocumented omit-ParentFolderId behaviour). Before the fix the driver hard-`raise`d "missing rd_parent_folder_id", permanently blocking every Aliyun sub-account request even though Aliyun never required the parameter. `rd_account_type` (`ResellAccount` default / `NormalAccount`) controls billing; `EntityAlreadyExists.Account` on a duplicate DisplayName is surfaced verbatim so admin renames and re-approves. **`PayerAccountId` = 结算账号 (fixed 2026-08-07)**: Aliyun treats it as optional and defaults to 「新成员自主付款」 when omitted, which Helix did — so every sub-account created via this path between 2026-06-03 and 2026-08-07 came out **self-paying** (caught on `data-integration-team-internal-alicloud`, AccountId `5083883504403318`, created 2026-07-08). A self-paying member sits outside the master's consolidated BSS bill that the per-scope cost/billing sync reads, so it under-reports, and it carries an unowned balance that can suspend the account. The driver now resolves the RD **管理账号** via `GetResourceDirectory` (`MasterAccountId`, same cached response as `RootFolderId`) and sends it as `PayerAccountId` = console 「使用管理账号为新成员付款」. Override with the optional `rd_payer_account_id`: a member AccountId → 「使用已有成员为新成员付款」; the literal `self` → deliberate 自主付款. Unresolvable master (or a non-numeric config value) **fails the request loud** rather than silently reverting to self-pay. **Existing self-paying members must be fixed BY HAND** — 阿里云控制台 → 资源目录 → 成员 → 「修改结算方式」 → 「使用管理账号为新成员付款」; `PayerAccountId` is settable only at creation time, so there is no Helix-side remediation path. **`AccountNamePrefix`** (seeds the auto-issued login email `<prefix>@<rd>.aliyunid.com`) is the **alphanumeric-only, Alibaba-brand-token-stripped** reduction of the request name, capped at Aliyun's 37-char limit. The live 2020-03-31 endpoint rejects a prefix containing the brand word `aliyun` (and `alibaba`/`alipay`/`hichina`) with `IllegalParameter.Account.AccountName` — **proven in prod 2026-06-03**: same request, prefix `devofflineafterworldaliyun` refused vs `devofflineafterworld` succeeded (AccountId 5660180459278784, login `devofflineafterworld@rd-grmjfa.aliyunid.com`). DisplayName keeps the readable name (brand word + hyphens both fine — the created DisplayName is `dev-offline-afterworld-aliyun`). If a name reduces to nothing after stripping, the prefix falls back to `helix`. **Debug recipe** for any future `IllegalParameter.Account.AccountName`: SSM into `compose-backend-1`, `build_driver_by_id("alicloud-intl-primary")`, monkeypatch `_acs3_call` to capture params, then make ONE real `CreateResourceAccount` with the candidate prefix (a failure is non-destructive; a success creates the desired account → reconcile via `manual_provision`). **Note**: only the integration whose credentials belong to the RD master can create sub-accounts — `alicloud-intl-primary` is the PingCAP RD master (18 sub-account scopes; both prod failures `dev-offline-afterworld-aliyun` + `alicloud-nexuscrm` routed here). If `GetResourceDirectory` returns no `RootFolderId` (member-account creds), `create_scope` fails loud with an actionable message instead of sending a blank folder.
   - For **Azure / GCP / Tencent / UCloud**: `create_scope` is NOT implemented. Admin must:
     1. Create the sub-account in the cloud console manually.
     2. Call `POST /api/v1/cmdb/scope-requests/{id}/manual-provision` with a JSON body `{"external_id": "<cloud_id>", "scope_type": "<optional>"}` — 前端即「回填云端 ID」弹窗。(它**不是** query 参数;此处曾长期写成 `?external_id=`。)`scope_type` 省略时按 provider 取默认,填了则必须落在 `TENANT_LEVEL_SCOPE_TYPES` 内,否则 422。
     3. Backend inserts `cmdb_cloud_scopes` with the requester as `owner_person_id` (Helix-internal ownership only — see step 4 for the cloud-side IAM contract).

4. **Initial ownership** (Helix-side + cloud-side, since 2026-05-13; AWS path added 2026-05-25): requester is written to `cmdb_cloud_scopes.owner_person_id`, AND a `CmdbCloudAccessGrant` row is automatically created with the per-provider owner-equivalent role. Per-provider map at `lifecycle_router._DEFAULT_OWNER_ROLE_BY_PROVIDER`:
   - **GCP** → static (`roles/owner`, "Owner"); bound via `cloudresourcemanager.projects.setIamPolicy`.
   - **AWS** → dynamic resolve via `_resolve_default_owner_role`: reads platform_setting `cloud.aws.scope_request_default_owner_role_name` (default `DBaaS-DevLeader-Role`), lists Identity Center permission sets, matches by display name, returns the resolved ARN. Bound via `sso-admin.create_account_assignment` in `AwsCloudDriver._grant_sso_sync`. Pre-requirements: the requester's `work_email` must already be a SCIM-synced user in AWS Identity Center (`_resolve_identity_store_user_id` looks them up); a permission set with the configured Name must exist in the instance. If the name has no match, the auto-grant audit-logs the skip and the requester can self-bind via owner-add.
   - **Aliyun / AliCloud 海外** → static (`admin`, `admin`) — the CloudSSO access configuration named `admin` (the driver's owner-equivalent; every existing sub-account's owner grant carries `role_key='admin'`). Bound via `cloudsso.CreateAccessAssignment` (requester must be a CloudSSO user in the directory; if not, the grant lands PENDING and the backfill cron retries). **Added 2026-06-03** — these entries were missing, so before the fix an Aliyun self-service requester became Helix-side owner with ZERO cloud IAM (finalize silently skipped the grant). Prod scope `32e44d04` (dev-offline-afterworld-aliyun) had its owner `admin` grant applied by hand via `provision_initial_owner_grant(role_key='admin')`.

   - **Azure / Tencent / UCloud** → static, **added 2026-08-17** (same bug class as the Aliyun entry above, three months later). These three have no `create_scope`, so every one of their sub-accounts arrives through the `manual-provision` rescue path — and the map's header comment used to say entries land "when their drivers grow `create_scope` support", so nobody added them. `_resolve_default_owner_role` returned `None`, `_post_provision_finalize` audit-logged `cloud.scope_request.auto_owner_grant status=skipped reason="no default owner role resolved"`, and the requester became Helix-side owner of a subscription with zero cloud IAM and an empty member list. Prod hits: `azure-ai-prod` (scope `7845b0f7`, 2026-08-17), `e7babe7b` (08-14), `ae0486fc` (08-11), tencent `7f4642bb` (08-12).
     - **Azure** → `/providers/Microsoft.Authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635` ("Owner", the built-in role whose definition GUID is stable across tenants). Bound via ARM `roleAssignments` PUT — the driver passes `role_key` straight through as `roleDefinitionId`. Requester must resolve in AAD (`_resolve_user_object_id` by email); otherwise the grant lands PENDING and the backfill cron retries.
     - **Tencent** → `AdministratorAccess` (CAM policy **name**, resolved to a PolicyId by the driver, then `AttachUserPolicy` onto the idempotently-created CAM sub-user). NOT the `Type:Name` form — that convention is domestic Aliyun's RAM impl only.
     - **UCloud** → `AdminPolicy` (IAM policy name). Inert until the first UCloud self-service ask; present so the next provider doesn't repeat this hole.
     - Deliberately still absent: **ksyun** (declares neither `owner_role_keys` nor a `grant_access` override — nothing to mint; its one integration row is disabled).
     - `tests/test_cmdb/test_auto_owner_role_map_coverage_2026_08_17.py` now fails CI if a driver declaring `owner_role_keys` has no map entry (and vice-versa), so "forgot to add the provider" can't ship silently a fourth time.

   Both writes happen inside a single per-integration `cloud_sync` Redis lock acquisition so they can't race the cron sweeps. If the driver's grant call fails (IAM transient blip, propagation delay), the grant row lands in PENDING and the requester can retry via the standard owner-add flow — the scope itself stays provisioned.

5. **Post-provision finalize** (added 2026-05-13, rewired 2026-06-02): `_post_provision_finalize(scope_id, trace_id=...)` is awaited inline from THREE call sites — `lifecycle_router.approve_scope_request` (admin HTTP), `lifecycle_router.manual_provision_scope_request` (admin HTTP, GCP/Azure/Aliyun/etc.), AND `scope_request_service.handle_feishu_callback` (the production happy-path, since prod approvals come from Feishu). All three sites pass their own `trace_id` so finalize audit rows correlate with the approve audit row that triggered them. The handler now (no longer `background.add_task`) which (a) calls `sync_roles_for_scope` to pre-warm `cmdb_cloud_roles` for the new scope — without this the "添加成员" dropdown sat at `暂无数据` until the next daily `cmdb.roles_sync` cron — and (b) drives the auto-Owner grant described in step 4. Both run under one acquisition of the `cloud_sync:{integration_id}` Redis lock so neither races the cron handlers (`run_scopes_sync` / `run_roles_sync` / `run_grants_sync`) nor an admin "sync now" click. The function wraps its work in `try/except LockBusy` + `try/except Exception` so an inline crash never propagates to the approve response. **Backstop (added 2026-06-02 same PR; hardened by Round-64 review-sweep same day)**: `cmdb.auto_owner_grant_backfill` cron runs every 10 min, re-invokes `_post_provision_finalize` for any active **self-service-provisioned** scope (provenance: `cmdb_cloud_scope_requests.created_scope_id`; sync-imported inventory is excluded — 2026-07-03 tightening) whose owner has no `source='helix'` **`active`** grant and no ACTIVE owner-equivalent grant of any source — catches inline LockBusy / Identity Center transients / driver exceptions. Round-64 fix: candidate/verify queries now key on `active` ONLY (was `active`/`pending`). A `pending` row means the driver IAM bind FAILED (no cloud-side access), so counting it as "done" both mis-reported success and permanently excluded the scope from retry — the exact silent-loss this cron exists to catch. `provision_initial_owner_grant` now **re-dispatches an existing stuck `pending` owner-auto-grant row** instead of raising `ConflictError`, so the safety net actually retries the failed bind until it lands `active`. **Why inline instead of bg task**: the original `background.add_task` wiring silently lost every firing in prod uvicorn (audit table `cloud.scope_request.auto_owner_grant` was empty from PR #279 launch on 2026-05-13 through 2026-06-02 — a ~20-day silent-loss window; every AWS scope had 0 owner grants). FastAPI BackgroundTasks runs after the response is sent and is not safe for 600s-lock + cloud-mutation work. Inline adds ~5-15s on top of the existing ~19s AWS CreateAccount poll — acceptable for the user-visible payoff.

6. **Failure**: if `driver.create_scope()` raises, `status='failed'` + `failure_reason`. **Re-approve** the same request (the recovery-retry branch in `service.approve()` accepts `failed`) — don't open a new request. Common failure on GCP: opaque `HttpError 403 "The caller does not have permission"` from `cloudresourcemanager.projects.create`. The message is identical for IAM-missing / VPC-SC / Org-Policy / **IAM-propagation-delay** (org-level grants can take 15+ min). Always pull GCP Cloud Audit Logs first (`protoPayload.serviceName="cloudresourcemanager.googleapis.com"` + `methodName=~"CreateProject"`, no email filter) to read `authenticationInfo.principalEmail` (actual SA) and `authorizationInfo[].granted` (real verdict) before guessing.

7. **Stuck-request reset playbook** (added 2026-05-25 after the `ClientRequestToken` outage): when a scope_request lands in `provisioning` or `failed` because of a transient driver bug — and the fix has already deployed — the recovery loop is:

   ```bash
   # ── Step 1. Identify the request ID by scope name + requester
   docker compose exec -T mysql mysql -uroot helix -e "
     SELECT id, status, failure_reason, raw_data
     FROM cmdb_cloud_scope_requests
     WHERE name = 'aws-pingcap-datalake'
       AND requester_person_id = (SELECT id FROM identity_persons WHERE work_email='shuai.feng@pingcap.com');"

   # ── Step 2. Pre-flight: confirm AWS has no zombie CreateAccountStatus
   #     for this AccountName — otherwise the new code will resume the dead
   #     in-flight RequestId instead of issuing a fresh create.
   #     (CreateAccountStatus rows expire after 30 days; usually clean.)
   aws organizations list-create-account-status \
       --states IN_PROGRESS SUCCEEDED \
       --query "CreateAccountStatuses[?AccountName=='aws-pingcap-datalake']"

   # ── Step 3. Pre-flight: confirm the configured Identity Center
   #     permission-set name actually exists. If it doesn't, auto-grant
   #     will warn-skip and the requester needs manual owner-add.
   aws sso-admin list-permission-sets --instance-arn <arn>
   #   then describe each ARN to find the one whose Name matches
   #   `cloud.aws.scope_request_default_owner_role_name` (default
   #   "DBaaS-DevLeader-Role"). Override via platform_settings if needed:
   #     INSERT INTO platform_settings(key, value, ...)
   #     VALUES ('cloud.aws.scope_request_default_owner_role_name',
   #             '"AlternateRoleName"', ...);

   # ── Step 4. Reset the request to feishu_approved (re-enters approve()
   #     via the recovery-retry path). Clear any stale alias so the new
   #     derive path takes over fresh. trace_id changes — that's fine,
   #     audit log preserves the original.
   docker compose exec -T mysql mysql -uroot helix -e "
     UPDATE cmdb_cloud_scope_requests
     SET status = 'feishu_approved',
         failure_reason = NULL,
         raw_data = JSON_REMOVE(COALESCE(raw_data, JSON_OBJECT()),
                                '$.owner_alias_email'),
         updated_at = NOW()
     WHERE id = '<request-id>';"

   # ── Step 5. Click "Approve" again in the admin UI (or POST
   #     /api/v1/cmdb/scope-requests/{id}/approve). The recovery-retry
   #     branch picks up the FEISHU_APPROVED state without re-asking
   #     Feishu, derives a fresh alias, calls GWS add_alias, and runs
   #     AWS Organizations CreateAccount.
   ```

   Production-specific: use SSM session manager + `sudo docker exec compose-worker-1 ...` per `reference_prod_ec2_ssm`. Never run the password-login / fake-account testing tools against prod (`feedback_no_prod_testing`).

8. **"立即同步" button contract** (admin scope detail page, since 2026-05-13): backend returns HTTP 202 with `{status: "scheduled", integration_id}` — the actual scopes + roles + grants sweep runs in a FastAPI `BackgroundTasks` under the `cloud_sync` lock. The page does NOT have synchronous summary counts to show; instead the frontend shows a "sync started, page refreshes in ~20s" toast and schedules a `setTimeout(load, 20_000)`. If the lock is held (a cron or another admin click is mid-flight), backend pre-flights with HTTP 409 `SCOPE_SYNC_ALREADY_RUNNING` and the frontend renders an info-level toast asking the operator to wait. The `_run_scope_sync_background` task also writes a `cloud.scope.sync_now status=skipped` audit row on `LockBusy` so the scheduled-jobs history UI reflects "didn't run" rather than appearing as silent success.

---

## 4. Investigate a missing resource

A user says "my server isn't showing up in Helix". Walk this:

> **2026-05-28 update — check the recycle bin first.** Migration 212 rescued the 116 UCloud servers that the legacy-CMDB → sync collision swept on 2026-04-24 (admins reported "10.9.84.114 / DLP-test missing"). The recycle-bin UI at `/admin/cloud/recycle-bin` lists every soft-deleted CMDB row across 6 resource types (server / domain / dns_record / load_balancer / cdn_domain / certificate) — start here before deep-diving the SQL below. If the row is in the recycle bin, click "恢复" / use `POST /api/v1/cmdb/recycle-bin/{resource_type}/{id}/restore` (or the MCP `restore_cmdb_resource` tool). The IP search on `/admin/cloud/servers` (and on the recycle-bin server tab) now matches `public_ips` / `private_ips` directly — paste the IP and the matching row surfaces immediately. Soft-delete protection has been hardened so this class of accidental sweep cannot recur (see §4d below for the invariants).


1. **Find the integration**: `SELECT id, slug, enabled, capabilities FROM integrations WHERE kind='cloud' AND provider='<provider>';`
2. **Confirm capability is enabled**: `'servers' IN capabilities`. If not, the user added the integration without enabling servers — fix.
3. **Confirm the cron is firing**:
   ```sql
   SELECT * FROM scheduled_jobs WHERE name LIKE 'cmdb.resource_sync%' AND config LIKE '%<integration_id>%' AND enabled=1;
   SELECT * FROM scheduled_job_runs WHERE job_id=? ORDER BY started_at DESC LIMIT 5;
   ```
4. **Inspect the latest sync job audit row**:
   ```sql
   SELECT id, status, total_upserted, total_errors, capability_results, error_message, started_at, finished_at
   FROM cmdb_sync_jobs
   WHERE integration_id=? AND JSON_OVERLAPS(resource_types, JSON_ARRAY('servers'))
   ORDER BY queued_at DESC LIMIT 5;
   ```
   If `status='partial_failure'` or `failed`, `capability_results['servers']['error']` has the cause.
   - **`status='failed'` with `partial_failures` covering every region/project** (AWS `UnauthorizedOperation × N`, GCP `http_403 × N`): the integration's IAM principal cannot enumerate that capability. Fix at the cloud console, NOT in Helix. For AWS: attach `AmazonEC2ReadOnlyAccess` (or `ec2:DescribeInstances` + `ec2:DescribeRegions`) to the IAM user/role behind the integration's access key. For GCP: grant the SA org-level `roles/compute.viewer` and `roles/storage.objectViewer` at the org node (per-project bindings work too but multiply N times). Wait 15+ min for propagation, then `POST /api/v1/integrations/{slug}/test` — the test now correctly flips `ok=False` with the leg breakdown when 100% of legs are still failing (was previously a false `OK · enumerated 0 servers`, fixed 2026-05-16).
   - **`status='partial_failure'` with a subset of legs**: the credential works on some legs but not others. Common shapes: org-level binding not yet propagated to newly-onboarded projects/subscriptions; regional SCP / VPC-SC blocking one AWS region; legacy projects intentionally excluded from the cross-project SA. The sync upserts the surviving legs, so data is partially fresh. Decide per case whether to widen the binding (then wait for propagation) or accept the gap.
4a. **Test the connection** before running another full sync: `POST /api/v1/integrations/{slug}/test`. The test connection now has three semantic states the admin should know how to read:
   - `ok=true` `message='OK · enumerated N servers'` (no leg suffix) — full success.
   - `ok=true` `message='OK · enumerated N servers · M leg(s) failed: ...'` — partial coverage; the credential is valid but doesn't see everything. The leg breakdown groups by reason (`UnauthorizedOperation × 5 [region:us-east-1, ...]`). This is the steady-state for many large orgs and may be intentional.
   - `ok=false` `message='0 servers enumerated · M leg(s) failed: ...'` — credential observably broken for this capability. Fix the IAM grant per the bullet above before retrying sync.
5. **Run a manual sync** if it's been 24h without one: `POST /api/v1/cmdb/sync?integration_id=...&capability=servers`. Watch the new row.
6. **Check soft-delete**: maybe the resource exists but has `deleted_at IS NOT NULL` (mark-and-sweep removed it):
   ```sql
   SELECT id, name, last_synced_at, deleted_at FROM cmdb_servers WHERE integration_id=? AND name LIKE '%<query>%';
   ```
   If `last_seen_sync_job_id < latest successful job's id`, the resource genuinely disappeared cloud-side. Confirm with the cloud console before further action.
7. **If the credentials are wrong**: decrypt + verify outside Helix. Use the AWS/Azure/GCP CLI with the same credential. Then re-insert via `PATCH /api/v1/integrations/{slug}` with new `secrets`.

---

### 4a. test_connection per-capability table (added 2026-05-17)

Clicking 测试 on a cloud integration now opens a per-capability breakdown modal (not a toast). Each row shows:

- **能力 (Capability)** — `servers` / `object_storage` / `iam` / `domains` / …
- **状态** — ✅ ok / ⚠️ 部分失败 / ⏭️ 跳过 / ❌ 失败
- **抽样行数** — sample row count returned by the cheapest list call
- **详情** — leg-failure reason summary (e.g. `auth_error × 5 [sub_account:506…, sub_account:553…]`) or skip reason

Read the table left-to-right. The top alert tells you the overall verdict; the table tells you which capability(ies) drove it.

**Three classes of "failed":**
- `0 rows + N legs failed with same auth-shaped reason` — the credential observably cannot reach this capability (typical: AliCloud-intl `object_storage` needs cross-account AssumeRole that's not implemented; GCP `object_storage` 403 on Apigee-managed `sys-*` projects; AWS `servers` UnauthorizedOperation in a region). The fix is on the cloud side (RAM / IAM policy + AssumeRole) — NOT a Helix bug. Once cloud side is fixed re-click 测试.
- `0 rows + 0 legs` — empty cloud (legitimate new tenant). Still passes as ok; verify by adding a test resource on the cloud side and re-clicking.
- `timeout (60s)` — fan-out too large or cloud API hung. Re-click; if it persists check the provider status page.

Money-cost capabilities (`billing`, `billing_daily`, `cost_facts`) and write-side capabilities (`user_revoke`) are deliberately excluded from the probe — BigQuery scans cost money and a write call must not fire from a health-check.

`last_test_message` on the integration row carries a one-line summary in the form `servers=ok(N) · object_storage=failed`. The full per-cap breakdown lives in the modal (and on the API response under `detail.capability_probes`); rerun 测试 to repopulate.

### 4b. GCP `sys-*` system projects auto-skipped (added 2026-05-17)

GCP auto-creates "system projects" with id prefix `sys-XXXXXXXXX` to back managed services (Apigee, AppEngine, Firebase, Workspace). Their IAM and Cloud Storage surfaces are walled off from external callers — even a project Owner cannot grant `resourcemanager.projects.getIamPolicy` or `storage.buckets.list` on them. Pre-2026-05-17, Helix listed them as scopes and the daily `cmdb.grants_sync` / `cmdb.resource_sync object_storage` cron tick spammed `partial_failure` with dozens of `sys-*` 403s.

Filter is applied at three layers (defense in depth):

1. `GcpCloudDriver.list_scopes` — `sys-*` projects never become `cmdb_cloud_scopes` rows in the first place.
2. `GcpProvider._list_active_projects_sync` — the resource-sync provider's `_fan_out_per_project` skips them.
3. `GcpCloudDriver._list_scope_grants_sync` — early-return `[]` if an orphan `sys-*` scope row still exists (created by an earlier driver version, will be auto-closed by the next `scopes_sync` orphan sweep).

If `cmdb.resource_sync gcp/object_storage` still reports `sys-*` 403s in `capability_results.partial_failures` after deploy, it's an orphan scope row that hasn't been swept yet — run `scopes_sync` manually (admin → 触发同步) or wait 24h. The runbook to verify a `sys-*` project is intentionally inaccessible: `gcloud projects get-iam-policy sys-XXXXX` returns `PERMISSION_DENIED` for any external SA — Google's design, not a Helix bug.

### 4d. Legacy-CMDB import ID-shape + sweep contract (added 2026-05-28, migration 212)

The 2026-04-24 prod incident: one UCloud sync silently soft-deleted 116 servers including the user-visible `10.9.84.114 / DLP-test`. Three independent fixes landed together to prevent recurrence.

**The bug**: `migrate_cmdb.py` synthesised `instance_id = f"legacy-{vendor}-{src_id}-{raw_iid}"` so cross-vendor PK collisions in the legacy `server` table couldn't collapse into one row. Live UCloud's driver writes the raw `uhost-...` ID. The unique key `(integration_id, instance_id)` never collided → first live sync after the migration treated all 116 migrated rows as missing → `_safe_sweep` marked them `deleted_at=NOW()`.

**Recovery (one-time, done)**: migration 212 walks every `cmdb_servers` / `cmdb_load_balancers` row with `instance_id LIKE 'legacy-%'`, strips the synthetic prefix, recovers the raw cloud ID, and either updates in-place (clearing `deleted_at`) or merges owner/app/remark onto a live-sync collision row before dropping the legacy one. True bare-metal rows (no raw_iid) move to a new `legacy-bare-metal` integration that's seeded `enabled=false` (cron never iterates it → no sweep ever reaches those rows).

**Going-forward invariants**:

1. **`migrate_cmdb.py` writes raw IDs.** Every upsert path now uses `raw_iid` directly as the unique-key column (servers/LB/DNS). When `raw_iid` is empty the row goes to `legacy-bare-metal` with key `legacy-bare-{vendor}-{src_id}`. The five `ON DUPLICATE KEY UPDATE` clauses all include `deleted_at=NULL` so a re-run resurrects tomb-stoned rows.
2. **`_safe_sweep` never sweeps rows the driver hasn't seen.** Pre-fix the predicate was `last_seen_sync_job_id IS NULL OR != current_job`; now it's `IS NOT NULL AND != current_job`. Migration / manual rows (`last_seen_sync_job_id=NULL`) are structurally outside the sweep. The `existing` baseline-count narrows to the same set, otherwise a freshly-imported batch trips the drop-ratio guard on every sync.
3. **Recycle bin gives admins a recovery surface.** `/admin/cloud/recycle-bin` lists soft-deleted server / domain / dns_record / load_balancer / cdn_domain rows; one-click restore + audit trail. HTTP routes `GET /api/v1/cmdb/recycle-bin/{resource_type}` + `POST /api/v1/cmdb/recycle-bin/{resource_type}/{id}/restore`, MCP twins `list_cmdb_recycle_bin` + `restore_cmdb_resource`.

**UCloud multi-project (paired fix)**: the user's DLP-test server lived in UCloud project `org-eneelp` while the integration was configured for a different project → the live sync never saw it. Now `credentials.fields['project_ids']` (CSV) drives the per-call `ProjectId` iteration in `list_servers` / `list_load_balancers` / `list_cdn_domains` / `list_object_storage` / `get_monthly_billing`. Legacy singular `project_id` still works for backward-compat. Configuration:

```bash
# Admin sets via /admin/integrations/ucloud-primary
# (or directly: UPDATE integrations SET secrets_encrypted=... in the encrypted blob)
project_ids = "org-pingcap,org-eneelp,org-finance"
```

Skipped-leg labels carry both project + region: `project:org-eneelp/region:cn-bj2` so admins can see WHICH (project × region) failed in the sync-history modal.

**Verify after configuring multi-project**:

```sql
-- Should show servers from BOTH projects
SELECT JSON_EXTRACT(raw_data, '$.project_id') AS project, COUNT(*)
FROM cmdb_servers
WHERE integration_id = (SELECT id FROM integrations WHERE slug='ucloud-primary')
  AND deleted_at IS NULL
GROUP BY project;
```

### 4c. Cross-account OSS / S3 known gap (AliCloud-intl, added 2026-05-17)

AliCloud-intl integrations whose root-tenant AK fans out across sub-accounts hit a structural gap for `object_storage`: OSS `ListBuckets` is sub-account scoped, the root AK can't list buckets owned by sub-accounts without an AssumeRole hop (not implemented). Pre-fix every cron tick reported `partial_failure` with `auth_error × N sub_accounts`.

Behaviour from 2026-05-17 on: when 100% of legs return an auth-shaped reason (`auth_error`, `buckets.list_http_403`, `buckets.list_http_401`, etc.) AND the capability upserted zero rows, the capability is degraded to `skipped` (NOT `partial_failure`) with a structured `reason` carrying the leg breakdown. Final job status lands `success`, the failure-notifier stays quiet, and the admin sync history shows `skipped` with the actionable explanation. A `logger.warning` is emitted once per cron tick for ops correlation (log aggregation surface only — no Feishu spam).

To restore full `object_storage` coverage, the fix is on the cloud side (configure cross-account AssumeRole and re-implement the driver's OSS calls to use STS-derived per-sub-account credentials) — not in Helix today. Until then the structural-skip is the right outcome.

## 5. Investigate a stuck sync

`cmdb_sync_jobs.status='running'` for hours.

1. **List currently-running jobs for the tenant**:
   ```sql
   SELECT id, resource_types, queued_at, started_at FROM cmdb_sync_jobs
   WHERE integration_id=? AND status='running';
   ```
2. **Check if it's a real overlap or a stale row**: if `started_at` is NULL but `queued_at` is hours old, it never started — likely the worker missed it. Look in worker logs for the job ID.
3. **Check JSON_OVERLAPS lock contention**: two runs targeting overlapping capabilities (`servers` + `servers`) serialize. Two disjoint (`servers` + `dns`) run in parallel. Confirm by looking at all running rows' `resource_types`.
4. **Manually fail a stale row** (only when you've confirmed the worker isn't going to finish it):
   ```sql
   UPDATE cmdb_sync_jobs SET status='failed', error_message='manual fail — stale', finished_at=NOW() WHERE id=?;
   ```
   Next cron tick creates a fresh job.
5. **If runs consistently take >10 min**: reduce batch size in the driver, or break the capability into smaller logical units. This is a code change, not an ops fix.

### 5b-fixed. The shared-lock starvation above is now structurally impossible — 2026-09-06

**Read §5b first; this is what happened to it.** `cmdb.scopes_sync`,
`cmdb.roles_sync` and `cmdb.grants_sync` all take the same
`cloud_sync:{integration_id}` advisory lock while running as three
independent crons scheduled minutes apart. That is why
`cmdb.scopes.aws-primary` silently did not run for 29 of 31 consecutive
days: roles starts at 04:09 and takes ~15 minutes, scopes fires at 04:18
into a held lock.

Migration **617** chains them per account:

```
cmdb.scopes.<account>   (cron, unchanged — a chain has to start somewhere)
  └─ cmdb.roles.<account>    (chained)
       └─ cmdb.grants.<account>  (chained)
```

They can no longer contend for the lock they share, because the successor
only starts once the predecessor has finished. The ordering is also the
real data dependency — roles are per scope, grants are per role.

**What this changes for you operationally:**

* `cmdb.roles.*` and `cmdb.grants.*` no longer have a cron. Their row in
  定时任务 shows 「接在「X」之后」 instead of a cadence, and the 容量与体检
  tab's 任务链 panel shows the whole chain with each step's measured
  duration. That is the number to look at when asking whether an
  account's nightly sweep still fits its window.
* **Disabling `cmdb.scopes.<account>` now stops that account's roles and
  grants too.** That is the honest semantics (there is no fresh scope
  list to enumerate roles against) and it is visible: the two successors
  show 「上游已停用」 in the ledger census rather than looking healthy.
  Re-enabling the root resumes them by itself.
* To run one member ad hoc, 「立即执行」 on its row still works — a manual
  trigger does not need a clock.
* If a chain stops advancing, `scheduler.chain_advance` (every 5 minutes)
  is the reconciler; its run summary lists what it released and what it
  deferred over the per-tick cap.

**Not chained, deliberately:** anything on the billing path
(`cost_facts_sync` / `billing_daily_sync` / month-close / recon) and
`efficiency_compute`. Every chain edge makes its successor start
*earlier* than today's gap allows, and those jobs pull the previous day's
vendor line items — starting sooner risks reading a batch the vendor has
not finished writing. `efficiency_compute` additionally has three
predecessors, which a single-parent chain cannot express. The full list
of decisions, with reasons, is
[`app/scheduled_jobs/ordering_inventory.py`](../../server/app/scheduled_jobs/ordering_inventory.py),
and a test fails the build if a new "runs after X" claim appears in a
handler description without a decision recorded there.

### 5c. A sync cron that is late, and how to tell "late" from "broken" — 2026-09-05

**Look at 定时任务 → 容量与体检 before touching any cadence.** The executor runs
at most `SCHEDULED_JOB_MAX_CONCURRENCY` handlers at once (default 8) with a
smaller sub-gate for `heavy` ones (default 4). A job that comes due while the
gate is full does not misfire — its APScheduler task is alive and it waits. The
wait is recorded on the run row as `result_summary._gate_wait_ms` and rolled up
in that tab.

Measured on prod 2026-09-05, before migration 613: **all cloud syncs together
demanded only 3.5 slot-hours a day against a 96-hour ceiling** — the pre-dawn
queueing was never a capacity problem, it was head-of-line blocking. At 04:00
Asia/Shanghai, 33 heavy rows came due in the same hour; three long ones (AWS
`roles` 1 055 s, AliCloud `server_metrics` 1 726 s, AliCloud `cost_facts` 759 s)
plus the every-15-minute directory refresh occupied all four heavy slots, so
`cmdb.grants.aws-sg` — which runs for **9 seconds** — averaged **1 384 seconds**
of queueing.

So the triage order is:

1. **Is it queueing or is it slow?** Capacity tab → 排队最久的任务. A large
   `avg_gate_wait_ms` with a small `avg_duration_ms` is queueing; move its start
   time, do not touch its cadence.
2. **Is anything hogging a slot?** 占用最多的任务 ranks by `runs/day ×
   avg duration`, which is the number that decides what to move. Single-run
   duration is the wrong lens — prod's biggest consumer ran for 7 minutes but
   96 times a day.
3. **Is it inert rather than late?** 台账体检 names five reasons a row can look
   healthy and do nothing. `integration_disabled` in particular is *not* a
   fault: the row skips every tick on purpose and resumes on its own when the
   integration is switched back on — since 2026-09-05 it skips **before** taking
   a slot, so it no longer costs the fleet anything.

**Opting a tenant into broad inventory**: `cmdb.resource_inventory_refresh` runs
daily (migration 613 moved it off hourly — every run for 100 consecutive ticks
reported `inventory_skipped=8, seen=0` because nobody had opted in). It is left
**enabled** on purpose so it discovers a new opt-in by itself within 24 h; raise
its cadence in 定时任务 if you want faster broad discovery after opting a tenant
in.

### 5b. A sync cron that is GREEN on every run but never actually ran (shared-lock starvation) — 2026-08-06

The nastiest shape, because every dashboard says healthy. Symptom as reported by
a user: "this AWS account was created in July, why did Helix only pick it up in
August — and only when I clicked sync manually?"

`cmdb.scopes_sync`, `cmdb.roles_sync` and `cmdb.grants_sync` all serialize on one
`cloud_sync:{integration_id}` Redis lock, but they are **three separate cron rows
scheduled minutes apart**. If the slowest sibling overruns the gap, the next one
finds the lock held. Before 2026-08-06 that path returned
`{"status": "skipped", "reason": "another sync is already running"}` — and a
handler that RETURNS instead of raising is recorded `status='ok'`, so neither the
failure notifier nor the admin UI ever showed a problem.

Production: `cmdb.roles.aws-primary` starts 04:09 and takes 731s avg / 882s max;
`cmdb.scopes.aws-primary` started 04:18. Result: **29 of 31 consecutive runs were
lock-skips**, so newly created AWS sub-accounts were invisible to Helix for over a
month.

**Detect** (works for any integration — this is the query that found it):

```sql
SELECT j.name,
       COUNT(*) AS total,
       SUM(CAST(r.result_summary AS CHAR) LIKE '%another sync is already running%') AS lock_skipped
FROM scheduled_job_runs r JOIN scheduled_jobs j ON j.id = r.job_id
WHERE r.started_at >= NOW() - INTERVAL 30 DAY
  AND j.handler_key IN ('cmdb.scopes_sync','cmdb.roles_sync','cmdb.grants_sync')
GROUP BY j.name HAVING lock_skipped > 0 ORDER BY lock_skipped DESC;
```

A handful of skips is normal (an admin "sync now" overlapping a cron). `lock_skipped`
approaching `total` means that tenant's sync is structurally starved.

**Fix**: post-2026-08-06 the handlers wait up to 900s for the lock and then RAISE,
so a starved run lands `failed` and pages instead of hiding. That is the safety
net — the real fix is spacing. Measure the siblings and re-space:

```sql
SELECT j.name, j.cron_expr, AVG(r.duration_ms)/1000 avg_s, MAX(r.duration_ms)/1000 max_s
FROM scheduled_jobs j LEFT JOIN scheduled_job_runs r ON r.job_id = j.id
WHERE j.name IN ('cmdb.scopes.<slug>','cmdb.roles.<slug>','cmdb.grants.<slug>')
GROUP BY j.name, j.cron_expr;
```

Then set each cron to start after the previous one's **max** duration plus margin.
(An earlier version of this step said to also "null `next_run_at` so the scheduler
recomputes" — that was always a no-op: `scheduled_jobs` never had a next-run cursor
the scheduler read, and the column is gone as of migration **620**. Editing
`cron_expr` is the whole change; the trigger is re-registered from it. Prefer
declaring "runs after X" as a chain — `trigger_type='chained'` + `chain_after_job_id`
— over hand-tuning gaps between cron minutes.) `cmdb.cost_facts.<slug>` uses
a *different* key (`cloud_sync:cost_facts:{id}`) and does not contend.

### 5a. A sync cron that is red on EVERY run, forever (scope-type mismatch) — 2026-07-30

Symptom shape: a `cmdb.*_sync` job fails every tick with a provider error naming something that is obviously **not** a tenant id, e.g.

```
cmdb.grants_sync int_sig=6310dc152f… partial_failure:
helix-aoai-20260729-5b25-pingcap-csp: RuntimeError: roleAssignments list 400:
{"error":{"code":"InvalidSubscriptionId","message":"The provided subscription
identifier 'helix-aoai-20260729-5b25-pingcap-csp' is malformed or invalid."}}
```

That string is an **Azure resource-group name**, not a subscription GUID. `cmdb_cloud_scopes` holds two kinds of row: tenant-level (`TENANT_LEVEL_SCOPE_TYPES` — `external_id` is a subscription GUID / account id / project id) and sub-tenant (today only `azure_resource_group`, one minted per approved AI-API-key request by the Azure OpenAI auto-grant flow, `external_id` = the RG name). Every driver call that puts `external_id` into a provider URL must be fed tenant-level rows only.

**This class of failure never self-heals.** Auto-grant-minted scope rows carry `last_seen_sync_job_id IS NULL`, which scopes_sync's mark-and-sweep deliberately excludes (correctly — they were not discovered by a sync). The bad row is permanent, so the cron stays red until the code is fixed. Do NOT wait it out, and do NOT delete the scope row — it is the cost-attribution anchor for that API key (a priority-10 `cmdb_cost_allocation_rules` row keys on its RG name, and `apikey_assignments.cloud_scope_id` FKs to it).

Triage:

```sql
-- Which rows in this integration are NOT tenant-level?
SELECT id, scope_type, external_id, display_name, status,
       last_seen_sync_job_id, owner_person_id
FROM cmdb_cloud_scopes
WHERE integration_id = ?           -- the int_sig in the error, with dashes restored
  AND deleted_at IS NULL
  AND scope_type NOT IN ('aws_account','azure_subscription','gcp_project',
                         'aliyun_account','standalone_iam','gcp_billing_account');
```

If the failing `external_id` shows up there, the fix is a code change in the **caller**: restrict that sync's scope list to `TENANT_LEVEL_SCOPE_TYPES` (the pattern all of grants / roles / billing_daily / cost_facts sync now follow), plus a driver-side `NotImplementedError` guard so the next caller that forgets gets a clean skip instead of a provider 400 recorded as a failure. Precedents: cost fan-out 2026-05-11, IAM fan-out 2026-07-30 (`sync_grants_for_integration` / `sync_roles_for_integration`, Azure `_require_subscription_scope`).

**The inverse symptom — a scope with NO discovered grants and an empty role dropdown, no error anywhere.** Same root cause seen from the other side: a scope row whose `scope_type` is not in `TENANT_LEVEL_SCOPE_TYPES` is skipped by all four fan-outs. Since 2026-07-30 both IAM syncs name what they skipped, so check the run summary before digging:

```sql
-- Last grants_sync summaries (same data the scheduled-jobs history UI
-- renders; `GET /cmdb/scopes/{id}/last-sync-summary` is the per-scope view):
SELECT r.started_at, r.status,
       JSON_EXTRACT(r.result_summary, '$.scopes_skipped_non_tenant') AS skipped
FROM scheduled_job_runs r
JOIN scheduled_jobs j ON j.id = r.job_id
WHERE j.handler_key = 'cmdb.grants_sync'
ORDER BY r.started_at DESC LIMIT 5;
```

A row listed there is skipped **by design** if it is an `azure_resource_group` (cost anchor, never IAM-addressable). Anything else — especially a typo like `azure_subscriptions`, or a bare provider name like `azure` — is a broken scope row from the era when `manual_provision` took `scope_type` as free-form text. Since 2026-07-30 scopes_sync **self-heals** it: the upsert refreshes `scope_type` from the driver, so one cron tick corrects the row. New manual provisions can no longer create one (`422 CLOUD_SCOPE_TYPE_INVALID`).

This actually happened, and it is the shape to recognise. Prod scope `97a6bfd7` carried `scope_type='azure'` while its `external_id` was a real subscription GUID (`7f070020-…`, `data-integration-team-internal-azure`), created 2026-07-08 by a hand-typed manual-provision. It had **921 roles + 7 live grants** and was syncing normally — until the IAM tenant-level filter shipped and silently dropped it. **The dangerous part was the roles sweep, not the skip**: 921 roles was 3% of that integration's 34,146-row catalog, i.e. *under* the 20% drop-tolerance floor, so the next roles_sync would have soft-deleted all 921 without tripping a single guard. To confirm the same shape isn't lurking:

```sql
-- Any scope row whose type is outside the known set (should be 0):
SELECT s.id, i.name, s.scope_type, s.external_id, s.status,
       (SELECT COUNT(*) FROM cmdb_cloud_roles r
          WHERE r.scope_id=s.id AND r.deleted_at IS NULL) roles,
       (SELECT COUNT(*) FROM cmdb_cloud_access_grants g
          WHERE g.scope_id=s.id AND g.status IN ('active','pending')) grants
FROM cmdb_cloud_scopes s JOIN integrations i ON i.id=s.integration_id
WHERE s.deleted_at IS NULL
  AND s.scope_type NOT IN ('aws_account','azure_subscription','gcp_project',
      'aliyun_account','standalone_iam','gcp_billing_account','azure_resource_group');
```

A GUID/account-id-shaped `external_id` with a non-canonical `scope_type` and a non-zero role/grant count is a live tenant scope wearing the wrong label — let scopes_sync heal it (run "Sync scopes" on the integration) rather than hand-editing, unless a sweep is imminent.

**Blast radius check while it is red**: grants_sync's drift mark-and-sweep is restricted to scopes whose driver call succeeded (`synced_scope_ids`), so a scope erroring out does NOT get its grants revoked — the cost of this bug is a permanently red cron plus the alert-fatigue it causes, not lost access. Confirm with the run summary's `drift_closed` and the `cloud.grant.drift_removed_sweep` audit rows before assuming otherwise.

---

## 6. Grant + revoke flow

### 6.0 Lifecycle timeline (audit-backed read surface)

Every state transition on a cloud scope-creation request OR an access
grant writes one `audit_logs` row. The timeline endpoints stitch
those rows into a per-record event stream — use them when a user (or
an owner, or you) asks "who applied for access to this subscription,
when, and what happened next."

**Endpoints:**

| URL | RBAC | Returns |
|-----|------|---------|
| `GET /api/v1/cmdb/scope-requests/{request_id}/timeline` | admin OR original requester | `submit / approve / approve_resume / cloud_created / provision / manual_provision / reject / withdraw / cancelled_via_feishu` |
| `GET /api/v1/cmdb/grants/{grant_id}/timeline` | admin / finance / grant holder / scope owner | `request / approve / reject / owner_add / renew / renew_reinstate / revoke / disable` |

**Response shape** (oldest-first):

```jsonc
{
  "data": [
    {
      "id": "...",                  // audit_logs row id
      "event_kind": "approved",     // stable i18n suffix (cloud.timeline.<domain>.<event_kind>)
      "action": "cloud.grant.approve",
      "status": "success",          // success | failure | skipped
      "occurred_at": "2026-05-18T...",
      "actor_user_id": "...",
      "actor_label": "Alice Chen · alice@example.com",
      "actor_type": "user",         // user | system | integration
      "trace_id": "...",
      "detail": { "ttl_days": 90 }  // operator-internal keys redacted
    }
  ]
}
```

**Source of truth.** This is purely a read surface on `audit_logs`.
Nothing dual-writes. The whitelist of actions surfaced is in
[`server/app/domains/cmdb/timeline_service.py`](../../server/app/domains/cmdb/timeline_service.py)
— internal retry / dispatcher noise (`revoke_attempt_failed`,
`revoke_retry`, `callback_ignored_state_advanced`) is intentionally
omitted. If you suspect events are missing, grep `audit_logs` by
`resource_id` directly:

```sql
SELECT created_at, action, status, actor_user_id, JSON_EXTRACT(detail, '$') AS detail
FROM audit_logs
WHERE resource_type IN ('cmdb_cloud_scope_request','cmdb_cloud_access_grant')
  AND resource_id = '<id>'
ORDER BY created_at;
```

**Performance.** Migration 180 (landed for the
`/api/v1/audit/resources/history` endpoint and reused by these
timeline routes) added `ix_audit_logs_resource_lookup` on
`(resource_type, resource_id, created_at)`. The lookup is a bounded
range scan; expect ms-level latency even on a multi-million-row
audit table.

**Frontend surfaces:**

* Scope-request detail drawer (`/my/cloud/scope-requests` row click,
  `/admin/cloud/scopes?tab=requests` row click) — timeline section
  auto-renders.
* "申请记录" tab on `MyCloudScopeDetailPage` and
  `CloudScopeDetailPage` — one row per grant on the scope, "view
  timeline" pivot opens the modal. This is the surface owners use to
  see who has applied for access to their subscription.
* Per-grant action column on the members tab's expanded sub-table
  and the HistoricalGrantsTab — "view timeline" button.

**MCP twins:** `get_cloud_scope_request_timeline(request_id)` and
`get_cloud_access_grant_timeline(grant_id)`. Same RBAC as HTTP.

---

### 6a. Employee requests access

1. Employee visits `/my/cloud/scope-requests` or `/my/cloud/scopes/{scopeId}` (if they know the scope) → "Request access".
2. Fills role (from `cmdb_cloud_roles WHERE is_promoted=true`) + reason + duration.
3. `POST /api/v1/cloud-lifecycle/grants/request` writes pending grant + opens approval flow if configured (Feishu) or notifies the scope owner.
4. Owner approves on `/admin/cloud/scopes/{id}` or via Feishu.
5. On approve: `CloudLifecycleService.grant_access()` → calls `driver.grant_access()`, sets grant `status='active'`, `expires_at = now + scope.default_grant_ttl_days` (or NULL if requester is `cloud_admin`), notifies requester via outbox.

**Production scopes are excluded from EVERY Helix grant flow
(added 2026-05-09; tightened in round-2 audit).** The block is
**symmetric across self-service and admin paths** so "self-service
never reaches prod" is a flat invariant that doesn't depend on the
caller's role:

* `GET /scopes?requestable=true` filters out
  `environment ∈ PROD_ENVIRONMENTS` (currently `{"prod"}`).
* `submit_access_request` raises `CLOUD_SCOPE_PROD_BLOCKED` (HTTP
  403) if a caller posts a prod scope_id directly (picker bypass via
  cached UI / direct curl / MCP tool).
* `owner_add_member` raises the **same** error code — admins who
  legitimately need to provision prod for someone go through the
  R&D security team's intake (point of contact: **Kidd.li**), not
  through any Helix endpoint. Both code paths call the same
  `_refuse_prod_self_service(scope)` helper so the two can never
  drift; the `test_owner_add_blocks_prod_scope_symmetric_with_submit`
  regression pins the parity.

The `/my/cloud/grants` modal renders an orange `Alert` banner
spelling the policy out so an employee doesn't waste a click. The
backend message is intentionally English-only — the frontend
catches `CLOUD_SCOPE_PROD_BLOCKED` and toasts the bilingual
`cloudLifecycle.myGrants.prodBlockTitle` i18n key, keeping the i18n
hard rule's single source of truth in the frontend bundles.

**Re-requesting after a prior revoke / expiry / disable** reuses
the existing `cmdb_cloud_access_grants` row in-place rather than
inserting a new one (added 2026-05-09 to fix a 500 — the unique
key `uq_cmdb_cloud_grants_scope_person_role` would otherwise reject
the INSERT). One shared helper
(`CloudLifecycleService._reuse_or_create_grant_row`) covers both
`submit_access_request` and `owner_add_member`, so the same
resurrection semantics apply regardless of which entry point
re-issued the grant. Behaviour:

* The row's `id` and `created_at` are preserved.
* Status flips back to `pending`; `granted_at` / `request_group_id`
  / `granted_by_person_id` are stamped fresh per the calling path.
* `detail` is **merged** rather than overwritten — the disable
  path's `disable_result` block (driver outcome + reason +
  timestamp) and the sync path's `last_error` stash survive so
  forensic reconstruction of "what happened to this grant between
  then and now" doesn't lose anything. New request fields
  (`reason`, `requested_ttl_days`, `primary_role_key`) layer on
  top.
* Resurrection markers (`_resurrected_from_status`,
  `_resurrected_from_revoke_reason`,
  `_resurrected_from_revoked_at`) are stamped in `detail` AND
  surfaced in the `cloud.grant.request` audit-row's
  `resurrected_grants` list, so a security auditor can answer "why
  is this `pending` now when last week it was `revoked`?" without
  correlating two audit rows by grant_id.
* Revoke-bookkeeping cleared:
  `revoked_at`/`revoked_by_person_id`/`revoke_reason`/`expires_at`/
  `last_renewed_at`/`last_notified_stage`/`last_login_at`/
  `employee_request_id`.
* **Source normalization**: a `source='discovered'` row that gets
  resurrected becomes Helix-issued going forward — `source` flips
  to `helix` and `last_seen_grants_sync_*` are cleared so the next
  drift sweep can't re-revoke a row now under Helix governance.
  Stale cloud-side metadata (`external_ref`,
  `external_principal_id`, `external_principal_email`) is also
  cleared so dispatch re-creates the cloud-side binding cleanly.
* **Concurrency**: the resurrection mutation is wrapped in the
  same Redis advisory lock (`grant_revoke:<grant_id>`) used by
  `revoke_grant` / `renew_grant`, so a concurrent admin revoke
  during a re-request serialises cleanly instead of producing a
  lost-update / split-brain. A `LockBusy` re-surfaces as a
  `ConflictError` (HTTP 409) with a "retry in a moment" message.
* **Race-after-lock guard**: under the lock, the helper re-checks
  `existing.status in GRANT_STATUSES_RESURRECTABLE` before the
  mutation. If a competing writer flipped the row to active /
  pending between the outer probe and the lock acquire, the
  helper raises `ConflictError` rather than silently overwriting
  the live grant.

### 6b. Renew an expiring grant — and restore an already-recycled one

**Renewal is owner-approved, not self-service.** The one-click
`POST /grants/{id}/renew` endpoint this section used to document was removed in #2098
(2026-07-30); anything still pointing at it is stale.

- **Still active, inside the reminder window** — employee opens `/my/cloud/grants`
  (`订阅` tab) → kebab → 「申请续期」 → `POST /scopes/{id}/renewal-requests`. Needs the
  scope Owner's approval — a single signature, at any length up to
  `GRANT_TTL_HARD_CAP_DAYS` (the >90-day cloud-admin countersignature was removed on
  2026-08-26). Renewal never SHORTENS an expiry (regression from 2026-07-15).
- **Already expired and recycled** — same endpoint, flagged `is_revival`. Reachable from
  the **`已结束` tab's action column** ("申请恢复访问"). Eligible only while
  `now - revoked_at <= RENEWAL_REVIVAL_MAX_AGE_DAYS` (**30 days**); past that the row shows
  `—` with reason `revival_window_expired` and the employee must file a fresh access
  request (which re-runs role governance). `status='revoked'` / `'disabled'` are never
  revivable — those were deliberate closures.
- **Why the revival path matters operationally**: after access is gone the holder usually
  cannot remember *which* of hundreds of accounts and roles they had. The `已结束` row
  already carries the account, role and end reason, so restoring from that row is the only
  path that does not require the employee to reconstruct it from memory. Until 2026-08-25
  the backend accepted revival but the tab rendered no button — if a user reports "there is
  no way to get my access back", check the UI before concluding the capability is missing.

**Reminder / notice cadence** (`sweep_expiring_grants`): T-14 / T-7 / T-1 reminders, then a
recycling notice at T+0 (the latter only exists since #2098 — grants recycled before
2026-07-30 got three warnings and then silence). Both cards render in the recipient's own
language via `resolve_lang(person)` since 2026-08-25; before that they were Chinese-only,
which is how a US-based holder lost access after three "delivered" warnings.

**Exempt from both passes** — do not treat these as bugs: the scope Owner's own grant
(`_is_owner_grant`), `prod` scopes (Helix does not manage production cloud access at all),
and perpetual-role holders (`cloud_admin` / `global_admin`, see 6d + 6g).

### 6c. Manual revoke

- Owner or admin on `/admin/cloud/scopes/{id}` clicks revoke on a grant.
- Calls `CloudLifecycleService.revoke_grant(grant, reason)` → driver revoke → `status='revoked'` + `revoked_at/by/reason`.

**Manual-revoke retry sweep (added 2026-05-16, pre-launch P1)** — when `revoke_grant` is called with `suppress_driver_error=True` (offboarding sweeps + the leaver-grant safety net) AND the driver call fails (5xx / throttle / IAM lock / propagation delay), the Helix row lands `status='revoked'` + `detail.requires_manual_cloud_revoke=true` while the cloud-side principal stays active. Pre-fix no cron retried — the flag was advisory. Now `cmdb.manual_revoke_retry_sweep` runs hourly at :15, picks up rows older than `min_age_seconds` (default 3600), re-invokes `driver.revoke_access`. To inspect:

```sql
-- Outstanding cloud-side residuals (waiting for the sweep to retry)
SELECT g.id, g.scope_id, p.work_email, i.slug,
       g.revoked_at, g.detail->>'$.last_error' AS last_error,
       g.detail->>'$.retry_succeeded_at' AS last_cleared_at
FROM cmdb_cloud_access_grants g
JOIN identity_persons p ON p.id = g.person_id
JOIN cmdb_cloud_scopes s ON s.id = g.scope_id
JOIN integrations i ON i.id = s.integration_id
WHERE g.status = 'revoked'
  AND JSON_EXTRACT(g.detail, '$.requires_manual_cloud_revoke') = true
ORDER BY g.revoked_at;
```

**Retry-storm cap + escalation (2026-06-10, execution-plane PR2)** — the sweep no longer re-hammers a structurally-broken revoke forever (the May-2026 Tencent `DetachUin` / UCloud account-not-exist storm produced 463 failed attempts over ~3 weeks with no page until the 7-day aging incident). Each failed retry increments `detail.manual_revoke_retry_count`; at **12 consecutive failures (≈ half a day of hourly ticks)** the grant ESCALATES once:

- one notification to every `cloud_admin` + `global_admin` (template `cloud.grant.manual_revoke_escalated`, deep-link to the scope page),
- audit row `cloud.grant.manual_revoke_escalated`,
- `detail.manual_revoke_escalated_at` stamped → the sweep's discovery query **skips the grant** from then on (`summary.escalated_pending` counts the parked backlog per tick).

The resume path is the **「重试云侧回收」 button** on the scope detail page's 已结束 tab (rows badge 云侧未回收 / 云侧未回收·已升级; `can_manage_cloud` only — `POST /cmdb/grants/{id}/retry-cloud-revoke`). It resets the counter + stamp (re-arming 12 fresh automated attempts), runs one immediate driver attempt under the `grant_revoke:{id}` lock, and reports `{cleared, last_error}`. The >7d aging compliance incident fires independently of escalation (escalated rows still match its query) — escalation is the fast page, the incident is the compliance backstop.

If a grant stays stuck (escalated, or just repeatedly failing):
1. Check the audit trail: `SELECT * FROM audit_logs WHERE resource_id=? AND action IN ('cloud.grant.revoke', 'cloud.grant.revoke_retry', 'cloud.grant.manual_revoke_escalated') ORDER BY created_at DESC;` to see the latest cloud error.
2. Fix the cloud side in the provider console, then click 「重试云侧回收」 on the 已结束 tab (preferred over raw SQL — it audits, resets the storm counter, and confirms convergence via the driver's `already_absent`).
3. If the error indicates a permanent failure with no cloud side left to fix (account closed cloud-side, principal already gone), the same button usually converges it too — the driver reports `already_absent` and the flag clears. Raw-SQL closing (`UPDATE cmdb_cloud_access_grants SET detail=JSON_SET(detail,'$.requires_manual_cloud_revoke',false,'$.manual_close_reason','<your reason>') WHERE id=?;`) remains the last resort.
4. For genuinely retry-worthy failures (IAM throttle window, intermittent 5xx), let the cron continue — it has 12 attempts before parking.

The `min_age_seconds` knob can be tightened to 60 for an urgent flush (`UPDATE scheduled_jobs SET config=JSON_OBJECT('min_age_seconds', 60) WHERE name='cmdb.manual_revoke_retry_sweep';` then trigger a one-off run via the admin "立即执行" button) — the 1 h default exists only to avoid racing the original revoke txn's commit and to give IAM/STS propagation time to settle.

### 6d. Perpetual administrator access

Both `cloud_admin` and `global_admin` are perpetual cloud-access roles. `global_admin` is already granted through the RBAC console; use `cloud_admin` sparingly for SRE/DevOps users who need permanent cloud authority without global platform administration.

1. Confirm the person exists: `SELECT id FROM identity_persons WHERE work_email=?;`
2. Ensure the role exists: `SELECT id FROM auth_roles WHERE name='cloud_admin';` (seeded in [`080_revive_cloud_admin_role.py`](../../server/alembic/versions/080_revive_cloud_admin_role.py)).
3. Assign: `INSERT INTO auth_user_roles (user_id, role_id, granted_by, granted_at) VALUES (?, ?, ?, NOW());`
4. Existing active grants are normalized automatically by the expiry sweep; migration 440 performs the same repair for global admins at deploy time. Emergency manual repair remains:
   ```sql
   UPDATE cmdb_cloud_access_grants SET expires_at=NULL, last_notified_stage=NULL
   WHERE person_id=? AND status='active';
   ```
   Future grants automatically use NULL for either role.

`cloud_admin` does NOT grant the right to mutate integrations or run cloud sync — that's `global_admin` (`can_manage_cloud`). It DOES grant the right to decommission scopes (`can_manage_cloud_scope_lifecycle`).

### 6g. `expires_at IS NULL` — the two reasons, and the one that expires (added 2026-08-18)

NULL is legitimate for exactly **two** populations. Anything else holding NULL
is invisible to both `sweep_expiring_grants` passes (they key on
`expires_at IS NOT NULL`) — a permanent grant nobody revisits:

| Holder | NULL legitimate? | On an ownership transfer |
|--------|------------------|--------------------------|
| `cloud_admin` / `global_admin` | yes — perpetual by ROLE (§6d) | stays NULL |
| The scope's **current** owner | yes — the owner always has access | — |
| The **previous** owner (backfill wrote it with `force_perpetual`) | **no** — the reason was "they are the owner" | re-stamped with a fresh TTL |
| Anyone else | **no** | re-stamped |

Two mechanisms keep this true, and both decide the exemption by the **role**,
never by the column being NULL:

* `_disarm_owner_handover_trap` — at the transfer, skips the outgoing owner
  only when they are in `load_perpetual_person_ids`.
* `access_expiry_sweep` **pass 0b** — nightly self-heal for rows already in the
  bad state. Stamps a TTL (never revokes) so the holder gets a full
  T-14 / T-7 / T-1 runway. Summary key `unreasoned_null_expiry_stamped`;
  **steady state is 0** — a non-zero value on a normal run means some writer is
  producing unreasoned NULLs again. **Prod scopes are excluded** (Helix does
  not manage prod cloud access at all).

Audit tell for the historical bug: `cloud.scope.assign_owner` rows carrying
`"prev_owner_grants_restamped": 0` on a scope where the outgoing owner did hold
grants. Production 2026-08-18: 7 such rows, all from the 2026-06-02
auto-owner-backfill batch.

---

### 6f. A certificate whose probe target vanished (added 2026-08-18)

`build_probe_targets` only takes DNS records with `status='enable'`, so a record
disabled or removed at the provider correctly leaves the probe target set — the
prober is NOT broken. The certificate row then stops being touched, and because
`status` is only written at probe time it freezes at whatever the last probe saw.

`cmdb.cert_probe_sweep` now flags these: `last_error` reads *"probe target no
longer in DNS…"* and the run summary carries `orphaned_targets`. Triage:

1. `SELECT probe_host, status, not_after, last_probed_at, last_error FROM
   cmdb_certificates WHERE last_error LIKE 'probe target no longer%';`
2. For each host, check the live DNS record —
   `SELECT name, zone_name, status, deleted_at, integration_id FROM
   cmdb_dns_records WHERE CONCAT(name,'.',zone_name) = ?;` Look at the row from
   the **live** DNS integration (腾讯云 DNS / DNSPod); the 腾讯云 and UCloud rows
   are soft-deleted legacy copies from retired integrations.
3. Then either **re-enable** the DNS record (the disable was a mistake — the
   sweep picks it back up the next night and clears `last_error` itself), or
   **retire the certificate row** (the endpoint is genuinely gone).

The sweep never changes `status` / `not_after` for these rows: which of the two
applies is a human call. Reading the row is safe — `status` is re-derived from
`not_after` on read (see `cert_service.effective_cert_status`) and the API
returns `status_stale: true` for it.

---

### 6e. How the health score reads grant policy (added 2026-08-18)

The Intelligence Center's 云授权治理覆盖率 does NOT count "grants with a future
`expires_at`". It scores conformance to the policy this section describes, so
the three legitimate exemptions are not deductions:

| State | Scored as | Why |
|-------|-----------|-----|
| `expires_at IS NULL` on a `cloud_admin` / `global_admin` | governed | §6d — perpetual by policy, via `load_perpetual_person_ids` |
| Scope owner's grant on their own scope | governed | `_is_owner_grant` — exempt from both expiry sweeps |
| Any grant on an `environment='prod'` scope | out of population | §8 / prod is owned by the R&D security team; both sweeps skip it |

Prod stays visible as `prod_grant_count` + `prod_overdue_grant_count` in the
dimension's aggregate facts (the same read-only counters admin-overview shows) —
excluded from scoring, never hidden.

So a governance deduction means one of exactly two things, both closable:
the holder is no longer active, or a managed grant has no valid expiry and no
policy reason to be perpetual. On 2026-08-18 that was 9 rows out of 1444
managed grants (99.38%).

---

## 7. Decommission a sub-account

This is **irreversible**. The cloud-side account is closed via the provider's account-close API; there is no reactivate.

1. **Confirm prerequisites**:
   - Scope has $0 spend in the last 30 days OR you have explicit approval to close anyway.
   - All grants have been audited — the close will revoke them but there's no take-back.
   - If the scope has IAM users (standalone_iam mode), they get `revoked_at` set automatically.
2. **Caller must hold** `global_admin` OR `cloud_admin` (`can_manage_cloud_scope_lifecycle`).
3. **Call the API**:
   ```
   POST /api/v1/cmdb/cloud-scopes/{scope_id}/decommission
   { "reason": "Project X wound down 2026-04. Cost zero past 60d. Approved by <person>." }
   ```
   Reason is ≤500 chars. Be specific — this is the audit trail.
4. **Backend flow**:
   - Calls `driver.close_scope_on_provider(scope)`.
   - Driver returns `CloseScopeResult(ok, provider_state, error)`. Per-provider `provider_state`:
     - AWS: `pending_closure` (90-day grace) or immediate `cancelled` for member accounts.
     - Azure: `disable_requested` (then `enabled=false` after 24h).
     - GCP: `delete_requested` (30-day grace; reversible during grace by `projects.undelete`).
     - Aliyun: `deleted` (deleted outright) or `deletion_silence_period` (Aliyun parks the member in its silence period, then deletes). See the Aliyun note below — it may take two clicks.
     - Tencent / UCloud / KSyun: no-op (cloud doesn't support API close — manual console action required, Helix only marks the scope closed in CMDB).
   - On `ok=true`: atomic update — `status='closed'`, `decommissioned_at=NOW()`, `decommissioned_by=admin_id`, `decommission_reason`, `decommission_provider_state=<from result>`. All active grants → `revoked` with `revoke_reason='offboarding'`.
   - On `ok=false`: error logged, no state change. Retry after fixing the cause (most common: insufficient creds for the close API).
5. **Verify cloud-side**: log into the cloud console, confirm the account is in the expected closed/cancelled state.
6. **You do not have to remember to come back.** `cmdb.decommission_reconcile` (04:20 daily, added 2026-09-01) re-reads the provider for every scope that is `closed` with a non-terminal `decommission_provider_state` and writes back what the cloud actually says — see the note below.

> **The admin's job ends at the confirm dialog (2026-09-02)** — one click is the whole interaction. Helix abandons every abandonable pre-check item so the deletion can actually proceed (withholding them is what killed prod account 5995487234252122 at `CheckFailed`), re-submits automatically if the provider lets the closure die (3 attempts, counted from `cloud.scope.decommission_auto_retry` audit rows), and notifies the person who clicked once the cloud confirms the account is really gone. **Do not tell an admin to re-click a failed closure** — if you are looking at one, the reconciler already owns it. The two states that genuinely need a human are `closure_stalled` (retries spent) and `active` (someone reversed the closure), and both alert on their own.
>
> **「订阅里还有资源」现在也是 Helix 的活,不是管理员的活(2026-09-07)** — Azure 拒绝取消还有资源的订阅(`400 ResourcesExistingOnSubscription`),GCP 拒绝删除挂着 lien 的项目。此前这两条都会变成一段带 `az group delete` / `gcloud … liens list` 的红色弹窗甩回给刚刚要求下线的人。现在:
>
> 1. 打开「下线」弹窗时先跑一次只读预检(`GET /cmdb/scopes/{id}/decommission-preflight`),把**会被一并删除的东西**列出来 —— 资源组 + 里面的资源名 + 管理锁(Azure),或 liens 及其 origin(GCP)。
> 2. 管理员确认一次(理由 + 输入 scope id,和以前一样),提交时把看到的 key 一起带上。
> 3. Helix 删锁 → 删资源组 → 重试关闭。云上删除是异步的,所以页面进入**「下线中 · 正在清理资源」**(`decommission_pending_since` 非空而 `status` 仍是 `active` —— 账号确实还活着),`cmdb.decommission_cleanup_resume`(`*/5`)推进到云端接受为止,然后给点击的人发一条完成通知。
>
> **排查一个卡住的下线**:`decommission_cleanup_state` 只有四个值。`cleaning` = Helix 还在干,**不要碰**;`needs_reconfirm` = 确认之后有人在这个 scope 里新建了东西,Helix 拒绝删没被确认过的东西 —— 让管理员重新点一次下线(弹窗会列出最新清单);`blocked` = 有 Helix 删不掉的东西(最常见是权限不够的 `CanNotDelete` 锁 —— 删锁需要 Owner / User Access Administrator,Contributor 不够),云控制台处理掉再回来;`stalled` = 72 次重试(≈6 小时)用尽。后三个都**已经通知过点击的人一次**并已离开重试队列,不会每 5 分钟再报一次。
>
> ```sql
> SELECT external_id, display_name, decommission_pending_since,
>        decommission_cleanup_state, decommission_cleanup_attempts,
>        decommission_cleanup_error, decommission_cleanup_keys
>   FROM cmdb_cloud_scopes
>  WHERE decommission_pending_since IS NOT NULL
>  ORDER BY decommission_pending_since;
> ```
>
> **不要手工去云控制台删「Helix 应该删的那些」来赶进度** —— 删完了 Helix 下一个 tick 会看到 blocker 空了并正常收尾,所以这么做是无害的;但反过来,**在一次下线在飞的时候往这个 scope 里新建资源**会把它打到 `needs_reconfirm` 并停下来。
>
> **$0 花费不等于订阅是空的。** 2026-09-07 的实例:`it-team-us` 本月花费 $0.00、下线理由填的「从未使用过」,而订阅里有一个活着的存储账号 `billingexport123` —— 一个几乎空的存储账号一个月就是几分钱。预检的第二个作用就是在决定还没落下之前反驳这个前提。
>
> **四家云还没有预检**(AWS / 阿里云 / 腾讯 / UCloud):弹窗会如实说「这家云不支持下线前预检」,而不是显示一个空清单 —— 空清单会被读成「里面是空的」。这四家的下线行为与 2026-09-07 之前完全一致。

> What Helix waved through is on the record: the decommission audit row's `close_detail.abandoned_check_ids` / `abandoned_checks` lists every Aliyun check item it passed on the admin's behalf, and each retry writes its own audit row naming the failure that prompted it.

> **A submitted closure is not a finished closure — the daily reconciler (2026-09-01)** — every cloud finishes asynchronously (AWS ~90d `PENDING_CLOSURE`, GCP ~30d `DELETE_REQUESTED`, Azure `Warned` → `Disabled` → ~90d `Deleted`, Aliyun check → silence period) and Helix writes `status='closed'` at submission time. `cmdb.decommission_reconcile` re-reads the provider each night and advances `decommission_provider_state`; when the closure **completes** it writes a terminal state + a `cloud.scope.decommission_confirmed` audit row and stops checking.
>
> **Two findings page you instead of resolving themselves**, both arriving as one aggregated Feishu/email alert to global_admin / it_admin plus a `cloud.scope.decommission_contradicted` audit row:
>
> * `deletion_failed` — the provider says the closure died (Aliyun `CheckFailed` / `DeleteFailed`). **Nothing retries it.** Clear whatever the alert names (the blocking items come straight from Aliyun's `FailReasonList`) and run 下线 again; the close path is resumable and will pick up from a fresh pre-check.
> * `active` — the provider says the scope is **usable again**. `projects.undelete` (GCP) and Azure subscription re-activation are self-service for the entire grace window and never tell Helix. This is the dangerous one: Helix already disabled every grant on that scope and every sweep skips closed scopes, so the account is now live with **no owner, no grants, no idle reminder, and no cost attribution**. Decide explicitly — close it again, or reopen it properly (re-run scope sync, restore an owner, re-issue grants).
>
> **Helix will not decide either of those for you.** The reconciler never flips `status` back and never re-issues a close: reversing an irreversible admin action on the strength of one provider read is a bigger call than the one being corrected. It also never reports a failed read as a closure — an unreadable provider leaves the row untouched and in the queue.
>
> Find everything currently in flight or contradicted:
> ```sql
> SELECT i.provider, s.external_id, s.display_name,
>        s.decommission_provider_state, s.decommissioned_at
> FROM cmdb_cloud_scopes s JOIN integrations i ON i.id = s.integration_id
> WHERE s.status = 'closed' AND s.deleted_at IS NULL
>   AND s.decommission_provider_state IS NOT NULL
>   AND s.decommission_provider_state NOT IN (
>     'deleted','deletesuccess','suspended','not_found','not_in_org',
>     'not_in_directory','removed','removesuccess','removed_from_org',
>     'users_evacuated','manual_console_close','closure_stalled')
> ORDER BY s.decommissioned_at;
> ```
> Rows at `active` or `closure_stalled` are **out** of the reconcile queue on purpose — they have been alerted once and a human owns them; re-alerting nightly is how an alert becomes noise. Re-run the job on demand from `/admin/scheduled-jobs` (`cmdb.decommission_reconcile` → 立即运行) rather than waiting for 04:20.

> **Azure `Warned` is a closure in flight, not a re-activated subscription (fixed 2026-09-02)** — an accepted `Microsoft.Subscription/cancel` parks the subscription in `Warned` before `Disabled`, and it can sit there for hours. The driver used to read `Warned` as `alive`, which was wrong twice over: it fired the "someone re-activated this" alert on a closure that was working, and — because `active` is a terminal provider state — retired the subscription from the reconcile queue, so its real `Disabled` → `Deleted` completion would never have been confirmed. `Warned` is now `pending` and shows as `取消已受理（转禁用中）` in 「已下线」. **Only `Enabled` / `PastDue` raise the alive alarm**; portal re-activation returns a subscription to `Enabled`, never to `Warned`, so nothing real was traded away.
>
> Measured on `918de8c4-…` (qa-efficiency), closed 2026-09-02 02:16 UTC: Helix's own pre-close snapshot read `Enabled`, ARM's activity log recorded `Microsoft.Subscription/cancel/action` **Succeeded** at 02:16:16, and the subscription still read `Warned` with 0 resources 48 minutes later. Confirm any Azure closure the same way — the activity log is the authoritative record, the `state` field alone cannot tell you whether a cancel was accepted:
> ```
> GET https://management.azure.com/subscriptions/<id>/providers/Microsoft.Insights/
>     eventtypes/management/values?api-version=2015-04-01
>     &$filter=eventTimestamp ge '<ISO8601>'
> # look for operationName.value == 'Microsoft.Subscription/cancel/action'
> # with status.value == 'Succeeded'
> ```
> (That ARM REST call is the one actually used to confirm this — the Helix Azure SP already holds the read it needs. `az monitor activity-log list --subscription <id> --offset 6h` reaches the same data if you have the CLI logged in.)

> **Aliyun: 下线 may legitimately need two clicks, and it used to not close anything at all (2026-09-01)** — Aliyun's `DeleteAccount` only accepts Resource Directory members typed `ResourceAccount`. A member typed `CloudAccount` is now switched back first via `UpdateAccount` (**this disables that member's root login**), then deleted. Aliyun then forces a deletion pre-check that its own docs put at **3–5 minutes**; Helix polls briefly and otherwise returns HTTP 409 `SCOPE_CLOSE_PRECHECK_RUNNING`. **That is not a failure** — nothing changed on the cloud or in Helix, and clicking 下线 again in a few minutes resumes from the finished check (it is not restarted). If the member's legal entity (实名信息) differs from the RD master's — an invited third-party account — Aliyun refuses the switch and Helix refuses the decommission with `foreign_legal_entity`; that account can only be cancelled (账号注销) by its holder from its own Aliyun console.
>
> **What this replaced, and what to go re-check.** Until 2026-09-01 a `CloudAccount` member took a `RemoveCloudAccount` fallback that only *unlinked* it from the Resource Directory: the account stayed alive, stayed billable, dropped out of `ListAccounts` (so Helix lost cost tracking, owner, and idle-sweep on it) — while Helix marked the scope `closed`, cascade-disabled its grants, and filed it under 「已下线」. **Any scope whose `decommission_provider_state = 'removed_from_org'` is one of these**: the cloud account behind it is almost certainly still running. Find them and reconcile against the Aliyun console:
> ```sql
> SELECT s.external_id, s.display_name, s.decommissioned_at, s.decommission_reason
> FROM cmdb_cloud_scopes s JOIN integrations i ON i.id = s.integration_id
> WHERE s.decommission_provider_state = 'removed_from_org'
>   AND i.provider IN ('aliyun', 'alicloud_intl');
> ```
> As of 2026-09-01 that query returns 0 rows on prod — the bug was found before anyone used the path — but it is the query to run if an Aliyun account shows unexplained spend after having been "closed" in Helix.

> **Resurrection guard (2026-05-21)** — `sync_scopes_for_integration`'s `on_duplicate_key_update` will NOT re-clear `deleted_at` when `decommissioned_at IS NOT NULL`. Pre-fix, a driver re-listing of a decommissioned scope cleared the soft-delete tombstone while `status` / `decommissioned_at` stayed (the deliberate "preserve admin-set status" rule). The half-revived row then leaked back into the 订阅列表 tab. Real incident: 10 GCP org-root projects decommissioned on 2026-05-18 reappeared on the 2026-05-19 20:24 sync. The code fix gates the un-soft-delete on `decommissioned_at IS NULL`; migration 196 normalized the legacy `status='decommissioned'` literal to canonical `'closed'` for 1018 GCP rows so the default list filter excludes them going forward. Operator implication: a decommissioned scope's `deleted_at` survives every future sync — you do NOT need to manually re-soft-delete after a routine sync run.

7. **For Tencent / UCloud / KSyun manual close**: do the cloud-console action FIRST, then mark closed in Helix:
   ```sql
   UPDATE cmdb_cloud_scopes SET status='closed', decommissioned_at=NOW(),
     decommissioned_by=?, decommission_reason=?, decommission_provider_state='manual_console_close'
   WHERE id=?;
   ```
   And revoke grants:
   ```sql
   UPDATE cmdb_cloud_access_grants SET status='revoked', revoked_at=NOW(),
     revoked_by_person_id=?, revoke_reason='offboarding' WHERE scope_id=? AND status='active';
   ```

---

## 8. Revoke cloud access for a leaving employee

> **Round-14 update (2026-05-08)**: `cloud_access_grant` blockers now carry `severity='advisory'` (migration 136). They no longer surface in the IT 「需 IT 介入」 red queue — they are listed under 「系统将自动处理」, since `ProcessAdvanceService._handle_one_blocker` calls `CloudLifecycleService.revoke_grant` automatically. IT can still revoke early via the offboarding admin page (same button) — it is informational only, not a gate. Operationally this means the dashboard's "blocked" stat no longer surfaces leavers whose ONLY outstanding work is auto-revoke; those land in "等系统自动处理".
>
> **Round-58 update (2026-05-30)**: `cloud_access_grant` is now part of `IMMEDIATE_REVOKE_RESOURCE_TYPES` and is revoked on the **first** advance tick that sees the leaver, **NOT** at the T+0+24h grace boundary. Trigger: P0788 / huangmeng@pingcap.com walk-through — both leavers had `cloud_access_grant` pending for hours because the grace window's "wait 24h for human action" rule was applied indistinguishable from physical-handover blockers (hardware / network ownership). IT ended up manually clicking revoke on huangmeng's grant at T+0 03:29 UTC just to break the deadlock. Round-58 carved out the 4 permission-grant types (`sso_grant` / `vpn_application` / `ai_tool_application` / `cloud_access_grant`) from the grace hold so they revoke immediately. The 24h grace push still fires for the leaver's remaining physical-handover blockers and surfaces the just-revoked permission grants in a separate "本次已立即回收" section so IT sees the full picture. See [`docs/architecture/PLATFORM.md`](../architecture/PLATFORM.md) §12.6 T+0 grace window for the partition rationale.
>
> **2026-06-02 — force-bypass-grace admin endpoint**. New `POST /api/v1/leaver-blocked/processes/{id}/force-bypass-grace` collapses the T+0 24h IT grace window to zero so physical force-transfer + cloud-revoke fire NOW. Stricter RBAC than `/advance-now`: `global_admin` / `it_admin` only — HR is rejected by `can_force_bypass_leaver_grace` because the grace window exists for IT. Required `reason` (≥ 30 chars) recorded in audit (`leaver_blocked.force_bypass_grace_advance`). **Canonical case**: HRIS-late-detection — termination_date weeks past, all T-N reminders skipped, IT has already drained the leaver's resources manually, no value in waiting another 24h. Forensic verification: `SELECT created_at, detail FROM audit_logs WHERE action='leaver_blocked.force_bypass_grace_advance' ORDER BY created_at DESC` returns one row per bypass with the actor_user_id + reason + summary. MCP twin: `leaver_blocked_force_bypass_grace` (mutates=True, global_admin only).

This runs automatically as part of offboarding (the offboarding scanner adds `cloud_access_grant` blockers + the lifecycle service revokes when offboarding advances). For manual / out-of-band cases:

1. **Find all active grants for the person**:
   ```sql
   SELECT g.id, g.scope_id, g.role_key, s.display_name, i.slug
   FROM cmdb_cloud_access_grants g
   JOIN cmdb_cloud_scopes s ON s.id=g.scope_id
   JOIN integrations i ON i.id=s.integration_id
   WHERE g.person_id=? AND g.status IN ('active','pending');
   ```
2. **Bulk revoke**: `POST /api/v1/cloud-lifecycle/persons/{person_id}/revoke-all`. Backend iterates, calls `revoke_grant` for each. Failures don't block remaining (per-grant transaction).
3. **Verify cloud-side**: next `cmdb.grants_sync` run (05:00 daily) will confirm. For urgent cases, run `POST /api/v1/cmdb/sync?integration_id=...&capability=grants` for each affected tenant.
4. **Standalone_iam clouds** (Tencent / UCloud / AWS SG): the IAM user is deleted on the cloud side; `cmdb_iam_users.revoked_at` records the action.
   > **UCloud `AttachPoliciesToUser` / `DetachPoliciesFromUser` (2026-05-14 round 3, P0092)**: the prod-verified shape is `{UserName, PolicyURNs.0, Scope}` where `Scope` MUST be `"All"` for the tenant-wide attachments Helix manages (the other legal value is `"Specified"` + `ProjectID`, used for project-scoped attachments — not Helix's model). Per [UCloud's docs](https://docs.ucloud.cn/api/iam-api/detach_policies_from_user). Legacy code used `Scope="Account"` which is NOT a legal value; current API rejects it with RetCode=220 / 230.
   >
   > Investigation log:
   > * Pre-fix: `parts[1]` truncated the URN to `ucs:iam`; UCloud's URN validator rejected it and returned `RetCode=220 Missing params [UserName or PolicyURNs or Scope]`. The error listed three fields generically; only PolicyURNs was actually bad.
   > * PR #295 round 1: fixed URN parse AND added `Scope="Account"` reading the literal RetCode=220 text → `RetCode=230 Params [Invalid Scope] not available`.
   > * PR #296 round 2: removed Scope entirely → regressed to RetCode=220 because Scope IS mandatory.
   > * PR #297 round 3 (this fix): per docs, `Scope="All"` is the correct value for tenant-wide. Applied symmetrically to Attach + Detach + revoke_account chain.
   >
   > **Diagnosis playbook**: if you see UCloud Attach or Detach errors, the params must include `Scope="All"`. RetCode=220 listing UserName/PolicyURNs/Scope generically usually means a malformed PolicyURN (check the URN against `ListPolicies` output) or a missing Scope. RetCode=230 "Invalid Scope" means the Scope value is wrong (`"Account"` is the historical wrong value; legal values are `"All"` and `"Specified"`).

   > **AliCloud International CloudSSO — `InvalidParameter.PolicyName.InvalidChars` (2026-05-14)**: CloudSSO assignments carry `external_ref="sso::{access_config_id}::{user_id}::{target_account_id}"` (or `…::group::{group_id}` for group-expanded synthetic rows). Pre-fix the intl driver inherited the RAM-based parent `revoke_access` which parsed `sso` as `UserName` + `:<rest>` as PolicyName, causing the InvalidChars error. `AlicloudIntlCloudDriver.revoke_access` now overrides: direct User refs → CloudSSO `DeleteAccessAssignment` (with `DeprovisionStrategy=DeprovisionForLastAccessAssignmentOnAccount`); group-expanded refs → `RemoveUserFromGroup` (per-person, doesn't affect other group members); missing SSO directory + `AccessAssignmentNotExist` → idempotent success. Non-`sso::` refs still fall through to the RAM parent. Parent has a defensive guard that surfaces a clear routing error if an `sso::` ref ever reaches it (driver mis-registration).

### 8a. IAM-user cleanup → cascade-revoke grants (added 2026-05-05)

`POST /cmdb/iam-users/{id}/cleanup` (disown branch) cascades synchronously, in the same transaction, to every active grant attributed to that IAM user. Two-pass:

1. **Discovered grants** matched by `external_ref LIKE 'iam-user:{user_name}::%'`. The prefix is the canonical FK back to the IAM user written by `_extract_iam_user_grants` at every upsert.
2. **Helix-issued grants** for the same person on the scopes those discovered rows live on. The IAM user was the auth handle that enacted those helix grants too — deleting the IAM user invalidates them in cloud reality, even if Helix is system-of-record for the row.

All cascades stamp `revoke_reason='iam_user_disowned'` (constant `GRANT_REVOKE_REASON_IAM_DISOWNED`) so forensics can distinguish from natural `user_released` / `inactive` / `drift_removed` revocations. Cascade count + ID list (truncated to 200) land in the `cmdb.iam_user.cleanup.disown` audit row's detail and in `cmdb_iam_users.raw_data._helix.cascade_revoked_grant_ids` for forensic queries.

**Why in-band cascade vs waiting for the next drift sweep.** Two pre-fix failure classes (user report 2026-05-05, AWS SG): (a) `source='helix'` rows are NEVER closed by the discovered drift sweep — Helix is system-of-record for those, so the sweep skips by design. Result: ghost helix grants forever for a deleted IAM user. (b) The drift sweep's 50%-safety-floor refuses to fire when a big batch of cleanups would close >50% of discovered grants. Mass IAM cleanup hits this floor and the discovered grants ALSO get stranded. The in-band cascade closes both in the same transaction as the IAM `DeleteUser` call so the post-cleanup picture is consistent without waiting for cron.

**Transfer mode is unaffected.** `transfer_to_person_id` set → driver is NOT called and grants are NOT cascade-revoked. The IAM user keeps working; only Helix-side ownership rebinds.

---

## 9. Cost watch

### 9a. Line-item cost facts (production-grade, migrations 139 + 140)

`cmdb.cost_facts.{slug}` (daily 04:xx SH, minute hashed from slug) — one cron per cloud integration, writes line-item cost rows to `cmdb_cost_facts` (one row per `date × resource × usage_type`). Drives `/admin/cloud/overview?tab=billing` resource排行 + multi-dim drilldown + chargeback + month-end forecast + IQR-based anomaly detection. Idempotent re-ingest via sha256(natural-key) `fact_key UNIQUE` (the natural key uses the cloud-side immutable `external_id` — Azure subId / GCP projectId / Aliyun member uid / AWS linked-account / Tencent ownerUin — NOT the mutable Helix `scope_db_id` UUID, so a re-ingest after `scopes_sync` catches up is a true upsert, not a duplicate). Backfill window via `config={"start_date":"YYYY-MM-DD","end_date":"YYYY-MM-DD"}`, or hit `POST /api/v1/cmdb/cost/backfill` directly. Drivers without `list_cost_facts` (UCloud, Kingsoft) return `not_supported` per scope — handler returns `ok` and the cron run is logged as success. Driver-load failures (credentials gone bad) and per-scope errors with zero upserts surface as `status=failed` / `partial_failure` and the handler raises so the failure-notifier fires.

Provisioning is owned by `scheduled_jobs_provisioning._build_lifecycle_rows` — the cost_facts row ships alongside `scopes` / `roles` / `grants` / `billing_daily` for every cloud integration. Worker boot reconciles missing rows; migration 140 also one-shot-seeds prod so the cron is live without a worker reboot.

**FX (CNY → USD)** — Aliyun + Tencent bill natively in CNY and need a rate to compute `amount_usd`. Single source of truth = `platform_settings.ai_spending.cny_to_usd_rate`, refreshed every Monday 05:00 UTC by `ai_spending.cny_rate_refresh` from frankfurter.dev (ECB-published, free, no auth). The same value also drives AI-spending USD totals — there's exactly one CNY/USD number across the platform.

* **Override per integration** (rare): set `config_public.cny_per_usd` on the cloud integration to pin a specific rate (e.g. finance wants a fixed monthly rate independent of the live ECB feed). The integration admin form has an optional "CNY→USD 汇率覆盖" field for this. Normally leave blank — platform value drives.
* **Stale-FX hard-fail** — a single transient frankfurter outage soft-skips (one missed run = 7 days, retry next Monday). If the stored rate is older than **14 days** (two missed runs) the cron raises and the failure-notifier alarms — silent staleness on a value driving every USD report would let dashboards drift unnoticed. Threshold = `_RATE_STALE_AFTER_DAYS` in [`ai_spending_handlers.py`](../../server/app/scheduled_jobs/ai_spending_handlers.py).
* **Manual override**: if frankfurter ever stops responding for >14 days, the cron will start firing failures. Set the rate manually on `/admin/ai-spending/settings` to silence; revisit the cron source once the upstream is fixed.
* **Resolution path** (in code) is [`app.core.fx_rate.resolve_cny_per_usd`](../../server/app/core/fx_rate.py): integration override → platform setting reciprocal → raise. Plausibility band on the platform side: 0 < USD-per-CNY < 10 (rejects hijacked / fat-fingered values).

### 9a-coverage. 一个订阅显示 $0 —— 先分清「真的没花钱」和「我们没取到账单」(2026-09-07)

抽屉 / 订阅详情 / 移动端 / MCP 的 `billing-detail` 响应里都带 `coverage` 块,它已经把这个判断做完了(实现与状态定义见 PLATFORM.md `cost_coverage` 段)。读它,别自己去猜:

| `coverage.status` | 含义 | 该做什么 |
|---|---|---|
| `complete` | 同云链路覆盖的每一天都有账单行 | 这个 $0 是真的,什么都不用做 |
| `gap` | 本订阅停在 `covered_through`,同集成已到 `expected_through`,且它是连续计费型 | **按下面的三步查**,总额被低估约 `estimated_missing_usd` |
| `pipeline_stale` | 整条链路滞后到不了所查月份 | 走 §5「stuck sync」,与单个订阅无关 |
| `no_history` / `indeterminate` | 从未计费 / 该订阅本来断续计费 | 判不了,别据此下结论 |

**`gap` 的三步定位**(2026-08-14 那次实战顺序,照抄):

1. **先确认不是整云问题** —— 同集成其它订阅的 `max(date)`:
   `select s.external_id, max(f.date) from cmdb_cost_facts f join cmdb_cloud_scopes s on s.id=f.scope_id where f.integration_id=:i and f.date>=:d group by 1 order by 2 desc`。形状会直接告诉你是一个订阅、一族订阅,还是整个云。
2. **去源头核**,不要停在库里的结论。GCP:拿 `config_public.billing_bq_table` 里的每张表查这个 project 的 `max(DATE(usage_start_time))`(带 `_PARTITIONTIME` 上下界,否则全表扫)。**库里断在哪天、源头也断在同一天 ⇒ 不是入库丢了,是当时就没拉到。**
3. **查这个订阅还在不在我们读的那个账单账号里**。GCP 一条命令:`gcloud billing projects describe <project>` —— 它给的 `billingAccountName` 如果不在 `billing_bq_table` 覆盖的账单账号里(表名后缀就是账单账号 ID),真因就是它。反过来,`billingEnabled: false` 说明计费真的关了,那个 0 是真的(`greg-kubernetes` / `dbaas-*` / `gcp-dm-dev` 属于这一类)。

**换了账单账号之后要做三件事**:GCP 侧为新账单账号开 BQ 导出(落同一个 dataset ⇒ SA 读权限现成)+ 给 SA `roles/billing.viewer`;把新表追加进 `billing_bq_table`;走 `execute_job` + `allow_past_window` 回填。**GCP 的 BQ 导出不回填历史** —— 从「挪账号」到「开导出」之间的那几天在 BQ 里永久缺失,只能从 Cloud Billing 报表或经销商发票另补。

**别把 `gap` 当断言读。** 「连续计费的订阅被真的关掉计费」和「导出表配漏了」在数据上形状完全相同(两条链路同源,没有第二个信源),所以判定只指方向,第 3 步那一条命令才是结论。

### 9a-azure-throttle. Azure Cost Management is throttle-paced — what "slow" and "429" mean here (added 2026-09-05)

**Baseline you should expect, not treat as an incident.** Azure's Cost Management Query API rate-limits our service principal hard enough that a 36-subscription Azure run is limiter-bound, not data-bound: on 2026-09-04 the worker logged **71 `Azure HTTP 429 list_cost_facts … sleeping 26-58s` lines in one day** (12 of them in the ordinary 20:36 UTC daily cron, the rest in the `cmdb.cost_facts_monthclose` fan-out). A 36-scope chunk taking 12-25 minutes is normal; roughly 80% of that is throttle sleep. Do **not** read a long Azure cost run as a hung job — check §5 (stuck sync) only if there are no `429 … sleeping` lines advancing.

**⚠️ Client-side pacing is OFF by default (`cost_pacing_enabled`, 2026-09-05).** It was enabled for a few hours on release day and the post-deploy verification tick **timed out at 1800s having reached only 27 of 36 subscriptions** (baseline 587-775s), with the pacing budget never exhausted — pacing was not preventing the 429s under that limiter, only spending headroom. Do not turn it on globally. To evaluate it: enable on ONE Azure integration, then compare a week of that job's `duration_ms` and `sudo docker logs compose-worker-1 | grep -c 'Azure HTTP 429'` against the baseline — and not within an hour of a manual run, which leaves the service principal's limiter hot for a long while.

**What the driver does about it (2026-09-05).** On by default: a 429 that carries **no** Retry-After header now backs off by the last cooldown the server *did* state this run (floor 20s) instead of restarting the exponential at 1s — the two 2s/8s rounds are what burned sub `7f070020`'s retry budget in the incident. Cumulative retry sleep per call is capped at 200s so a hard-throttled scope reports `429` rather than being killed mid-sleep by the 300s per-scope timeout. `AzureCloudDriver._cm_pacer` also *learns* cooldowns across scopes, but only **acts** on them (proactive waits) when `cost_pacing_enabled` is set — see the warning above.

**Reading an Azure 429 that did surface as `partial_failure`.** The error text now carries the evidence:

```
… status=429 throttle_headers=entity=45,clienttype=57 body='{"error":{"code":"429", …}}'
```

* `throttle_headers=none` → Azure gave no cooldown at all; the driver fell back to the last cooldown it learned this run.
* `entity=` → per-subscription limiter (that one sub is hot). `clienttype=` → our SP across the whole tenant. `tenant=` → tenant-wide, the most aggressive.
* Worker log lines tag each wait with `source=server` (a header said so) or `source=learned/backoff`.

**Operator action:** normally none — one throttled scope out of ~36 re-ingests cleanly on the next daily tick, and the window is a rolling 3 days, so nothing is lost. Act only if the SAME subscription fails on consecutive days (then it is not throttling — check credentials / subscription state per §5a), or if a whole run fails. A scope that burns its 200s cumulative throttle budget reports `status=429` rather than `status='timeout'`; that distinction is deliberate — `timeout` would mean a hung driver, `429` means Azure said no.

If a whole Azure cost run ever *slows down* after this change, look for `Azure Cost Management pacing budget spent` in the worker log — that line means the driver hit its 600s per-run pacing budget and reverted to reactive 429 handling, which is the safety valve doing its job, not a bug.

**Do not "fix" this by raising the retry budget or the per-scope timeout.** Retrying harder inside a closed window is what the pacer exists to stop; the budget is sized to stay inside `cost_service`'s 300s `_COST_FACTS_PER_SCOPE_TIMEOUT_SEC`. If Azure genuinely needs to go faster, the lever is `billing_source: exports` (Cost Exports, §9a) — a storage read with no query-API limiter at all.

### 9a-bedrock. AWS Bedrock cost attribution (2026-05-12)

Bedrock is unique among the cloud AI products: its on-demand `InvokeModel` has NO taggable billable resource (the foundation model is shared across customers, so resource-level user-defined tags don't reach Cost Explorer). AWS exposes IAM **principal tags** as a parallel cost-allocation surface — but the admin MUST activate the chosen tag key in AWS Billing Console first, and Helix MUST tag IAM users with that key at create time. Misalignment at any step silently drops every Bedrock dollar into the `(unallocated)` bucket.

**End-to-end flow** (autograny path):

1. **AWS Billing Console activation (admin one-time action, per AWS account)**: `Billing & Cost Management → Cost allocation tags → User-defined cost allocation tags → activate the key `Owner`**. Activation is forward-looking: cost from before activation stays `(unallocated)` in Cost Explorer; only future spend gets grouped by the tag. Cost-allocation-tag activation is a Billing-root-account-only operation (member accounts cannot toggle it).
2. **Helix-side ingestion-mode flip**: migration 142 seeds `platform_settings.cost_facts.aws_principal_tag.aws-sg = "Owner"` so the AWS driver's `_list_cost_facts_principal_tag_sync` path fires instead of the default per-resource path. The driver fans out per-(date × service × Owner-tag-value) by calling `GetCostAndUsage` with `GroupBy=[SERVICE, TAG:iamPrincipal/Owner]` (**NOT `aws:PrincipalTag/Owner`** — that's the IAM-policy evaluator namespace; CE silently returns empty owner values for it, collapsing every row into `(unallocated)`. The 2026-05-12 production audit found this confusion was the root cause of 100% of Bedrock cost landing unallocated even after admin had correctly activated the tag in Billing Console). Emits `resource_id="aws-principal:Owner=<value>|service=<svc>"`. Untagged spend lands in `Owner=(unallocated)`.
3. **IAM user tagging at AK grant time**: [`bedrock_autogrant.py`](../../server/app/domains/apikey/bedrock_autogrant.py) tags every minted IAM user with `Owner=<applicant_work_email>` (was `Owner=<project_name>` pre-PR-220 — wrong value, dropped per-employee chargeback granularity).
4. **Per-AK panel resolver**: [`apikey/spend.py::_resolve_bedrock_owner_tag_value`](../../server/app/domains/apikey/spend.py) reads the Owner value via a 3-step cascade: `cmdb_iam_users.tags['Owner']` → `IdentityPerson.work_email` → `parse_iam_user_name(arn)`. Matches `cost_facts.resource_id LIKE 'aws-principal:Owner=<value>|%'`.
5. **Allocation rule for cross-dashboard chargeback**: autogrant writes a priority-10 `CmdbCostAllocationRule` keyed on `match_tag_key="Owner", match_tag_value=<email>, assign_person_id=<applicant>` in the same durable commit as `granted=True`.

**Triage runbook** — "Bedrock AK shows $0 but I know it's spending":

| Symptom | Check | Fix |
|---|---|---|
| Panel shows `$0 / $X (0.0%) 正常`, no caveat | DB query: `SELECT resource_id FROM cmdb_cost_facts WHERE integration_id=<aws-sg id> AND service LIKE '%Bedrock%' AND date>=CURDATE()-INTERVAL 7 DAY LIMIT 10` — if every row is `Owner=(unallocated)`, AWS-side tag isn't activated OR IAM users aren't tagged | Activate `Owner` in AWS Billing Console (forward-only); for historic IAM users without `Owner` tag, run `scripts/backfill_bedrock_owner_tags.py --prod --dry-run` then drop `--dry-run` |
| Panel shows the unallocated-caveat banner | The Owner cascade resolved a value for THIS AK, but every fact in the integration is `Owner=(unallocated)` | Same as above — admin needs to activate in Billing Console |
| Panel shows `data_available=False` with no-Owner caveat | IAM binding exists but no Owner tag value resolvable | Run the backfill script; if the assignee has no work_email, fix HRIS first |
| Panel shows `data_available=False` with no-IAM caveat | `iam_user_arn` AND `iam_user_id` both NULL on the assignment | Re-run the autogrant via `/admin/api-keys` rebind, or manual fill |
| Detail page shows `—` (not `$0`) + `bedrock_owner_shared` caveat naming another key | Two+ live Bedrock keys of the same person share one `Owner` tag; the total is pinned to the earliest-created one (#1776). Confirm: `SELECT request_number, created_at FROM api_key_assignments WHERE assigned_to=<person> AND status IN ('active','approved','expired','pending_release')` | Expected, not a bug — the caveat links to the key carrying the number. For real per-key numbers, do the per-key tag activation below. Revoking a no-longer-used sibling also promotes the survivor to canonical |
| Real spend exists but allocation-rule chargeback is missing | `SELECT * FROM cmdb_cost_allocation_rules WHERE match_tag_value='<email>'` — empty | Run backfill script; it writes missing rules + stamps `detail.bedrock.allocation_rule_id` |

**Backfill script** ([`server/scripts/backfill_bedrock_owner_tags.py`](../../server/scripts/backfill_bedrock_owner_tags.py)) — idempotent; safe to re-run. Per AK: re-tags AWS IAM user, mirrors into `cmdb_iam_users.tags['Owner']`, writes missing `CmdbCostAllocationRule`, stamps `detail.bedrock.owner_tag_value` + `allocation_rule_id`. Always exercise `--dry-run` against prod first; the output enumerates every action per AK. Requires `HELIX_PROD_DB_URL_ASYNC` env var.

**Structural limitation — Owner tag is per-PERSON, not per-KEY (#1776, 2026-07-10)**: the `Owner` cost-allocation tag value is the applicant's `work_email`. AWS Cost Explorer therefore aggregates a person's ENTIRE Bedrock spend under one `Owner=<email>` row — it cannot split spend between two IAM users tagged with the same email. So when one person holds **>1 live Bedrock key** (a rare-but-real case: a second key issued before the first is revoked), a naive `Owner=<email>|%` match reports that person's *combined* Bedrock total on *each* key — an N× double-count in the fleet rollup (`analytics.py`'s `total_mtd`) and false budget breaches on every one of the keys.

- **Helix-side fix (shipped, live)**: [`apikey/spend.py::_bedrock_owner_canonical`](../../server/app/domains/apikey/spend.py) detects the shared-Owner collision among the person's spend-eligible Bedrock keys and pins the Owner total to a single deterministic **canonical key** (earliest-created; id tiebreak). The canonical key reports the real number; the siblings report `data_available=False` with a caveat naming the canonical key. This keeps the Owner total counted exactly once fleet-wide and stops the false breaches. No per-key isolation is claimed — because AWS genuinely cannot provide it from this data.
- **Forward-looking per-key isolation — LIVE IN PROD since 2026-07-29 (#2093, migration 435)**: the Helix side is done; per-key numbers appear as soon as AWS starts returning the tag. Steps 2–4 of the old plan are implemented, and the tag from step 1 already exists on every IAM user Helix mints — `bedrock_autogrant.py` has always written **`helix-request-number=<AK number>`** and **`helix-assignment-id=<uuid>`** (the old text here said `HelixAssignment=<assignment_id>`; that tag key was never real — verified against production `cmdb_iam_users.tags` for `bedrock-investibook-20260710-B5F7`).

  **As executed 2026-07-29** (kept as the runbook for any other Bedrock account):

  1. **AWS Billing Console** (management account of the Bedrock linked account): Billing & Cost Management → Cost allocation tags → User-defined → activate **`helix-request-number`**; it appears under the IAM-principal namespace as `iamPrincipal/helix-request-number`. **On `841162690310` this was ALREADY `Active`** — check before scheduling the work (`ce.list_cost_allocation_tags()` through the connector's own creds; `Owner`, `UsedBy`, `iamPrincipal/helix-assignment-id`, `iamPrincipal/helix-owner-email`, `iamPrincipal/helix-project-name`, `iamPrincipal/helix-request-number` were all Active). Everyone had assumed this step was pending for months; it was not. Note the account is its own payer — the ljq key's account `239023060546` is a MEMBER of org `o-dc5dyo0oct` (payer `317766989874`) and gets `AccessDeniedException: Linked account doesn't have access to cost allocation tags`, so use the SG connector creds, not ljq.
  2. **Verify CE really groups by it before flipping anything** — "Active in Billing" and "CE returns groups" are different facts. One `GetCostAndUsage` with `GroupBy=[SERVICE, TAG:iamPrincipal/helix-request-number]` filtered to the linked account, per month, showed real per-key values from **2026-05** onward and 100% `(unallocated)` for 2026-04 and earlier.
  3. **Helix setting**: extend the profile-axis setting from a string to a list —
     `platform_settings.cost_facts.aws_profile_tag.aws-sg = ["UsedBy", "iamPrincipal/helix-request-number"]`.
     The setting accepts a bare string (single axis, the pre-2026-07-29 shape) or a list (one CE fan-out per axis). `UsedBy` MUST stay first-class in the list — dropping it silently empties the per-AK detail page's workload breakdown.
  4. **Backfill every month the 6-month chart can reach**, not just the current one. `_per_key_monthly_series` reads ONLY per-key rows, so a key that switched axis shows `$0` for any month you didn't backfill even though it spent — `AK-20260430-DA42` would have looked like it started spending in July. Ran `sync_bedrock_profile_facts_for_integration` for 2026-07 (391 rows), 2026-06 (374) and 2026-05 (516). Don't bother going earlier than the first month CE has per-key data for; it is honestly empty, and every key on the axis here was created end-of-April or later.
  5. **Verify per key against a pre-flip baseline.** Snapshot every live Bedrock AK's `mtd_usd` / `data_available` / `data_caveat_code` BEFORE the flip and diff after — a key whose IAM user carries the tag for only part of the window would silently under-report. Observed diff (2026-07): `AK-20260710-B5F7` `$0.00`/pinned → **`$10.67`**/`bedrock_per_key_tag_shared`; `AK-20260430-DA42` `$51.48` (holder's Bedrock total) → `$53.04` (its own tagged total), code `bedrock_per_key_tag`; `AK-20260428-B9C1` unchanged at `$1,645.58` with its transfer-epoch caveat and its 6 `by_profile` workload rows intact (proof the per-key axis stays out of the workload breakdown); all 20 other AKs byte-identical.

  **What then changes in the UI**: `apikey/spend.py`'s Bedrock branch checks the per-key axis BEFORE the canonical pin. A key with rows in the requested window reports its OWN `mtd_usd` / daily series / model table (all four reductions share `_per_key_where`, so they cannot disagree), with the informational caveat `bedrock_per_key_tag_shared` naming the key that still carries the person-level Owner total. **The gate is "rows exist in this window", not "the setting is configured"** — a month that predates activation keeps the honest Owner pin instead of reporting a bogus `$0`.

  **Do NOT sum a per-key number with the canonical key's Owner total.** The per-key rows are a drill-down OF that total (same dollars, finer axis), exactly like the `UsedBy` workload rows. `cmdb_cost_facts` is untouched by this axis, so every dashboard / rollup / chargeback aggregate keeps counting the Owner total exactly once.

  **Keys that stay on the Owner pin**: IAM users created outside Helix carry no `helix-request-number` tag (production example: `investibook`, hand-created pre-Helix, `detail.bedrock.manual_backfill=true`). To bring one onto the per-key axis, tag its IAM user with `helix-request-number=<its AK number>` — an AWS write, so confirm with the key's owner first; it is forward-only like every cost-allocation tag.

  **Executed 2026-07-30 — every SG IAM user now carries the tag set.** All 14 IAM users in `841162690310` were audited (SSM one-off through the SG connector creds; mirror updated in `cmdb_iam_users.tags` in the same pass). Backfilled: `investibook` → `helix-request-number=AK-20260428-B9C1` + `helix-assignment-id` + `helix-project-name`; `mem9` → full `helix-*` set for its legacy-format AK `202604150071` (its `Owner=mem9` was deliberately LEFT ALONE — the active assignment's panel resolves through the mirror's `Owner` value and its historic CE days all sit under `Owner=mem9`; re-tagging would orphan them with no epoch stitch); `zhaiyl@pingcap.com` → full `helix-*` set for `AK-ADOPT-690310`; console users `chen.du@pingcap.com` / `shuai.feng@pingcap.com` had non-email `Owner` values (`duchen` / `冯帅`) fixed to work emails (no assignments bound, so no attribution continuity to break) and all non-Helix pingcap console users got `helix-owner-email`. `yuyaobj@amazon.com` (external AWS SA) untouched. Consequence to expect: once CE starts returning per-key groups for `investibook` (~1–2 days), `AK-20260428-B9C1` flips from the Owner pin to the per-key axis — its model table starts showing real SERVICE-dimension model names instead of `(uncategorised)` (the Owner-axis ingest has no model dimension: CE caps GroupBy at 2, spent on SERVICE × Owner-tag). ~~and its `UsedBy` workload panel goes empty BY DESIGN~~ **(SUPERSEDED 2026-08-04 by the per-key × UsedBy cross below — the per-key key now carries its OWN UsedBy split)**. The `UsedBy` axis itself is a RESOURCE tag on Bedrock application inference profiles, not an IAM-principal tag — a key whose calls bypass tagged inference profiles lands in the workload table's `(unallocated)` row regardless of IAM tags.

- **Per-key × UsedBy cross — LIVE since 2026-08-04 (migration 452)**: each per-key key now shows its OWN `UsedBy` workload split instead of the shared person-level one. **Why it was needed**: before this, the per-key detail page's "by UsedBy" table was the person-level `UsedBy` axis (owner-filtered), so two keys held by one person rendered an IDENTICAL workload table (xuhuaiyu reported it on B5F7 vs B9C1). **What makes it possible**: CE's `UsedBy` resource tag and the `helix-request-number` IAM-principal tag cross at the CUR line-item level — `Filter=account+key, GroupBy=(UsedBy × SERVICE)` returns each key's own workload split, and the per-key crosses sum to the owner total to the cent (prod-verified 2026-08-04: B5F7 `$50.4772` all `dev-external` + B9C1 `$115.4180` across `aiops`/`clinic`/… = `$165.8952` = xuhuaiyu's Owner total). **How**: `cost_facts.aws_profile_tag.aws-sg` already lists `iamPrincipal/helix-request-number`; the sync now sends that axis to the driver's `list_bedrock_per_key_profile_facts` (one account-wide `GroupBy=(key × owner)` discovery call, then per spending key the `(UsedBy × SERVICE)` cross), landing rows with a new `sub_profile` column (= the `UsedBy` value). The read side (`_per_key_by_profile`) makes `by_profile_scope` = `self` — the rows sum to the per-key headline, no "holder" caveat. **No AWS change** (tags already Active, setting already a list). **Transition**: existing per-key rows have `sub_profile=NULL` until the next `cost_facts_sync` re-ingests the rolling 3-day window; until then the UsedBy table shows a single `(unallocated)` row = headline. Backfill older months with `cost_backfill` if the 6-month chart needs the split. **CE-call cost**: +2 requests/day/scope (fan-out covers only keys with spend in the window, not all 14 IAM users). **Timeout**: the profile-facts per-scope leg is now bounded by `_BEDROCK_PROFILE_PER_SCOPE_TIMEOUT_SEC=120s` (wraps only the vendor fetches, not the DB ingest) — it runs inside the `cloud_sync:cost_facts:<id>` lock (ttl=900), and previously a wedged CE call could hold that lock to expiry and let the next cron tick race the window DELETE.

### 9a-revoke. Revoke disables the allocation rule (2026-05-12)

When an autograny AK is revoked (Bedrock / Azure OpenAI / Bailian), [`ApiKeyService.revoke_key`](../../server/app/domains/apikey/service.py) now disables (sets `enabled=False`, not delete) every `CmdbCostAllocationRule` referenced by `assignment.detail.<bucket>.allocation_rule_id`. Disable, not delete: forensics + the dashboard's "未分摊" KPI rely on the rule row remaining inspectable. The disable adds a description suffix `[disabled by revoke of <req#> on <UTC-date>]` and the revoke audit row's `detail.allocation_rules_disabled` field carries the summary `[{bucket, rule_id, status, rule_name}]` for SRE forensics. Idempotent: re-revoking returns `status='already_disabled'` (no suffix re-stamp); admin-deleted rule returns `status='not_found'` (revoke continues — cloud teardown is past the point of no return). Pre-2026-05-12 rules accumulated indefinitely on revoke and silently mis-charged departed employees' email tags on any future AWS principal sharing the value.

### 9a-aws-bq. AWS billing-source = BigQuery CUR for the PingCAP main AWS Org (2026-05-15)

The prod main AWS Org runs entirely on a CUR-to-BigQuery pipeline — Cost Explorer was never granted to the IAM user (`arn:aws:iam::317766989874:user/PingCAP`) and never will be. All three billing surfaces on the AWS driver — `list_cost_facts`, `list_billing_daily`, `list_billing_by_dimension` — route to BQ via the `config_public.billing_source` switch. AWS SG (single-account IAM mode, separate credentials with CE access, no CUR export) stays on Cost Explorer; the principal-tag mode at §9a-bedrock continues to fire for SG and is mutually exclusive with BQ mode.

**Config fields on the AWS integration** (`integrations.config_public`):

| Key | AWS main (prod) | AWS SG | Notes |
|---|---|---|---|
| `billing_source` | `"bigquery"` | unset (default `cost_explorer`) | Drives the driver's routing branch |
| `billing_bq_dataset` | `"gcp-digital-bi.aws_billing_no_payer"` (current 2026-05-19; was `aws_nonprod_billing` pre-CUR-2.0 cutover) | unset | Fully-qualified `<project>.<dataset>` — must contain a single dot |
| `billing_bq_credentials_integration_id` | UUID of the Helix GCP cloud integration (prod: `c1a4f301-adc2-46de-8525-fb8212268c46`) | unset | The driver does NOT store the SA JSON; it reads at call time from the referenced GCP integration's decrypted secrets — single rotation point |
| `billing_bq_table_pattern` | `"aws_billing_{yyyymmdd}"` (per_month default; legacy per_account default is `"{account_id}_aws_billing_data"`) | unset | Template MUST contain the placeholder matching the layout: `{account_id}` for `per_account`, `{yyyymmdd}` for `per_month`. The driver substitutes verbatim — no general format-string evaluation |
| `billing_bq_table_layout` | `"per_month"` (CUR 2.0) | unset (default `per_account`) | Selects table-resolution / partition-pruning / scope-filter strategy. `per_account` (CUR 1.0): one EXTERNAL Parquet table per linked AWS account, Hive `year`+`month` INT partitions, scope→route to the per-account table. `per_month` (CUR 2.0 / Data Exports): one native TABLE per month with all linked accounts merged, native DAY partition on `line_item_usage_end_date`, scope→`WHERE line_item_usage_account_id=?` filter on every UNION leg |
| `billing_bq_tag_schema` | `"nested"` (CUR 2.0) | unset (default `flat`) | Tag projection strategy. `flat`: one column per tag (`resource_tags_user_<sanitised_key>`); schema heterogeneity across per-account tables triggers per-pick column-presence drops. `nested`: tags live in a single `resource_tags.key_value` `ARRAY<STRUCT<key STRING, value STRING>>` column; element keys carry the `user_` prefix (`user_owner` etc.); schema is uniform across every per-month table |
| `billing_bq_fact_grain` | `"per_product"` (default, both layouts) | unset (default `per_product`) | Aggregation grain for ``cmdb_cost_facts``. `per_product` (default): GROUP BY (day × account × product × usage_type × currency); `line_item_resource_id` is NULL'd in SELECT and Python synthesises `aws-account:<acc>:<product>` — ~hundreds of facts per scope per day. `per_resource` (opt-in): adds `line_item_resource_id` to GROUP BY → ~thousands of facts per scope per day (proportional to active-resource count). CUR 2.0 makes `line_item_resource_id` present on every row (vs CUR 1.0 where it was absent for many per-account tables), so the cutover would 100-1000x cost_facts row volume unless `per_product` is set. **2026-05-19 prod incident root cause**: deploying CUR 2.0 with default `per_resource` produced 66k facts for a single 3-day window on one PingCAP account (vs 245 in CUR 1.0 same window), exhausting TiDB upsert throughput. Default flipped to `per_product`. Operators wanting per-resource attribution for chargeback rules opt into `per_resource` explicitly. |

**`cmdb.cost_facts_sync` per-scope timeout** (`cost_service._COST_FACTS_PER_SCOPE_TIMEOUT_SEC = 300s`, 2026-05-19 follow-up): each scope's `driver.list_cost_facts` call is wrapped in `asyncio.wait_for`. When the timeout fires the scope's per-scope entry records `status='timeout'` + an `errors[]` line; the cron moves to the next scope rather than wedging the whole run in `running` state. Bounded by the BQ-side `job_timeout_ms = 5min` cap in `_bq_query_job_config`, so an orphan worker thread (the asyncio cancel doesn't reach the to_thread thread directly) cleans up within the same 5-minute envelope. Pre-fix a single CUR 2.0 wedge on scope #43 hung the whole cron for 47+ min before manual intervention.

**Service account** — `pch-resource-manager@gcp-digital-infra.iam.gserviceaccount.com`, already on the prod GCP cloud integration as `service_account_json`. It has `bigquery.dataViewer` on the dataset and `bigquery.jobUser` on its home project. **Scope is `https://www.googleapis.com/auth/bigquery`** (NOT `.readonly` — that one grants metadata listing only and 403s on `Jobs.InsertJob`).

**How resolution works in code** — every billing call site (`cost_service.sync_cost_facts_for_integration`, `lifecycle_service.sync_billing_daily_for_integration`, `lifecycle_router.list_billing_by_dimension`) calls [`_resolve_aws_bq_billing(session, integration)`](../../server/app/domains/cmdb/lifecycle_service.py) before invoking the driver. The helper returns `None` for non-AWS / AWS-on-CE integrations (the driver's `_billing_source` property short-circuits past the BQ branch in those cases) and raises `RuntimeError` loudly for any half-configured BQ source — silent fall-through to CE would yield 60 AccessDenied errors identical to the original prod failure. Resolved kwarg shape: `{"credentials_json": <SA dict>, "dataset": "<proj>.<ds>", "table_pattern": "<pattern>", "table_layout": "per_account"|"per_month", "tag_schema": "flat"|"nested", "fact_grain": "per_product"|"per_resource"}`.

**CUR 1.0 vs CUR 2.0 cheat sheet** (2026-05-19, dataset cutover from `aws_nonprod_billing` → `aws_billing_no_payer`):

| Dimension | CUR 1.0 (`aws_nonprod_billing`) | CUR 2.0 (`aws_billing_no_payer`) |
|---|---|---|
| Table type | EXTERNAL (Parquet on GCS) | Native TABLE |
| Layout | One table per linked account (`{account_id}_aws_billing_data`, ~60 tables) | One table per month (`aws_billing_{yyyymmdd}`, ~one per month since 2025-10), all accounts merged |
| Partition | Hive `year` INT + `month` INT columns | Native DAY on `line_item_usage_end_date` |
| Account filter | Route by table name | `WHERE line_item_usage_account_id = ?` injected into every UNION leg |
| Tag storage | 13 flat columns `resource_tags_user_<sanitised>` | `resource_tags.key_value` `ARRAY<STRUCT<key STRING, value STRING>>` |
| Tag key naming | Column suffix is sanitised lowercase (`resource_tags_user_owner`) | Array element key carries `user_` prefix (`user_owner`) |
| `product_<field>` (`product_product_name`, `product_region`, `product_instance_type_family`) | Flat top-level column | Nested in `product.key_value` `ARRAY<STRUCT>` — keys are the bare names (`product_name`, `region`, `instance_type_family`) |
| Schema heterogeneity | Per-account-table column drift (CUR-without-resource-ids, tag-keys-never-set); driver introspects `Table.schema` and drops absent columns from SELECT/GROUP BY | Uniform schema across every per-month table (no per-pick drop) |
| History coverage | Multi-year | ~8 months since 2025-10 (per PingCAP's CUR-2.0 export start) |
| Helix config | `billing_bq_table_layout="per_account"` + `billing_bq_tag_schema="flat"` | `billing_bq_table_layout="per_month"` + `billing_bq_tag_schema="nested"` |

**Totals divergence (CUR 1.0 vs CUR 2.0)** — observed against PingCAP linked account 385595570414 over 2026-05-01..05-03: per-day totals differ by ~$500-700 (CUR 1.0 reads higher). **Root cause confirmed 2026-05-20 (BQ direct probe both datasets, line_item_type breakdown side-by-side)**: the new CUR 2.0 export surfaces two enterprise-discount line types that the legacy CUR 1.0 export was created WITHOUT: `EdpDiscount` (PingCAP's negotiated Enterprise Discount Program, ~-$566/day for account 385595570414, ~-$26K/day org-wide × 93 linked accounts) and `PrivateRateDiscount` (per-SKU rate adjustments, ~-$45/day for this account, ~-$24K/day × 48 accounts). 5/01 example: CUR 1.0 total $2687.81 (= Usage $2671.68 + SP CoveredUsage $1269.50 + FlatRate $16.13 + Tax $0 − SP Negation $1269.50); CUR 2.0 total $2056.69 (= same five rows − EdpDiscount $583.81 − PrivateRateDiscount $47.31). CUR 1.0 was over-reporting list-price by ~23%; CUR 2.0 reflects the actual net invoiced amount, matching what PingCAP pays AWS per invoice. Not a Helix bug, not an export bug — it's the deliberate v2 export-config decision to include discount detail. **Action**: 2026-05-20 PR adds `'EDP Discount'` / `'Private Rate Discount'` CASE buckets to both `_service_select` and `_select_for_pick` so the per-scope donut + cost_facts surface labels them by name rather than collapsing into an anonymous `Other` slice. Backfill historical cost_facts (2025-10 → today) once after PR ship to apply the new bucket names retroactively.

**Schema landmarks** (CUR built WITH `Include resource IDs`, verified prod 2026-05-15):
- `line_item_usage_account_id` STRING — 12-digit linked-account
- `line_item_line_item_type` STRING — filter `!= 'SavingsPlanNegation'`
- `line_item_usage_start_date` TIMESTAMP — bucket via `DATE(...)`
- `line_item_product_code` STRING — filter `!= 'ComputeSavingsPlans'`
- `line_item_resource_id` STRING — per-resource cost; NULL on aggregate rows
- `line_item_unblended_cost` FLOAT
- `line_item_currency_code` STRING — USD on standard partitions, CNY on Sinnet/NWCD (BQ mode same handling as CE: non-USD lands `amount_native` + `amount_usd=0`)
- `product_product_name` STRING — display name for `service`
- `product_region` STRING
- `resource_tags_user_owner` / `_application` / `_environment` / `_project` / `_department` STRING — replace the iamPrincipal/Owner CE breakdown for the BQ codepath's `owner_breakdown`
- `year` INT + `month` INT — Hive AUTO partition columns; the driver pins these via `_bq_month_predicate` for partition pruning

**Tenant-wide queries** (`scope=None` in `list_billing_daily`'s reconciliation pass / `list_billing_by_dimension`) discover tables via `<dataset>.INFORMATION_SCHEMA.TABLES WHERE table_name LIKE @pat` then UNION ALL the per-table aggregates. Tables not matching the 12-digit-id pattern (e.g. `557537366020_aws_grouped_o11y_billing_cost`) are skipped to avoid double-counting.

**Per_month month-boundary scan widening** (2026-05-19 PR-#404-followup, `aws_cloud.py:_bq_resolve_tables`): the per_month enumeration covers `[start, end + 2 days]` rather than `[start, end]`. AWS CUR rows with `start_date` in month-N but `end_date` in month-(N+1) (typical for midnight-rollover SP/RI recurring fees) land in month-(N+1)'s per-month table by the native DAY partition on `line_item_usage_end_date`, while their day-grain cost belongs to month-N (computed from `start_date`). Without the +2-day pad, a month-N window would enumerate month-N's table only, miss the boundary row, and silently undercount the last day of every month by ~0.1-1% (PingCAP prod sample). The pad symmetrically aligns with `_bq_partition_predicate`'s per_month end-pad so the next-month table's scan is clamped to just the boundary-straddling rows; cost overhead is bounded to ~the first 2 days of the next month per query.

**Missing tables vs missing dataset — two distinct failure classes** (`_bq_resolve_tables` + `_bq_probe_dataset_exists`, 2026-05-19 PR-#404-followup):

- **Missing table** = the dataset itself exists but this specific table isn't in it. Two legitimate causes: (a) a newly-added linked account whose CUR isn't onboarded yet (per_account layout), (b) a backfill window extending past the first CUR-export month (per_month layout). The driver's `client.get_table(...)` probe returns `NotFound` and the call returns `[]` rather than failing the whole sync. Visible in `scheduled_job_runs` as `status=ok` with `facts: 0` for that scope/month. Operator action: add the account to the CUR export config on AWS side if coverage matters, OR widen the window OR ignore.
- **Missing dataset** = `billing_bq_dataset` is typo'd, the SA has lost access, or the dataset was deleted. Pre-2026-05-19 this fell through identical-to-missing-table behaviour and produced `status=ok` with `facts: 0` for every scope, the cron's failure-notifier stayed silent, and a 12-month backfill could land zero rows before anyone noticed. The driver now runs an upfront `client.get_dataset(dataset)` probe (cached per dataset-string on the driver instance so 60-scope cron ticks don't pay 60 round trips) BEFORE any per-table enumeration; `NotFound` is rewrapped as a loud `RuntimeError` that lands in `scheduled_job_runs.errors[]` and fires the failure-notifier. Forbidden / 5xx propagate unchanged so transient throttles surface with their real error class. Operator action: re-check `config_public.billing_bq_dataset` value vs the BQ console, re-grant the SA on the dataset (legacy ACL or IAM dataViewer per [§9a-aws-bq SA setup](#9a-aws-bq-aws-billing-source--bigquery-cur-for-the-pingcap-main-aws-org-2026-05-15)).

**Troubleshooting**:

| Symptom | Check | Fix |
|---|---|---|
| `cmdb.cost_facts_sync` returns `status=failed`, reason mentions `bq_billing` config | `SELECT JSON_EXTRACT(config_public, '$.billing_*') FROM integrations WHERE provider='aws'` — verify all 3 BQ fields present | Re-fill the missing field via SQL or admin UI; ensure `billing_bq_credentials_integration_id` is a real enabled GCP cloud integration UUID |
| Per-scope `status=ok` but `facts=0` for some accounts | Check BQ has the table: `SELECT table_name FROM \`<dataset>.INFORMATION_SCHEMA.TABLES\` WHERE table_name = '<acc>_aws_billing_data'` | If missing, the linked account isn't in CUR export — work with cloud infra team to add it OR accept the gap (rare for non-billing-bearing accounts) |
| `cmdb.cost_facts_sync` returns `status=failed`, error reads `dataset 'X' was not found by the BigQuery API` | Compare `config_public.billing_bq_dataset` literal vs the dataset name in the BQ console; check the GCP integration's SA has READER on the dataset (legacy ACL `bq show --format=prettyjson <dataset>.access` or IAM `roles/bigquery.dataViewer`) | Fix the typo OR re-grant the SA — see [§9a-aws-bq](#9a-aws-bq-aws-billing-source--bigquery-cur-for-the-pingcap-main-aws-org-2026-05-15) for the prod grant procedure. The probe deliberately fails loudly here rather than the pre-2026-05-19 silent-skip-everything behaviour. |
| Some tag values missing from `CostFact.tags` | Only the curated subset (`owner` / `application` / `environment` / `project` / `department`) is harvested — adding others requires extending the `tag_columns` list in `_list_cost_facts_bq_sync` | PR to add the column-alias pair; coordinate with the chargeback rule-set since new tags affect rule matching |
| `RuntimeError: principal_tag_keys is incompatible with billing_source=bigquery` | Both `platform_settings.cost_facts.aws_principal_tag.<slug>` AND `config_public.billing_source=bigquery` are set on the same integration | The two modes never coexist — SG sets the platform-setting, main AWS sets billing_source. Pick one. |
| `RuntimeError: ... billing_source changed from 'X' → 'Y'; refusing to ingest ...` | `cost_service.ingest_cost_facts` detected `integration.config_public.billing_source` differs from `platform_settings.cost_facts.last_billing_source.<integration_id>`. Toggling `cost_explorer` ↔ `bigquery` causes two `cmdb_cost_facts` rows per logical day×resource (CE emits `usage_type=NULL`, BQ emits the real `line_item_usage_type` — `compute_fact_key` includes `usage_type`) → `SUM(amount_usd)` double-counts. | **Toggle procedure** (added 2026-05-16 pre-launch follow-up): 1) decide the window to purge (typically all months since the last toggle); 2) `DELETE FROM cmdb_cost_facts WHERE integration_id='<uuid>' AND date BETWEEN '<start>' AND '<end>';` 3) `DELETE FROM platform_settings WHERE \`key\`='cost_facts.last_billing_source.<integration_id>';` 4) flip `config_public.billing_source` via admin UI or SQL; 5) re-run `cmdb.cost_facts_sync` manually OR wait for the next 04:xx tick. The error message from the guard includes the exact SQL with the correct window-bounds for your case. |

### 9a-efficiency-grain. 云效率「每实例可省」对 AWS / GCP 恒为 $0 — 计费 grain 太粗,成本无法按实例归属 (added 2026-07-25)

**Symptom.** On `/admin/cloud/efficiency` (and `/my/cloud/efficiency`) the KPI 「预估可省 / 月」(`totals.estimated_monthly_savings_usd`) and the per-cloud savings breakdown (`by_cloud[]`) show **$0 for every AWS integration** — and for GCP integrations wired to the *standard* billing export — even for boxes the utilization engine has flagged idle/oversized. The paired KPI 「已归属月成本」(`totals.attributed_monthly_cost_usd`) is likewise $0 for those clouds, and per-instance `cmdb_server_efficiency.monthly_cost_usd` is `NULL`. Utilization verdicts (idle / oversized / healthy) still render — only the **money** is missing.

**Root cause (verified at file:line, not a heuristic).** The efficiency engine attributes spend by matching each server's identity against the cost-fact table:

- [`efficiency.load_cost_map`](../../server/app/domains/cmdb/efficiency.py) builds `{cmdb_cost_facts.resource_id → trailing-30d USD}` (`COST_LOOKBACK_DAYS = 30`), and [`efficiency.match_cost`](../../server/app/domains/cmdb/efficiency.py) compares it against `[server.instance_id, raw_data["id"], server.name]` — exact/case-insensitive, then an unambiguous `/`- or `:`-anchored suffix. No match → `monthly_cost = None` → `estimate_savings` returns `$0`. This is by design for Azure-style id/name skew; the bug is that **the AWS/GCP cost rows carry no per-instance id to match against.**
- **AWS** runs on the BigQuery-CUR path (§9a-aws-bq) with `config_public.billing_bq_fact_grain` defaulting to **`per_product`** ([`aws_cloud._select_for_pick`](../../server/app/integrations/drivers/aws_cloud.py) emits `CAST(NULL AS STRING) AS resource_id`; the Python then synthesises `resource_id = "aws-account:<acct>:<product-or-discount-bucket>"` in [`aws_cloud.py`](../../server/app/integrations/drivers/aws_cloud.py)). That synthetic id can **never** equal an `i-…` instance id, so `match_cost` always returns `None`. `per_resource` grain projects the real `line_item_resource_id` instead ([`aws_cloud.py`](../../server/app/integrations/drivers/aws_cloud.py) — the CUR-2.0/nested branch always has the column; CUR-1.0/flat has it only when present in that table's schema).
- **GCP** standard export (`gcp_billing_export_v1_…`) has no `resource` column, so [`gcp_cloud._select_for_table`](../../server/app/integrations/drivers/gcp_cloud.py) substitutes `''` for `resource_name` (gated by [`GcpProvider._is_detailed_export`](../../server/app/integrations/drivers/gcp_cloud.py)). Only the **detailed** export (`gcp_billing_export_resource_v1_…`) sets `resource.name`, which is what feeds `resource_id`. Standard-export tenants therefore have the same no-id-to-match problem.

**Why this is NOT a code change — and specifically not a default flip.** Do **not** change the `per_product` default in `aws_cloud.py` / `lifecycle_service.py`. That default exists because of the 2026-05-19 prod incident (see §9a-aws-bq `billing_bq_fact_grain` row): CUR-2.0 with `per_resource` produced **66k facts for a single 3-day window on one linked account** (vs 245 in CUR-1.0), exhausting TiDB upsert throughput and wedging the cron. Flipping the default would inflict that 100–1000× row-volume blow-up on **every** AWS tenant, including the many that never use efficiency attribution. The fix is a **per-integration opt-in** for the integrations whose efficiency numbers you actually want, plus a bounded backfill.

---

#### AWS — enable per-instance attribution on one integration

1. **Identify the integration UUID.**

   ```bash
   mysql ... -e "SELECT id, name, JSON_EXTRACT(config_public,'$.billing_source') AS src, JSON_EXTRACT(config_public,'$.billing_bq_fact_grain') AS grain FROM integrations WHERE provider='aws' AND kind='cloud' AND deleted_at IS NULL"
   ```

   Proceed only for rows with `src = "bigquery"` (Cost-Explorer/SG integrations don't have this bug and don't have the knob).

2. **Flip that one integration to `per_resource`** — admin UI (Integrations → the AWS integration → Billing → *BigQuery Fact Grain* = `per_resource`) or SQL:

   ```sql
   UPDATE integrations
   SET config_public = JSON_SET(config_public, '$.billing_bq_fact_grain', 'per_resource')
   WHERE id = '<aws-uuid>';
   ```

   No migration, no schema change. Takes effect on the next `cmdb.cost_facts_sync` for that integration.

3. **Estimate the blast radius BEFORE backfilling (isolated validation).** Dry-run the backfill for *only* this integration over *only* the 30-day efficiency window, and eyeball the plan:

   ```bash
   cd server
   HELIX_PROD_DB_URL_ASYNC='mysql+aiomysql://...@tidbcloud.com:4000/helix?ssl=true' \
     uv run python scripts/backfill_cost_facts.py --prod --dry-run \
     --integration-id <aws-uuid> --start 2026-06-25 --end 2026-07-25
   ```

   Row volume will grow roughly in proportion to the account's active-resource count. Watch two limits during the real run: the per-scope `cmdb.cost_facts_sync` timeout (`_COST_FACTS_PER_SCOPE_TIMEOUT_SEC = 300s`, bounded by the BQ `job_timeout_ms = 5min` cap — §9a-aws-bq) and TiDB upsert throughput. A very large single account may hit the per-scope timeout; when it does, that scope records `status='timeout'` in `scheduled_job_runs` and the run moves on with **partial** data for that scope — re-run narrowed to that scope if so.

4. **Backfill exactly the efficiency window** (`COST_LOOKBACK_DAYS = 30`; a wider window just adds volume the efficiency engine won't read). Drop `--dry-run`:

   ```bash
   HELIX_PROD_DB_URL_ASYNC='...' \
     uv run python scripts/backfill_cost_facts.py --prod \
     --integration-id <aws-uuid> --start 2026-06-25 --end 2026-07-25
   ```

   The re-ingest is a clean replace: `cost_service` deletes existing `cmdb_cost_facts` rows for each `(integration_id, scope_id, date)` before re-upserting ([`cost_service.py`](../../server/app/domains/cmdb/cost_service.py)), so the old `aws-account:*` per-product rows for the window are swapped for per-resource rows with no residue and no double-count. Keep the post-backfill `recompute_allocation` (do **not** pass `--skip-recompute`): the newly-fine-grained rows need `cost_owner_*` re-attributed too.

5. **Recompute efficiency** so the engine re-reads the now-attributable cost facts. Trigger the `cmdb.efficiency_compute` job for this integration (Admin → Scheduled Jobs → 云效率评估 → run with `{"integration_id": "<aws-uuid>"}`), or wait for the daily tick. It rewrites `cmdb_server_efficiency.monthly_cost_usd` + `estimated_monthly_savings_usd` from `cmdb_cost_facts`.

#### GCP — switch to the detailed export

GCP has no `fact_grain` knob; per-instance attribution depends entirely on **which export table** the integration points at.

1. Inspect `config_public.billing_bq_table`:

   ```bash
   mysql ... -e "SELECT id, name, JSON_EXTRACT(config_public,'$.billing_bq_table') AS tbl FROM integrations WHERE provider='gcp' AND kind='cloud' AND deleted_at IS NULL"
   ```

   If the table name contains `gcp_billing_export_v1_` (standard) rather than `gcp_billing_export_resource_v1_` (detailed), attribution is impossible — `_is_detailed_export` returns false and `resource.name` is projected as `''`.

2. The tenant must have the **detailed usage cost export** enabled in GCP billing (it's a *separate* export sink from the standard one; enabling it is a GCP-console action, not a Helix action). Once it exists, point the integration at it:

   ```sql
   UPDATE integrations
   SET config_public = JSON_SET(config_public, '$.billing_bq_table', '<project>.<dataset>.gcp_billing_export_resource_v1_XXXXXX')
   WHERE id = '<gcp-uuid>';
   ```

3. Backfill 30 days + recompute exactly as AWS steps 3–5 (same `backfill_cost_facts.py` + `cmdb.efficiency_compute`).

#### Isolated-validation checklist (before declaring the numbers trustworthy)

This change alters cost-fact **row volume** and the reported **attributed** cost, so validate on one integration before rolling to others:

| Check | SQL | Expected |
|---|---|---|
| **Total spend is UNCHANGED** (grain must re-resolve the same money, not create/lose it) | `SELECT ROUND(SUM(amount_usd),2) FROM cmdb_cost_facts WHERE integration_id='<uuid>' AND date BETWEEN '2026-06-25' AND '2026-07-25';` — run before the flip and after the backfill | Within a few $ of the pre-flip total. A material move means over/under-attribution — investigate before trusting savings. |
| Row volume grew as expected (not pathologically) | `SELECT COUNT(*) FROM cmdb_cost_facts WHERE integration_id='<uuid>' AND date BETWEEN '2026-06-25' AND '2026-07-25';` before/after | Grows ≈ active-resource count ×; cross-check against the 2026-05-19 66k/3-day/account caution — a wildly higher multiple suggests a runaway UNION or an un-pruned partition scan. |
| No scope silently truncated | `SELECT scope, status FROM scheduled_job_runs WHERE job_key='cmdb.cost_facts_sync' ORDER BY started_at DESC LIMIT 20;` (or the per-scope `errors[]`) | No `status='timeout'` for this integration's scopes; if any, re-run narrowed to them. |
| resource_id shape is matchable | `SELECT DISTINCT resource_id FROM cmdb_cost_facts WHERE integration_id='<uuid>' AND resource_id LIKE '%i-%' LIMIT 10;` | Bare `i-0abc…` or an ARN ending `…:instance/i-0abc…` — `match_cost` handles both (exact + `/`/`:`-anchored suffix). |

#### Did it work (post-verify)

| Check | SQL / UI | Expected |
|---|---|---|
| Per-instance cost now attributed | `SELECT COUNT(*) FILTER (WHERE monthly_cost_usd IS NOT NULL) AS attributed, COUNT(*) AS total FROM cmdb_server_efficiency WHERE integration_id='<uuid>';` (MySQL/TiDB: `SUM(monthly_cost_usd IS NOT NULL)`) | `attributed` jumps from ~0 toward `total` (won't be 100% — genuinely un-billed/terminated boxes stay NULL). |
| KPIs populate | `/admin/cloud/efficiency` | 「已归属月成本」(`attributed_monthly_cost_usd`) rises; 「预估可省 / 月」and the `by_cloud[]` row for this integration become non-zero. |

#### Rollback

Flip the config back (`JSON_SET(... '$.billing_bq_fact_grain','per_product')` / repoint GCP at the standard table) and re-backfill the same window — the `(integration, scope, date)` delete-then-upsert cleanly restores the coarse rows and row volume returns to baseline. Efficiency savings for that cloud drop back to $0 on the next `cmdb.efficiency_compute`.

### 9a-aws-sp. Savings Plan discount accounting fix (2026-05-17, PR #339)

**One-time backfill after merge.** PR #339 changed the AWS CUR SQL to stop excluding `SavingsPlanNegation` rows / `ComputeSavingsPlans` product-code rows from the `SUM(line_item_unblended_cost)`. Existing rows in `cmdb_cost_facts` and `cmdb_cloud_billing_daily` were written by the **pre-fix** SQL and remain ~30–40% too high for every AWS scope. Reproduced against prod account 552185537300 (April 2026: Helix $8,360.87 vs invoice $5,505.88, 34% over). The daily cron tick after merge only rewrites yesterday's data, so historical months stay wrong until backfilled.

**Run after the PR is deployed to prod**:

```bash
# Trailing 12 months, AWS only (other providers don't have the SP-discount bug).
# Identify the AWS cloud integration UUID first:
mysql ... -e "SELECT id, name FROM integrations WHERE provider='aws' AND kind='cloud' AND deleted_at IS NULL"

# Dry-run the backfill plan:
HELIX_PROD_DB_URL_ASYNC='mysql+aiomysql://...@tidbcloud.com:4000/helix?ssl=true' \
  uv run python scripts/backfill_cost_facts.py --prod --dry-run \
  --integration-id <aws-uuid> --months 12

# When dry-run output looks right, drop --dry-run:
HELIX_PROD_DB_URL_ASYNC='...' \
  uv run python scripts/backfill_cost_facts.py --prod \
  --integration-id <aws-uuid> --months 12
```

The script walks 30-day chunks through both `sync_billing_daily_for_integration` and `sync_cost_facts_for_integration`, so it overwrites the bad rows in both tables in one pass. Idempotent via the natural-key UNIQUE constraints — safe to re-run if interrupted.

**Validate after backfill**:

| What to check | SQL | Expected |
|---|---|---|
| Per-scope monthly total matches AWS invoice | `SELECT date_format(date,'%Y-%m') AS m, ROUND(SUM(total_cost_usd),2) FROM cmdb_cloud_billing_daily WHERE scope_id IN (SELECT id FROM cmdb_cloud_scopes WHERE external_id='<aws-account-id>') GROUP BY m ORDER BY m DESC LIMIT 12;` | Each month within $1 of AWS Cost Explorer "Net unblended cost" for that account |
| cost_facts has a `'Savings Plans'`-bucketed line per scope per active month | `SELECT scope_id, COUNT(*) AS sp_rows, ROUND(SUM(amount_usd),2) AS sp_total FROM cmdb_cost_facts WHERE service='Savings Plans' AND date >= '<window-start>' GROUP BY scope_id;` | Linked accounts: sp_total is **negative** (the discount applied to that account). Payer account: sp_total is **positive** (the SP commitment fee). |
| Daily totals agree between cost_facts and billing_daily | `SELECT b.date, b.total_cost_usd, ROUND(c.cf,2) FROM cmdb_cloud_billing_daily b LEFT JOIN (SELECT scope_id, date, SUM(amount_usd) AS cf FROM cmdb_cost_facts GROUP BY scope_id, date) c ON c.scope_id=b.scope_id AND c.date=b.date WHERE b.date>='<window-start>' AND b.scope_id='<id>' ORDER BY b.date DESC;` | Each row matches to within $0.01 — same upstream BQ query feeds both paths |

**If totals are still off after the backfill**: the source CUR table may be missing SP negation rows for that linked account (CUR export config), or the account is a payer with its SP-Recurring-Fee row genuinely landing as positive cost. Compare against AWS Cost Explorer → Group by None → Charge type = "Net unblended" for the same window before assuming a Helix bug.

**Side-effects to handle the same day as the backfill**:

| Side-effect | Action | Why |
|---|---|---|
| Budget thresholds set against the pre-fix inflated numbers | Audit every `platform_settings.billing.budget_usd.<integration_slug>` row for AWS integrations. Compare each to the post-backfill rolling 3-month average from `cmdb_cloud_billing_daily`. Lower thresholds proportionally (~30% drop is typical). | Pre-fix budgets are over-tuned by the same factor as the inflation. Post-fix MTD will be ~30% lower → budgets stop firing exactly when finance most needs them. The dashboard at `/admin/cloud/billing` shows the per-integration ratio. |
| First live cron tick after backfill may fire `daily_spike` for the wrong day | Schedule the backfill to COMPLETE BEFORE the next 04:00 UTC cron tick (the entire 12-month window runs in ~10-20 minutes against prod TiDB — well within a single overnight maintenance window). | `_alert_spike` compares "yesterday" vs the trailing 7-day mean. After backfill all 7 days are post-fix, the comparison is consistent, no false spike fires. The risk is only if the backfill is STILL running when 04:00 UTC arrives — the 7d window then mixes pre-fix and post-fix values. Per-day dedup uses `notification_outbox.(recipient_id, related_resource_id)` keyed on grant/day, NOT a `scheduled_job_dedup_state` table; if a spurious spike does fire, suppress the next day's legitimate alert by inserting a manual outbox row (see [`_alert_spike` in lifecycle_service.py](../../server/app/domains/cmdb/lifecycle_service.py)). The cleanest answer is "don't let the backfill straddle the cron". |
| Per-scope drawer's "Savings Plans" donut callout | No action — the frontend already renders the SP discount as a separate "Discounts applied · Savings Plans: -$X" badge below the donut. Negative slices are NOT fed to the ECharts pie series; the gross-services donut + the discount badge sum visually to the rendered total. | Behaviour added in PR #339 — `ScopeBillingPanel.tsx::serviceChart` memo splits the SP entry out via the cost_by_service value sign. |
| `/admin/cloud/billing/overview` top-services list | No action — the SP "Savings Plans" entry is pinned into top-services regardless of where it sorts by absolute cost. | Without the pin, on a linked account the SP row was the MOST negative entry and dropped off the top-10 cliff, leaving finance unable to explain the gap between gross service total and reported scope total. PR #339 adds the pin in `/cmdb/billing/overview` AND `/cmdb/billing/summary`. |
| Per-person / per-department / chargeback reports | No action — SP rows are stamped `auto_owner_source='savings_plan'` and `cost_owner_person_id=NULL` so they fall into the "Unallocated" bucket on every per-person facet. Per-scope totals still net correctly because they aggregate over all rows including SP. | Without this guard, scope owner X's chargeback line would carry a negative SP discount as if Helix had paid X. PR #339 short-circuits both `ingest_cost_facts` and `recompute_allocation` for `service='Savings Plans'`. |

### 9a-alicloud-intl-billowner. AliCloud `list_billing_daily` BillOwnerId fix (2026-05-18, Round-32)

**One-time backfill after merge.** Round-32 added `BillOwnerId=<scope.external_id>` to `QueryAccountBill` in both [`aliyun_cloud.py::list_billing_daily`](../../server/app/integrations/drivers/aliyun_cloud.py) and the intl override [`alicloud_intl_cloud.py::list_billing_daily`](../../server/app/integrations/drivers/alicloud_intl_cloud.py). Pre-fix Aliyun BSS returned ONLY the calling account's own bill — on PingCAP's Resource Directory master that's just cross-account fees (Data Transfer / Container Service K8s / Cloud Monitor / ApsaraMQ Kafka, ~4 services total). Every per-resource ECS / EBS / OSS / SLB line item owned by RD member sub-accounts was missing from `cmdb_cloud_billing_daily`. Prod evidence (alicloud-intl-primary 18 sub-account scopes, May 1-18, 2026): cost_facts daily total ~$8K/day vs billing_daily ~$50/day, ratio ~165–200×. cost_facts was correct all along — `DescribeInstanceBill` (the API behind cost_facts) is RD-consolidated by default and was unaffected. **Only the daily-billing surface is wrong**; the daily cron tick after merge only rewrites yesterday's data, so historical months stay wrong until backfilled.

**Run after the PR is deployed to prod**:

```bash
# alicloud_intl only — other providers don't have this bug.
# Confirm the integration UUID:
mysql ... -e "SELECT id, name FROM integrations WHERE provider='alicloud_intl' AND kind='cloud'"
# → 62d312e2-589a-4fa2-894c-f38b6ac59f87 (alicloud-intl-primary)

# Dry-run the plan:
HELIX_PROD_DB_URL_ASYNC='mysql+aiomysql://...@tidbcloud.com:4000/helix?ssl=true' \
  uv run python scripts/backfill_cost_facts.py --prod --dry-run \
  --integration-id 62d312e2-589a-4fa2-894c-f38b6ac59f87 --months 6 --skip-recompute

# Drop --dry-run when output looks right:
HELIX_PROD_DB_URL_ASYNC='...' \
  uv run python scripts/backfill_cost_facts.py --prod \
  --integration-id 62d312e2-589a-4fa2-894c-f38b6ac59f87 --months 6 --skip-recompute
```

Why `--skip-recompute`: cost_facts data is unchanged by this fix (it was correct); only `cmdb_cloud_billing_daily` rows get rewritten. Allocation rules drive `cost_owner_*` columns from cost_facts, so recompute would be a no-op CPU burn. Why `--months 6`: Aliyun BSS retention is 6–12 months; `--months 12` is fine if Aliyun still has the rows, but for the backfill-then-validate iteration 6 months is a faster end-to-end. The script walks 30-day chunks through both `sync_billing_daily_for_integration` (`dispatch_alerts=False` — no Feishu cards during replay, see [`backfill_cost_facts.py:396`](../../server/scripts/backfill_cost_facts.py:396)) and `sync_cost_facts_for_integration` (cost_facts is idempotent + already-correct, so this leg is a no-op-with-API-quota-spend; can't be skipped from the current CLI but causes zero data change).

**Validate after backfill**:

| What to check | SQL | Expected |
|---|---|---|
| Daily totals converge to cost_facts | `SELECT cf.date, ROUND(SUM(cf.amount_usd),2) AS cf_usd, (SELECT ROUND(SUM(bd.total_cost_usd),2) FROM cmdb_cloud_billing_daily bd WHERE bd.integration_id=cf.integration_id AND bd.date=cf.date) AS bd_usd FROM cmdb_cost_facts cf WHERE cf.integration_id='62d312e2-589a-4fa2-894c-f38b6ac59f87' AND cf.date >= CURDATE() - INTERVAL 7 DAY GROUP BY cf.date ORDER BY cf.date;` | Each row's `bd_usd` within ±20% of `cf_usd`. Pre-fix `bd_usd` was ~0.5% of `cf_usd`. |
| Per-scope service breadth in billing_daily | `SELECT scope_id, JSON_LENGTH(JSON_KEYS(cost_by_service)) AS svc_count FROM cmdb_cloud_billing_daily WHERE integration_id='62d312e2-589a-4fa2-894c-f38b6ac59f87' AND date=CURDATE()-INTERVAL 1 DAY ORDER BY svc_count DESC LIMIT 5;` | Top scopes should report 5+ services (ECS, EBS, OSS, SLB, NAT, …). Pre-fix EVERY scope reported ≤ 4 services and most reported 0–2. |
| One scope's daily total matches its line-item sum | `SELECT bd.date, bd.total_cost_usd AS bd, ROUND((SELECT SUM(amount_usd) FROM cmdb_cost_facts WHERE scope_id=bd.scope_id AND date=bd.date),2) AS cf FROM cmdb_cloud_billing_daily bd WHERE bd.scope_id=(SELECT id FROM cmdb_cloud_scopes WHERE external_id='5901860086223871') ORDER BY date DESC LIMIT 10;` | `bd ≈ cf` within ±20%. Discrepancy explained by `QueryAccountBill` vs `DescribeInstanceBill` aggregation tier differences (sub-day rounding, prepaid amortization timing); both reflect the same real bill. |

**Side-effects to handle the same day as the backfill**:

| Side-effect | Action | Why |
|---|---|---|
| Budget thresholds trained on pre-fix undercounted numbers | After backfill, re-tune `platform_settings.billing.budget_usd.alicloud-intl-primary` and any per-scope alert_rules thresholds. Numbers will jump ~200× → previously over-permissive thresholds suddenly silent; previously well-tuned thresholds now flag prod normally. | Pre-fix daily ~$50, post-fix daily ~$8000. The threshold needs to reflect post-fix reality. The same gotcha as PR #339 (AWS SP fix) but in the opposite direction — there budgets had to come DOWN, here they come UP. |
| First live cron tick after backfill may fire `daily_spike` | Schedule the backfill to COMPLETE BEFORE the next 04:xx UTC `cmdb.billing_daily_sync` tick (single-integration 6-month window runs in ~3–5 min against prod TiDB + Aliyun BSS, well within an overnight window). `_alert_spike` compares "yesterday" vs trailing 7-day mean: post-backfill all 7 days are post-fix → consistent comparison → no false spike. Only risk is straddling the cron with a half-finished backfill. | The Feishu spike-detection card would rightly say "alicloud-intl-primary jumped 200× overnight" — looks alarming on the first post-deploy day; explain to finance / IT admin in advance that this is the long-undercount finally getting corrected. |
| Page-vs-alert agreement on `scope_billing_detail` drawer | No action — drawer reads cost_facts which was already correct. Post-backfill billing_daily now matches; the `summary.data_source='cost_facts'` path stays primary and the fallback path (PR #339) will only fire on cost_facts ingestion outages, which is the intended design. | Pre-fix the fallback path on alicloud_intl scopes would have shown the ~200× undercounted billing_daily total when cost_facts was missing. Post-fix the fallback is honest. |
| `last_seen_billing_daily_at` per-scope freshness on `/admin/cloud/scopes` | Refreshes naturally — the backfill calls `sync_billing_daily_for_integration` per chunk which updates the scope's `last_seen_*` timestamps. No manual reset needed. | The scope-staleness banner ("last billed 12 days ago") was misleading pre-fix because billing_daily was running but landing wrong numbers; post-fix the timestamp and content are both honest. |

**If totals are still off after backfill** (cf vs bd > ±30%): pull a sub-account's raw bill via Aliyun BSS console (https://account.aliyun.com → 费用账单 → 账单总览 → 账单分组 = 资源拥有人) for the same date, compare the per-product sum to the cost_facts breakdown. If console matches cost_facts but Helix billing_daily still under-counts, suspect a missing `BillOwnerId` propagation in a third call site (the fix only patches `list_billing_daily`; `list_billing_by_dimension` on aliyun is a planned follow-up if finance dashboards need cross-cloud GROUP-BY on alicloud_intl).

### 9a-backfill. Historical cost backfill (multi-month, all integrations) — 2026-05-15

The default cost crons (`cmdb.billing_daily_sync` + `cmdb.cost_facts_sync`) only pull **yesterday's** data, so a fresh deploy or a long-stale integration silently shows $0 for every historical month even when the vendor side has the rows. The user-visible symptom: open `/my/cloud/grants` or `/admin/cloud/overview`, pick a 3-month-old billing month, every subscription shows `$0`. The fix is **one-shot** `server/scripts/backfill_cost_facts.py`.

**What the script does** — walks every enabled `cloud`-kind integration over the requested window in 30-day chunks, calling the same `sync_billing_daily_for_integration` + `sync_cost_facts_for_integration` paths the daily cron uses. The cron-shared distributed lock (`cloud_sync:billing_daily:{id}` / `cloud_sync:cost_facts:{id}`) prevents race with the 04:xx tick. UPSERT via `fact_key UNIQUE` → idempotent re-runs are a no-op. After all integrations finish, runs a single `recompute_allocation` over the full window so denormalised `cost_owner_*` columns reflect every active `CmdbCostAllocationRule` AND the up-to-date `cmdb_cloud_scopes.owner_person_id` fallback.

**Local-dev run** (default 6-month window):

```bash
cd server
uv run python scripts/backfill_cost_facts.py
```

**Prod run** — REQUIRES the explicit env var, refuses to fall back to `DATABASE_URL`:

```bash
HELIX_PROD_DB_URL_ASYNC='mysql+aiomysql://...@tidbcloud.com:4000/helix?ssl=true' \
  uv run python scripts/backfill_cost_facts.py --prod --dry-run
# dry-run prints the plan but issues no vendor billing API calls.

# When the dry-run output looks right, drop --dry-run:
HELIX_PROD_DB_URL_ASYNC='...' uv run python scripts/backfill_cost_facts.py --prod
```

**Window controls**:

* `--months N` — trailing N months from today (default 6). Capped at ~14 months (AWS Cost Explorer retention floor); older windows are refused.
* `--start YYYY-MM-DD --end YYYY-MM-DD` — explicit window; both required if either is.
* `--integration-id <uuid>` — restrict to one integration (debugging).
* `--dry-run` — print plan only, no vendor calls.
* `--skip-recompute` — skip the post-backfill allocation recompute (rarely useful — without it backfilled rows keep stale owner columns).

**Vendor-side retention** (script enforces 425-day floor, but per-vendor caps still apply mid-window):

| Vendor | Retention horizon | Notes |
|---|---|---|
| AWS Cost Explorer (SG) | 14 months default (~425d) for `GetCostAndUsage`; only 14 days for resource-level | Driver sub-chunks resource-level internally |
| AWS BigQuery CUR (main) | Bounded by the export table's own retention (admin-controlled, typically 12-24 months) | See §9a-aws-bq |
| Azure Cost Management | 13 months for most subscription types | No hard chunk limit; 30-day chunks are still preferred for response size |
| Aliyun BSS | 6-12 months depending on plan | |
| GCP BigQuery export | Bounded by the export table | Standard vs detailed flavours; driver auto-detects |

**Exit codes**:

* `0` — all integrations processed (some may have been skipped with `status=skipped` — that's reported, not a failure).
* `1` — at least one integration errored. Summary lists which.
* `2` — environment / config error caught before any sync call (no prod URL, no enabled cloud integrations, bad date format, etc.).

**Per-integration result summary** (printed at end as JSON):

```json
[
  {
    "name": "aws-sg",
    "provider": "aws",
    "status": "ok",
    "billing_daily_chunks": 6,
    "cost_facts_chunks": 6,
    "errors": []
  },
  ...
]
```

**Common follow-ups**:

* After backfill, verify the new data: `SELECT MIN(date), MAX(date), COUNT(*) FROM cmdb_cost_facts WHERE integration_id=<uuid>` — should now span the requested window.
* The `subscriptions.cost_ranking` panel on `/admin/cloud/overview?tab=subscriptions` uses the new data immediately on next page load.
* The per-scope `ScopeBillingPanel` (embedded on `/my/cloud/scopes/:id` and `/admin/cloud/scopes/:id`; row click on `/my/cloud/grants` or the admin Subscriptions tab navigates to the corresponding detail page) renders the new data on next page open.
* AWS main account (`billing_source=bigquery`) reads CUR-in-BQ — backfill works the same way; the driver routes through BQ instead of Cost Explorer. CUR historical data is bounded by what's been in the BigQuery dataset since pchub built the FinOps pipeline.

**Alert safety (2026-05-16, post-incident)**: the backfill script ALWAYS passes `dispatch_alerts=False` to `sync_billing_daily_for_integration`. Pre-fix, every backfill chunk's `end_date` was treated as "today" by `evaluate_scope_alert_rules` + `_alert_spike`, so a 6-month replay fired one Feishu card per scope per crossed threshold per chunk — prod 2026-05-16 saw ~7 cards per scope across ~60 scopes (Nov / Dec / Jan / Feb / Mar / Apr / May × scope). With `dispatch_alerts=False` the `cmdb_cloud_scope_alert_fired` slots STILL claim (so the next live cron tick doesn't re-page on historical-month data) but Feishu/email aren't sent. **You should not need to do anything special at the operator level** — the script is now safe to run during business hours without spamming subscription owners. The cost_facts side of the script does not fire alerts (no equivalent path) and never has.

If you must hot-fix prod alert state directly (e.g., another wave of false-positive cards landed and you need to stop the bleed before deploying the code fix), the runbook is:

```sql
-- Stamp every applicable scope's legacy last_fired map for current month + every
-- historical month whose total exceeded the rule's threshold. This works as
-- both a deploy-day back-compat skip and a defensive guard.
UPDATE cmdb_cloud_scopes SET alert_rules = JSON_SET(alert_rules,
  '$[*].last_fired', JSON_OBJECT('monthly:2026-05', true, /*…*/))
WHERE alert_rules IS NOT NULL AND JSON_LENGTH(alert_rules) > 0
  AND deleted_at IS NULL;
-- (See server/scripts/ for a Python-driven version that computes the right
-- months per scope based on actual cmdb_cloud_billing_daily totals.)
```

The new `cmdb_cloud_scope_alert_fired` table is the **authoritative** dedup post-migration-171. Pre-deploy the JSON map is still readable as a deploy-day fast-path skip — see PLATFORM.md §6.1 "Claim-first dedup".

### 9a-billing. Billing daily sync: unregistered accounts + owner breakdown (added PR #161, 2026-05-13)

**Unregistered linked account reconciliation** — `sync_billing_daily_for_integration` now runs a second pass after the per-scope loop. It calls `driver.list_billing_daily(scope=None)` to retrieve ALL Cost Explorer-visible linked accounts, then filters out accounts already in `registered_external_ids`. Remaining unregistered accounts are written as `scope_id=NULL, linked_account_id=<acct_id>` rows. Idempotency: DELETE all `scope_id IS NULL` rows for the integration + date range before INSERT (MySQL/TiDB treats NULL != NULL in unique indexes so ON DUPLICATE KEY UPDATE would never fire). The billing overview aggregates these rows into `unregistered_cost_usd`. **Symptom before fix**: Helix billing total was ~$1,170 lower than AWS Cost Explorer for AWS SG May 2026 ($979.48 vs $2,150.36).

**iamPrincipal/Owner tag breakdown** — AWS driver third CE pass queries `GroupBy=[TAG:iamPrincipal/Owner]` per registered scope when `include_owner_breakdown=True`. Important: the correct CE tag namespace is `iamPrincipal/Owner`, NOT `aws:PrincipalTag/Owner` (the IAM-policy evaluator namespace — CE silently returns empty strings for it). Tag values strip the `"iamPrincipal/Owner$"` prefix. Results stored in `cmdb_cloud_billing_daily.owner_breakdown` (JSON). `BillingOverview.owner_breakdown` aggregates the per-day maps across all rows in the date range.

**Triage runbook — "Helix billing total still lower than CE"**:

| Symptom | Check | Fix |
|---|---|---|
| `unregistered_cost_usd > 0` in billing overview | Expected — these are accounts not yet registered in Helix | Register the linked account as a `cmdb_cloud_scopes` row via `/admin/cloud/scopes` |
| Total still < CE after next sync | `SELECT linked_account_id, SUM(total_cost_usd) FROM cmdb_cloud_billing_daily WHERE scope_id IS NULL GROUP BY 1` — compare to CE | If rows missing, check that `list_billing_daily(scope=None)` didn't raise (look at `scheduled_job_runs.errors`) |
| `owner_breakdown` is empty for an AWS scope | CE `iamPrincipal/Owner` tag not activated in Billing Console, or no resources tagged | Activate in AWS Billing Console `Cost allocation tags → User-defined → Owner`; forward-only, historical spend stays unallocated |

### 9b. Daily spike alerts (scope-level rollup)

`cmdb.billing_daily_sync` (06:00) writes per-scope daily cost into `cmdb_cloud_billing_daily`, then evaluates:

```
spike_ratio = (yesterday_cost / avg_7d_cost) - 1
if spike_ratio > cloud.daily_spike_multiplier  # default 2.0
  → fire Feishu alert to scope owner + IT
```

Configure threshold in `platform_settings.cloud.daily_spike_multiplier`. Alert webhook in `platform_settings.feishu.alert_webhook_url`.

### 9c. Monthly budget alerts

`cmdb.billing_sync` (02:00 on 2nd of month) compares:

```
total_cost_usd > coalesce(
  platform_settings.billing.budget_usd.<integration_slug>,
  platform_settings.billing.budget_usd.default
)
```

If exceeded → Feishu alert with % overage. No state change; purely notification.

### 9d. Per-scope alert rules (`alert_rules` JSON array on `cmdb_cloud_scopes`)

Each scope can carry custom rules — daily spike %, monthly % of budget, absolute USD threshold. Evaluated alongside global rules. UI: scope detail page → "Alerts" tab.

**At-most-one-alert-per-period guarantee (2026-05-16 pre-launch hardening)**: tuning a threshold value mid-period (e.g. monthly $500 → $100) does NOT re-page the same scope. The dedup table `cmdb_cloud_scope_alert_fired` has UNIQUE-key including `threshold_usd` for audit archaeology, but `evaluate_scope_alert_rules` / `sync_billing_daily_for_integration` consult `_period_already_alerted(scope_id, rule_type, period_key)` BEFORE attempting a claim — that helper ignores `threshold_usd`. WITHIN-run tier suppression (3 thresholds at 50%/80%/100% all crossed in one run → all 3 slots claim, only the highest dispatches) is preserved via a per-call SNAPSHOT taken before the rule loop: snapshots see only PRE-RUN history, so within-run claims by earlier iterations don't short-circuit later ones.

**Empty-recipient signal (2026-05-16 pre-launch hardening)**: if a fired rule (or spike) lands on a scope with NO owner + NO billing-class grant holders + NO rule `cc_person_ids`, the dedup slot is consumed (otherwise every cron re-evaluates an empty recipient set — log spam without action) but a `logger.warning("cloud.scope_alert dispatched to empty recipient set: …")` or `cloud.spike_alert empty recipient set: …` line is emitted. Searchable in the log aggregator:

```bash
# In the log aggregator (Splunk / Datadog / cloud equivalent)
grep "cloud.scope_alert dispatched to empty recipient set" --last-30d
grep "cloud.spike_alert empty recipient set"               --last-30d
```

**Remediation per affected scope**: open `/admin/cloud/scopes/<id>` → set an owner OR `/admin/cloud/scopes/<id>/grants` → grant a billing-class role (e.g. `BillingViewer` / `BillingAdmin` per provider) to whoever should receive alerts. Next period (different `period_key`) the alert re-evaluates and the slot rolls fresh; mid-period the slot stays consumed so admin shouldn't expect a retroactive card.

**Refund / credit day handling**: cloud refunds / credits write as negative `total_cost_usd`; the spike detector adds `today_usd > 0` to the trigger condition so a post-refund return-to-normal (avg negative, today positive) doesn't fire a spurious "spike high" on regular usage. No operator action — purely defensive.

### 9e. Investigate a cost spike

1. Open `/admin/cloud/billing` → drill into the scope.
2. Look at `cmdb_cloud_billing_daily.cost_by_service` for the spike day — it's a JSON dict of service-level cost.
3. Cross-reference with `cmdb_servers` / `cmdb_object_storage` first-seen timestamps near the spike — often a forgotten resource.
4. If it's AI usage, hit `/admin/ai-spending` → filter by scope.

---

## 10. Access expiry: notify, revoke, escalate

`cmdb.access_expiry_sweep` (08:00 daily) walks every active grant. **cloud_admin holders skipped.** **prod scopes fully unmanaged — no reminder, no revoke, no notification (2026-07-16).** **scope owners skipped on their own grants (2026-05-12).**

| Stage | Trigger | Action | Idempotency marker |
|-------|---------|--------|---------------------|
| t14 | `expires_at - 14d <= now < expires_at - 7d` | Email/Feishu reminder to grant holder. | `last_notified_stage='t14'` |
| t7 | `expires_at - 7d <= now < expires_at - 1d` | Reminder to holder + cc owner. | `t7` |
| t1 | `expires_at - 1d <= now < expires_at` | Final reminder. | `t1` |
| t0 (default) | `expires_at <= now` | `driver.revoke_access()` → `status='expired'`. Notify holder + owner. | `t0` |
| any stage (prod scope) | `scope.environment == 'prod'` | **Skipped at every stage — no reminder, no revoke, no notification (member or owner).** Prod cloud access is not managed by Helix; the R&D security team owns it. | counted in `prod_reminder_skipped` / `prod_skipped` summary keys |
| t0/t* (owner row) | `grant.person_id == scope.owner_person_id` | **Skipped at every stage.** No reminder, no revoke. Owner manages their own access. | counted in `owner_skipped` summary key |

**Prod is fully unmanaged (2026-07-16).** Production scopes (`environment='prod'`) are excluded from *every* stage of the sweep: no T-14/7/1 reminder, no T+0 revoke, and no owner/member notification of any kind. Prod cloud access is never auto-recycled and the prod cloud-write lock is default-LOCKED, so a "一键续期" card would both be false ("will be auto-recycled") and a dead-end (the renew click 403s on the lock, surfacing as "链接无效或已过期"). Before this fix the reminder pass did NOT exclude prod (only the revoke pass did) and a prod-overdue owner ping fired — both are now gone. The one-click renew endpoint also short-circuits any *already-delivered* prod card to a reassuring "生产访问长期有效,无需续期" page instead of the lock error. To actually change prod access, the R&D security team acts directly (break-glass the prod write lock if a Helix-side edit is needed). The internal log line + `prod_skipped`/`prod_reminder_skipped` summary counts remain for oncall visibility.

If you find an expired grant the user says shouldn't have expired:
1. Check the scope environment: `SELECT environment FROM cmdb_cloud_scopes WHERE id=?;` — if prod, the sweep never revokes; the row just shows expired because nobody renewed.
2. Check `scope.owner_person_id` against the grant's `person_id` — if they match, the sweep also never touches it.
3. Check `last_notified_stage` — if it's `t0`, the sweep already revoked.
4. Renew if the user still has business need: `POST /api/v1/cloud-lifecycle/grants/{id}/renew` (this re-issues the grant cloud-side too, not just bumps the timestamp).

**Per-grant commit (2026-07-10, P0 login outage).** The sweep used to run its whole pass in ONE DB transaction. On 2026-07-10 08:00 CST a T-14 reminder storm (907 scanned / 821 notified, 20.6-minute run) held TiDB FK locks on every notified employee's `identity_persons` row for the entire run (`notification_outbox.recipient_person_id` FK check locks the parent row until COMMIT) — every affected employee's login (`UPDATE identity_persons SET last_login_at`) hit 1205 lock-wait timeouts, so Google/Feishu login 500'd from 08:21 to 08:35 until the sweep committed. Same "long transaction across external I/O" bug class as the 2026-05-27 billing_daily/roles_sync/grants_sync fix (§ above) — the expiry + inactivity sweeps were missed in that round. Both now `commit()` after every processed grant/group; the stage markers in the table above are the idempotency guards that make per-item durability safe. **Operationally**: a sweep crash mid-run no longer rolls back already-sent notifications' stage markers (no duplicate reminders on rerun), and mass-reminder days no longer affect login.

**Owner-exemption rationale (2026-05-12).** The scope's owner is the human who approves every other member's grant on the scope. Auto-revoking the owner's own grant creates a chicken-and-egg deadlock: owner can't approve their own renewal (the approve action requires owner role, which they just lost when the sweep revoked their grant), owner can't approve any other member's renewal either, and "re-request access to a scope you own" is busywork that solves nothing. Same rationale and stage-by-stage shape as the existing `cloud_admin` privileged-role exemption — but **keyed on the (scope, person) pair, NOT globally**: a person who owns scope A and holds a regular grant on scope B keeps the scope-B grant on the normal expiry clock. Frontend renders the chip as a green "Owner · 不过期" pill (see `ExpiryChip` in `web/src/pages/cloud/`) instead of a misleading countdown.

---

## 11. Inactivity sweep — REMOVED 2026-05-09

The `cmdb.scope_inactivity_sweep` cron (originally documented here as the
"60+ days no login → notify owner → escalate IT → auto-disable" pipeline)
was removed in migration 138 + the matching handler delete. The TL;DR
of why is the same TL;DR for why a daily false-positive alert kept landing
in scope owners' inboxes:

* **The `last_login_at` signal was unreliable.** The sweep depended on
  `CmdbCloudAccessGrant.last_login_at`, which `refresh_person_last_login_in_scope`
  populated from each cloud's IAM-side login telemetry (AWS
  `PasswordLastUsed` + `AccessKeyLastUsed`, Aliyun RAM equivalent, etc.).
  In practice the providers we ship don't expose a queryable signal for
  the SSO-class logins our users actually do — AWS Identity Center +
  Azure portal SSO sign-ins aren't surfaced through the IAM lastUsed
  APIs we wired up. So `last_login_at` stayed NULL on most active
  grants, and the sweep's NULL-fallback branch (`granted_at < cutoff`)
  ended up firing "60+ days inactive" on people who actively used
  their access daily.
* **Group-by + privileged-role + 50%-safety-floor patches helped, but
  didn't fix the data-source problem.** PR #175 introduced the
  group-by aggregation; PR review-pass added the privileged-role
  exemption and the Stage-2 helper split; PR #201 hardened the
  drift-sweep precursor. None of those patches changed the fact that
  the sweep was reading from a column we don't reliably populate.

**What replaced it.** Nothing on the inactivity axis. Going forward the
platform recycles cloud grants exclusively via the *expiry* axis —
[`cmdb.access_expiry_sweep`](#10-cloud-access-expiry-sweep) hits
`expires_at` deterministically, T-14 / T-7 / T-1 nudges the user to
renew, T+0 auto-revokes (prod scopes excepted — held with owner
notification). That's an honest, evidence-based gate; the inactivity
sweep was a guess dressed up as policy. If a person doesn't renew they
lose access at expiry, which is the same end-state the inactivity
sweep was approximating.

**What's still in code.** The lifecycle-service helpers
(`sweep_inactive_grants`, `refresh_person_last_login_in_scope`,
`disable_grant_due_to_inactivity`) remain in
`server/app/domains/cmdb/lifecycle_service.py` so the existing test
suite (`test_grants_inactivity_and_drift.py`) still exercises the
disable-on-inactivity branch. They're no longer wired to any cron and
no caller in handlers / routers reaches them. Treat them as
"infrastructure for a future, evidence-based inactivity signal" rather
than active code.

**Settings.** The `cloud.inactivity_threshold_days /
inactivity_escalation_days / inactivity_disable_days` rows in
`platform_settings` were intentionally left in place. Removing them
would mean a follow-up migration to put them back on resurrection;
the cost of the orphan rows is zero and the rows document the intended
thresholds if a future sweep ever returns.

**Migration 138** (`138_remove_scope_inactivity_sweep.py`) deletes the
`scheduled_jobs` row keyed on `handler_key='cmdb.scope_inactivity_sweep'`,
cascading the run history. Without that delete, the scheduler runner
would log "Handler not registered" every tick and migration 137's
failure-dedup would push exactly one in-app card per fresh signature.

---

## 11a. Discovered-grant drift sweep

`cmdb.grants_sync` (05:00 daily) ends with a per-scope mark-and-sweep over `cmdb_cloud_access_grants`:

```
WHERE scope_id IN (synced_scope_ids)
  AND source = 'discovered'
  AND status = 'active'
  AND last_seen_grants_sync_job_id != current_run_id
```

Matched rows flip to `status='revoked'` with `revoke_reason='drift_removed'`, `revoked_at=NOW()`, `last_notified_stage=NULL`. One audit row per integration-run with `action='cloud.grant.drift_removed_sweep'` carrying `{integration_id, provider, closed_count, scopes_scanned}`. Summary exposes `drift_closed: int`.

**Why it exists.** Pre-2026-05-05, when an admin detached a policy / deleted an IAM user / revoked a permission-set assignment cloud-side, the corresponding Helix `cmdb_cloud_access_grants` row stayed `active` forever. The inactivity sweep would then keep firing `[云账号不活跃]` alerts for permissions that didn't exist anymore (Lingtao's `IAMUserChangePassword` scenario). The drift sweep keeps the local picture honest by closing rows the cloud's authoritative response stopped including.

**Why scope-isolated.** A driver exception for one scope is NOT eligible for drift sweep — only scopes where `driver.list_scope_grants(scope)` returned successfully (or where the IAM-user fallback ran) are passed in `synced_scope_ids`. A transient outage cannot snowball into mass revoke across the integration.

**Helix-issued rows are immune.** `source='helix'` rows are never closed by the drift sweep — Helix is system-of-record for those, and cloud-side disappearance is a compliance signal handled by the existing `discovered_grant_drift_removed` `compliance_incidents` flow elsewhere, not an auto-revoke.

**Self-service AWS managed policies.** `IAMUserChangePassword` / `IAMUserSSHKeys` / `IAMSelfManageServiceSpecificCredentials` are filtered at synthesis time inside `_derive_grants_from_iam_users` — they never become tracked grants in the first place. AWS auto-attaches them to every IAM user; tracking them as grants generated dozens of false alerts per leaver. Match by ARN suffix AND by exact policy name (Tencent / RAM fallback path stores name only). Adding a new self-service policy → update both `_AWS_SELF_SERVICE_MANAGED_POLICIES` in `lifecycle_service.py` AND the `_SELF_SERVICE_ROLE_KEYS` list in migration 118 (or write a fresh migration if the bad rows are already in production).

### 50% safety floor + admin override (added 2026-05-06)

The drift sweep refuses when `existing_active >= 5 AND matched_active / existing_active < 50%`. The run summary surfaces an `errors[]` entry of the form:

```
drift_sweep refused: matched 9 of 37 existing discovered grants (24%) is
below the 50% safety floor — likely driver pagination truncation or transient
outage. Investigate before re-running.
```

The check exists because driver pagination bugs (boto3 truncation, API-throttle mid-stream, AWS regional outage) silently return a partial enumeration that looks identical from outside Helix to "the admin really did delete most users". Without the floor, one bad enumeration would close the entire integration's discovered-grant set.

**The floor's failure mode.** Once tripped on a real bulk cleanup, the sweep deadlocks: the floor refuses → stale rows can't close → match rate stays low → next sync trips the floor → forever. Every scheduled run after the bulk-cleanup logs the same `drift_sweep refused` error.

**Diagnosis.** When `drift_sweep refused` shows up in the run history:

1. Confirm cloud-side state directly: log into the AWS / Aliyun / Tencent / Azure / GCP console and verify the user/role count for the affected scope.
2. Spot-check a few `last_seen_grants_sync_job_id != current_run_id` rows in `cmdb_cloud_access_grants WHERE source='discovered' AND status='active'` — does the corresponding cloud principal still exist with that role attachment?
3. If the cloud-side picture matches Helix's `matched_active` count → the drift is real (admin-driven cleanup). Use the override.
4. If the cloud-side principals still exist but Helix didn't enumerate them → driver bug. **Do NOT override.** Investigate the driver: check rate-limit headers, region-failover, IAM permission changes (`iam:ListUsers` / `iam:ListAttachedUserPolicies` for AWS).

**The override.**

```
POST /api/v1/cmdb/integrations/{integration_id}/grants-sync/force-drift-sweep
```

Cloud-admin only. Holds the same `cloud_sync:<integration>` distributed lock the cron uses (cannot race a regular sync). Returns the full sync summary including `drift_closed_grant_ids: list[str]` so the caller can show exactly what was closed. Writes a `cloud.grant.drift_sweep_force_override` audit row attributing the decision to the admin user; the inner sweep audit (`cloud.grant.drift_removed_sweep`) gains `force_override: True`, `matched_active`, `existing_active` in detail. The MCP equivalent is `force_cloud_grants_drift_sweep(integration_id)`.

**UI affordance.** The admin runs page surfaces this directly — no need to remember the curl. Both `ScheduledJobsPage` per-job runs drawer (Admin → Scheduled Jobs → AWS SG / Aliyun / … → 历史) and `ScheduledJobRunsPage` global view (Admin → Scheduled Jobs → 全部执行日志) detect the `drift_sweep refused` error on a failed run and render a yellow warning banner with `[已确认漂移真实，强制扫尾]`. Click → Popconfirm → backend call → toast shows `已关闭 N 条漂移的 grant`. Both pages refresh automatically after success so the failure goes green.

### Why the UI calls drift-closed rows "云端关闭" not "已撤销" (added 2026-05-17, widened 2026-05-17 Round-23)

The DB stores `drift_removed` rows as `status='revoked'` (the grant is no longer effective), but rendering that literally as "已撤销 / Revoked" in the UI misleads IT admins into thinking **Helix** revoked the permission. Helix did NOT — the cloud-side admin did, and the drift sweep only updated the local ledger. Three closure reasons fall in this "cloud-side-initiated" bucket: `drift_removed`, `iam_user_disowned`, `orphan_principal_swept`. Every UI surface displaying a cloud grant's status badge now special-cases them:

| Surface | File | Notes |
| --- | --- | --- |
| Admin overview → Permissions → 最近回收记录 | [PermissionsTab.tsx](../../web/src/pages/cloud/cloudOverview/PermissionsTab.tsx) `CLOUD_SIDE_REASONS` | Top-30 most recent closures. |
| Admin scope detail → 已结束 (历史) | [CloudScopeDetailPage.tsx](../../web/src/pages/cloud/CloudScopeDetailPage.tsx) `HistoricalGrantsTab` | Status filter exposes `cloud_closed` as a separate option; `revoked` predicate excludes cloud-side rows. |
| Admin scope detail → 成员 → expand-row | [CloudScopeDetailPage.tsx](../../web/src/pages/cloud/CloudScopeDetailPage.tsx) `perGrantActionColumns` | Per-role sub-table for one member. |
| Owner scope detail → expand-row | [MyCloudScopeDetailPage.tsx](../../web/src/pages/cloud/MyCloudScopeDetailPage.tsx) `perGrantActionColumns` | Same view from the owner's self-service surface. |
| Employee → My Cloud Grants → 已结束 | [MyCloudGrantsPage.tsx](../../web/src/pages/cloud/MyCloudGrantsPage.tsx) status column | Dominant-status pill collapses to `cloud_closed` when every revoked sibling is cloud-side initiated. |
| IAM-user revoked tooltip | [CloudIamUsersPage.tsx](../../web/src/pages/cloud/CloudIamUsersPage.tsx) | Tooltip text translated via `cloudLifecycle.grants.endedReason.*`; badge stays "已撤销" because `CmdbIamUser.revoked_at` is always Helix-initiated. |

**Software seat parallel (Round-23)**: same bug class applies to SaaS seats. Slack / Zoom / Google / Jira / Feishu `seat_sync` crons mark `seat.status='revoked'` and stamp `seat.reason` with `[auto-revoked by <vendor>.seat_sync: ...]` when the upstream vendor deletes the user. Render override via [`web/src/utils/seatStatus.ts`](../../web/src/utils/seatStatus.ts) `displaySeatStatus(status, reason)` — vendor-drift seats render `vendor_closed` ("厂商已删除 / Closed in vendor", cyan, tooltip). Applied at [`SoftwareCostPage.tsx`](../../web/src/pages/asset/SoftwareCostPage.tsx) seat status column and [`UserItUniverseTab.tsx`](../../web/src/pages/admin/UserItUniverseTab.tsx) `renderSeatStatusTag` helper (two render sites). The `[auto-revoked by leaver_block_guard ...]` / `approval callback` / requester-cancelled markers do NOT match — those are genuine Helix-initiated actions and stay "已撤销".

Origin: user incident 2026-05-17. Six rows on `dbaas-prod-*` and `pingcap-dbaas` (`source='discovered'`, all six) showed as "已撤销 / 云端漂移清理" in the recent recoveries table 19-20h after the cloud-side admin removed them. The IT admin reasonably interpreted this as Helix auto-revoking prod permissions. Prod audit confirmed 15/15 recent `drift_removed` rows were `source='discovered'`, AWS sweep closed 5/1241 (99.6% match), AliCloud 1/230 (99.6%), Azure 1/449 — all far above the 50% floor, all healthy small drift. Round-23 (later same day) extended the fix to the 4 sibling UI surfaces missed in Round-1 and to the analogous SaaS-seat path.

---

## 11a-bis. Resource-sync safety-floor refused — admin override runbook (added 2026-05-14)

**The class of failure.** `cmdb.resource_sync` per-(integration × capability) cron sees the cloud return fewer rows than Helix has on record. Helix's `_safe_sweep` refuses to soft-delete the missing rows when either:

* `upserted == 0 AND existing > 0` (provider returned nothing but the table has rows — almost always a partial driver failure that swallowed an exception and returned `[]`); or
* `upserted < existing × (1 - tolerance/100)` (drop exceeds the configured tolerance — same family of suspicion, just with some rows returned). This is evaluated as an exact ratio, without rounding the required row count down: at 20%, 2→1 refuses, 5→4 is the exact allowed boundary, and 6→4 refuses. Tolerance is read from `platform_settings.cmdb_sync.sweep_drop_tolerance_percent` (default 20 — was hardcoded 5 pre 2026-05-19). **Edit the row in Platform Settings admin page; effective on the next sweep call, NO worker restart needed.** `.env` `CMDB_SYNC_SWEEP_DROP_TOLERANCE_PERCENT` remains the fallback if the row is missing / malformed.

Both raise `ProviderTransientError`. The capability is marked `failed`, the next cron tick retries, and the failure-notifier pages an IT admin. The error string is `<capability>: provider returned N rows vs M existing (>X% drop) — refusing to sweep (likely partial driver failure).` (X = the configured tolerance) or `<capability>: provider returned 0 rows but M exist for this integration — refusing to sweep ...`.

**Why this is needed.** A driver hiccup that returns `[]` would otherwise mass-soft-delete every server / domain / IAM user the integration ever discovered. Soft-deletes ARE reversible (`UPDATE … SET deleted_at = NULL`), but mass-deleting then mass-restoring 1000+ rows across several capabilities is noisy at best, dangerous at worst (every soft-delete that intersects a Helix grant or seat allocation triggers downstream business logic — cleanup-on-disown, billing recomputation, alert recipient filtering).

**The floor's failure mode.** Once tripped on a *real* bulk cleanup (admin manually closed 11 of 20 IAM users in the AWS console; a region truly emptied out; a whole cloud project was deleted) the sweep deadlocks: the next sync still sees N < 95% of M, still raises, still marks failed. Every cron tick logs the same error. Clicking 「标记为已确认」 only hides the row from the default red queue; the underlying state (M rows in DB, only N returned from cloud) stays unchanged, and the failure-notifier re-pages on every subsequent failure because acked-prior-failure + new-failure is the dedup gate's notify trigger (intentional: ack means "I saw this, show me when it happens again"). The user-perceived symptom: "I clicked acknowledge but the alert keeps coming back" — a perceived dead loop. Bug report 2026-05-14.

**Diagnosis.** When `refusing to sweep` shows up in the run history for `cmdb.resource_sync`:

1. Confirm cloud-side state directly: log into the AWS / Aliyun / Tencent / Azure / GCP / UCloud / AliCloud-Intl console and verify the actual count for the affected capability + region(s).
2. Cross-check Helix's count: `SELECT COUNT(*) FROM cmdb_<capability> WHERE integration_id = '...' AND deleted_at IS NULL;` (use `cmdb_iam_users`, `cmdb_servers`, `cmdb_domains`, etc).
3. If the cloud-side count matches what the sync returned → the drop is real (admin-driven cleanup; region offline; project deleted). Use the override.
4. If the cloud-side count is closer to Helix's existing count → driver is paginating wrong or auth-failing mid-stream. **Do NOT override.** Investigate the driver: rate-limit headers, IAM permission revocation (`iam:ListUsers`, `iam:ListAttachedUserPolicies`, `RAM:ListUsers`, equivalents), regional outage. Once the driver returns the full set the next sync resolves on its own.

**The override.**

```
POST /api/v1/cmdb/sync/{integration_id}/force-sweep
   body: { resource_types: ["iam" | "servers" | "domains" | ...], reason?: "..." }
```

`global_admin` only (via `can_manage_cloud`; `cloud_admin` is a perpetual-grant marker, not an authority). `resource_types` is required and must be non-empty — the admin picks WHICH capabilities to force-sweep, no wildcard. The endpoint:

* validates each capability is supported by the provider (400 otherwise);
* creates a `cmdb_sync_jobs` row with `triggered_by='force_sweep'` + audit row `cmdb.sync.force_sweep_override` BEFORE entering the lock;
* holds the `cloud_sync:<integration>` distributed lock (same lock the scope sync / grants force-drift-sweep / regular cron use) — 409 `CLOUD_SYNC_ALREADY_RUNNING` if contended;
* runs `run_sync(force_sweep=True)` synchronously inside the lock so the response carries the final status;
* returns the refreshed `SyncJob` whose `capability_results[cap]` contains `forced_sweep: true`, `forced_sweep_guard` (`"drop_ratio"` | `"zero_rows"`), `forced_sweep_existing` (the pre-sweep count), and for drop-ratio also `forced_sweep_upserted`.

The audit detail carries `integration_id`, `provider`, `resource_types`, `reason`, `job_id`. Soft-deletes from the sweep are reversible by clearing `deleted_at`; the audit row is the forever record of WHO chose to bypass.

---

## 11a-ter. Scope / role sync safety-floor refused (added 2026-05-16, pre-launch P0)

**The class of failure.** `cmdb.scopes_sync` and `cmdb.roles_sync` cron jobs see the cloud return fewer rows than Helix has on record AND the gap exceeds the configured tolerance (both lifecycle_service.py call sites `await get_sweep_drop_tolerance_percent(session)` from [sweep_settings.py](../../server/app/domains/cmdb/sweep_settings.py), same DB-backed knob as `sync.py::_safe_sweep`; default 20, was hardcoded 5 pre 2026-05-19). The matching `sync_scopes_for_integration` / `sync_roles_for_integration` in [`lifecycle_service.py`](../../server/app/domains/cmdb/lifecycle_service.py) `raise RuntimeError(...)` on:

* `upserted == 0 AND baseline > 0` (driver outage returned `[]`), or
* `upserted < baseline × (1 - tolerance/100)` (drop exceeds tolerance).

The corresponding handler now `raise`s on `result.status='failed'` OR non-empty `errors[]` so `scheduled_job_runs.status` lands `failed` and `failure_notifier` actually pages. Error string examples:

* `scope sync: provider returned 0 scopes but baseline is 47 active scopes for integration 'aws-primary' — refusing close-sweep (likely partial driver failure).`
* `role sync: every scope's list_scope_roles failed (errors=12) but baseline is 158 roles for integration 'aws-primary' — refusing soft-delete sweep (likely transient driver failure).`

**Why this is needed.** Before this layer, a single Feishu-side / cloud-API blip would:

* close every `active` scope in the tenant (billing UI / self-service grant flow blank, every IAM grant orphaned),
* soft-delete every role catalog row (request modal dropdown empty across the org),

while the cron history quietly showed green (because the inner function returned `{"status":"failed"}` and the handler returned that dict without raising — `feedback_handler_status_must_raise` violation that bit the codebase 5+ times pre-launch).

**Diagnosis.** When `refusing close-sweep` / `refusing soft-delete sweep` shows up:

1. Direct cloud-console check: are the missing scopes / roles really gone, or did the cron call out into a permission error?
2. Cross-check the inner `errors[]` list in the run summary — if non-empty AND content references the same `external_id` repeatedly, the driver is broken for that specific scope (auth scope drift, permission revoked).
3. If the cloud-side count matches the sync's returned count → drop is real (admin closed accounts cloud-side; reorg). Trigger the regular admin "force re-sync" flow via the API once you've audited the diff.
4. If the cloud-side count is closer to Helix's existing baseline → driver / auth issue. Fix the upstream cause first; the next cron tick will succeed once enumeration completes.

**GCP scope-partition completeness (2026-08-02).** GCP projects and billing accounts come from independent APIs, so their discovery completeness is evaluated independently before the generic count floor. Project discovery is complete only after the configured organization hierarchy has been recursively traversed with parent-bounded `projects.list` / `folders.list`; billing discovery is complete only under an explicit `billing_scope_mode`: `projects_only` produces an authoritative empty partition, while `allowlist` retrieves and organization-verifies every configured account. Any 403/429/503, malformed identity/parent, repeated token/node, traversal cap, missing policy, or foreign billing parent marks the relevant partition incomplete. **Every incomplete partition blocks before any upsert or close-sweep, including a brand-new integration**; there is no longer a never-seen billing exception. Existing scopes and active/pending grants remain unchanged, and a healthy sibling partition never makes the run authoritative. Operator action: inspect `scope_partitions`, fix the organization/folder list IAM or billing allowlist/policy, run the integration's read-only connection test, then rerun normally. Do not raise the percentage tolerance: partition incompleteness is a hard fail-closed condition and is intentionally outside that override.

**Manual unblock for scopes / roles** — there is no dedicated force-sweep API for these (unlike `cmdb.resource_sync`'s `/sync/{id}/force-sweep` or `grants_sync`'s `/grants-sync/force-drift-sweep`); the scope/role syncs are lower-volume and historically the right answer has been "fix the driver, then let the cron resolve". If a genuine bulk-close happened and the next cron keeps tripping, options are:

* **(a) Raise tolerance temporarily.** Edit `platform_settings.cmdb_sync.sweep_drop_tolerance_percent` via the Platform Settings admin page — effective on the next sweep tick, NO restart. Pick a value that lets the next sweep proceed: tolerance `T` allows drops up to `T%`, i.e. the next sweep proceeds when `upserted >= baseline × (1 - T/100)`. If you have 9 of 20 active (55% drop) you need `T >= 56` (round up to be safe — 60 is a common pick). **Revert to 20 immediately after the unblocking sweep completes.**

* **(b) Manually close affected scopes BEFORE the next sweep.** `UPDATE cmdb_cloud_scopes SET status='closed' WHERE id IN (...);` — once Helix's `status='active'` count drops to a level that satisfies `upserted >= baseline × (1 - T/100)`, equivalently `baseline <= upserted × 100 / (100 - T)`, the floor stops tripping and the sweep proceeds. With the default `T = 20` you need `baseline <= upserted × 1.25` (e.g. close 5 rows manually when cloud reports 12 and Helix has 20).

Most chronic role-sweep refusal cases are upstream-driver problems (an integration's IAM permissions silently lost `iam:ListRoles` equivalent, every per-scope `list_scope_roles` returns auth error). Fix the upstream — do NOT bypass via raw SQL.

**No MCP tool.** Deliberately omitted, same precedent as the grants force-drift-sweep. Admin-override semantics are kept off every PAT-reachable surface so a stolen token cannot bypass the safety-floor guard. The `^force_` denylist on `test_mcp_tools_surface._FORBIDDEN_TOOL_NAME_PATTERNS` would have failed CI on a `force_cloud_resource_sweep` tool name anyway.

**UI affordance.** Admin → Scheduled Jobs → 全部执行日志 (or open the per-job drawer on `ScheduledJobsPage`) → expand the failed row. When the error is `refusing to sweep` AND the handler is `cmdb.resource_sync`, a warning banner appears with the body explaining the trade-off (parameterised on the capability — `iam`, `servers`, etc) and a Popconfirm-gated `[已确认数据真实，强制以最新同步为准]` danger button. Click → Popconfirm → backend call → toast `强制同步完成：新增/更新 N 条，软删 M 条` (or the en-US equivalent `Force sync complete: N upserted, M soft-deleted`). The failure run is auto-acknowledged on success (the issue is resolved; leaving it in the red queue would mislead the next admin). Sibling-page coverage: both the per-job drawer in `ScheduledJobsPage` and the global view in `ScheduledJobRunsPage` carry the affordance.

**Rollback path.** If the override soft-deleted rows that turn out to have been wrongly absent (driver was actually mid-failure and resumed afterwards), restore them with:

```sql
UPDATE cmdb_<capability>
SET deleted_at = NULL
WHERE integration_id = '<integration_uuid>'
  AND deleted_at >= '<the_force_sweep_timestamp>';
```

Then re-trigger the sync. The next `run_sync` will upsert any rows the cloud now returns, leaving cleared `deleted_at` on the rows that match (the `ON DUPLICATE KEY UPDATE` clause carries `deleted_at=None` on every refresh).

---

## 11b. Cloud-side orphan-principal cleanup (added 2026-05-07)

**The class of stale rows the drift sweep cannot see.** The drift sweep in §11a closes grants Helix once successfully discovered. AWS Identity Center has a different shape of stale row: a permission-set assignment whose `PrincipalId` has been deleted from the SCIM-synced Identity Store. The console renders these as `User not found (UUID)`. `aws_cloud._list_scope_grants_sso_sync` silently drops them (`describe_user` returns None → row is skipped), so they never made it into Helix's discovered set in the first place — drift sweep is blind. The result is exactly the symptom the user reported on `tidbcloud-marketplace`: `DBaaS-Prod-Admin-Role` still bound to a leaver's principal_id long after offboarding ran.

**Detection (continuous + read-only).** `sync_grants_for_integration` calls `scan_orphan_assignments` after the main commit and surfaces the count as `orphan_assignments_detected` in the run summary. The scan walks every active scope's permission-set assignments, classifies a principal as orphan only when `IdentityStore.DescribeUser` returns `ResourceNotFoundException`, and treats throttling / 5xx / auth errors as alive (err on the side of safety). NEVER auto-deletes — the entry-point for actual deletion is the admin endpoint below.

**Cleanup (admin-confirmed).**

```
GET  /api/v1/cmdb/cloud-grants/orphan-scan?integration_id={id}
POST /api/v1/cmdb/cloud-grants/orphan-cleanup
       body: { integration_id, targets: [{scope_id, role_key, principal_id, ...}, ...] }
```

Both gated on `can_manage_cloud` (cloud-admin / global-admin). The scan returns a candidate list with a best-effort `recovered_email` looked up from past `cmdb_cloud_access_grants` rows whose `external_ref` ended in `::<principal_id>` (the AWS driver's canonical ref shape is `account_id::permission_set_arn::principal_id`, so any past helix-issued or discovered grant for the same principal carries that suffix). The cleanup endpoint per-target calls `sso-admin.DeleteAccountAssignment`, inserts a `cmdb_cloud_access_grants` row with `status='revoked'`, `revoke_reason='orphan_principal_swept'`, `person_id=NULL`, populated `external_principal_id` / `external_principal_email`, and writes a `cloud.grant.orphan_principal_swept` audit row.

**UI affordance.** Admin → 云子账号管理 → click any AWS scope → 已结束 tab → 扫描云端孤儿权限 button (visible only when `provider === 'aws'`). Modal auto-runs the scan, pre-selects every row, lets the admin uncheck rows to keep, and requires a final `Modal.confirm` before deletion. Cleaned rows surface in the same 已结束 tab on the next reload (filterable by `revoke_reason='orphan_principal_swept'`).

**Provider scope (v1).** AWS only. Azure / GCP / Aliyun / Tencent drivers raise `NotImplementedError` from `list_scope_orphan_assignments`. The frontend `orphanScanSupported` whitelist hides the button for unsupported providers. Adding a new provider is mechanical: implement `list_scope_orphan_assignments` + `delete_orphan_assignment` on the driver, drop the provider into the whitelist.

**MCP**: `scan_cloud_orphan_assignments(integration_id, scope_id?)` (read) and `cleanup_cloud_orphan_assignments(integration_id, targets)` (write). Same auth gate (`global_admin` only — `cloud_admin` is a perpetual-grant marker, not an authority), same per-target outcome contract.

**Idempotency** (two layers):
1. `delete_orphan_assignment` returns `already_absent=True` on `ResourceNotFoundException`; cleanup service skips the history-row insert in that case (happy path — re-running on a clean integration is a no-op).
2. Migration 131 adds `uq_cmdb_cloud_grants_scope_role_extprincipal (scope_id, role_key(191), external_principal_id)` as belt-and-braces. NULL `external_principal_id` makes the constraint a no-op for normal helix-issued / discovered rows; for orphan-cleanup history rows, a second cleanup against the same orphan that fails to return `already_absent` (transient driver glitch) hits the UQ and rolls back the duplicate insert.

**Per-target durability**: each target is committed BEFORE the next target's driver call. A mid-batch DB failure on target #50 doesn't roll back targets 1-49 whose cloud-side state is already deleted. On a per-target Helix-side write failure AFTER a successful cloud delete, a `CRITICAL` log line names the orphan (integration / scope / role / principal_id / recovered_email / trace_id) — SRE manually inserts a recovery history row using that trace.

**RBAC + PII redaction**: `global_admin` only. The history rows surface `external_principal_id` and `external_principal_email` (the swept leaver's UUID + recovered work email). `_serialize_grants` redacts both for non-privileged readers; scope owners and members see a generic `(orphan principal)` placeholder in `person_label` instead of a leaver's identifier.

**Paired: offboarding cc-strip hook**: `LeaverBlockedGuard.flip_to_left` calls `strip_person_from_alert_cc` after the orphan-seat sweep, removing the leaver's UUID from every `cmdb_cloud_scopes.alert_rules[*].cc_person_ids`. Without this hook, leavers' UUIDs accumulated as `未知用户` tags in the alert config panel because `/scopes/{id}/pickable-people` filters out `status='left'`. Same root-cause class as Identity Center "User not found" residue — the platform mutated the principal's lifecycle but didn't sweep the references that pointed at it. Audit: `cloud.alert.cc_leaver_stripped` (success per scope modified) / `cloud.alert.cc_leaver_strip_failed` (status='failed' on the rare DB error path). Failure is non-fatal — runtime routing in `_resolve_billing_alert_recipients` already filters out missing IdentityPerson rows. **One-shot data cleanup ran 2026-05-07 against local + prod**: 145 local + 135 prod scopes had stale cc_person_ids from the pre-PR-#182 era when the cc list was used as the "all members" recipient. Wipe trace ids: `alert-cc-wipe-20260507T144543` (local) / `alert-cc-wipe-20260507T144802` (prod), each scope's pre-wipe cc list preserved in the `cloud.alert.cc_wiped` audit row's detail for restore reference.

---

## 11c. Admin Cloud Overview workspace (added 2026-05-08)

**Where**: `/admin/cloud/overview` — multi-tab dashboard styled after the operational-insights workspace. Replaces the prior single-scroll page that gave admins no visibility into "how many subscriptions / how many active grants / how many recovered permissions". Driven by user feedback after PR #200 shipped orphan-principal cleanup: the headline question "回收了多少个权限" had no UI surface, only audit grep.

**Tabs**: 概览 (Overview) / 主机 (Hosts) / 网络 (Network) / 子帐号 (Subscriptions) / 帐号与权限 (Permissions) / 身份 (Identity) / 报表 (Reports). Tenant filter is global — selecting a tenant narrows EVERY tab. URL `?tab=` deep-link sync; `destroyInactiveTabPane` so hidden ECharts don't keep rendering.

**Endpoints**:
- `GET /api/v1/cmdb/analytics?integration_id=...` — existing payload; powers Overview / Hosts / Network / Reports.
- `GET /api/v1/cmdb/overview/admin?integration_id=...` — NEW admin-only payload; powers Overview / Subscriptions / Permissions / Identity. Implementation in `server/app/domains/cmdb/admin_overview_service.py`. Five sections: `subscriptions`, `permissions`, `iam_users`, `alerts`, `filter`. Every aggregation joins through `CmdbCloudScope` so the integration filter cannot accidentally widen results.

**RBAC**: gated by `can_manage_cloud` — `global_admin` only. `cloud_admin` is a perpetual-grant marker per `app/core/rbac.py`, not an authority for this dashboard. Frontend route also gated by `RoleRoute roles={['global_admin']}` — two-layer enforcement.

**Headline metrics** the dashboard answers:
- 子帐号 distribution: by status / environment (dev/staging/prod) / scope_type / provider / integration; owner_health (owned / unowned / **owner_left_org** — joins `identity_persons.employment_status != 'active'`); recent-90d decommissions with actor attribution + provider state.
- 权限 (grants) distribution: active / pending / expired / revoked / disabled; distinct active holders; expiring T-1 / T-7 / T-14 / T-30 buckets; **active past-due** (sweeper hasn't run / prod-exempt).
- 回收 (recovered) — the user's "回收了多少个权限" ask: total, last-30d, last-90d, by-reason 30d (`user_released` / `expired` / `inactive` / `offboarding` / `owner_removed` / `admin_removed` / `drift_removed` / `iam_user_disowned` / `orphan_principal_swept`), orphan-swept totals, recent-30 history table. Orphan rows (`person_id IS NULL`, `revoke_reason='orphan_principal_swept'`) render with a special tag and fall back to `external_principal_email` for attribution.
- 身份 (IAM users): active / mapped-to-Helix / unmapped / **mfa_disabled** (high-signal compliance gap); revoked-lifetime count.
- Alerts strip on Overview: unowned scopes, prod-overdue grants, active past-due grants, owner-left-org count — each a clickable chip linking to the matching admin drill-down page.

**Edge cases handled**:
- Unknown `integration_id` → 404 (not silent zeros).
- Malformed UUID → 422 at validation gate (not a deep-SQL crash).
- Closed scopes excluded from owner_health (decommission flow intentionally orphans them).
- Owner-left-org filter restricts to `status != 'closed'` so dead scopes don't pollute the metric.

**Read-only**. The workspace surfaces lifecycle state but does not initiate revocations / decommissions / IAM cleanup — those still live on their respective admin pages (`/admin/cloud/scopes/:id`, `/admin/cloud/iam-users`, etc.) where the cloud-side mutation gates already exist.

---

## 12. Idle scope sweep

`cmdb.scope_idle_sweep` (09:00 Mondays). Identifies scopes where `status='active' AND 0 active grants AND $0 spend past 30 days`. Builds an email digest (top 20) to IT admins.

**No auto-decommission.** Action is human — review the list, pick which scopes to actually close, run §7 for each.

---

## 13. Domain expiry alert

`cmdb.domain_expiry_alert` (daily). Scans `cmdb_domains.expiry_date <= today + warn_days` (default 30) or already expired. Posts via `feishu.alert_webhook_url`. Config knobs in `platform_settings.cmdb.domain_expiry_*`.

---

## 13.5 Leaver grant sweep (offboarding safety net)

`cmdb.leaver_grant_sweep` (01:30 daily). The daily safety net behind the T+0 force-transfer in `hr_lifecycle.advance_service` for cloud_access_grants — closes a class of residual that no other cron reaches.

### What it does

SELECTs every row in `cmdb_cloud_access_grants` where:
- the holder's `identity_persons.employment_status = 'left'`,
- the holder is not soft-deleted,
- the grant `status IN ('active', 'pending')`.

For each row, calls `CloudLifecycleService.revoke_grant(reason='offboarding', suppress_driver_error=True, actor_role_names=[GLOBAL_ADMIN_ROLE])` in its own per-grant DB session + commit. Helix-side row flips to `revoked` regardless of cloud outcome (the offboarding-invariant precedence — a leaver MUST NOT hold an active grant in our DB). When the provider call fails, `requires_manual_cloud_revoke=True` lands on the row so the drift / orphan-principal sweeps retry the provider call later; Helix UI / audit don't lie about a cloud-side that's still active.

### Why it's needed (the residuals it targets)

The primary revoke path is `hr.process_advance` (every 15 min, per-blocker). That works for any leaver still being processed. But two residual classes slip through:

1. **Pre-PR #214 historical leavers.** The T+0 cloud_access_grant revoke branch in `advance_service._handle_one_blocker` only landed 2026-05-08. Any person who completed offboarding before then has a `cmdb_cloud_access_grants` row that no cron ever cleaned up — `hr.post_leaver_verify` would surface them as `LEAVER_ORPHAN` compliance incidents but never acted, and `recent_days=7` meant they fell off the alert window entirely.

2. **Drift grants minted AFTER `left`.** If `cmdb.grants_sync` discovers a cloud-side role assignment that was created manually in the provider console post-departure, the row lands `source='discovered'` against a `left` person. No offboarding process is running for them anymore; nothing else picks it up.

This sweep handles both — no `recent_days` cutoff (those residuals are exactly the ones the recent-only verify can't see).

### Counters

The cron's `last_result_summary` carries:
- `scanned` — grants matching the SELECT.
- `revoked` — Helix-side rows successfully flipped (driver may or may not have succeeded).
- `cloud_side_failed` — subset of `revoked` where `requires_manual_cloud_revoke=True` (cloud-side rejected; awaiting drift retry).
- `errors` — per-grant exception traces from anything that didn't reach the lifecycle service cleanly.

### Schedule choice

01:30 Asia/Shanghai is half an hour BEFORE `hr.post_leaver_verify` at 02:00, so the alert-only verify sees a clean cloud-grant state and doesn't repeatedly surface what's been cleaned. Don't move the sweep AFTER the verify — you'd get a 30-minute window where each daily run starts with a fresh batch of false-positive `LEAVER_ORPHAN` incidents.

### Admin escape valves

- **One-off retry**: hit `POST /api/v1/scheduled-jobs/{id}/run-now` on the cmdb.leaver_grant_sweep row. Idempotent.
- **Per-blocker admin revoke** (the manual lever, also new 2026-05-14): if an offboarding process still exists for the leaver, the offboarding console's per-blocker "回收" button now works for `cloud_access_grant` blockers (pre-2026-05-14 it 400'd with `UNSUPPORTED_RESOURCE_TYPE`). Hits the same `CloudLifecycleService.revoke_grant` but with `suppress_driver_error=False` — admin-initiated, fail loud, surfacing cloud errors in the UI instead of silently flipping the row.

### Common scenarios

**"`cmdb.leaver_grant_sweep` summary shows `cloud_side_failed > 0` — what now?"**

The Helix row flipped to `revoked` (audit + UI consistent on our side), but the provider call did not succeed. Check the run's `errors` for the per-grant cause:

- Azure throttle / quota → wait for the next `cmdb.grants_sync` drift sweep to retry.
- Provider 5xx → same as above; drift sweep retries.
- Permission error on the credential → the sweep credential lost the right to revoke. Re-test with `POST /api/v1/integrations/{slug}/test`; if the credential is stale, rotate it and the next sweep clears the `requires_manual_cloud_revoke` flag.

**"Grant on a leaver isn't getting swept"**

Check the leaver's `identity_persons.employment_status` — if it's still `leaver_pending` (not yet `left`), the sweep deliberately skips them because the primary `hr.process_advance` path still owns the case. If it's `deleted_at IS NOT NULL`, the sweep also skips (soft-deleted persons are out of scope). Otherwise the grant should land — check `cmdb.leaver_grant_sweep` run history for errors.

---

## 14. Common scenarios

### "A grant disappeared from Helix but the user still has access cloud-side"

= the grant was discovered (`source='discovered'`), then the next sync didn't see it, so `cmdb.grants_sync` marked it removed. Possibilities:
- The cloud-side principal's role was changed (different `role_key` now). Check current cloud-side state.
- API permissions on the tenant credential dropped. Re-test with `POST /api/v1/integrations/{slug}/test`.
- The grants_sync handler has a bug for that capability/provider — check `compliance_incidents` for `discovered_grant_drift_removed`.

### "User says they got an expiry email but their grant expires in 6 months"

Check `last_notified_stage`. The sweep is idempotent per stage; if marked but `expires_at` doesn't match the stage window, somebody manually edited `expires_at` after the notification fired. Clear `last_notified_stage` to NULL if you want the sweep to re-evaluate.

### "Scope has no owner and grants_sync isn't auto-filling"

= no candidates hold an owner-equivalent role AND no candidates have any active grant. The fallback chain (owner_role → active_grant → first by person_id) failed because both pools are empty.

When candidates DO exist, `_pick_auto_owner_person` picks one deterministically: largest privilege wins, then earliest `granted_at`, then lowest `person_id` lexicographically (see [`lifecycle_service.py`](../../server/app/domains/cmdb/lifecycle_service.py)). The previous version of this runbook said it required exactly one candidate — that was wrong; the code has always picked one of N when N > 1.

Either way, override via `PATCH /api/v1/cmdb/cloud-scopes/{id}` with `owner_person_id` if the auto-pick is wrong.

### "Decommission failed with `account_close not supported`"

= You hit a Tencent/UCloud/KSyun scope where the cloud doesn't have an account-close API. Follow §7 step 6 — close in cloud console, mark closed in Helix manually.

### "Cost is right in admin but wrong in `/my/cloud/...`"

The employee view filters `cmdb_cloud_billing_daily` by their scope_ids (those they hold an active grant on). If they were just granted access, today's billing row hasn't synced yet — wait for tomorrow's `cmdb.billing_daily_sync` or run it manually.

### "`cmdb.resource_sync ... auth error: {Code: CdnServiceNotFound, ...}`" (Aliyun)

= the Aliyun account never opened the targeted service (CDN / OSS / SLB / DNS / RAM, depending on the failing capability). Aliyun's gateway returns 403 against an unsubscribed service because the AK technically lacks permission, and the pre-2026-05-05 `_call` blindly mapped any 401/403 to `ProviderAuthError` — the scheduled-job history went red as if the credential was bad. Post-fix, `_call` inspects the body's `Code` field; codes in `ALIYUN_SERVICE_NOT_SUBSCRIBED_CODES` (currently `CdnServiceNotFound` + `CdnService.NotFound`) raise `CapabilityNotSupported` instead, the sync framework lands the capability in the `skipped` bucket (not `failed`), and historical rows for that capability survive untouched. **No action required** when the account legitimately has not subscribed to the service. To enable: subscribe the service in the Aliyun console and the next sync will start ingesting normally — the SNF mapping does not negative-cache. **A new SNF code that surfaces in production should be added to `ALIYUN_SERVICE_NOT_SUBSCRIBED_CODES`** in [`server/app/domains/cmdb/providers/aliyun.py`](../../server/app/domains/cmdb/providers/aliyun.py) — keep the list narrow and explicit (verbatim observed Code from a real prod failure, no regex / fuzzy match, no doc/SDK speculation) so a real auth fault never gets silently swallowed.

**Trust assumption (added 2026-05-05 review pass)**: classification reads the gateway's response body integrity-as-trusted. We talk to Aliyun over TLS pinned to Aliyun's PKI, so the body is trustworthy in our threat model. An attacker capable of MITM'ing the Aliyun gateway could in theory forge a `{"Code":"CdnServiceNotFound"}` body to downgrade a real auth failure into "skipped" — but if the TLS path to Aliyun can be tampered with, capability skipping is the least of the platform's problems (cloud writes, billing, IAM mutations all share the same trust path). Documented for completeness, not a known exploit vector.

### "I rotated `CREDENTIAL_ENCRYPTION_KEY` and now everything 503s"

= every `integrations.secrets_encrypted` row is encrypted with the OLD key. As of 2026-05-03, callers see a typed `CredentialDecryptionError` ("decryption failed (InvalidTag) — likely CREDENTIAL_ENCRYPTION_KEY rotation without re-encryption") wrapped in an `AppException` with code `CLOUD_INTEGRATION_CREDENTIAL_INVALID` (HTTP 500) — no more opaque base64 traceback. Either:
- Roll back the env var.
- OR: write a one-off script that decrypts with the old key, re-encrypts with the new key, atomically swaps env var. There's no built-in key-versioning — you need downtime. Plan accordingly.

### "Tencent IAM resource_sync fails with `UnknownParameter `P``"

The Tencent CAM API rejects `Page` / `Rp` parameters on `ListUsers`, `ListAttachedUserAllPolicies`, and `ListGroupsForUser` despite the SDK request classes exposing those fields, AND its error parser truncates the rejected field name to a single character (you'll see `未定义参数 P` for `Page`, same shape as the `Month` → `Onth` billing bug). Helix omits Page/Rp on all three calls and defensively force-nulls `_Page`/`_Rp` on `ListUsersRequest` so the call survives older `tencentcloud-sdk-python` versions whose `__init__` populated those defaults.

If you see this error after a Tencent SDK upgrade, check whether the new version re-introduced default values on any CAM request class — `vars(req)` should be empty after construction; if it has `_Page` / `_Rp` keys, the SDK regressed and Helix's defensive null-out is the only thing keeping the call alive. Filing a Tencent ticket is also fair, but expect them to insist the param is "valid per the docs" — production behaviour disagrees.

The default page size returns up to 200 policies / groups per call, which covers every CAM user we sync. If you encounter a tenant with >200 attached policies on a single user, switch to `Keyword`-based filtering rather than re-introducing pagination.

### "Sync says `refusing to sweep — likely partial driver failure`"

`_safe_sweep` saw the provider return 0 rows (or fewer than half of what we previously had) for an integration that was non-empty before. The capability is marked `failed` and historical rows are preserved. Diagnose by:

1. Look at the failed `cmdb_sync_jobs.capability_results` for the actual `ProviderTransientError` message.
2. Re-test the integration credentials (`POST /api/v1/integrations/{slug}/test`).
3. If the provider really is empty (rare — admin nuked everything cloud-side intentionally), set the integration's `_helix_force_sweep` flag (admin override, not yet exposed in UI; manually update the row's `config_public` JSON) for one run, then re-clear.

### "`status='running'` rows are blocking my sync"

The hourly `cron.cleanup_stale_running` job sweeps `cmdb_sync_jobs` + `scheduled_job_runs` rows stuck in `status='running'` for >6h and flips them to `failed`. If you don't want to wait an hour:

```sql
UPDATE cmdb_sync_jobs
SET status='failed', finished_at=NOW(),
    capability_results = JSON_OBJECT('_helix_auto_fail', 'manual sweep')
WHERE status='running' AND started_at < NOW() - INTERVAL 1 HOUR;
```

Then re-trigger the sync.

### "利用率采集失败：`(2013, 'Lost connection to MySQL server during query')`"

Fixed 2026-08-11 — an alert with this shape from **before** that deploy needs no
action beyond a re-run; the next nightly tick is idempotent and re-fills the
window anyway.

What used to happen: `cmdb.server_metrics_sync` read the fleet, then ran the
cloud-monitoring pull **inside that read transaction** (AliCloud 海外: 24–27
min, 583 instances × 7 CloudMonitor metrics × master + every RD member). The
first statement after the pull — the preload `SELECT cmdb_server_metrics_daily
...` — hit a socket TiDB Serverless had recycled under it, so a *completed*
24-minute pull was discarded. `pool_pre_ping` cannot catch this: it probes at
checkout, and that connection was checked out 24 minutes earlier.

Now the pull holds no connection (fleet read commits first), the write phase is
a chunked `INSERT ... ON DUPLICATE KEY UPDATE` that needs no preload at all, and
a drop *during the write* retries the write only — never the pull. If you see
this errno again on any job, the question to ask is "what is this job holding a
connection across?", not "is TiDB flaky?".

Related: the same night produced a **duplicate** run 28 min after the first one
succeeded — the hourly scheduler reload treated a still-running cron as a missed
beat (see the `_inflight_job_ids` guard in `app/worker.py`). A duplicate 24-min
pull burns vendor API quota for nothing; if you see two scheduler-triggered runs
of one daily cron in `scheduled_job_runs`, check whether the worker predates
2026-08-11.

### "Daily-spike alert fired twice for the same scope/day"

Pre-2026-05-03 only. The dedupe was claimed in the docstring but not enforced; from migration 113 onward, `cmdb_cloud_scopes.last_spike_alert_date` is the at-most-once-per-(scope, date) idempotency stamp. If you see duplicates after that date, check that the migration ran and the scope row has a non-NULL value after a spike fires.

---

## 15. Fast-reference: where to look in code

| Concern | Path |
|---------|------|
| Driver capability matrix (live source) | [`server/app/integrations/drivers/`](../../server/app/integrations/drivers/) and [`server/app/domains/cmdb/providers/`](../../server/app/domains/cmdb/providers/) |
| Lifecycle orchestration | [`server/app/domains/cmdb/lifecycle_service.py`](../../server/app/domains/cmdb/lifecycle_service.py) |
| Scope request flow | [`server/app/domains/cmdb/scope_request_service.py`](../../server/app/domains/cmdb/scope_request_service.py) |
| Sync architecture + handlers | [`server/app/domains/cmdb/sync.py`](../../server/app/domains/cmdb/sync.py), [`server/app/scheduled_jobs/handlers.py`](../../server/app/scheduled_jobs/handlers.py) (search `cmdb.`) |
| Billing alerts | [`server/app/domains/cmdb/billing_alerts.py`](../../server/app/domains/cmdb/billing_alerts.py) |
| Credential encryption | [`server/app/domains/cmdb/credential_crypto.py`](../../server/app/domains/cmdb/credential_crypto.py) |
| RBAC predicates | [`server/app/core/rbac.py`](../../server/app/core/rbac.py) (`can_manage_cloud*`, `is_cloud_admin`, `is_cloud_scope_owner_of`) |
| Provisioning auto-job creation | [`server/app/domains/cmdb/scheduled_jobs_provisioning.py`](../../server/app/domains/cmdb/scheduled_jobs_provisioning.py) |

---

## 16. Non-obvious behaviors (read this once, save yourself a debugging session later)

1. **Per-capability lock via `JSON_OVERLAPS`**: `cmdb_sync_jobs.resource_types` is a JSON array. Two runs targeting overlapping capabilities serialize; disjoint run in parallel. If you see weird "second sync didn't start" behavior, this is why.

2. **prod scopes never auto-revoke** at expiry. They notify the owner with `last_notified_stage='prod_overdue'` once, then go silent. Production access expiring silently is too disruptive — manual action required.

3. **cloud_admin and global_admin grants are forever** (`expires_at = NULL`). Both expiry sweep and inactivity sweep skip them. If a person loses every perpetual role, existing NULL grants stay perpetual unless you manually rewrite `expires_at`.

4. **Discovered grants drift detection emits compliance incidents.** `cmdb.grants_sync` sets `last_seen_grants_sync_job_id`; rows where this lags behind = drift. Helix-sourced drift = serious; discovered drift = backfill cleanup.

5. **Scope owner auto-fill is deterministic but tie-breaks silently.** If at least one person holds an owner-equivalent role (or any active grant as a fallback) at sync time, `_pick_auto_owner_person` picks one: largest privilege wins, then earliest `granted_at`, then lowest `person_id` lexicographically. The earlier "only fills with exactly one candidate" wording was wrong — the code has always picked one of N. Admins should still verify the auto-pick on first sight; PATCH the scope's `owner_person_id` if it's not the right human.

6. **Renewal resets the clock from now**, not old_expiry + delta. A grant expiring tomorrow, renewed today with `cloud.grant_renewal_days=90`, becomes 90 days from today (not 91 days from the originally-scheduled expiry). This is intentional.

7. **standalone_iam scopes have one row** per integration (`scope_type='standalone_iam'`). Grants in this mode = IAM users. Revoke = delete user; grant = create user. There's no "scope-level access" abstraction.

8. **Aliyun BSS Billing API doesn't expose token counts** for Bailian / DashScope. `bailian_usage_daily.input_tokens` / `output_tokens` are often 0 — rely on `cost_*` fields. Token counts are a future API add.

9. **AWS + Aliyun/AliCloud-海外 have a working `create_scope()`** (AWS Organizations `create_account`; Aliyun Resource Directory `CreateResourceAccount`, `rd_parent_folder_id` optional → resolves RD Root via `GetResourceDirectory` when blank, fixed 2026-06-03). Azure / GCP / Tencent / UCloud still require manual provision via `/scope-requests/{id}/manual-provision?external_id=...`.

10. **Scheduled job changes take up to 1h to apply.** APScheduler reload runs every 3600s. If you flip enable/disable in the UI, expect up to an hour before it actually starts/stops firing.

---

## 17. Concurrency & locking (post-2026-05-05 5th audit)

Cloud-lifecycle paths that need single-flight semantics now use **Redis-backed advisory locks** ([`app/core/distributed_lock.py`](../../server/app/core/distributed_lock.py)) instead of holding MySQL `FOR UPDATE` locks across multi-second cloud-HTTP calls. The previous shape — `SELECT ... FOR UPDATE` on a row, then call AWS Organizations.CloseAccount (30+s) before commit — would hold the InnoDB row lock long enough to trip `innodb_lock_wait_timeout=50s` on every concurrent sync upsert against that row.

Lock keys + TTLs in use:

| Key pattern | TTL | Used by |
|------------|-----|---------|
| `decommission_scope:{scope_id}` | 600s | `decommission_scope` (covers AWS account-close polling, up to 6 min) |
| `approve_grant:{grant_id}` | 300s | `approve_grant`, `reject_grant` (multi-role group dispatch) |
| `grant_revoke:{grant_id}` | 120s | `revoke_grant`, `renew_grant` (single grant cloud round-trip) |
| `cloud_sync:{integration_id}` | 600s | `/scopes/{id}/sync-now` background dispatch |

**Wrapping pattern is always**:

```python
try:
    async with distributed_lock(key, ttl=...):
        ...
except LockBusy:
    ...handle 409 / no-op / skip-with-info-log...
```

`LockBusy` raises from `__aenter__` of the `@asynccontextmanager`, i.e. it surfaces from the `async with` line, **NOT** from the bare `distributed_lock(...)` call. A `try: cm = distributed_lock(...) except LockBusy:` is dead code — the `LockBusy` flows out of the subsequent `async with`. (This was a bug in the first round of fixes; now documented to prevent recurrence.)

`revoke_grant` is the only site that swallows `LockBusy` as a graceful no-op (admin double-click → second click sees the row already terminal and returns it as-is). Every other site converts `LockBusy` to a 409 `ConflictError` so the user sees explicit "already in progress" feedback.

Release: Lua-scripted compare-token-then-del so a TTL-expired lock that's been re-acquired by another holder isn't accidentally released. Holder crash → lock auto-clears via TTL.

Inspect live locks: `redis-cli -u $REDIS_URL keys 'helix:lock:*'`. Non-empty during normal operation is fine; persistent stale entries 10+ minutes after the last admin click warrant investigation.

---

## 17.5 Owner governance — semiannual confirm + zero-spend alert (added 2026-05-15)

Two crons, one timeline table, one detail-page tab. Targets **non-prod scopes only** (`environment ≠ prod`). The intent is: owners take responsibility for ongoing spend on their scope, and idle scopes don't accumulate forever.

**Cron 1 — `cmdb.owner_semiannual_confirmation`** (daily 09:00 Asia/Shanghai):
- Selects active non-prod scopes whose `last_owner_confirmed_at < now − 180 days` (NULL counts as past-due) AND scope was created at least 30 days ago.
- Skips scopes with an outstanding `semiannual_pending` row of the same kind (idempotent re-runs).
- Sends a Feishu card to the owner with: last 30/180-day spend + trend, current member count, last-confirmed date, top 3 active members. Three buttons: ✅ I own this / 🔄 Transfer to someone else / 📊 View details.
- **Same cron re-sends every unanswered prompt to its OWNER, once a day, until they answer** (both kinds). `reminder_count` / `last_reminded_at` on the row are the bookkeeping; the card's subtitle becomes "自 {date} 起尚未确认 · 第 N 次提醒". **There is no IT escalation** — retired 2026-08-24, see 17.5a. Nothing is capped by default; `config_override={"reminder_max_per_run": 20}` throttles one tick and the summary reports `deferred_over_cap`.
- Three cases never get a daily card: subscription gone/decommissioned → prompt closed `voided_scope_closed`; owner changed while the prompt was open → closed `voided_owner_changed` (the cadence cron then asks the new owner); **owner has left the company → skipped, prompt left open** (offboarding's force-transfer re-points the scope shortly and the previous case closes it).

**Cron 2 — `cmdb.zero_spend_alert`** (monthly 1st 09:00 Asia/Shanghai):
- Resolves last calendar month relative to today (Jan → previous year December).
- Selects active non-prod scopes where `SUM(cmdb_cost_facts.amount_usd)` for the month = 0 AND `scope.created_at` is before the month start.
- Skips scopes with an outstanding `zero_spend_pending` row created since the month boundary.
- Sends a Feishu card with: last-month spend ($0), last 90/180-day spend (to confirm long-term idle), member count + top 3, scope age. Three buttons: ✅ Keep (write reason) / 🗑 I'll clean up / 🆘 Hand to IT.
- An owner who picks "keep" with a reason is NOT silenced — next month's $0 will fire again. Repeated keeps + drift to IT is the intentional escalation path.

**Where to look:**
- Timeline table: [`cmdb_cloud_scope_owner_confirmations`](../../server/app/domains/cmdb/models.py). One row per event; 8 `kind` values share the schema. Snapshot columns (`spend_recent_30d_usd`, etc.) capture what the owner saw when prompted — independent of later cost-facts re-ingest.
- Service: [`OwnerConfirmationService`](../../server/app/domains/cmdb/owner_confirmation_service.py) — `list_scopes_due_semiannual` / `list_scopes_with_zero_spend_last_month` / `create_semiannual_pending` / `create_zero_spend_pending` / `respond_semiannual_confirm` / `respond_zero_spend` / `mark_transferred` / `escalate_overdue`.
- Detail-page tab: "Owner 与治理记录" in [`CloudScopeDetailPage.tsx`](../../web/src/pages/cloud/CloudScopeDetailPage.tsx) (admin) and [`MyCloudScopeDetailPage.tsx`](../../web/src/pages/cloud/MyCloudScopeDetailPage.tsx) (`/my`), both rendering the shared [`OwnerGovernanceTab.tsx`](../../web/src/pages/cloud/OwnerGovernanceTab.tsx). Newest first; pending rows show inline action buttons for the current owner. The tab label carries the outstanding-prompt count. **2026-08-24**: this pane had been moved off the tab bar into a drawer behind a 12px "治理与确认 · 查看" link in the right-hand action rail — restored to a real tab (the rail link and the Feishu `?ownerAction=` deep-links now switch to it instead of opening a drawer).

**Side effects of transfer (changed in this PR):**
- **Offboarding force-transfer also bumps it (2026-08-24).** The successor already got the "[自动转让] … 已转给你" push naming the subscription, so the handover counts as their confirmation and the 180-day clock starts there. Before this they were told once and then asked again on the next cadence tick about the same handover. Migration 532 back-stamped the historical ones (9 of 11 ever force-transferred had a NULL clock).
- `assign_scope_owner` now bumps `last_owner_confirmed_at = now()` (the new owner restarts the 180-day clock; cron won't re-prompt them tomorrow).
- Any outstanding `semiannual_pending` row is marked `semiannual_transferred` so the at-most-one-outstanding invariant holds for the new owner.
- The new owner gets a Feishu welcome card (previous owner name, 30-day spend snapshot, next confirmation due date). This used to be silent — owners only learned they were on the hook when the cron pinged them six months later.

**Manual interventions:**
- "Owner did not respond and IT didn't escalate fast enough" — the escalation runs every day, so the worst-case wait is one cron tick. If an admin needs to ping the owner sooner, the detail-page tab shows the pending row's `notified_at`; they can DM the owner directly or use `respond_cloud_scope_semiannual_confirm` (MCP) to force-confirm on the owner's behalf (writes an audit row attributing the action to the admin).
- "Cron fired against the wrong recipient (leaver)" — the cron filters `employment_status ∈ (left, leaver_pending)` already. If an active owner is mid-transition (Helix lag from HRIS), nuke the pending row + re-run the transfer flow.
- "Zero-spend keep loop forever" — repeated keep responses are visible in the timeline. IT decides whether to manually decommission the scope. There is no automated cleanup — the cleanup decision stays human.

### 17.5a Owner history is missing / nobody is being reminded (2026-08-24)

Two failure modes found on `apac-presales-demo-env`, both fixed — keep the diagnostics, they generalise.

**"The Owner 与治理记录 tab shows one card and nothing about how the current owner got here."** The timeline table is only written by the confirmation machinery and by `assign_scope_owner`. Two owner-change paths bypass both: the sync's initial auto-claim (`cloud.scope.auto_owner`) and offboarding's force-transfer (`cmdb.cloud_scope.owner_force_transferred`, written by `hr_lifecycle/advance_service.py`). Those are now merged into the timeline **at read time** from `audit_logs` and shown as read-only rows (`derived: true`, kinds `owner_auto_assigned` / `owner_force_transferred`). To check the raw facts for one scope:

```sql
SELECT created_at, action, status, detail
FROM audit_logs
WHERE resource_type = 'cmdb_cloud_scope' AND resource_id = '<scope_id>'
  AND action IN ('cloud.scope.auto_owner',
                 'cmdb.cloud_scope.owner_force_transferred',
                 'cloud.scope.owner_confirmation_voided')
ORDER BY created_at DESC;
```

An `auto_owner` row with `reason = active_grant_fallback:first_by_person_id` means the owner was **guessed** — first active-grant holder sorted by person id, `owner_pool_size: 0`. Treat that owner as unverified until a semiannual confirm comes back.

**"Nobody is being reminded / escalation audit rows said `admin_candidate_count: 0` every single night."** The old sweep escalated to IT after 14 days and resolved admins by the bare legacy role name `it_admin`, which is **not a row in `auth_roles`** (canonical: `global_admin`). It therefore matched nobody, refused to flip the prompt (correctly — it requires ≥1 delivery), and re-ran identically the next night for three months.

**That whole policy is retired.** An unanswered prompt is now re-sent to its owner daily; no admin is paged. Diagnostics for the current design:

```sql
-- Is the daily reminder sweep landing anything?
SELECT DATE(created_at) d, status, COUNT(*) n
FROM audit_logs
WHERE action = 'cloud.scope.owner_confirmation_reminded'
GROUP BY d, status ORDER BY d DESC LIMIT 14;

-- What is still open, how long, and how many times has it been asked?
SELECT s.display_name, p.work_email AS owner, oc.kind,
       oc.notified_at AS first_asked, oc.reminder_count, oc.last_reminded_at
FROM cmdb_cloud_scope_owner_confirmations oc
JOIN cmdb_cloud_scopes s ON s.id = oc.scope_id
LEFT JOIN identity_persons p ON p.id = oc.owner_person_id
WHERE oc.status = 'pending'
ORDER BY oc.notified_at;
```

Read it this way:

- **`reminder_count` stuck at the same value across days** → the sweep is not reaching this row. Check the cron ran, then check the row's owner: an inactive addressee is skipped on purpose (`skipped_owner_inactive` in the run summary) and never pushed to.
- **`reminder_count` climbing, owner still silent** → working as designed. There is no automatic escalation by design; if a specific subscription needs a human decision, an admin can force-confirm or reassign (below).
- **`last_reminded_at` moving twice in one day** → should be impossible (`REMINDER_MIN_INTERVAL_HOURS = 20`). If you see it, someone is calling the service outside the cron.

Historical note: rows with `escalated_at` set and audit rows on the retired `cloud.scope.owner_confirmation_escalated` action are from before 2026-08-24. Migration 531 re-opened every `status='escalated'` row to `pending` — that status was a dead end, since the respond endpoints only accept a PENDING row, so neither the owner nor an admin could ever answer one.

**Stuck-forever prompts.** A `_pending` row addressed to a superseded owner blocks its scope permanently: `list_scopes_due_semiannual` skips any scope with an open pending row, so the real owner is never prompted. To find them:

```sql
SELECT s.display_name, s.external_id, po.work_email AS card_sent_to,
       po.employment_status, co.work_email AS current_owner, oc.notified_at
FROM cmdb_cloud_scope_owner_confirmations oc
JOIN cmdb_cloud_scopes s ON s.id = oc.scope_id
LEFT JOIN identity_persons po ON po.id = oc.owner_person_id
LEFT JOIN identity_persons co ON co.id = s.owner_person_id
WHERE oc.status = 'pending' AND s.status = 'active' AND s.deleted_at IS NULL
  AND oc.owner_person_id <> s.owner_person_id
ORDER BY oc.notified_at;
```

The daily sweep voids these itself (`cloud.scope.owner_confirmation_voided`, `via=reminder_sweep_stale_addressee`), so the expected steady state is **zero rows**. If rows persist across two ticks, the sweep is not running — check the cron, not the data.
- **"First sweep on a fresh tenant returns 0 every day"** — the 30-day newborn guard hides scopes whose `created_at` is within the last 30 days, so a tenant whose scopes were all bulk-imported recently (e.g. fresh prod cutover) will see `scopes_prompted=0` until the oldest scope ages past 30 days. Bypass for the **init sweep** only: trigger the cron via `POST /api/v1/scheduled-jobs/{job_id}/trigger` with body `{"config_override": {"min_age_days": 0}}` — handler reads the override (default 30) and surfaces every eligible scope. Default scheduled ticks remain unchanged. Audit row records the override (`scheduled_job.trigger` detail.config_override). One-shot only — after the first init sweep, the normal 30-day guard guarantees re-runs only target genuinely aged scopes. Configurable knob added 2026-05-19 — see [`cloud_owner_handlers.py:run_owner_semiannual_confirmation`](../../server/app/scheduled_jobs/cloud_owner_handlers.py).

**Migration**: 168 creates the table + adds the `last_owner_confirmed_at` column. 169 idempotently seeds the two cron rows (NOT EXISTS gate so re-runs / fresh dev DBs don't blow up). Worker handler imports are wired in `app/worker.py::_run()` AND `app/main.py` (per the worker-handler-imports rule).

## 17.5.1 Cloud-server semiannual owner confirmation (added 2026-05-29, migration 216)

Pair to 17.5 but operating on individual SERVERS aggregated per owner (one card per owner, listing every due server). The motivation came from the user's prod observation: there's no mechanism to remind owners to re-confirm individual VMs, so long-idle VMs accumulate forever.

**Cron — `cmdb.server_owner_semiannual_confirmation`** (weekly Mon 09:00 Asia/Shanghai):
- Selects `cmdb_servers` where `last_owner_confirmed_at < now − 180d` OR `last_owner_confirmed_at IS NULL` (never confirmed; COALESCE to 1970 in the SQL handles this branch).
- **Skip rules** (per user direction):
  - `owner_person_id IS NULL` — no one to ping.
  - `owner.employment_status ∈ (left, leaver_pending)` — offboarding's leaver sweep already touches these servers; sending a card to a leaver is double-noise.
  - `first_seen_at >= now − 90d` — newborn guard, don't ask for confirmation on a server that landed last week.
  - `status ∈ (stopped, decommissioned)` — already surfaced via the new 长期停机 Top 10 panel; pinging the owner is redundant.
- Groups eligible servers by owner_person_id, opens one per-owner DB session, creates a `cmdb_server_owner_confirmations` row (UNIQUE(owner, period_start) makes same-day re-runs a no-op), sends the Feishu card.
- **Re-prompt interlock**: owners with a still-PENDING row from a prior tick are excluded — otherwise a weekly cron would keep minting fresh cards (each with a new `period_start`, so the UNIQUE doesn't dedup) for the same unresponded owner. Once the owner responds (or admin manually closes the row), the next eligible cycle reopens.
- **Per-owner cap = 500 servers** (`MAX_SERVERS_PER_CARD`): bounds the snapshot JSON column, the IN-list in card hydration, and the Feishu card body. Owners above the cap get a card for the first 500; remaining servers pick up on the next weekly tick after they respond. Was unbounded in the original draft — 50K-machine owners would have landed a 50K-element IN clause in three places.

**Card content** ([owner_confirmation_cards.py / NotificationService](../../server/app/notifications/service.py)):
- Title: `☁️ 云服务器半年盘点 — 共 N 台`
- Body: bulleted list of server name + region (capped at 10 visible, "...还有 M 台" hint when over).
- Two CTAs: `[✅ 全部确认仍在使用]` (deep-link `?confirmationId=<id>&action=confirm_all` — auto-commits on landing) + `[📋 前往逐台管理]` (opens the drawer for per-server actions).

**Reconcile — `cmdb.server_owner_confirmation_reconcile`** (daily 09:30 Asia/Shanghai, migration 416; also runs as step-0 of the weekly card cron):
- Snapshots are frozen at card time, but the fleet keeps moving. The 2026-07-14 incident: the first real UCloud sync swept (soft-deleted + replaced) every scan-seeded server **52 minutes after** 18 owners got their cards — leaving pending rows whose every referenced server was dead. Those rows (a) nag owners about machines they can't act on, (b) are **unfinishable** (a deleted/re-assigned server can never be stamped, so `_all_snapshot_confirmed` never covers), and (c) permanently suppress the owner in `list_due_owners` via the re-prompt interlock — the owner would never get another card.
- The reconcile re-applies the scan-time eligibility rules to every PENDING snapshot: drops servers that are now missing / soft-deleted / `status ∈ (stopped, decommissioned)` / re-assigned to another owner / under an archival tenant. Emptied rows close as **`auto_resolved`** (no `last_owner_confirmed_at` stamp — a stopped machine that comes back becomes due again next scan). Rows whose remaining servers were all already owner-confirmed close as `responded` (the earlier clicks are honoured). Every mutation writes a system audit row `cmdb.server_owner_confirmation.reconcile` with the dropped ids + reasons.
- `GET /cmdb/server-confirmations/mine` additionally filters soft-deleted servers out of the drawer hydration, covering the window between a sync and the next reconcile tick.
- **Provenance:** this fix (#1984) was silently reverted when arcade PR #1992 — branched before #1984 merged — was squash-merged; restored on top of head via migration 416 (the original seed migration claimed revision 412, which collided with arcade's `412_arcade_trivia_bank` — the same collision that also left prod's 412 stamped-but-unapplied, fixed separately in #1999's migration 415).

**Where to look:**
- Service: [`ServerOwnerConfirmationService`](../../server/app/domains/cmdb/server_owner_confirmation_service.py) — `list_due_owners` / `create_pending_for_owner` / `respond_confirm_all` / `respond_per_server` / `list_pending_for_owner` / `reconcile_pending`.
- Table: `cmdb_server_owner_confirmations` (status=pending/responded/escalated/auto_resolved; `response_kind` = `confirmed_all` | `partial`; `server_ids` is the snapshot JSON array — **shrunk in place** by the reconcile when snapshot servers become ineligible, with the dropped ids preserved in the audit row).
- Drawer: [`MyServerConfirmationDrawer.tsx`](../../web/src/pages/cloud/MyServerConfirmationDrawer.tsx) on [`MyCloudServersPage.tsx`](../../web/src/pages/cloud/MyCloudServersPage.tsx). Manual re-entry via the toolbar `Badge(count)` button.
- HTTP routes: `GET /cmdb/server-confirmations/mine` + `POST /cmdb/server-confirmations/{id}/respond` (owner-scope only, no admin override).
- MCP: `list_my_server_owner_confirmations` + `respond_server_owner_confirmation`.

**Manual interventions:**
- "Owner ignored the card, what now?" — there is **no escalation cron in v1** (deferred follow-up). The pending row stays as-is; admins can see it via the audit log (`cmdb.server_owner_confirmation.*`). The conscious choice was to ship the loop end-to-end first and add escalation once we see real response rates.
- "Cron fired for a leaver" — should be impossible (skip rule above), but if it does happen because of HRIS lag, delete the pending row from `cmdb_server_owner_confirmations` and re-run the offboarding force-transfer to attach the servers to the replacement owner. The next weekly tick will ping the new owner.
- "Owner clicked 'confirm all' but a server is actually idle" — owner's call; the cron will re-fire in another 180 days. If admin wants to force a re-check sooner, manually clear `cmdb_servers.last_owner_confirmed_at = NULL` for that row and the next cron tick picks it up.
- "Owner re-assigned a server mid-period" — the snapshot is fixed at card creation. `respond_confirm_all` only stamps servers whose `owner_person_id` STILL matches the confirmation row's owner — re-assigned servers are silently skipped, and the new owner gets their own card next cron tick.

**Migration**: 216 adds `cmdb_servers.last_owner_confirmed_at` + creates `cmdb_server_owner_confirmations` (with UNIQUE constraint on (owner_person_id, period_start)) + seeds the cron row. Worker handler imports wired in both `app/worker.py::_run()` and `app/main.py`. (Originally written as 213; the same revision slot landed on main first via PR #561's `213_seed_jira_recently_revoked_audit_job.py`, then 214 + 215 were claimed by PR #563's Feishu downstream sync, so this migration was renumbered to 216 — `down_revision="215"` — per the [alembic-revision-race rule](../../CLAUDE.md).)

## 17.5.2 Self-service DNS / server owner transfer (added 2026-06-02)

Two HTTP endpoints + one shared FE modal. The mechanism replaces "find IT and ask them to re-bind owner" — a tax that was triggered most often by departures and project handoffs.

**Endpoints**:
- `POST /api/v1/cmdb/dns-records/batch-owner-transfer` — body `{record_ids: [uuid], new_owner_person_id: uuid, reason?: str}`. Caller must own every record (admin bypass via `can_manage_cloud`). Recipient must be `employment_status='active'`. Returns `{requested, transferred, not_found, not_owned}`.
- `POST /api/v1/cmdb/servers/batch-owner-transfer` — same shape over `server_ids`.

**No approval, on purpose**: handing off rows the caller already owns does not escalate privilege; the receiver is already an active employee. The 4-eyes gate on the existing self-service `/dns-record-requests` flow exists because those endpoints DRIVE A CLOUD WRITE (create / update / disable). Transfer is pure DB mutation against rows the caller already controls — modelling it as an approval would only delay the obvious answer.

**Side effects**:
- DNS path clears `cmdb_dns_records.last_owner_review_at` + `last_recertify_nag_at` so the new owner enters the next `cmdb.dns_owner_recertify` cron pass (mirrors `update_dns_record_in_db` Round-30 dedup-clear so the recipient gets a recertify nag promptly, not the previous owner's stale exemption).
- Per-row `cmdb_owner_history` write with `change_reason=self_service_transfer` — distinct from `batch_owner` (admin cleanup) and `offboarding_force_transfer` (T+0) so the audit timeline reads cleanly.
- Notification: new owner gets a Feishu DM + email via `NotificationService.dispatch` ("张三 把 N 条 DNS 记录 / N 台服务器转给你了"). Failure of dispatch never blocks the transfer commit — the outbox row is persisted before any channel is tried, so a transient Feishu / SMTP outage produces a retryable row.

**Manual interventions**:
- "Wrong recipient picked" — the original owner cannot un-transfer (the ownership check fails because they no longer own the row). The new owner has to transfer back, OR an admin uses the same endpoint with `can_manage_cloud` bypass.
- "Recipient is `leaver_pending`" — endpoint refuses with 400 (matches the `_resolve_owner_input` guard in `update_dns_record_in_db`). The leaver's T+0 force-transfer will run the cleanup automatically — don't try to pre-fill it.
- "Notification didn't arrive" — check `notification_outbox` for a row with `related_resource_type='cmdb_dns_record' / 'cmdb_server'` and the transfer time. The outbox status (`pending/dead/delivered`) and `last_error` explain why.
- "Audit log missing" — search `audit_logs` for `action='cmdb.dns.self_service_transfer'` or `'cmdb.server.self_service_transfer'`; the `resource_id` carries the first 50 transferred ids (CSV), the `detail.summary` carries the count.

**Why not approval-routed**: see [DECISIONS.md] — owners hand off resources they already control to other active employees. Approval here would be ceremony, not security. The audit row + the recipient notification together provide the trail: if the transfer was malicious, the recipient sees they received it and can flag it.

## 17.6 Round-22 pre-launch hardening (2026-05-17)

Released alongside the 500-employee general-availability rollout. Operational consequences worth knowing:

- **Orphan cleanup now re-scans the target scope at confirm time** — Between opening the admin orphan-scan modal and clicking 确认清理, SCIM mid-sync or a re-onboarding event can move a flagged principal back into IdentityStore. Helix now calls `driver.list_scope_orphan_assignments(scope_ref)` per scope at confirm time and refuses to delete principals that are no longer in the live orphan set. Operator-visible: a target may return `ok=False` with `principal is no longer orphan in the live cloud scan — refusing to delete. Re-open the orphan scan and reconfirm.` The fail-closed behavior is intentional — re-opening the scan is cheap; touching a real employee's access is not.
- **`gcp.revoke_account` refuses on project-count drift > 10%** — When `list_scopes()` returns materially fewer projects than `cmdb_cloud_scopes` has on record (SA lost `resourcemanager.projects.list`, IAM propagation race, parent folder IAM mid-flux), the revoke is refused with an error pointing at the discovery drift. SRE action: wait for org-level IAM to stabilize (≤15 min per memory `GCP 403 — audit log is source of truth`), then re-trigger the revoke. Helix will retry on the next leaver-grant-sweep cron tick anyway.
- **`tencent_cloud.revoke_account` step 2 refuses on `ListGroupsForUser` truncation** — Tencent CAM caps single-call group enumeration at 200 rows. When a leaver belongs to >200 groups, Helix now returns `ok=False` with the truncation reason instead of attempting DeleteUser (which would fail with the misleading "residual group memberships" + a partial-detach side-effect). Operator action: manually remove overflow group memberships in the CAM console, then re-trigger the revoke.
- **`scope_request` enforces a 5-open-request cap per `(requester, integration)`** — Submit returns `429 CLOUD_SCOPE_REQUEST_QUOTA_EXCEEDED` when the requester already has ≥5 open requests (PENDING + FEISHU_APPROVED + PROVISIONING) on the same integration. Withdraw stale ones via `/cmdb/scope-requests/{id}/withdraw` or wait for approvals to drain.
- **Billing alert card now includes a `数据源: cmdb_billing_monthly (fetched ...)` line + a `对账明细:/admin/cloud/billing?month=...&integration_id=...` deep link** — Finance reading the Feishu card and the dashboard simultaneously can cross-check the exact source row; previously a weekly FX-rate refresh caused 1-2% mismatches that looked like Helix lying.
- **CSV export emits a `# WARNING: result truncated at 100000 rows` line + a `# rows_emitted=N window=...` trailer** — When a YTD export crosses the 100k cap, finance can no longer mistake the truncated dump for the full set.

## 17.7 Cloud efficiency / cost-optimization (added 2026-07-20, migration 419)

Read-only utilization + cost-efficiency insight. **Performs NO cloud writes** — verdicts are advisory; any downsize/stop stays a human action via the existing server lifecycle controls.

- **Two new per-integration crons, auto-provisioned for AWS / Aliyun / Azure / GCP** (any integration whose provider class declares `ProviderCapability.METRICS`): `cmdb.server_metrics_sync` (daily ~03:xx — pulls the trailing 14 days of hourly CPU/network from the cloud's monitoring API into `cmdb_server_metrics_daily`, prunes >90 days) and `cmdb.efficiency_compute` (daily ~07:xx — recomputes `cmdb_server_efficiency` from metrics + `cmdb_cost_facts` + the rule engine). Both `weight=heavy` (go through the heavy concurrency sub-gate). Tencent / UCloud / Ksyun get NO efficiency jobs (no metrics adapter yet). On-demand: use 立即执行 in `/admin/scheduled-jobs`.
- **IAM the monitoring read needs** (the connector's existing credential must also carry these, else that cloud degrades to "no utilization" without failing the sync): AWS `cloudwatch:GetMetricData`; Aliyun CloudMonitor `DescribeMetricList` (Cms activated); Azure `microsoft.insights/metrics/read` (Monitoring Reader); GCP `monitoring.timeSeries.list` (roles/monitoring.viewer — implied by `cloud-platform.read-only`).
- **CPU + network are the idle signal; memory/disk are best-effort.** Only CPU and network land agent-free on all four clouds. Memory / disk *usage* require an in-guest agent (CloudWatch agent / CloudMonitor agent / Azure diagnostics / GCP Ops agent); when absent they stay NULL and the rule engine treats them as "unknown", never as 0%/idle. A low-CPU box with live memory is classified memory-bound (healthy), never idle — so we don't recommend stopping a memory-heavy workload.
- **"A box shows `no_metrics` / `insufficient_data`"** — `no_metrics`: the monitoring API returned nothing for the window (stopped instance emits no CPU; agent-only metric with no agent; the cloud's metrics IAM missing → check `result_summary.partial_failures` on the `server_metrics_sync` run). `insufficient_data`: fewer than `min_data_days` (default 3) sampled days — a freshly-synced or intermittently-running box; it converges as days accumulate.
- **"Savings is blank but utilization shows"** — the billing line-item `resource_id` in `cmdb_cost_facts` couldn't be matched to the instance (`resource_id` is the vendor's billing identifier, NOT FK-linked to `cmdb_servers.instance_id`; matched by exact / ARN-suffix). Utilization + verdict still render; only the monthly-cost / estimated-savings numbers are withheld rather than guessed.
- **Tuning thresholds without a deploy** — `PUT /api/v1/cmdb/efficiency/rules` (or the admin UI) writes the `cloud_efficiency.rules` platform setting; any subset of keys overrides the code defaults (idle/oversized CPU cutoffs, idle network floor, memory veto, downsize savings ratio, window/min-data days). `can_manage_cloud` only.
- **RBAC** — admin-tier reads (`/summary`, `/instances`, `/instances/{id}`) are `can_read_cloud` (IT admin / cloud admin / finance / auditor / management); an `employee` sees only servers they own (self + subordinates); threshold edits are `can_manage_cloud`.

## 17.8 续期与到期回收运维（2026-07-29 重写，迁移 436/437/438）

**读这一节之前先记住一件事**：`expires_at IS NULL` 在这张表里**不是「缺数据」，而是「永久」**。
2026-07-29 的事故就是有个写入点把 NULL 当成缺失并回填了 90 天。

### 谁不会被到期回收（三类豁免，判定源各不相同）

| 豁免 | 数据表现 | 判定源 | 备注 |
|---|---|---|---|
| 云/全局管理员（`cloud_admin` / `global_admin` / legacy `it_admin`） | `expires_at IS NULL` | `load_perpetual_person_ids()` | **唯一判定源**。写入侧（审批/Owner 加人/续期/发现）与 `sweep_expiring_grants` 两个 pass 全部走它；迁移 440 归一存量全局管理员权限。 |
| 订阅 Owner 自身 | `expires_at` 照常有值，读时豁免 | `scope.owner_person_id == grant.person_id` | 值会「僵死」在过去（生产有 140 条），**这是正常的**，危害只在交接那一刻，已在 `assign_scope_owner` 处补 TTL 治本 |
| 生产环境 | `expires_at` 照常有值，读时豁免 | `_is_prod_environment()` | 不提醒、不回收、不通知任何人（#1961）。研发安全端到端负责 |

**排查「某人的权限为什么被回收了 / 为什么没被回收」的顺序**：
1. `sweep_expiring_grants` 的运行摘要 —— `perpetual_skipped` / `owner_skipped` / `prod_skipped` /
   `perpetual_normalized` 四个计数直接告诉你他落在哪一类；
2. `perpetual_normalized > 0` 且**不是**刚部署完 → 有写入点还在给永久行打 TTL，去查是哪条路径；
3. `audit_logs` 里该 grant 的 `cloud.grant.renew` 行看 `detail.approved_request_id` ——
   有值=走审批来的，`None`=云管理员 break-glass；`cloud.grant.renew_admin_override` 是**专门**
   为「哪些续期绕过了审批」准备的，一条 SQL 就能查全。

### 续期不再是自助（这是审批流，不是按钮）

* 员工路径：`POST /cmdb/scopes/{id}/renewal-requests` → Owner 在订阅详情页「续期审批」批准。
  一次提交覆盖该订阅下**全部**可续角色（一人一决策，不是一角色一决策）。
* `POST /cmdb/grants/{id}/renew` **只剩云管理员 break-glass**（值班被锁在外面这种场合）。
  普通持有人调它是 403 `CLOUD_RENEWAL_NEEDS_APPROVAL`。每次调用单独审计。
* **Owner 一签即生效**，不论天数 —— 2026-08-26 起取消了「>90 天需云管理员复核」这一档
  （2026-07-30 #2098 引入，用户决定移除：订阅负责人就是该订阅的权威，硬顶已经兜住风险）。
  >180 天（`GRANT_TTL_HARD_CAP_DAYS`）仍然直接拒；表单默认 90 天（`GRANT_TTL_DEFAULT_DAYS`）。
  历史 `pending_admin` 行仍可被 Owner 或云管理员直接终结（生产上取消时为 0 行）。
* 已回收的权限 30 天内（`RENEWAL_REVIVAL_MAX_AGE_DAYS`）可申请恢复，同样要审批；超期只能重新提权限申请。
* **`open_for_grant_id` 上的 UNIQUE 是并发闸**：一个 grant 同时只能有一条待审申请。
  看到 409「已有一条待审批的续期申请」是**设计行为**，不是 bug。关单（批准/驳回/超期/撤回）
  会把这列置 NULL 释放名额 —— 如果哪天出现「怎么都提不了续期」，第一件事是查这条是不是没被置 NULL。

### 两个 cron 的先后顺序不能改

```
00:15 UTC  cmdb.access_expiry_sweep     # 归一化 → 提醒 → 回收 → 通知被回收的人
01:00 UTC  cmdb.renewal_lapse_sweep     # 先提醒即将超期的审批，再关超期的单
```

`renewal_lapse_sweep` **必须**排在 expiry sweep 之后：反过来会把一个刚在夜里被批准的申请
在同一轮里当成「未决」关掉。同一 handler 内部也是先提醒后关单，理由相同。

### 通知口径（改过，别按老印象排查）

* 到期提醒**按人聚合**：一人一天一张卡，列出他全部即将到期项。
  历史上是逐条发 —— 2026-07-15 一次 cron 861 张卡给 198 人，单人单日最高 63 张。
  所以「某人只收到一张卡但他有 12 条权限要到期」是**正确**的。
* 回收当天**会**通知本人（`revoke_notified` 计数）。这条 2026-07-29 之前不存在，
  在此之前用户抱怨「权限没了没人告诉我」是真的。
* 审批超期前 3 天提醒 Owner **一次**（`owner_reminded_at` 保证不重复）。

### 到期日抖动

发现即打 TTL 现在是「`now + ttl` 再确定性向前抖动 ≤14 天、不低于半程」（BLAKE2b，非随机）。
目的是治「一个云账号接入 → 全员同一秒到期」（385595570414 有 164 条挤在同一秒，
随后在 T-14 引发单日提醒风暴、T+0 引发回收悬崖，且自助续期会把这个队列原样复制到 3 个月后）。
**排查时注意**：同一批入库的 grant 到期日现在**本来就该**是散的，看到不一致不必怀疑数据问题。

---

## 18. Doc-update rule

Per [CLAUDE.md](../../CLAUDE.md): **every PR touching cloud lifecycle (decommission / IAM cleanup / scope sync / cloud drivers) MUST update this file in the same PR**.

Bar is "kept current," not "exhaustive." Update the affected runbook section + the capability matrix in §1 if a driver gains/loses something. Run `git diff docs/ops/cloud-lifecycle-ops.md` before opening the PR — empty diff on a cloud-lifecycle change means you forgot.

## 18. 成本双口径切换运维手册（added 2026-07-28，PR4/6）

研发面全面切换到「业务折前成本」（list，白名单口径）；实付（invoice）
只留在保密圈（finance / cloud_admin / global_admin）的专属面（成本
分析页的口径切换 + `/admin/finance/cloud-waterfall`）。**部署顺序
非常重要**：

1. **部署代码**（PR1–PR4 均已合并即可）。此刻研发页面读
   `COALESCE(list, invoice)` —— 历史行 list 为 NULL，页面显示与切换
   前完全一致（无空窗、无泄露变化）。
2. **历史回填**：对每个已接入云跑一次 cost_facts + billing_daily 的
   backfill（`POST /api/v1/cmdb/cost/backfill` 或 admin 的回填入口；
   AWS BQ/GCP 建议按月窗口分批，阿里云受 BSS API 保留窗口 ~18 个月
   限制）。**回填期间页面自动逐步切到 list 值**（COALESCE 语义）。
3. **验收断言**（只读）：
   ```bash
   uv run python scripts/verify_cost_dual_measure_backfill.py \
       --provider gcp --start 2026-05-01 --end 2026-05-31 \
       --expect-list 910010.34 --tolerance 1.0
   uv run python scripts/verify_cost_dual_measure_backfill.py \
       --provider aws --start 2026-06-10 --end 2026-06-16 \
       --expect-list 1414157 --tolerance 5.0
   ```
   期望值来自 2026-07-27 对云厂商账单原始数据的只读核验（详见
   CHANGELOG 对应条目）。脚本还会做白名单纯度与零线体检。
4. **预算 + 告警阈值换算**（方案确认项④）。**顺序不可颠倒**：
   `历史 list 回填` → `alembic upgrade head`（迁移 433） → `--detect` 核对
   → `--apply`。两步之间不要隔着 22:18 的 billing_daily cron。
   ```bash
   uv run python scripts/convert_cloud_budgets_to_list_measure.py --detect   # 只读核对
   uv run python scripts/convert_cloud_budgets_to_list_measure.py            # dry-run
   uv run python scripts/convert_cloud_budgets_to_list_measure.py --apply
   ```
   按最近完整月的 list/invoice 比例缩放每个 scope 的月度预算**与
   `alert_rules[*].threshold_usd`**（不只预算），写 audit
   (`cmdb.scope.budget_measure_converted` / `…alert_rules_measure_converted`)
   并落两种标记：规则 dict 的 `"measure": "list"`，和 scope 的
   `budget_measure='list'` 列。比例出 [0.5, 5] 带或口径未回填的 scope 会跳过
   并列出，需人工处理。

   **⚠️ 双重换算风险(2026-07-29)**：换算是**乘法**。#2079 生产已跑过一次的
   那一版**不落标记**，它改过的行是「list 数字 + 零标记」，再跑 `--apply` 会
   第二次 ×ratio（实测 675 → 1822.5 → 4920）。修复不在脚本里：**迁移 433**
   从那次运行留下的 audit 回填两种标记，回填后 `--apply` 对那批 scope 一律
   `SKIP already-list`（有回归测试钉住）。因此 **`--detect` 报出任何
   `⚠ converted-but-unmarked` 行都说明迁移 433 没跑或没生效 —— 此时禁止
   `--apply`**。
   阈值被管理员改过、与 audit 记录的换算后数值不再相等的规则，迁移**故意不打
   标记**（那个数字是管理员按自己在看的口径填的，实付是保守默认）。这类 scope
   会在评估器日志里出现 `cloud.scope_alert measure ambiguity`，并在账号详情
   「告警」页的口径徽标上可见，需人工在页面重填。
   旧的 `platform_settings.cloud.budget_measure_list_scope_ids` 已被迁移 433
   消费并删除，不要再手工回填。

   **修正一个已是 list 口径的预算**时，PATCH 必须同时带 `budget_measure`
   (`'list'` 保持折前口径 / `'net'` 表示填的是发票数)，否则
   `422 CLOUD_BUDGET_MEASURE_REQUIRED`。控制台里人类 scope 归属人可**陈述**
   当前口径但不能**移动**它(`403 CLOUD_BUDGET_MEASURE_FORBIDDEN`)；对外 API
   的 `cloud.subscriptions:write` 可以移动(IT 签发的服务主体)。

5. **迁移 434 —— cost_facts 非 usage 桶行重键**（数据修复，2026-07-29）：
   - **跑之前**留基线，好让修复可审计：
     ```sql
     SELECT cost_bucket, COUNT(*), SUM(amount_usd) FROM cmdb_cost_facts
      WHERE cost_bucket <> 'usage' GROUP BY cost_bucket;
     SELECT date, integration_id, resource_id, usage_type, cost_bucket, COUNT(*)
       FROM cmdb_cost_facts WHERE cost_bucket <> 'usage'
      GROUP BY 1,2,3,4,5 HAVING COUNT(*) > 1;   -- 重复行探针
     ```
   - **跑**：正常情况就是 `alembic upgrade head`。若非 usage 行数大到让部署
     窗口没把握，**先离线跑**（本仓库有过「长回填被部署杀掉」的事故）：
     ```bash
     uv run python scripts/repair_cost_fact_bucket_keys.py --dry-run  # 只报数,回滚
     uv run python scripts/repair_cost_fact_bucket_keys.py --apply    # 按批 commit
     ```
     该脚本调用的就是迁移**自己**的 `repair_bucket_fact_keys`（不是平行实现
     —— 平行实现会漂移，而这里漂移等于错误重键），它只是替你封装了「文件名
     以数字开头、不能直接 import」那一层。跑完之后 `alembic upgrade head` 里的
     434 只剩一次纯扫描（每行都重算出与已存 `fact_key` 不匹配 → 跳过）。
     `--apply` 被打断直接再跑一次即可；`--dry-run` 把全部写入压在一个事务里
     再回滚，所以在很大的写集上可能撞 TiDB 事务大小上限 —— 那个失败的含义是
     「真跑确实有活要干」，不是数据有问题，直接走 `--apply`（它才是有界的那条）。
   - **跑之后**：迁移在 logger `alembic.runtime.migration` 下打印
     `rekeyed / deduped / scope_recovered / not_reconstructible`。重跑上面的
     重复行探针，必须为空。**`not_reconstructible > 0` 是预期且无害的** ——
     那些行的 scope 已被硬删，因此不可刷新、但也永不被重新摄取（该 scope 不再
     是摄取目标），不构成双计，无需处理。
   - **中途失败**（TiDB 事务大小 / `tidb_mem_quota_query` / 被部署杀掉）：表是
     部分修复但**一致**的，`alembic_version` 停在 433，重跑 `alembic upgrade head`
     即续。**不要手工 stamp 434**（本仓库有过「stamp 跳过 upgrade」的事故）。
   - **租户级 integration**：部署后它们的 cost_facts 窗口每次同步会被完整替换
     （此前 `if scope_db_id:` 门控使预窗口 eviction 永不执行，陈旧行永久残留）。
     部署后第一跳可能删掉上游已不再上报的行 —— **这是修复在生效**，也正是让
     表合计与发票对得上的原因。

**口径消费面速查**：Owner 预算/波动提醒（scope.alert_rules 路径）→
list；IT/财务告警 webhook（billing_alerts 的月度预算卡）→ 刻意保持
invoice（保密圈渠道，与发票对账）；效率洞察每实例成本/可省 → list
（SP 覆盖机器不再显示 ≈$0）；已实现节省 baseline → list；CSV 导出/
MCP 读 → 跟随请求者口径（默认 list，invoice 有闸）。

**已知边界**：AWS 的 CUR 导出不含 payer 账号（无权限），Support 与
AWS 侧 credit 多记在 payer —— 财务对账须计入该范围差；CE 链路（SG）
无公开牌价度量（public 恒 NULL）且未来买 RI 须切 CUR；Azure 等 Cost
Exports 开通后由 PR6 接入，期间 Azure 研发面显示的仍是折后实付
（现状既有面，口径页如实披露）。

### 18.0 回填实战记录（2026-07-28 生产全量回填，务必先读）

第一次真跑（AWS BQ / AWS SG / GCP / 阿里云国际站，窗口 2025-10/11 →
2026-07-27）踩到的四件事，下次照做可省一整天：

1. **不要在 `compose-backend-1` 里 `docker exec -d` 跑长回填** ——
   期间任何一次部署都会重建容器，把回填进程连日志一起抹掉（本次
   08:00 UTC 的部署就杀掉了第一轮）。用**独立容器**跑：
   ```bash
   sudo docker run -d --name helix-backfill --network compose_helix \
     --env-file /opt/helix/env/.env.production -w /app \
     <ECR>/helix-server:<sha> sh -c \
     "/app/.venv/bin/python scripts/backfill_cost_facts.py --start ... --end ... --integration-id ..."
   ```
   容器里跑脚本要带 `PYTHONPATH=/app`（镜像未把 app 装成包）。
2. **300s per-scope 闸会静默吃掉大账号**。`sync_cost_facts_for_integration`
   的 `_COST_FACTS_PER_SCOPE_TIMEOUT_SEC=300` 是按**每日 cron 的 1–3 天
   窗口**定的；30 天回填块上，两个最大的 AWS 账号（`632330501419`
   dbaas-prod-dataplane-nextgen、`385595570414` PingCAP）每块都超时 →
   该块记 `partial_failure`，这两个账号的 `amount_usd_list` 全程留
   NULL。修复后脚本会**点名超时 scope 并以退出码 1 结束**（见
   `backfill_cost_facts.py` 的 `_collect_chunk_failure`），重跑用
   `--chunk-days 7` 收窄块长，**不要**去调平台的 per-scope 闸（那会
   放宽每日 cron 的护栏）。
3. **验收断言要分清粒度**。定稿数字是 BQ / CUR 级别的，行级
   `cmdb_cost_facts` 天然略低（净额为 0 的分组不落行；日汇总重取窗口
   更宽）。实测可接受区间：AWS 行级比日汇总低 0.7–1.0%；GCP 2026-05
   行级 $905,442 vs 日汇总 $905,819.91 vs BQ 白名单总额 $910,010.34
   （差额 $4,190 是无项目归属行，落不到子账号 —— 见「成本口径」页
   GCP 条目的已知边界）。所以**先看 `verify_cost_dual_measure_backfill.py`
   的白名单纯度 + 零线体检两项，再看金额是否落在区间内**，不要拿
   BQ 总额当行级期望值。
4. **阿里云 BSS 必须串行回填**。2026-07-29 试过 4 路并行（按月份切分）
   —— 阿里云用 **400 + `Code=Throttling.User`** 表达限流，触发 219 次，
   4 个 worker 合计只落 5,596 行。同日已给 `_acs3_call` 补了退避重试
   （见 CHANGELOG），但**并行度仍需实测后再放开**；默认单 worker。
   注意自研补数脚本要**先取数、失败即跳过、再删旧行**（本次正是这个
   顺序救了命：219 次失败没有删掉任何既有数据）。
5. **耗时预期**（生产实测，单账号窗口 9–10 个月）：AWS BQ ~2h、
   GCP ~2.5h、阿里云 BSS **每 30 天块约 1.5h**（按天分页，最慢，
   billing_daily + cost_facts 两遍要预留一整天）。四个集成并行跑会
   互相抢 TiDB 写入，建议**串行链式**（`cmd1 && cmd2 && cmd3`）。

6. **同一云的 daily 与 facts 是两条驱动代码路径，验收要分别看**。
   阿里云踩过：PR3 给国内父类的 `list_billing_daily` 加了双口径列，但
   生产走的是 intl 子类的**覆盖实现**（Reseller master 必须用
   `DescribeInstanceBill`），覆盖漏改 → `billing_daily.total_cost_usd_list`
   9 个月全 NULL，而同一云的 `cost_facts` 因为没有覆盖、继承了父类实现
   所以是好的。**逐月对数时两张表都要看 `with_list` 计数**，别只看财务
   页读的那一张（facts）。修复见 2026-07-29 CHANGELOG 条目。

回填后必做：`verify_cost_dual_measure_backfill.py` 两组断言 → 逐月
facts↔billing_daily 对数（缺口 >2% 就是有 scope 没写进去）→ 预算换算
→ 清 Redis `cost:waterfall:*` / `cost:methodology:*`。

### 18.2 红色 `Deploy to EC2` ≠ 生产没更新（2026-07-29 实战,一天内踩两次）

`Auto Deploy to EC2` 这个 job 等 SSM 命令**最多 15 分钟**,超时就把 job 判失败:

```
##[error]SSM command did not finish within 15 min (last status: InProgress)
##[error]Deployment failed with status: InProgress
```

但宿主机上的 `deploy.sh` **不会被杀**,它会继续跑完 Step 3 迁移 → Step 4 重建容器 → Step 5 健康检查。所以一个红色 job 完全可能对应一次**成功**的部署 —— 只是晚了几十分钟。`image-release.yml` 的失败分支还会自动开一个 `Image release failed: <run-url>` 的 issue,同样可能是假警报。

**判断生产到底更新了没有,按顺序看这三样**(别看 job 颜色):

```bash
cat /opt/helix/releases/current     # 只在 healthcheck.sh 通过后才写 → 内容即证据
sudo docker inspect compose-backend-1 --format '{{.Config.Image}}'
#  ⚠ `docker ps` 对 backend 常显示裸 image id(如 6bbe66e609ab)而不是带 tag 的名字,
#    用 grep SHA 去匹配 ps 输出会得出「没部署」的错误结论 —— 必须 inspect .Config.Image
```
```sql
SELECT version_num FROM alembic_version;   -- 迁移有没有跟上镜像
```

**在上一条 SSM 命令还 InProgress 时重新触发部署,是最有害的动作** —— 两个 `alembic upgrade head` / `docker compose up` 会互相踩。先 `aws ssm list-commands` 看有没有 InProgress 的,再 `ps -eo pid,etime,cmd | grep deploy.sh`。

**为什么会超过 15 分钟**:长数据修复迁移在部署关键路径上。本次是 #2090 的迁移 **434**(给 `cmdb_cost_facts` 里 305 万非 `usage` 行重算 `fact_key`,全表 867 万行),实测跑了约 40 分钟。诊断姿势:`docker stats <迁移容器>` 看 CPU(54–84% = 在算,不是死锁)+ 数据库侧看写入量(30 分钟只有 11 行 `updated_at` 变化 = 绝大多数行早已修好、每行都落到「有界前像搜索」全都 miss,纯 CPU 烧 sha256)。**它是分批 commit 的,可中断可续跑**,所以正确处理是**让它跑完**,不是杀掉。

**这个盲区的真实代价**:同一天 #2090 自己那次部署也撞了 15 分钟闸,job 判红没人跟 —— 生产**用 #2090 的代码跑了约一小时,而它的迁移 434 从未应用**(`alembic_version` 停在 433)。镜像上线、迁移没跟上,且**零告警**。看到红色 deploy job 一定要按上面三步确认一遍,别假设「红=没上」也别假设「红=只是超时」。

### 18.1 Azure Cost Exports 白名单（PR6，规则已实现 · 拉取器待接）

- **现状**：Azure 集成默认 `billing_source` 未配置（= `cost_management`），
  走 Cost Management Query API（EA/MCA 折后实付），`list` 恒 NULL。
- **已实现**（[azure_cloud.py](../../server/app/integrations/drivers/azure_cloud.py)）：
  Exports 白名单分类器 + 数据面适配层。规则（云管理定稿）：`list =
  payg_cost_in_billing_currency`，白名单 `charge_type='Usage'`（OnDemand/
  Spot/RI/SP 覆盖行都保留，RI/SP 覆盖行以 PAYG 价还原）+ 非承诺类
  Purchase（pricing_model ∉ {Reservation, SavingsPlan} 且 reservation_id/
  benefit_id 为空）；排除 RI/SP 购买行、Unused*、Refund/Credit/
  RoundingAdjustment/税费、Microsoft Support（单独统计→support 桶）；
  **禁用列** `cost_in_billing_currency`/`effective_price`/`unit_price`
  （含合同价）；PAYG 缺失行绝不用实际成本兜底（list NULL + WARN 计数）。
- **待云管理开通 Exports 后的接入步骤**：
  1. 确认导出落点与读取方式（storage account 的 CSV/parquet；SAS vs
     RBAC、目录布局、每日 vs 月度覆盖式导出）。
  2. 实现 `AzureCloudDriver._fetch_exports_rows`（现为显式
     NotImplementedError）——解析导出文件表头后过
     `_normalize_exports_row` 归一（PascalCase/camelCase → snake_case
     契约），变换/聚合层（`_list_cost_facts_exports` /
     `_list_billing_daily_exports`）零改动直接可用。
  3. admin UI 给该 Azure 集成配置 `billing_source=exports` +
     `exports_storage_account` / `exports_container` /
     `exports_path_prefix`。
  4. 按本节主流程回填 + 验收（Azure 期望值届时用真实账单核验后补进
     verify 脚本），口径说明页 Azure 条目由「规则已实现」翻「已核验」。
- **护栏**：`billing_source=exports` 在拉取器未接前开启不会产出静默
  空数据 —— cron 会收到 NotImplementedError 并记 per-scope error（与
  其它驱动失败同路径，failure notifier 可见），所以开关只在拉取器落
  地后再翻。