# Config-Center Operations Guide / 运维操作手册

> Last updated: 2026-04-17

## ctl Command Reference / ctl 命令参考

`ctl` is mainly used for **oncall emergency operations** and **NextGen config migration**. Day-to-day management (Dedicated version sync, template editing, dict editing) is done through the ops portal.

`ctl` 主要用于 **oncall 紧急操作** 和 **NextGen 配置迁移**。日常管理（Dedicated 版本同步、模板编辑、字典编辑）通过 ops portal 完成。

```
ctl --config-center <host:grpc_port> <command> [flags]
```

### Template Commands / 模板命令

| Command | Description / 描述 |
|---------|-------------------|
| `ctl template search --cluster-type X --config-type Y --version V --cloud C --instance I` | Search default template for given SKU / 搜索给定 SKU 的默认模板 |
| `ctl template get --id N` | Get template by ID / 按 ID 获取模板 |
| `ctl template delete --id N` | Delete template / 删除模板 |
| `ctl template usage --id N` | Show which clusters and default bindings use this template / 显示模板被哪些集群和默认绑定使用 |
| `ctl template clean` | Purge unused templates / 清理未使用的模板 |
| `ctl template list-binding` | List all default template bindings / 列出所有默认模板绑定 |

### Dict Commands / 字典命令

| Command | Description / 描述 |
|---------|-------------------|
| `ctl dict list --config-type X --version V` | List dict items with pagination / 分页列出字典项 |
| `ctl dict create --config-type X --version V --file data.json` | Batch create dict items from JSON / 从 JSON 批量创建字典项 |
| `ctl dict import2 --config-type X` | Import param keys & types from configdumper output (code) / 从 configdumper 导入参数键和类型（代码源） |
| `ctl dict import --config-type X` | Fill description, hot_reloadable, default_value from doc source / 从文档源填充描述、热重载、默认值 |
| `ctl dict copy --from V1 --to V2` | Copy dict from old version to new version / 从旧版本复制字典到新版本 |
| `ctl dict diff --config-type X --version V` | Compare dict with configdumper output / 与 configdumper 输出对比 |
| `ctl dict diff-exist V1 V2 --config-type X` | Compare two versions in DB / 对比数据库中两个版本 |
| `ctl dict purge --config-type X --version V` | Delete unreferenced dict items / 删除未引用的字典项 |

### Migration Commands (ctl-only) / 迁移命令（仅 ctl）

| Command | Description / 描述 |
|---------|-------------------|
| `ctl migrate base` | Full migration: templates + images + init SQL from gitops / 完整迁移：从 gitops 导入模板+镜像+init SQL |
| `ctl migrate image` | Migrate component images only / 仅迁移组件镜像 |
| `ctl migrate template --force` | Migrate default templates from tier config / 从 tier config 迁移默认模板 |
| `ctl migrate initsql` | Migrate INIT_SQL template / 迁移 INIT_SQL 模板 |
| `ctl migrate ng-base` | NextGen shared pool defaults (config + images) / NextGen 共享池默认配置和镜像 |
| `ctl migrate ng-cdc` | NextGen TiCDC config / NextGen TiCDC 配置 |
| `ctl migrate ng-br` | NextGen BR (backup & restore) config / NextGen BR 配置 |
| `ctl migrate byoc-init-sql` | BYOC-specific INIT_SQL / BYOC 专用 INIT_SQL |

### Cluster Commands (oncall use) / 集群命令（oncall 使用）

| Command | Description / 描述 |
|---------|-------------------|
| `ctl cluster-config list --cluster-type X --cluster-id Y` | List cluster configurations / 列出集群配置 |
| `ctl cluster-config get <filename>` | Get a specific config file / 获取指定配置文件 |
| `ctl cluster-config autocreate <filename>` | Auto-create config file (selects template by SKU) / 自动创建配置文件 |
| `ctl cluster-config update <filename>` | Update config file / 更新配置文件 |
| `ctl cluster-config autoupdate <filename>` | Auto-update config file / 自动更新配置文件 |
| `ctl cluster-config upsert <filename>` | Upsert custom params / 增量更新自定义参数 |
| `ctl cluster-config delete` | Delete config file / 删除配置文件 |
| `ctl cluster-component list` | List cluster components / 列出集群组件 |
| `ctl cluster-component get <component>` | Get component info / 获取组件信息 |
| `ctl cluster-component create <component>` | Create component / 创建组件 |
| `ctl cluster-component update <component>` | Update component / 更新组件 |
| `ctl cluster-component check` | Check component consistency / 检查组件一致性 |

### Other Commands / 其他命令

| Command | Description / 描述 |
|---------|-------------------|
| `ctl meta` | MetaProfile operations / MetaProfile 操作 |
| `ctl available-version list` | List available versions / 列出可用版本 |
| `ctl default-version get/list/update/delete` | Default version rule management / 默认版本规则管理 |
| `ctl component-image get/update` | Component image registry / 组件镜像注册表 |

## Dict Management Pipeline / 字典管理流水线

```
1. configdumper (external repo: github.com/kaaaaaaang/configdumper)
   │  Reflect Go structs or parse tikv --config-info
   │  从 Go 结构体反射或解析 tikv --config-info
   ▼
2. JSON files (param_key + type)
   │
   ▼
3. ctl dict import2 --config-type <type>
   │  Import keys & types to param_dict table
   │  导入键和类型到 param_dict 表
   ▼
4. ctl dict import --config-type <type>      (optional)
   │  Fill description, hot_reloadable, default_value from docs
   │  从文档填充描述、热重载、默认值
   ▼
5. Ops Portal UI
   │  TSE edits risk_level, safety flags
   │  TSE 编辑风险等级、安全标志
   ▼
6. param_dict table ready for use
```

**Dict data sources by component / 各组件字典数据源:**

| Component | Key & Type Source | Other Fields |
|-----------|------------------|--------------|
| tidb, pd, tiproxy, ticdc, tidb-lightning | configdumper (Go reflection) | docdumper (from TiDB docs) |
| tikv | configdumper (`tikv --config-info`) | docdumper |
| tiflash | docdumper (v8.5+), QADB (older) | — |
| tiflash_learner | QADB | — |
| backup, restore | Manual SQL insert | — |

## Version & Config Release Process / 版本和配置发布流程

### Dedicated: Version Sync / Dedicated 版本同步

Ops portal calls config-center's `gitopsfile` package directly (not via ctl). The flow:

Ops portal 直接调用 config-center 的 `gitopsfile` 包（不通过 ctl）。流程：

1. Prepare gitops PR in `tidbcloud/tidb-configuration`, merge to `3layer-prod`
2. On ops portal config-center version page, click "Sync from Gitops"
3. Set min/max version, submit and approve the ticket
4. Ops portal calls `gitopsfile.ImportBaseFromGitops()` → imports templates, images, init SQL
5. Wait ~5min for completion / 等待约 5 分钟完成

### Dedicated: Version Release (v8.5.2+) / Dedicated 版本发布

Also managed via ops portal:

同样通过 ops portal 管理：

1. Create VersionRelease on ops portal (source version → target version)
2. Modify Templates page: review and batch-modify per-SKU templates
3. Modify Image page: set component image tags
4. Release to dev → staging → prod (with approval flow)

### NextGen: Version & Config Management / NextGen 版本和配置管理

- **Version & images**: Managed via ops portal (same as Dedicated) / 通过 ops portal 管理（与 Dedicated 相同）
- **Default config (templates)**: Updated via ctl, because NG default templates are **node_instance-independent** (unlike Dedicated which has per-SKU templates). Update `cmd/ctl/migrate_ng.go` and run `ctl migrate ng-base` / 通过 ctl 更新，因为 NG 默认模板**与机型无关**（不同于 Dedicated 的按 SKU 模板）。更新 `cmd/ctl/migrate_ng.go` 后执行 `ctl migrate ng-base`

## Common Operations / 常见操作

### Add new SKU (Dedicated) / 新增 SKU

1. Add config to gitops for applicable versions
2. Click "Sync from Gitops" on ops portal
3. For Azure: also update instance mapping code in config-center, then rebuild & deploy

### Add new component / 新增组件

1. Add constants to `proto/config_center/const.go`: component name, config type, format mapping, component↔config mapping
2. Regenerate proto if needed
3. Add migration data via ctl or ops portal
4. No ops portal frontend change needed (config-center is component-agnostic by design)

### Fix wrong config in production / 修复生产环境错误配置

1. Prefer using ops portal config change flow (with approval) / 优先使用 ops portal 配置变更流程（带审批）
2. If ops portal cannot handle, use direct SQL (record in Manual Change History doc on Lark) / 如果 ops portal 无法处理，使用直接 SQL（记录在飞书 Manual Change History 文档中）
3. After manual SQL, ensure `cluster_config_texts` is consistent — if you modify `cluster_config_params`, you must also regenerate `cluster_config_texts` (or trigger a config update through API) / 手动修改 SQL 后，确保 `cluster_config_texts` 一致

### Regional server management / Regional 服务管理

Run these from inside a **global** config-center pod (`kubectl exec -it <global-pod> -- sh`), hitting `localhost:8081` directly — no proxy, so no `/inner` prefix and no auth.

在 **global** config-center pod 内执行（`kubectl exec -it <global-pod> -- sh`），直接打 `localhost:8081`——不经 proxy，因此无需 `/inner` 前缀、无需认证。

```bash
REGION=us-west-2

# Register a new region / 注册新 region
# (after regional is deployed & its gateway/DNS is ready; triggers async global→regional sync)
# (regional 部署完成、gateway/DNS 就绪后；会触发异步 global→regional 同步)
curl -fsS -X POST -H "Content-Type: application/json" \
  "localhost:8081/config-center/cluster-types/SHARED_POOL/regional-servers" \
  -d "{\"logical_region\":\"$REGION\",\"server_address\":\"config-center-regional.$REGION:8081\"}"

# Check sync status / 查看同步状态  (sync_status: pending → syncing → synced | failed)
curl -fsS "localhost:8081/config-center/cluster-types/SHARED_POOL/regional-servers/$REGION"

# Manually retry sync if failed / 同步失败时手动重试
curl -fsS -X POST "localhost:8081/config-center/cluster-types/SHARED_POOL/regional-servers/$REGION/sync"

# List all regions / 列出所有 region
curl -fsS "localhost:8081/config-center/cluster-types/SHARED_POOL/regional-servers"

# Offline a region / 下线 region
curl -fsS -X DELETE "localhost:8081/config-center/cluster-types/SHARED_POOL/regional-servers/$REGION"
```

Register `SHARED_POOL` first, wait for `synced`, then repeat for `LOGICAL_CLUSTER`. See client-guide "Open a new region" for the full flow.

先注册 `SHARED_POOL`，等 `synced` 后再对 `LOGICAL_CLUSTER` 重复。完整流程见 client-guide「开区」一节。

## Troubleshooting / 故障排查

### "cannot list config in ops" / ops 无法列出配置

**Root cause:** Missing row in `cluster_config_texts` table. This happens when config is manually inserted into `cluster_config_info` without generating the text cache.

**根因：** `cluster_config_texts` 表缺少行。当手动向 `cluster_config_info` 插入配置但未生成 text 缓存时发生。

**Fix:** Insert empty text row, then trigger a config update via API to regenerate.

### Wrong template after scale-up / 升配后模板错误

**Root cause:** `node_instance` in `cluster_components` not updated during scale-up (only configurationFiles were updated).

**根因：** 升配时 `cluster_components` 中的 `node_instance` 未更新（仅更新了 configurationFiles）。

**Fix:** Ensure caller uses `UpdateClusterComponent` with correct `node_instance` during scale-up. Config-center now auto-updates via `update_related_configuration_files: true`.

### pd-ctl change lost after an ops config update / pd-ctl 修改在 ops 配置更新后丢失

`pd-ctl` modifies PD's **dynamic (runtime) values directly** — it does not go through config-center, so config-center never learns about the change. Later, when ops pushes any config update, config-center **regenerates the whole config file** (template + custom params) and that full file is re-applied to the cluster, overwriting the dynamic values `pd-ctl` had set. The pd-ctl change is therefore silently reverted.

`pd-ctl` **直接修改 PD 的动态（运行时）值**——不经过 config-center，因此 config-center 完全不知道这次改动。之后只要 ops 触发任何配置更新，config-center 会**重新生成整个配置文件**（模板 + custom params），整份文件重新下发到集群，覆盖掉 pd-ctl 设置的动态值。pd-ctl 的修改就被悄悄回退了。

**Solution:** Make persistent config changes through ops portal (config-center) so they live in the config file, not only as runtime values set by pd-ctl.

**解决：** 需要持久化的配置改动通过 ops portal（config-center）下发，让其落到配置文件里，而不是仅靠 pd-ctl 设置运行时值。

## Audit / 审计

Mutating operations are logged to `audit_logs` table with full request/response. Audited APIs include ConfigurationFile mutations (Update/BatchUpdate/AutoUpdate/BatchAutoUpdate/BatchUpsert), ClusterComponent updates (Update/BatchUpdate), ClusterMeta ops (Init/Update/AutoUpdate), and default template changes (Update/BatchUpdate/DeleteDefaultConfigurationTemplate). Query:

变更类操作都会记录到 `audit_logs` 表，包含完整的请求/响应。被审计的 API 包括：ConfigurationFile 变更（Update/BatchUpdate/AutoUpdate/BatchAutoUpdate/BatchUpsert）、ClusterComponent 更新（Update/BatchUpdate）、ClusterMeta 操作（Init/Update/AutoUpdate）以及默认模板变更（Update/BatchUpdate/DeleteDefaultConfigurationTemplate）。查询：

```sql
SELECT * FROM audit_logs
WHERE cluster_type = 'DEDICATED_TIDB' AND cluster_id = '12345'
ORDER BY request_ts_ms DESC LIMIT 10;
```