# add unrecognized option to TidbCluster (by pausing the cluster) **Refer to [https://github.com/pingcap/tidb-operator/issues/3234,](https://github.com/pingcap/tidb-operator/issues/3234,) which may add support for a different workflow to address this situation.** To configure cluster components (PD, TiDB, TiKV, TiFlash, etc.) for a TiDB Cluster managed by tidb-operator, you set options in the TidbCluster CRD: * [https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#cluster-topology](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#cluster-topology) * [https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md#tidbcluster](https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md#tidbcluster) Unfortunately, **configuration items not present in the source code definition of the CRD are silently ignored**. This refers to the **source code of the CRD** (manifests/crd.yaml in tidb-operator source), not to your own _instance_ of a CRD (your tidb-cluster.yaml file used to apply a TidbCluster configuration to a k8s cluster). If you need to use a configuration option that is not supported by the CRD, the solution is to **pause the cluster** so that it will "not be processed by the controller" and then **edit the ConfigMap directly**. 1. `kubectl edit tc` and set `spec.paused` to `true` 2. `kubectl edit cm` the most-recent ConfigMap for the relevant component to add the configuration items you want 3. `kubectl delete` the component pods so that they’re recreated with the new configuration ## Background For example, [the documentation has this example](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#configure-tidb-parameters): ```yaml apiVersion: pingcap.com/v1alpha1 kind: TidbCluster metadata: name: basic spec: .... tidb: image: pingcap/tidb:v4.0.4 imagePullPolicy: IfNotPresent replicas: 1 service: type: ClusterIP config: split-table: true oom-action: "log" requests: cpu: 1 ``` The config section sets split-table and oom-action, which are top-level items in the [tidb.toml configuration file read by tidb-server](https://docs.pingcap.com/tidb/stable/tidb-configuration-file). To set more complex configuration options, you have to use more complex yaml structures, like [this](https://github.com/pingcap/tidb-operator/blob/master/deploy/aws/manifests/db.yaml.example#L42): ```yaml tidb: affinity: {} annotations: tidb.pingcap.com/sysctl-init: "true" baseImage: pingcap/tidb config: log: level: info performance: max-procs: 0 tcp-keep-alive: true ```  That will generate a ConfigMap that looks like this: ```yaml apiVersion: v1 kind: ConfigMap data: config-file: | \[log\] level = "info" \[log.file\] max-backups = 3 \[performance\] max-procs = 0 tcp-keep-alive = true startup-script: | ... ``` You can see that the **`config-file` of the ConfigMap is just what you'd create as a `tidb.toml` file**. Note that tidb-operator automatically adds max-backups = 3 ([PR#2405](https://github.com/pingcap/tidb-operator/pull/2045)). Note that hierarchical items expressed in TOML using section names with dots in them (such as log.file) are expressed in yaml as nested objects. Note that the [TiFlash configuration](https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md#tiflashconfig) is a little different than [other components](https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md#tidbspec), because it has multiple sub-parts that each use their own configuration file: ```yaml apiVersion: pingcap.com/v1alpha1 kind: TidbCluster metadata: name: basic spec: ... tiflash: config: config: logger: count: 5 level: information ## Issues ``` Because the CRD manages the configuration, each config item has to be included in both the [CRD](https://github.com/pingcap/tidb-operator/blob/master/manifests/crd.yaml) _and_ in the [source code of tidb-operator itself](https://github.com/pingcap/tidb-operator/blob/master/pkg/apis/pingcap/v1alpha1/tidb_config.go#L36). There have been a couple issues ([#3075](https://github.com/pingcap/tidb-operator/issues/3075), [#3198](https://github.com/pingcap/tidb-operator/issues/3198)) where customers have tried to set config items that are not supported by the CRD. ## Solution The solution to this issue is to "pause" the TidbCluster so it will "not be processed by the controller". This means that you can directly edit the ConfigMap and restart Pods so they see the new configuration. When the cluster is paused: * The configuration will not go through the CRD, so unrecognized YAML elements will not be stripped out. * The configuration will not be processed by tidb-operator for conversion into TOML, so there will not be any errors about data types or unexpected items. * **No changes to the CRD will be propagated to the cluster!** Here are the steps to pause the TidbCluster and update a ConfigMap to contain an unrecognized item: 1. `kubectl edit tc` and set `spec.paused` to `true` 2. `kubectl edit cm` the most-recent ConfigMap for the relevant component to add the configuration items you want 3. `kubectl delete` the component pods so that they’re recreated with the new configuration ## Example It's difficult to demonstrate the root symptoms of this issue, because unrecognized configuration items are simply discarded from the YAML when it's saved, and the user is surprised to find that they're simply missing if the edit the TidbCluster CRD again. Nevertheless, here's a simple example walking through trying to apply an unsupported config option: ```bash $ kubectl get tc -n poc basic -o json | jq '.spec.tidb.config' { "log": { "file": { "max-backups": 3 } } } $ kubectl edit tc -n poc basic … spec: configUpdateStrategy: RollingUpdate discovery: {} enablePVReclaim: false imagePullPolicy: IfNotPresent pd: baseImage: pingcap/pd config: {} maxFailoverCount: 3 replicas: 1 requests: storage: 1Gi pvReclaimPolicy: Delete tidb: baseImage: pingcap/tidb config: **unexpected: unknown** log: **format: json** file: max-backups: 3 maxFailoverCount: 3 replicas: 1 service: type: ClusterIP … $ kubectl get tc -n poc basic -o json | jq '.spec.tidb.config' { "log": { "file": { "max-backups": 3 }, "format": "json" } } ``` As you can see, the `unexpected` item I added is simply not there, even while `log.format` is added as expected. If we try to edit the ConfigMap directly, it will be overwritten by the tidb-controller-manager. But let's try it anyway. ```bash $ kubectl get cm -n poc NAME DATA AGE basic-pd-6130373 2 16h basic-tidb-3761386 2 16h basic-tidb-6565316 2 8m59s basic-tikv-3137346 2 16h basic-tikv-6330643 2 16h ``` Note that there are multiple ConfigMap objects for both tidb and tikv. This is because old confitgurations are kept, since they're may still be used by old pods and since it may be necessary to revert to them. Here's a way to see how the `config-file` section of each ConfigMap differ for a specific cluster component: ```bash $ kubectl get cm -n poc -l app.kubernetes.io/component=tidb -o NAME | while read -r cm; do echo "$cm:" kubectl get "$cm" -n poc -o json | jq -r '.data."config-file"' done configmap/basic-tidb-3761386: \[log\] \[log.file\] max-backups = 3 configmap/basic-tidb-6565316: \[log\] format = "json" \[log.file\] max-backups = 3 ``` If you're going to edit a ConfigMap, always edit the most-recent one. ```bash $ kubectl edit cm basic-tidb-6565316 -n poc # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: config-file: | **unexpected = "unknown"** \[log\] format = "json" \[log.file\] max-backups = 3 startup-script: | ``` If you look at the new ConfigMap right away, you might see the change you've made: ```bash $ kubectl get cm basic-tidb-6565316 -n poc -o json | jq -r '.data."config-file"' **unexpected = "unknown"** \[log\] format = "json" \[log.file\] max-backups = 3 ``` However, the tidb-controller-manager re-syncs the cluster every 30 seconds, and your ConfigMap will be overwritten. You can see the resync happening in the logs of the tidb-controller-manager pod: ```{output} I0902 17:55:02.725237 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:02.729739 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:02.791510 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:02.794503 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:02.929373 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:02.958690 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.023366 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.027585 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.027827 1 controller\_utils.go:519\] controller poc/basic of poc/basic-discovery not found, ignore I0902 17:55:03.075943 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.076508 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.080610 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync I0902 17:55:03.143390 1 reflector.go:241\] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:96: forcing resync ``` After the cluster is resynced, the changes to ConfigMap are lost: ```bash $ kubectl get cm basic-tidb-6565316 -n poc -o json | jq -r '.data."config-file"' \[log\] format = "json" \[log.file\] max-backups = 3 ``` So, let's follow the recommended steps to **pause the cluster**. After `kubectl edit tc` and adding `paused: true` to the `spec`, you should see this in the log output of the tidb-controller-manager pod when the next resync is triggered: ```{output} I0902 17:20:41.603088 1 pd\_member\_manager.go:123\] tidb cluster poc/basic is paused, skip syncing for pd service I0902 17:20:41.603144 1 pd\_member\_manager.go:167\] tidb cluster poc/basic is paused, skip syncing for pd headless service I0902 17:20:41.609770 1 pd\_member\_manager.go:222\] tidb cluster poc/basic is paused, skip syncing for pd statefulset I0902 17:20:41.609814 1 tikv\_member\_manager.go:149\] tikv cluster poc/basic is paused, skip syncing for tikv service I0902 17:20:41.613186 1 tikv\_member\_manager.go:208\] tikv cluster poc/basic is paused, skip syncing for tikv statefulset I0902 17:20:41.613887 1 tidb\_member\_manager.go:160\] tidb cluster poc/basic is paused, skip syncing for tidb headless service I0902 17:20:41.613905 1 tidb\_member\_manager.go:286\] tidb cluster poc/basic is paused, skip syncing for tidb service I0902 17:20:41.616146 1 tidb\_member\_manager.go:216\] tidb cluster poc/basic is paused, skip syncing for tidb statefulset ``` Now, let's try again to edit the ConfigMap ```bash $ kubectl edit cm basic-tidb-6565316 -n poc # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: config-file: | unexpected = "unknown" \[log\] format = "json" \[log.file\] max-backups = 3 startup-script: | … $ sleep 60 $ kubectl get cm basic-tidb-6565316 -n poc -o json | jq -r '.data."config-file"' **unexpected = "unknown"** \[log\] format = "json" \[log.file\] max-backups = 3 ``` So, even after the resync was triggered by the controller-manager, the ConfigMap still has our updates. However, nothing is managing the pods now, so no restart has been triggered to propagate the configuration change: ```bash $ kubectl get po -n poc NAME READY STATUS RESTARTS AGE basic-discovery-59c77754b4-rtrfb 1/1 Running 0 17h basic-pd-0 1/1 Running 0 16h basic-tidb-0 2/2 Running 0 46m basic-tikv-0 1/1 Running 0 16h ``` Because we edited the ConfigMap currently in use by the pods, after some time we may see our changes in the TOML file mounted inside the container: ```bash $ kubectl exec -n poc basic-tidb-0 -c tidb -- cat /etc/tidb/tidb.toml **unexpected = "unknown"** \[log\] format = "json" \[log.file\] max-backups = 3 ``` However, we must restart the tidb-server process in order for it to see the new configuration. This means restarting the Pods. Normally, the easiest way to do that is to delete the StatefulSet (sts) that manages the tidb pods, but since the cluster is not being managed by tidb-controller-manager right now, the StatefulSet will not be re-created! So, **do not delete the StatefulSet!** ```bash $ kubectl get sts -n poc NAME READY AGE basic-pd 1/1 17h basic-tidb 1/1 17h basic-tikv 1/1 17h $ kubectl delete sts -n poc basic-tidb statefulset.apps "basic-tidb" deleted $ kubectl get sts -n poc NAME READY AGE basic-pd 1/1 17h basic-tikv 1/1 17h $ kubectl logs -n pingcap tidb-controller-manager-54cfcbff7b-wt949 … I0902 18:07:02.559670 1 tidb\_member\_manager.go:216\] tidb cluster poc/basic is paused, skip syncing for tidb statefulset ``` Instead, you must delete the component pods one-by-one. The StatefulSet will re-create them. ```bash $ kubectl get po -n poc -l app.kubernetes.io/component=tidb -o NAME pod/basic-tidb-0 $ kubectl delete pod/basic-tidb-0 -n poc pod "basic-tidb-0" deleted $ kubectl get po -n poc -l app.kubernetes.io/component=tidb NAME READY STATUS RESTARTS AGE basic-tidb-0 2/2 Running 0 30s $ kubectl logs -n poc basic-tidb-0 -c tidb start tidb-server ... /tidb-server --store=tikv --advertise-address=basic-tidb-0.basic-tidb-peer.poc.svc --host=0.0.0.0 --path=basic-pd:2379 --config=/etc/tidb/tidb.toml --log-slow-query=/var/log/tidb/slowlog **config file /etc/tidb/tidb.toml contained invalid configuration options: unexpected**; check TiDB manual to make sure this option has not been deprecated and removed from your TiDB version if the option does not appear to be a typo ``` So, there you can see that tidb-server has seen our invalid configuration option "`unexpected`" and has complained about it. (Remember that you can use the `-config-strict` command-line option to tidb-server to cause it to "enforce config file validity (default false)".) * Remember that **no changes to the CRD will be propagated** to the cluster as long as it's paused. * If/**when you unpause the cluster, changes to the ConfigMap will be lost**. * Any configuration items missing from the CRD/tidb-operator should be considered a bug. * **[File a bug report](https://github.com/pingcap/tidb-operator/issues/new?template=bug-report.md)** * Try to fix it yourself. You'll need to edit manifests/crd.yaml as well as the appropriate API file, such as pkg/apis/pingcap/v1alpha1/tidb\_config.go. * After trying your fix, run hack/update-all.sh to auto-generate the API references and interfaces * Use hack/local-up-operator.sh to start a local cluster under kind to test your changes.