# 2021-07-23-16:38:29+0800 shrink cloudchat tikv replicas from 21 to 18 - Cause: https://jira.tidbcloud.com/projects/TCOC/issues/TCOC-144 - Operation: 1. delete backup job k delete backup ab-1311285575352848384-{auto_increase_id} 2. find the tikv which will be deleted ```sql select * from component_machines cm left join components c on cm.component_id = c.id where c.component_name='tikv' and cluster_id = 1311285575352848384; ``` you will get: ``` +---------+---------------------+---------------------+--------------+------------+---------+------+---------------------+---------------------+----------------+---------------------+ | id | created_at | updated_at | component_id | machine_id | status | id | created_at | updated_at | component_name | cluster_id | +---------+---------------------+---------------------+--------------+------------+---------+------+---------------------+---------------------+----------------+---------------------+ | 1424 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1424 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1425 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1425 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1426 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1426 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1427 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1427 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1428 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1428 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1429 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1429 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1430 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1430 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1431 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1431 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1432 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1432 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1433 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1433 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1434 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1434 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1435 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1435 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1436 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1436 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1437 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1437 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1438 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 1438 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 360012 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 360012 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 360013 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 360013 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 360014 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 360014 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 990011 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 990011 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 990012 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 990012 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 990013 | 2020-09-30 12:43:38 | 2021-08-04 16:46:53 | 610 | 990013 | active | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1620001 | 2020-09-30 12:43:38 | 2021-07-23 18:22:02 | 610 | 1620001 | deleted | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1620002 | 2020-09-30 12:43:38 | 2021-07-23 18:22:05 | 610 | 1620002 | deleted | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | | 1620003 | 2020-09-30 12:43:38 | 2021-07-23 18:22:08 | 610 | 1620003 | deleted | 610 | 2020-09-30 12:43:38 | 2021-04-06 06:50:09 | tikv | 1311285575352848384 | +---------+---------------------+---------------------+--------------+------------+---------+------+---------------------+---------------------+----------------+---------------------+ ``` we select 990011, 990012, 990013 tikv to be deleted 3. exeucte the following sql against metadb: ```sql START TRANSACTION; INSERT INTO `cluster_operations` (`created_at`,`updated_at`,`cluster_id`,`type`,`completed_at`,`status`,`content`) VALUES ('2021-08-13 02:00:00.463','2021-08-13 02:00:00.463','1311285575352848384','scale',NULL,'created',''); select id from `cluster_operations` where cluster_id = '1311285575352848384'; INSERT INTO `machines` (`created_at`,`updated_at`,`name`,`instance_id`,`machine_profile_id`,`zone_id`,`status`,`id`) VALUES ('2021-04-07 23:34:10','2021-04-06 14:50:09','1311285575352848384-tikv','','8','174665','running','990011') ON DUPLICATE KEY UPDATE `id`=`id`; UPDATE `component_machines` SET `machine_id`='990011',`status`='deleted',`updated_at`='2021-08-13 02:00:02.211' WHERE `id` = '990011'; INSERT INTO `machines` (`created_at`,`updated_at`,`name`,`instance_id`,`machine_profile_id`,`zone_id`,`status`,`id`) VALUES ('2021-04-07 23:34:10','2021-04-06 14:50:09','1311285575352848384-tikv','','8','174667','running','990012') ON DUPLICATE KEY UPDATE `id`=`id`; UPDATE `component_machines` SET `machine_id`='990012',`status`='deleted',`updated_at`='2021-08-13 02:00:02.211' WHERE `id` = '990012'; INSERT INTO `machines` (`created_at`,`updated_at`,`name`,`instance_id`,`machine_profile_id`,`zone_id`,`status`,`id`) VALUES ('2021-04-07 23:34:10','2021-04-06 14:50:09','1311285575352848384-tikv','','8','204666','running','990013') ON DUPLICATE KEY UPDATE `id`=`id`; UPDATE `component_machines` SET `machine_id`='990013',`status`='deleted',`updated_at`='2021-08-13 02:00:02.211' WHERE `id` = '990013'; # INSERT INTO `scale_operation_trackings` (`created_at`,`updated_at`,`component_id`,`cluster_operation_id`,`status`,`replicas`) VALUES ('2021-08-13 02:00:00.093','2021-08-12 02:00:00.093',610,'120001','scale-init',18); # UPDATE `clusters` SET `status`='scaling',`updated_at`='2021-08-13 02:00:02.211' WHERE id = '1311285575352848384' AND status NOT IN ('deleted','deletecomplete','cleared'); INSERT INTO `audit_logs` (`created_at`,`tenant_id`,`user`,`content`) VALUES ('2021-08-13 02:00:02.211','1311265642040528896','xufei02@pingcap.com','scale cluster: 1311285575352848384, detail: tikv replicas: 18, '); COMMIT; ``` change the configuration of TiKV, set `backup.num-threads` to 6 ``` kubectl edit tc -n tidb1311285575352848384 db ``` - Impact: the TiKV cluster will be scaled to 19 and the TiKV cluster will be rolling-updated - Recovery operation: N/A - Time window: 2021-08-13 02:00-08:00