## モジュール 1: CSV ファイルの TiDB へのインポート ### 手順 1. 仮想マシンにログイン + ここをクリック - [ターミナル 1](ssh://monitor1-ssh#1) で、ハンズオン演習用に割り当てられた仮想マシンに接続された新しいターミナルを開きます。 ### 手順 2. Lightning タスク用ユーザーの作成 ```bash source hosts-env.sh ``` ```bash mysql -h ${HOST_DB1_PRIVATE_IP} -uroot -P4000 ``` ```sql CREATE USER ltask@'%' IDENTIFIED BY 'q1w2e3R4_'; ``` ```output mysql> CREATE USER ltask@'%' IDENTIFIED BY 'q1w2e3R4_'; Query OK, 0 rows affected (0.05 sec) ``` ```sql GRANT ALL PRIVILEGES ON *.* TO ltask@'%'; ``` ```output mysql> GRANT ALL PRIVILEGES ON *.* TO ltask@'%'; Query OK, 0 rows affected (0.05 sec) ``` ```sql EXIT; ``` ```output mysql> EXIT; ``` ### 手順 3. `monitor1` に `TiDB Lightning` をインストール ```bash tiup install tidb-lightning ``` ### 手順 4. データの展開 ```bash unzip -u stage/tidb-admin-dataset-v2.zip -d stage/ ``` ```output Archive: tidb-admin-dataset-v2.zip creating: TiDB-Administration-exercise-data-v2/ ... ``` ### 手順 5. CSV データディレクトリの内容の一覧表示 ```bash ls ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-CSV-File ``` ```output emp.sal.000000000.csv emp.sal.000000002.csv emp.sal.000000004.csv emp.sal.000000006.csv emp.sal.000000008.csv emp.sal.000000010.csv emp.sal.000000001.csv emp.sal.000000003.csv emp.sal.000000005.csv emp.sal.000000007.csv emp.sal.000000009.csv metadata ``` ### 手順 6. `emp.sal.000000000.csv` ファイルの表示 ```bash head -n 10 ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-CSV-File/emp.sal.000000000.csv ``` ```output "emp_no","salary","from_date","to_date" 10001,60117,"1986-06-26","1987-06-26" 10001,62102,"1987-06-26","1988-06-25" 10001,66074,"1988-06-25","1989-06-25" 10001,66596,"1989-06-25","1990-06-25" 10001,66961,"1990-06-25","1991-06-25" 10001,71046,"1991-06-25","1992-06-24" 10001,74333,"1992-06-24","1993-06-24" 10001,75286,"1993-06-24","1994-06-24" 10001,75994,"1994-06-24","1995-06-24" ``` ```bash wc -l ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-CSV-File/emp.sal.000000000.csv ``` ```output 275696 /home/ec2-user/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-CSV-File/emp.sal.000000000.csv ``` ### 手順 7. Lightning の設定ファイルの確認 + `solution-lightning-csv.toml` を確認してください。以下に表示されるIPアドレスは異なる場合があります。 ```bash cd cat solution-lightning-csv.toml ``` ```output [lightning] # The concurrency number of data. It is set to the number of logical CPU # cores by default. When deploying together with other components, you can # set it to 75% of the size of logical CPU cores to limit the CPU usage. # region-concurrency = 1 # Logging level = "info" file = "tidb-lightning-csv.log" [tikv-importer] # Sets the backend to the "local" mode. backend = "local" # Sets the directory of temporary local storage. sorted-kv-dir = "/tmp/sorted-kv-dir" [mydumper] # Local source data directory data-source-dir = "/home/ec2-user/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-CSV-File" # Configures the wildcard rule. By default, all tables in the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, and INSPECTION_SCHEMA system databases are filtered. # If this item is not configured, the "cannot find schema" error occurs when system tables are imported. filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*'] [[mydumper.files]] # Import only 00001~05000 data files and ignore other files pattern = '(?i)(?:[/]*/)*emp\.sal\.(00000000[0-1]|000000001)\.csv' schema = "emp" table = "sal" type = "csv" [mydumper.csv] # Separator between fields. Must not be empty. separator = ',' # Quoting delimiter. Empty value means no quoting. delimiter = '"' # Line terminator. Empty value means both "\n" (LF) and "\r\n" (CRLF) are line terminators. terminator = "" # Whether the CSV files contain a header. # If `header` is true, the first line will be skipped. header = true # Whether the CSV contains any NULL value. # If `not-null` is true, all columns from CSV cannot be NULL. not-null = false # When `not-null` is false (that is, CSV can contain NULL), # fields equal to this value will be treated as NULL. null = '\N' # Whether to interpret backslash escapes inside fields. backslash-escape = true # If a line ends with a separator, remove it. trim-last-separator = false [tidb] # Configuration of any TiDB server from the cluster. host = "" port = 4000 user = "ltask" password = "q1w2e3R4_" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 # Address of any PD server from the cluster. pd-addr = "10.90.2.84:2379" ``` ### 手順 8. TiDB に接続 ```bash mysql -h ${HOST_DB1_PRIVATE_IP} --port 4000 -u root ``` ### 手順 9. スキーマの準備 ```sql DROP DATABASE IF EXISTS emp; ``` ```output mysql> DROP DATABASE IF EXISTS emp; Query OK, 0 rows affected (0.21 sec) ``` ```sql CREATE DATABASE emp; ``` ```output mysql> CREATE DATABASE emp; Query OK, 0 rows affected (0.07 sec) ``` ```sql USE emp; ``` ```output mysql> USE emp; Database changed ``` ```sql CREATE TABLE `sal` ( `emp_no` INT(11) NOT NULL, `salary` INT(11) NOT NULL, `from_date` DATE NOT NULL, `to_date` DATE NOT NULL, PRIMARY KEY (`emp_no`,`from_date`)); ``` ```output mysql> CREATE TABLE `sal` ( `emp_no` INT(11) NOT NULL, `salary` INT(11) NOT NULL, `from_date` DATE NOT NULL, `to_date` DATE NOT NULL, PRIMARY KEY (`emp_no`,`from_date`)); Query OK, 0 rows affected (0.08 sec) ``` ```sql SELECT COUNT(*) FROM emp.sal; ``` ```output mysql> SELECT COUNT(*) FROM emp.sal; +----------+ | COUNT(*) | +----------+ | 0 | +----------+ 1 row in set (0.00 sec) ``` ```sql EXIT; ``` ```output mysql> EXIT; ``` ### 手順 10. Lightning を使ったデータのインポート + `nohup` コマンドを使用して `tidb-lightning` プロセスを開始します。 (Enter を押して `tidb-lightning` をバックグラウンドで動作させます): ```bash nohup tiup tidb-lightning -config solution-lightning-csv.toml > nohup.out & ``` ```output [1] 17481 nohup: ignoring input and redirecting stderr to stdout ... [1]+ Done nohup tiup tidb-lightning -config solution-tidb-lightning-csv.toml > nohup.out ``` ### 手順 11. Lightning の出力の表示 + `TiDB Lightning` が終了していることを確認するために `nohup.out` ファイルの内容を表示します。: ```bash cat nohup.out ``` ```output Verbose debug logs will be written to tidb-lightning-csv.log +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | # | CHECK ITEM | TYPE | PASSED | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 1 | Source csv files size is proper | performance | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 2 | checkpoints are valid | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 3 | table schemas are valid | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 4 | Cluster is available | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 5 | Lightning has the correct storage permission | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 6 | sorted-kv-dir:/tmp/sorted-kv-dir and data-source-dir:/home/ec2-user/sample/csv are in the same disk, may slow down performance | performance | false | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 7 | local disk resources are rich, estimate sorted data size 20.53MiB, local available is 5.15GiB | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 8 | Cluster available is rich, available is 54.48GiB, we need 61.58MiB | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 9 | Cluster doesn't have too many empty regions | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 10 | Cluster region distribution is balanced | critical | true | +----+-----------------------------------------------------------------------------------------------------------------------+-------------+--------+ tidb lightning exit successfully ``` + エラーが発生した場合、`tidb-lightning-csv.log` ファイルの内容を参照してエラーの原因を確認してください。 ```bash less tidb-lightning-csv.log ``` + データを再びインポートする前に lightning チェックポイントを削除する必要があるかもしれません。以下のコマンドは、チェックポイントが `/tmp/` に保存されていると想定しています。 ```bash rm /tmp/tidb_lightning_checkpoint.pb ``` ### 手順 12. TiDB でのデータの表示 ```bash mysql -h ${HOST_DB1_PRIVATE_IP} --port 4000 -u root ``` ```sql SELECT COUNT(*) FROM emp.sal; ``` ```output mysql> SELECT COUNT(*) FROM emp.sal; +----------+ | COUNT(*) | +----------+ | 551395 | +----------+ 1 row in set (0.20 sec) ``` ```sql EXIT ``` ```output mysql> EXIT; Bye ```