# Log Analysis Tool This command-line tool is designed to help you download raw log files and statistics from object storage (S3, GCS, Azure Blob Storage) within specified time ranges, and to compare log statistics between different periods. # Quick Start ## Prerequisites The tool uses a GitHub Personal Access Token to authenticate and obtain necessary observability platform permissions. 1. Visit the [GitHub Token Creation Page](https://github.com/settings/tokens/new). ![GitHub Token Example](docs/images/github-token.png) 2. Enable these scopes: * `read:user` * `read:org` 3. Generate the token and save it. You will be prompted to input this token when you run the tool for the first time. ## Quick Install Clone the repository and run the install script: ```Bash git clone https://github.com/tidbcloud/observability-client.git cd observability-client bash install.sh ``` The install script will: 1. Create a Python virtual environment and install dependencies 2. Check that required tools (`curl`, `jq`, `aws`, `duckdb`) are available 3. Install the O11Y Data skill for Claude Code and Cursor AI assistants 4. Optionally configure your GitHub PAT ## How to Run from Source 1. Clone the repository ```Bash git clone https://github.com/tidbcloud/observability-client.git cd observability-client ``` 2. Set up Python Virtual Environment It's highly recommended to use a Python virtual environment to manage dependencies. ```Bash python3.10 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` 3. Run the tool to download logs ```Bash PYTHONPATH=. python app/logcli/main.py logs download \ --cluster 10601809957776321988 \ --vendor aws \ --region ap-northeast-1 \ --components tidb tikv \ --start 2025052305 \ --end 2025052306 \ --dest local ``` 4. Run the tool to download statement logs, if not set start and end, it will download the latest 2 hours logs ```Bash PYTHONPATH=. python3 app/logcli/main.py statement download \ --tenant 1372813089209061633 \ --cluster 10297819991689593990 \ --vendor aws \ --region us-east-1 ``` 5. Run the tool to download slowlogs as CSV using DuckDB syntax. Suggest adding date filter in query to improve performance. For more information, see [DuckDB](https://duckdb.org/). ```Bash PYTHONPATH=. python3 app/logcli/main.py slowlog sql \ --vendor aws \ --region us-east-1 \ --tenant 1372813089209061633 \ --cluster 10324983984131567830 \ --query "COPY (SELECT * FROM slowlogs WHERE time > epoch(TIMESTAMP '2026-02-03 10:00:00') and date='20260203') TO 'slowlogs.csv' WITH (HEADER true, DELIMITER ',')" ``` ## Parameter Reference * --cluster: TiDB Cluster ID. * --vendor: Cloud provider (e.g., `aws, gcp, azure`). * --region: Cloud region (e.g., `us-east-1, ap-northeast-1`). * --components: List of components (e.g., tidb tikv). If not specified, the default components are pd, tikv, tidb, and tiflash. * --levels: Log levels to filter (e.g., `ERROR WARN`), only valid for stats. If not specified, all valid log levels (ERROR, WARN, INFO, DEBUG) are included by default. * --start: Start time in `YYYYMMDDHH` UTC format (e.g., 2025052205). If not specified, it defaults to two hours before the current UTC time. * --end: End time in `YYYYMMDDHH` UTC format (e.g., 2025052206). If not specified, it defaults to the current hour of UTC time. * --dest: Local destination path to save downloaded logs. If not specified, the logs will be saved to the "local" directory. * --comparison-start: (Compare only) The start time for the comparison period, in `YYYYMMDDHH` UTC format (e.g., `2025052300`). This parameter is required. * --comparison-end: (Compare only) The end time for the comparison period, in `YYYYMMDDHH` UTC format (e.g., `2025052305`). This parameter is required. # Development Guide This guide provides an overview of the `observability-client` project, a command-line tool designed for downloading raw log files and generating statistics from specified TiDB Cloud clusters within given time ranges. It aims to help new contributors set up their development environment, understand the project structure, and contribute effectively. ## Table of Contents - [Observability Client Development Guide](#Development-guide) - [Table of Contents](#table-of-contents) - [Project Overview](#project-overview) - [Features](#features) - [Getting Started](#getting-started) - [Clone the Repository](#clone-the-repository) - [Set up Python Virtual Environment](#set-up-python-virtual-environment) - [Install Dependencies](#install-dependencies) - [Project Structure](#project-structure) - [How to Run from Source](#how-to-run-from-source) - [Log Download Example](#log-download-example) - [Log Statistics Example](#log-statistics-example) - [Building the Executable](#building-the-executable) - [Key Modules and Functionality](#key-modules-and-functionality) - [`app/logcli/main.py`](#applogclimainpy) - [`logs/download.py`](#logsdownloadpy) - [`logs/stats.py`](#logsstatspy) - [`logs/compare.py`](#logcomparisonpy) - [`objstore/objstore.py`](#objstoreobjstorepy) - [`auth/client_auth.py`](#authclient_authpy) - [Contributing](#contributing) ## Project Overview The `observability-client` is a client-side command-line interface (CLI) tool for interacting with observability data, specifically focusing on logs from TiDB Cloud clusters. It supports downloading raw log files and generating aggregated statistics. The tool is designed to abstract away the complexities of different cloud storage providers (AWS S3, Google Cloud Storage, Azure Blob Storage) and provides a unified interface for data retrieval and analysis. ## Features * **Raw Log Download**: Download merged log files from specified TiDB Cloud clusters within a given time range and filter by components. * **Log Statistics**: Generate and display statistics for logs, aggregated by component and log level. * **Log Comparison**: Compare log statistics between two distinct time periods, highlighting log entries that appeared in the latter period but not the former, or where the total count in the latter period is significantly higher than in the former. * **Multi-cloud Support**: Integrates with AWS S3, Google Cloud Storage, and Azure Blob Storage for log retrieval. * **Local Storage Support**: Ability to interact with a local filesystem as an object store for development and testing. * **GitHub Token Authentication**: Simplifies access by using a GitHub token for initial authentication to obtain cloud platform credentials. * **Concurrent Operations**: Utilizes thread pools for efficient concurrent downloads and listings from object stores. * **Flexible Filtering**: Supports filtering logs by cluster ID, vendor, region, components, log levels, and time ranges. ## Getting Started ### Clone the Repository ```Bash git clone [https://github.com/tidbcloud/observability-client.git](https://github.com/tidbcloud/observability-client.git) cd observability-client ``` ### Set up Python Virtual Environment It's highly recommended to use a Python virtual environment to manage dependencies. ```Bash python3 -m venv venv source venv/bin/activate ``` ### Install Dependencies All required Python packages are listed in requirements.txt. ```Bash pip3 install -r requirements.txt ``` ### Project Structure ``` . ├── app/ │ └── logcli/ │ └── main.py # Main CLI entry point for log operations ├── auth/ │ └── client_auth.py # Handles GitHub token authentication and credential retrieval ├── build/ │ └── build_logcli_mac_v1.sh # Script to build the macOS executable ├── logs/ │ ├── download.py # Logic for downloading merged log files │ └── stats.py # Logic for downloading log statistics and merging them │ └── compare.py # Logic for comparing log statistics between two distinct time periods ├── objstore/ │ └── objstore.py # Cloud storage abstraction layer (S3, GCS, Azure, Local) ├── slowlogs/ │ ├── detector.py # Slowlog anomaly detection │ ├── ruler.py # Delta Lake interaction and detection rules │ ├── sql_client.py # Interactive DuckDB SQL client for slowlogs │ └── result.py # Result formatting ├── skills/ │ └── o11y-data/ │ ├── SKILL.md # AI skill definition for O11Y data access │ ├── o11y-inspect.py # Full cluster inspection script │ ├── o11y-data.sh # Shell helpers (auth, API queries, S3 access) │ └── o11y-s3-explore.py # Python S3 data explorer ├── requirements.txt # Python dependencies ├── install.sh # One-step installation script ├── demo_download.sh # Example script for downloading logs ├── demo_stats.sh # Example script for getting log statistics ├── demo_compare.sh # Example script for comparing log statistics └── README.md # This file ``` ## How to Run from Source After setting up the virtual environment and installing dependencies, you can run the tool directly from the source code. The main entry point is app/logcli/main.py. You need to set PYTHONPATH=. so that the Python interpreter can find the custom modules (e.g., logs, objstore, auth). ### Log Download Example To download raw logs: ```Bash PYTHONPATH=. python3 app/logcli/main.py logs download \ --cluster 10601809957776321988 \ --vendor aws \ --region ap-northeast-1 \ --components tidb tikv \ --start 2025052305 \ --end 2025052306 \ --dest local ``` * --cluster: TiDB Cluster ID. * --vendor: Cloud provider (e.g., aws, gcp, azure, local). * --region: Cloud region (e.g., us-east-1, ap-northeast-1). * --components: Comma-separated list of components (e.g., tidb, tikv). If empty, pd, tikv, tidb, and tiflash are used by default. * --start: Start time in `YYYYMMDDHH` format (e.g., 2025052205). * --end: End time in `YYYYMMDDHH` format (e.g., 2025052206). * --dest: Local destination path to save downloaded logs. Refer to `demo_download.sh` for a quick example. Output: All downloaded data will be saved in {dest}/logs/{cluster_id}. ### Log Statistics Example To get log statistics: ```Bash PYTHONPATH=. python3 app/logcli/main_v2.py logs stats \ --cluster 10601809957776321988 \ --vendor aws \ --region ap-northeast-1 \ --components tidb \ --levels WARN \ --start 2025052305 \ --end 2025052310 \ --dest local ``` * --levels: Filter specific log levels (comma-separated, e.g., ERROR, WARN, INFO, DEBUG). Refer to `demo_stats.sh` for a quick example. Output: ``` ================================================================================ LOG STATISTICS SUMMARY ================================================================================ ### Component: tidb ------------------ #### Level: ERROR ------------------------------------------------------------- | COUNT | MESSAGE | LOCATION | SAMPLE | ------------------------------------------------------------- | 11 | auto analyze failed | autoanalyze.go:481 | ... | Full Sample: 2025-05-23T06:11:25.739770364Z;ip-192-168-162- 2.ap-northeast- 1.compute.internal;tidb10601809957776321988;db- tidb-1;tidb [2025/05/23 06:11:25.739 +00:00] [ERROR] [autoanalyze.go:481] ["auto analyze failed"] [category=stats] [sql="analyze table `dquery_dex`.`dex_tx_base` partition `p_202603` index `address`"] [cost_time=76.315µs] [error="[schema:1146]Table 'dquery_dex.dex_tx_base' doesn't exist"] ------------------------------------------------------------- ``` All downloaded stats data will be saved in {dest}/stats/{cluster_id}. ### Log Comparison Example (New) To compare log statistics between two time periods: ``` PYTHONPATH=. python3 app/logcli/main.py logs compare --cluster 10601809957776321988 --vendor aws --region ap-northeast-1 --components tidb tikv --levels ERROR WARN --comparison-start 2025052305 --comparison-end 2025052308 --start 2025052405 --end 2025052408 --dest local ``` * --comparison-start, --comparison-end: The comparison period. * --start, --end: The log period. * --components: Optional, used to filter components for comparison. * --levels: Optional, used to filter log levels for comparison. Output: The tool will print a detailed comparison report, highlighting newly appearing log entries in the second period and log entries with significantly increased counts. ``` ################################################################################ LOG STATISTICS COMPARISON ################################################################################ ### NEW LOG ENTRIES (Present in Period B, Not in Period A) ------------------------------------------------------------ #### Component: tidb ##### Level: ERROR ------------------------------------------------------ | COUNT | MESSAGE | LOCATION | SAMPLE | ------------------------------------------------------ | 12 | encountered error | terror.go:324 | ... | Full Sample: 2025-05-24T08:28:35.699628709Z;ip-192- 168-162-2.ap-northeast- 1.compute.internal;tidb10601809957776321 988;db-tidb-1;tidb [2025/05/24 08:28:35.699 +00:00] [ERROR] [terror.go:324] ["encountered error"] [error="write tcp 192.168.162.2:4000- >192.168.217.99:20868: write: broken pipe"] ------------------------------------------------------ ##### Level: WARN ------------------------------------------------------------------------------- | COUNT | MESSAGE | LOCATION | SAMPLE | ------------------------------------------------------------------------------- | 7 | failed to check the user authplugin | conn.go:610 | ... | Full Sample: 2025-05-24T07:30:52.183977339Z;ip-192-168-162-2.ap- northeast-1.compute.internal;tidb10601809957776321988;db- tidb-1;tidb [2025/05/24 07:30:52.183 +00:00] [WARN] [conn.go:610] ["failed to check the user authplugin"] [conn=1663665252] [error="[server:1045]Access denied for user 'admin'@'192.168.185.82' (using password: YES)"] ------------------------------------------------------------------------------- ### SIGNIFICANTLY INCREASED LOG COUNTS (Count in Period B > 5x Period A) -------------------------------------------------------------------------- #### Component: tikv ##### Level: WARN ------------------------------------------------------------------ | COUNT (A) | COUNT (B) | MESSAGE | LOCATION | SAMPLE | ------------------------------------------------------------------ | 57 | 293 | txn deadlocks | errors.rs:432 | ... | Full Sample: 2025-05-24T05:59:49.376372803Z;ip-192-168-194- 216.ap-northeast- 1.compute.internal;tidb10601809957776321988;db- tikv-1;tikv [2025/05/24 05:59:49.376 +00:00] [WARN] [errors.rs:432] ["txn deadlocks"] [err="Error(Txn(Error(Mvcc(Error(Deadlock { start_ts: TimeStamp(458245115427946609), lock_ts: TimeStamp(458245115506589753), lock_key: [116, 128, 0, 0, 0, 0, 0, 15, 44, 95, 114, 3, 128, 0, 0, 0, 0, 0, 0, 14, 1, 48, 120, 49, 54, 98, 57, 97, 56, 255, 50, 56, 57, 49, 51, 51, 56, 102, 255, 57, 98, 97, 56, 48, 101, 50, 100, 255, 54, 57, 55, 48, 102, 100, 100, 97, 255, 55, 57, 100, 49, 101, 98, 48, 100, 255, 97, 101, 0, 0, 0, 0, 0, 0, 249], deadlock_key_hash: 16060650039329636718, wait_chain: [txn: 458245115506589753 wait_for_txn: 458245115427946609 key_hash: 16060650039329636718 key: ? resource_group_tag: ?, txn: 458245115427946609 wait_for_txn: 458245115506589753 key_hash: 1083668901582211720 key: ? resource_group_tag: ?] })))))"] [thread_id=129] ------------------------------------------------------------------ ``` ## Building the Executable The project includes a script to build a standalone executable using PyInstaller. This is useful for distributing the tool without requiring users to set up a Python environment. To build the macOS executable: ```Bash sh build/build_logcli_mac_v1.sh ``` This script will: 1. Create a temporary Python virtual environment. 2. Install pyinstaller and other dependencies from requirements.txt. 3. Bundle the application into a single executable named logcli in `bin/logcli/cmd`. 4. Create a `logcli_v1.tar` archive in the bin directory containing the executable. The `build_logcli_mac_v1.sh` script currently builds for macOS. For other operating systems, you might need to adjust the PyInstaller commands accordingly. ## Key Modules and Functionality `app/logcli/main.py` This is the command-line interface (CLI) entry point. * It uses argparse to define and parse command-line arguments. * It supports only one data type: `logs`. * For `logs`, it supports two actions: download and stats. * It validates datetime formats for `--start` and `--end` parameters. * Dispatches calls to `logs.download.download_merged_logs` or `logs.stats.download_stats_and_merge` based on user input. `logs/download.py` Handles the downloading of raw merged log files. * `download_merged_logs()`: Orchestrates the download process. It determines the S3 bucket and key prefixes based on cluster ID, vendor, region, and time range. * Uses `objstore.pick_object_store()` to get the appropriate object storage client. * Generates a list of download tasks (S3 key to local path mappings). * Employs ThreadPoolExecutor for concurrent downloads of log files. * `_download_single_file()`: A helper function for downloading a single file, checking for local existence before downloading. `logs/stats.py` Manages the downloading and merging of log statistics. * * `download_stats_and_merge()`: Downloads hourly log statistics SQLite database files, merges them, and prints a formatted summary. * It leverages objstore for downloading the SQLite `.db` files containing log statistics. * `merge_stats()`: Connects to multiple SQLite files, reads log statistics, and aggregates counts based on component, level, and message digest. It filters by specified log levels and components. * `print_formatted_log_stats()`: Formats and prints the merged log statistics to the console in a human-readable table, prioritizing high-occurrence logs and organizing by component and level. `objstore/objstore.py` Provides an abstraction layer for various cloud object storage services and local filesystem. Defines an ObjectStore abstract base class with methods like list, lists, get_object, get_objects, upload_file, and download_file. Concrete implementations: * `S3`: For AWS S3, using boto3. Handles authentication with credentials obtained from `auth.client_auth`. Supports gzip content. * `pick_object_store()`: A factory function that returns a singleton instance of the appropriate ObjectStore implementation based on the vendor string. `auth/client_auth.py` Handles user authentication and credential management. * `get_github_token()`: Prompts the user for a GitHub token and saves it to a secure local file (`~/.observability/.github`). * `authenticate()`: Sends the GitHub token to a TiDB Cloud authentication service endpoint to obtain temporary AWS credentials (access key ID, secret access key, session token). * Uses `cryptography` for encryption/decrypting of credentials (though not actively used for credential storage in the provided snippet, the framework is there). * `get_credentials()`: The main function to retrieve AWS credentials for `boto3`. ## O11Y Data Skill (AI Assistant Integration) The repository includes an AI skill (`skills/o11y-data/`) that enables Claude Code and Cursor AI assistants to directly access O11Y observability data. It provides: - **Cluster Inspection**: One-command full health report (metrics + logs + slowlogs) - **Authentication**: Obtain Auth0 tokens and AWS STS credentials via GitHub PAT - **Cluster Info API**: Query cluster metadata (o11y_id, tenantID, region, version, etc.) - **Metrics API**: Run Prometheus queries against any cluster - **S3 Data Access**: Browse and download logs, log-stats, slowlogs, and statement logs - **Multi-tier Support**: Handles Dedicated/Premium and Essential/Starter data layouts ### Cluster Inspection Run a full health check on any cluster with one command: ```bash # Inspect last 1 hour (auto-detects region and bizType) python skills/o11y-data/o11y-inspect.py # Inspect last 6 hours with region hint python skills/o11y-data/o11y-inspect.py --hours 6 --region us-east-1 # Save to specific file python skills/o11y-data/o11y-inspect.py -o report.json ``` The inspection collects: - **Metrics**: QPS (by type), P99 latency, connections, TiKV CPU, region counts - **Log Stats**: ERROR/WARN counts by component, top 20 error messages with samples - **Slowlogs**: Total count, P99 query time, top 10 slow queries by avg time - **Anomaly Detection**: Automatic flagging (high connections, latency, errors, slowlogs) ### Standalone Usage (without AI assistant) The shell helpers can be used directly in any terminal: ```bash # Load helpers source skills/o11y-data/o11y-data.sh # Authenticate (requires GITHUB_PAT in .env or interactive input) o11y_auth # Query cluster info o11y_cluster_info aws us-west-2 "" # Run Prometheus query O11Y_ID=$(o11y_cluster_info aws us-west-2 "" | jq -r '.[0].id') o11y_metrics_query "$O11Y_ID" 'sum(rate(tidb_server_query_total[5m])) by (type)' # List S3 data o11y_s3_list "diagnosis/data//" us-west-2 # Download log-stats o11y_s3_download "diagnosis/data//log-stats/1h/2026030500.db" ./stats.db # Explore S3 with Python python skills/o11y-data/o11y-s3-explore.py list-clusters us-west-2 python skills/o11y-data/o11y-s3-explore.py cluster-dirs us-west-2 python skills/o11y-data/o11y-s3-explore.py log-stats 2026030500 us-west-2 ``` ### Available Tools | Tool | Description | |------|-------------| | `o11y-inspect.py` | Full cluster inspection (metrics + logs + slowlogs + anomaly detection) | | `o11y-data.sh` | Shell helpers (auth, cluster info, metrics, S3 access) | | `o11y-s3-explore.py` | S3 data explorer (list clusters, browse dirs, query log-stats) | ### Shell Functions (o11y-data.sh) | Function | Description | |----------|-------------| | `o11y_auth` | Authenticate and export credentials (Auth0 + AWS STS) | | `o11y_cluster_info` | Query cluster info from GS API | | `o11y_metrics_query` | Run Prometheus query against a cluster | | `o11y_s3_list` | List S3 objects under a prefix | | `o11y_s3_download` | Download a file from S3 | | `o11y_s3_bucket` | Get bucket name for a region | | `o11y_resolve_biz_type` | Map deployType to O11Y bizType | ### S3 Data Layout **Dedicated / Premium** clusters store data under: ``` s3://o11y-prod-shared-{region}/diagnosis/data/{clusterId}/ ├── log-stats/1h/{YYYYMMDDHH}.db # SQLite: hourly log statistics ├── slowlog-stats/1h/{YYYYMMDDHH}.db # SQLite: hourly slowlog stats ├── merged-logs/{YYYYMMDDHH}/{comp}/ # Raw log files └── slowlog-merged/ # Merged slowlogs ``` **Essential / Starter** clusters store data under: ``` s3://o11y-prod-shared-{region}/0/{tenantId}/{projectId}/{clusterId}/logs/ ├── tidb/ # Component logs (.log.gz) ├── tikv/ ├── slowlog/ # Slow query logs └── statementlog/ # SQL statement logs ``` **Delta Lake slowlogs** (all tiers): ``` s3://o11y-prod-shared-{region}/deltalake/{tenantId}/{o11yId}/slowlogs/ ``` For full documentation, see [`skills/o11y-data/SKILL.md`](skills/o11y-data/SKILL.md). ### Contributing We welcome contributions to the `observability-client` project! Fork the repository. 1. Create a new branch for your feature or bug fix: `git checkout -b feature/your-feature-name`. 2. Implement your changes. Ensure your code adheres to the existing style and best practices. 3. Write and run tests if applicable. 4. Update documentation (e.g., this README, function docstrings) to reflect your changes. 5. Commit your changes with a clear and concise message. 6. Push your branch to your forked repository. 7. Open a Pull Request to the `main` branch of the original repository.