## Module 3: Ensuring Agent Structured Data Persistence This is actually a bonus module designed to show you just how easy it is to provision a dedicated Starter database cluster for your Agent using the TiDB Cloud CLI. While Agents often prefer a "local-first" lightweight database setup, it's worth considering that a small, cloud-based, persistent database is actually a better choice for Agents. It serves as the database equivalent of the Agent's filesystem: "create once, always available, mount anywhere." The filesystem is ideal for storing code, build artifacts, and restorable workspaces, while databases are better suited for persisting and querying structured state. In this module, we'll use `ti db` to create a TiDB Cloud Starter instance, then combine it with TiDB Cloud Filesystem to provide Hermes with a complete Unified Agent Storage Layer. ### Step 1: Create a new Starter cluster for your agent - Go to **[Terminal 2](ssh://monitor1-ssh#2)**, and execute the following command to create a Starter cluster, create a database named `sol`, prepare some SQL user credentials, and generate a database URL for your agent. > [!TIP] > TiDB Cloud Starter cluster is a serverless MySQL compatible database cluster. Leverage `ti` CLI's abilities for `--wait`, `--query`, and `--output` flags makes automation easy. Multiple steps can be combined into a single scripted step for streamlined orchestration. ```bash # Create the TiDB Cloud Starter read DB_CLUSTER_ID <<< "$(ti db create-db-cluster \ --db-cluster-name hermes-data \ --db-cluster-type starter \ --region aws-${LAB:REGION_CODE} \ --wait \ --query "id" \ --output text \ )" # Create SQL users with predefined roles - admin, read_only, and read_write ti db create-db-sql-users \ --db-cluster-id $DB_CLUSTER_ID \ --region aws-${LAB:REGION_CODE} # Create a database for your agent, the database name is ‘sol’ ti db execute-sql-statement \ --db-cluster-id $DB_CLUSTER_ID \ --sql "CREATE DATABASE sol" \ --admin \ --transport mysql \ --region aws-${LAB:REGION_CODE} # Generate a Database Connection URL for Agent read DATABASE_URL <<< "$(ti db format-db-connection-string \ --db-cluster-id $DB_CLUSTER_ID \ --read-write \ --database sol \ --query "connection_string" \ --output text \ )" # Pass the DATABASE_URL to Agent environment echo DATABASE_URL=$DATABASE_URL >> .hermes/.env ``` ```output { "cluster_id": "xxxxxxxxxxxxxx", "users": { "admin": { ... }, "read_only": { ... }, "read_write": { ... } } } { "fields": [], "rows": [], "row_count": 0, "transport": "mysql", "access_mode": "admin", "cluster_id": "xxxxxxxxxxxxxxx" } ``` ### Step 2: Let the Agent Access the Database - Return to **[Terminal 1](ssh://monitor1-ssh#1)** and start Hermes. > [!NOTE] > To make it easier for Hermes to experiment and troubleshoot, we have intentionally lowered the security level so that the password in the DATABASE URL can be viewed by Hermes. ```bash export HERMES_REDACT_SECRETS=false hermes chat ``` - Check the environment variables again. This time you shall see the `DATABASE_URL` variable. ```text print env ``` ```output ... - DATABASE_URL: mysql://XXXXXX.ti_rw:***@gateway01.${LAB:REGION_CODE}.prod.aws.tidbcloud.com:4000/sol?ssl-mode=VERIFY_IDENTITY ... ``` - Because Starter is MySQL-compatible, the Agent doesn't need to learn anything special—having `DATABASE_URL` is enough for the Agent to use it. ### Step 3: The Challenge - Here goes the final challenge. Please instruct Hermes to migrate the data structures and data that support the Solar System program from Module 2 into the newly created database named `sol`. - It's a mini database migration project, give your agent some time to figure it out, please be patient. - Feel free to develop your own approach if you prefer. - Here is a suggested solution: ```text Mount the TiDB Cloud FS to /workspace, then use a Python program (accepting a database URL) to migrate the Solar Python program's data and schema to the sol database. ``` > [!NOTE] > Based on the LLM model we use for this lab, it might take 5 minutes to complete the misson. - After receiving confirmation of a successful migration from your agent, switch to **[Terminal 2](ssh://monitor1-ssh#2)**. Check which tables have been created by your agent. The results you see may differ. Pay attention to the tables your agent'd created. The sample output includes `moons` and `planets`, but your results may vary. ```bash read DB_CLUSTER_ID <<< "$(ti db list-db-clusters \ --db-cluster-type starter \ --region aws-${LAB:REGION_CODE} \ --query "clusters[?display_name=='hermes-data'].id" \ --output text \ )" ti db execute-sql-statement \ --sql "SHOW TABLES FROM sol" \ --region aws-${LAB:REGION_CODE} \ --admin \ --db-cluster-id $DB_CLUSTER_ID \ --output text ``` ```output +---------------+ | Tables_in_sol | +---------------+ | moons | | planets | +---------------+ ``` - If you also confirm that there is a table called planets, use a simple query to check the rows. ```bash ti db execute-sql-statement \ --sql "SELECT * FROM sol.planets" \ --admin \ --db-cluster-id $DB_CLUSTER_ID \ --output text ``` > [!NOTE] > This command may occasionally fail, since it assumes the table is named `planets`. If your agent used a different table, you may need to explore the database schema yourself using SQL queries, or go back to your Hermes session and ask the agent for the correct structure. ```output +---------+--------------------------------------------------------------------------------------------------------------+... | name | description |... +---------+--------------------------------------------------------------------------------------------------------------+... | Mercury | Mercury has no natural satellites. | | Venus | Venus has no natural satellites. | | Earth | Earth has one natural satellite, the Moon, which is the fifth largest moon in the Solar System. | | Mars | Mars has two small moons, Phobos and Deimos, which are thought to be captured asteroids. | | Jupiter | Jupiter has 95 confirmed moons, including the four large Galilean moons: Io, Europa, Ganymede, and Callisto. | | Saturn | Saturn has 146 confirmed moons, including Titan, the second-largest moon in the Solar System. | | Uranus | Uranus has 27 known moons, with the five largest being Titania, Oberon, Umbriel, Ariel, and Miranda. | | Neptune | Neptune has 14 known moons, with Triton being the largest and the only large moon with a retrograde orbit. | +---------+--------------------------------------------------------------------------------------------------------------+... 8 rows in set ``` - Alternatively, you can also visit the [web console](https://tidbcloud.com/tidbs) to explore the `sol` schema your agent created. Click into a cluster called `hermes-data`, then `SQL Editor`. ### Step 4: Brainstroming - Return to **[Terminal 1](ssh://monitor1-ssh#1)** where Hermes is still running. - Now that your Hermes Agent has gained two zero-infrastructure, scalable, and durable storage capabilities, ask it if it has any ideas for improving its memory mechanism using these new abilities. ```text By using TiDB Cloud Filesystem and TiDB Cloud Starter, what ideas do you have for improving your memory management mechanism? ``` - Here is a random sample. It is actually practical. Your results may differ. ```output ╭─ ⚕ Hermes ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ Excellent question! Using TiDB Cloud Filesystem and TiDB Cloud Starter together provides powerful opportunities to create a significantly improvement. 1. Multi-Tier Memory Architecture Current Limitations: - Memory is limited to 2,200 characters total - Single flat structure without organization - No separation by priority or recency - No ability to query or filter memories Enhanced Architecture using TiDB Cloud: ┌─────────────────────────────────────┐ │ TIER 1: ACTIVE MEMORY │ │ (In-context, 2,200 chars) │ │ High-priority, frequently accessed │ └─────────────────────────────────────┘ │ ┌─────────────────────────────────────┐ │ TIER 2: STRUCTURED MEMORY │ │ (TiDB Cloud Starter Database) │ │ Categories, metadata, relationships │ └─────────────────────────────────────┘ │ ┌─────────────────────────────────────┐ │ TIER 3: ARCHIVED MEMORY │ ... Here's how we could implement this enhancement: Phase 1: Database Integration ... Phase 2: Filesystem Integration ... Phase 3: Smart Memory Management ... Phase 4: Advanced Features ... Key Benefits of This Enhanced System: 1. Unlimited Storage: No more 2,200 character limit 2. Intelligent Retrieval: Get relevant memories, not everything 3. Version Control: Rollback to previous memory states 4. Relationship Mapping: Understand how memories connect 5. Self-Optimizing: Automatic cleanup and reorganization 6. Cross-Session: Share learnings across different Hermes instances 7. Audit Trail: Track memory creation, modification, and usage ``` - This concludes the lab. You should now understand the diagram below pretty well. ```mermaid --- title: Unified Storage Layer for Coding Agents --- flowchart LR subgraph region["Cloud Region"] style region stroke-dasharray: 5 5 subgraph temporary["Short-Lived Layer - Sandbox for Agent"] FS_CLIENT["Files"] DB_CLIENT["Data"] end subgraph persistent["TiDB Cloud Persistent Layer - Scalable, Always-On"] FS["Filesystem"] DB["Starter"] end FS_CLIENT -->|FUSE & POSIX| FS DB_CLIENT -->|SQL| DB end ``` - Please proceed to the cleanup section.