--- title: AGENTS.md - drive9 development guide for AI coding agents --- ## Repository overview drive9 is a Go agent-native filesystem server. This repository maintains only server-side code: the HTTP API, tenant provisioning, metadata store, object storage integration, search/embedding, vault/journal, and server-local dev mode (`DRIVE9_TENANT_PROVIDER=local`). Client code (CLI, FUSE mounts, language SDKs, plugins) lives elsewhere and is intentionally absent here. Module: `github.com/mem9-ai/drive9` Go version: 1.25.1 (see `go.mod`) Server entrypoints: - `cmd/drive9-server/` — production server (also local via `DRIVE9_TENANT_PROVIDER=local`) ### DB9 provider naming `db9` is a historical provider name. In the server's supported production path, DB9 tenant databases are TiDB Cloud endpoints accessed through the MySQL protocol and MySQL/TiDB SQL dialect. The server does not support PostgreSQL as a tenant runtime. Do not infer PostgreSQL support from legacy `pgx` bootstrap code under `pkg/tenant/db9/`; active cold opens, schema repairs, migrations, and regression fixtures for `ProviderDB9` must use TiDB/MySQL-compatible DSNs and SQL. - `cmd/drive9-billing-worker/` — hourly metering/billing worker (leader-elected; ledger + SQS outbox) --- ## Build commands ```bash make build # build server → bin/drive9-server and billing worker → bin/drive9-billing-worker make build-server # server only make build-billing-worker # billing worker only ``` Direct Go equivalents: ```bash go build -o bin/drive9-server ./cmd/drive9-server ``` All binaries are built with `CGO_ENABLED=0`. --- ## Test commands ```bash # full suite make test # single package make test TEST_PKGS='./pkg/datastore/...' # single test make test TEST_RUN='TestInsertAndGetNode' TEST_PKGS='./pkg/datastore/...' ``` TiDB-backed tests require a container runtime or an explicit DSN: ```bash # Use an existing TiDB instance DRIVE9_TEST_TIDB_DSN='root@tcp(127.0.0.1:4000)/drive9_test?parseTime=true' make test ``` If `DRIVE9_TEST_TIDB_DSN` is unset and `podman` is available, `make test` auto-configures testcontainers via `scripts/test-podman.sh`. Otherwise a Docker-compatible runtime is used. If a direct `go test` run fails with `rootless Docker not found`, retry with `make test` so the project can use `scripts/test-podman.sh` to route testcontainers through Podman. **E2E:** public HTTP / CLI / FUSE smokes live in `mem9-ai/drive9`. `make e2e-local` grafts this repo's `bin/drive9-server` onto that e2e (`RUN_TOKENS_SMOKE=1` `RUN_SSE_SMOKE=1` `RUN_JOURNAL_SMOKE=1` `RUN_POSIX_SMOKE=1` by default). Shared-schema stays in `e2e/` and is the Local E2E job. Read `e2e/AGENTS.md` first. **Blackbox:** POSIX / perf / workflow modules also live in `mem9-ai/drive9` (`blackbox/`). `make blackbox-local` grafts this repo's `bin/drive9-server` onto `python3 blackbox/run.py --server-mode local` (builds the drive9 CLI from the upstream checkout). Default group is `community`. CI is `.github/workflows/blackbox.yml` (manual) plus daily/weekly wrappers. ```bash make e2e-local make blackbox-local DRIVE9_UPSTREAM_DIR=/path/to/drive9 make blackbox-local make blackbox-local BLACKBOX_ARGS="--module community.pjdfstest" bash e2e/shared-smoke-test.sh ``` --- ## Lint and format ```bash make lint # golangci-lint run (v2.5.0, installed to bin/) make fmt # golangci-lint run --fix ``` golangci-lint is auto-installed to `bin/golangci-lint` on first `make lint`. There is no `.golangci.yml`; linter runs with default settings. CI (`code-ci.yml`) enforces lint before tests on every PR to `main`. ## OpenAPI documents Three OpenAPI 3.0.3 documents live under `docs/api/` and are generated, not hand-written: ```bash make swagger # regenerate + verify docs are up to date make swagger-check # verify only (CI-friendly, exits non-zero when stale) ``` - `docs/api/control-plane-aksk.openapi.json` — control plane authenticated with TiDB Cloud API keys (`X-TiDBCloud-Public-Key` / `X-TiDBCloud-Private-Key`). - `docs/api/control-plane-jwt.openapi.json` — control plane authenticated with a TiDB Cloud user JWT (`X-TiDB-Cloud-Authorization: Bearer `), every operation carries the required `orgId` query parameter. - `docs/api/data-plane.openapi.json` — data plane authenticated with a drive9 API key (`Authorization: Bearer ` or `X-API-Key`). The single source of truth is the endpoint registry in `pkg/server/openapi_registry.go` (it references the handlers' own request/response structs; `pkg/openapi` reflects them into JSON Schema components). `cmd/swagger-gen` renders the three files. When changing any HTTP API in `pkg/server`, update the registry and run `make swagger` so the committed documents stay in sync with the code. --- ## Local dev server ```bash export DRIVE9_LOCAL_DSN='root@tcp(127.0.0.1:4000)/drive9_local?parseTime=true' make run-server-local # DRIVE9_TENANT_PROVIDER=local, listen 127.0.0.1:9009 ``` Requires TiDB on `127.0.0.1:4000` (or let `make e2e-local` start `pingcap/tidb`). Suites `POST /v1/provision` for a JWT; do not use `local-dev-key`. --- ## Project layout ``` cmd/drive9-server/ Server entrypoint (`provider=local` for single-machine dev) cmd/drive9-billing-worker/ Hourly metering/billing worker (ledger + SQS outbox, leader-elected) .github/ISSUE_TEMPLATE/ GitHub issue templates (bug report / enhancement / feature request) pkg/ backend/ AGFS FileSystem implementation (Drive9Backend) billing/ Hourly usage collector (meta-DB ops deltas + storage snapshots) and SQS outbox sender buildinfo/ Version/hash ldflags injection datastore/ Core metadata store (TiDB/MySQL) embedding/ Embedding provider integration encrypt/ Encryption helpers gitcache/ Server-side Git workspace cache journal/ Append-only journal models leader/ Leader election helpers logger/ Structured logging (zap) meta/ Metadata/search models metrics/ Metrics recording migrate/ Tenant schema split tooling mysqlutil/ MySQL connection pooling helpers pathutil/ Path canonicalization and validation s3client/ S3 interface (AWS + local mock) schemaspec/ Schema SQL dump helpers semantic/ Durable background task types server/ HTTP server (/v1/fs/{path} router) slockoauth/ Lock/OAuth client helpers tagutil/ Tag validation tenant/ Tenant schema management tenantctx/ Tenant context plumbing testtidb/ TiDB test helpers (shared across packages) traceid/ Trace ID helpers vault/ Vault (scoped secrets) implementation e2e/ Shared-schema smoke; public e2e/blackbox is grafted from mem9-ai/drive9 scripts/ Shell helpers for local dev and test (e2e-local, blackbox-local) docs/ Design documents ``` When creating a new GitHub issue, follow the templates under `.github/ISSUE_TEMPLATE/` to keep issue structure and required context consistent. --- ## Code style guidelines ### Package structure - One package per directory; package name matches directory name. - Package-level doc comment on the first file: `// Package foo provides ...` - Each package has a focused responsibility — avoid cross-cutting concerns. ### Imports Group imports in three blocks separated by blank lines: ```go import ( "context" // stdlib "fmt" "go.uber.org/zap" // third-party "github.com/mem9-ai/drive9/pkg/logger" // internal ) ``` Use an import alias only when there is a naming collision: ```go pathpkg "path" // disambiguates from a local "path" variable ``` ### Naming conventions - Packages: short, lowercase, no underscores (`datastore`, `pathutil`, `s3client`). - Exported types/functions/consts: PascalCase (`FileNode`, `StorageType`, `RRFMerge`). - Unexported: camelCase (`smallFileThreshold`, `newBaseBackend`). - Sentinel errors: `ErrFoo` pattern (`ErrNotFound`, `ErrPathConflict`). - Constructor functions: `New(...)` or `NewWithConfig(cfg Config)`. - Config structs: `type Config struct { ... }` passed to `NewWithConfig`. - Test helpers: accept `*testing.T` as first arg, call `t.Helper()` at top. ### Types and constants - Use typed string constants for domain enums: ```go type StorageType string const ( StorageDB9 StorageType = "db9" StorageS3 StorageType = "s3" ) ``` - Prefer `*T` return from constructors; embed only when there is a strong behavioral reason. - Struct fields that can be absent: use pointer (`*int64`, `*time.Time`). ### Error handling - Return errors; do not panic in library code. - Wrap with context: `fmt.Errorf("insert node %s: %w", path, err)`. - Sentinel errors defined at package level with `errors.New(...)`. - Check `errors.Is(err, datastore.ErrNotFound)` at call sites; do not compare strings. - Ignore errors explicitly: `_ = s.Close()` (not silent discard). ### Logging Use `go.uber.org/zap` exclusively. Never use `log` or `fmt.Print*` in library code. ```go logger.With(zap.String("path", path)).Error("failed to insert node", zap.Error(err)) ``` Obtain a logger from `pkg/logger` or accept `*zap.Logger` via `Config`. ### Testing - Test files use `package foo` (same package, not `foo_test`) for white-box access. - Shared TiDB setup via `pkg/testtidb`; call `testtidb.ResetDB(t, db)` to clean state. - Test helper constructors (`newTestStore`, `newTestServer`) accept `*testing.T`, call `t.Helper()`, register cleanup with `t.Cleanup(func() { ... })`. - Use `t.Fatal` / `t.Fatalf` for setup failures; use `t.Errorf` for assertion failures. - No external assertion library — plain `if got != want { t.Fatalf(...) }`. - `TestMain` in `testmain_test.go` wires up the shared DSN for each package. - Server tests that need to call the HTTP API use the raw-HTTP helpers in `pkg/server/apitest_test.go`; do not reintroduce a client SDK for tests. ### Failpoint testing - Use failpoint only for high-value concurrency and failure-path boundaries (lease expiry, renew/stop races, panic cleanup, finalize ack/retry ownership checks), not as a blanket replacement for ordinary polling or simple mocks. - Put failpoint-backed tests in `*_failpoint_test.go` files with `//go:build failpoint`. - Prefer injection points at state-transition boundaries so tests can deterministically control ownership windows without production-only branching. - Scope failpoint callbacks narrowly by task/resource/action so one test cannot accidentally perturb unrelated work in the same package. - Always pair `failpoint.Enable(...)` / `failpoint.EnableCall(...)` with `t.Cleanup(...)` that disables the failpoint. - Run failpoint tests through `python3 scripts/run_failpoint_tests.py` or `make test-failpoint`. Do not run them in parallel with ordinary `go test` commands or `make lint`: `failpoint-ctl enable/disable` rewrites source files during the run and can break concurrent non-failpoint builds and type-checking. - Keep failpoint-off behavior identical to the non-instrumented code path. - When a timing-sensitive test still needs synchronization, prefer channels plus failpoint gating over sleeps that guess at scheduler timing. ### Context - All I/O and DB calls accept `context.Context` as their first parameter. - Pass context through; do not store it in structs. ### Concurrency - Protect shared mutable state with `sync.Mutex` (named `mu`); prefer fine-grained locking. - Background goroutines use `sync.WaitGroup` + a cancel context for clean shutdown. ### HTTP server patterns - Route on `*http.ServeMux`; handler methods on `*Server`. - Response helpers write JSON with `encoding/json`; set `Content-Type: application/json`. - Auth checked in a thin middleware layer (`pkg/server/auth.go`). ### Path conventions - All drive9 paths are absolute, UTF-8, NFC-normalized, no backslashes, no `..` segments. - Directories always end with `/`; files never do. - Use `pkg/pathutil` for all path normalization — never manipulate raw strings directly. ### Schema synchronization - `pkg/tenant/schema/tidb_auto.go`, `pkg/tenant/schema/tidb_app.go`, and `pkg/tenant/db9/schema.go` are the source of truth for tenant init schema SQL. - If you change schema shape in those files — columns, indexes, generated columns, constraints, defaults, or table definitions — you must also update externally managed schema definitions using the exported SQL from: ```bash drive9-server schema dump-init-sql --provider tidb_zero drive9-server schema dump-init-sql --provider tidb_cloud_native drive9-server schema dump-init-sql --provider db9 drive9-server schema dump-init-sql --provider tidb_cloud_native_shared drive9-server schema dump-init-sql --provider local ``` - The `tidb_cloud_native_shared` provider dumps the new shared (multi-tenant) schema shape: all 30 tenant tables in one physical database, each carrying an `fs_id BIGINT NOT NULL` discriminator column. - Do not maintain a second handwritten copy of those init SQL statements when a command can export the exact runtime source of truth.