# Codebase Restructure Design ## Background: The Next.js + `agentkit` Problem ### What Happened The current repository root doubles as the Next.js project root. `tsconfig.json` uses a broad glob pattern: ```json "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ...] ``` This inadvertently pulled `agentkit/plugins/ya-opencode-telemetry/index.ts` (an OpenCode plugin written in TypeScript/Bun) into the Next.js compilation graph. Next.js tried to compile it as frontend code, causing type errors and build failures because the plugin is built for a completely different runtime (Bun, not browser/Node). ### The Interim Fix (now resolved) `agentkit` was added to the `exclude` list in `tsconfig.json`: ```json "exclude": ["node_modules", "agentkit"] ``` This was a band-aid. **Phase 1 of this restructure (Frontend Isolation) has since been completed**: all Next.js code now lives under `frontend/`, so the root is no longer a Next.js workspace and this exclusion is no longer needed. ### Why This Was a Structural Problem The original root mixed three distinct concerns without clear boundaries: | Directory | Nature | |-----------|--------| | `app/`, `components/`, `lib/`, `public/` | Next.js frontend | | `agentkit/` | Core agent service (skills, MCP, plugins) | | `backend/` | Python FastAPI service | | `cli/` | Go binary | | `next.config.ts`, `package.json`, `tsconfig.json`, `pnpm-workspace.yaml` | Next.js config at root | Adding any new TypeScript project required manually patching `tsconfig.json`. **This is now resolved**: all Next.js code has been moved to `frontend/` (Phase 1 complete). The backend `main.py` is also a **2900+ line monolith** that handles authentication, workspace CRUD, ACP session management, SSE streaming, file serving, source preparation, and admin routes — all in one file with no internal package structure. This remains to be addressed in Phase 2. --- ## Target Directory Structure Legend: ✅ already in place, 🔲 planned. ``` investibook/ ← repo root ├── frontend/ ✅ ← Next.js app (self-contained) │ ├── app/ ← App Router pages │ ├── components/ ← React components │ ├── lib/ ← Client-side utilities & types │ ├── public/ ← Static assets │ ├── next.config.ts │ ├── tsconfig.json ← Clean tsconfig, no exclude hacks │ ├── eslint.config.mjs │ ├── postcss.config.mjs │ ├── package.json │ └── pnpm-lock.yaml │ ├── backend/ ✅ ← Python FastAPI service │ ├── investibook/ 🔲 ← Main Python package (Phase 2) │ │ ├── __init__.py │ │ ├── main.py ← FastAPI entry point (thin) │ │ ├── api/ ← Route handlers by domain │ │ │ ├── auth.py ← /api/auth/* │ │ │ ├── workspaces.py ← /api/workspaces/* + /api/workspace/* │ │ │ ├── chat.py ← /api/chat/stream │ │ │ ├── sources.py ← /api/sources/* │ │ │ ├── files.py ← /api/workspace/*/files, /file, /screenshot │ │ │ └── admin.py ← /admin/* │ │ ├── services/ ← Business logic │ │ ├── db/ ← SQLite access layer │ │ └── models/ ← Pydantic request/response models │ ├── main.py ✅ ← Current monolith (to be decomposed in Phase 2) │ ├── services/ ✅ ← Existing service modules │ ├── tests/ ✅ ← Pytest suite │ ├── scripts/ ✅ ← Entrypoint, init scripts │ ├── data/ ✅ ← Runtime data dir (gitignored) │ ├── Dockerfile ✅ │ └── pyproject.toml ✅ │ ├── agentkit/ ✅ ← Core agent service (independent, backend depends on this) │ ├── config/ ✅ ← Agent runtime config templates │ │ └── opencode.json.template ← OpenCode config template for workspace sessions │ ├── docs/ ✅ ← Agent-specific documentation │ ├── mcp/ ✅ ← MCP server implementations │ │ ├── aiops-mcp-demo/ ← AIOps MCP demo server │ │ └── resource-worker-mock-mcp/ ← Resource worker mock MCP server │ ├── skills/ ✅ ← Agent skills │ │ ├── builtin/ ← Built-in skills (grafana-screenshot, time-conversion, etc.) │ │ └── nutshell-skills/ ← Nutshell skill definitions (clinic-api, o11y-*, tidb-perf, etc.) │ ├── plugins/ ✅ ← OpenCode/agent runtime plugins │ │ └── ya-opencode-telemetry/ ← OpenCode telemetry plugin (Bun) │ ├── template/ 🔲 ← Workspace scaffold templates (Phase 3, replaces backend/templates/) │ │ ├── default/ ← Template for blank workspaces │ │ └── example/ ← Template for the example workspace │ ├── global-service/ 🔲 ← Shared long-running services for all workspaces (Phase 3) │ ├── credentials/ 🔲 ← Per-workspace auth/token storage (Phase 3, gitignored) │ │ └── workspace-{id}.json ← Injected credential context for one workspace session │ └── env/ 🔲 ← Environment variable management (Phase 3, gitignored) │ ├── global.env ← Shared vars for all workspace agent sessions │ └── workspace-{id}.env ← Per-workspace overrides (API keys, MCP tokens, etc.) │ ├── cli/ ✅ ← Go CLI ├── deployment/ ✅ ← Docker Compose + Kustomize manifests ├── docs/ ✅ ← All documentation ├── scripts/ ✅ ← Root-level ops scripts ├── tools/ ✅ ← Auxiliary tools (connection-api-key, mdbook) ├── AGENTS.md ✅ ├── CHANGELOG.md ✅ ├── README.md ✅ └── Makefile ✅ ``` --- ## Migration Plan ### ✅ Phase 1 — Frontend Isolation (COMPLETE) **Goal**: Move all Next.js code under `frontend/` so the project root is no longer a Next.js workspace. All steps completed: - `app/`, `components/`, `lib/`, `public/`, `next.config.ts`, `tsconfig.json`, `eslint.config.mjs`, `postcss.config.mjs`, `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` moved to `frontend/`. - `@/*` path alias preserved, `tsconfig.json` path root updated. - `"exclude": ["agentkit"]` removed from `tsconfig.json`. - `Makefile` updated to `cd frontend && ...`. - `AGENTS.md` updated. **Result**: `tsconfig.json` is clean. Root is no longer a Next.js workspace. ### Phase 2 — Backend Package Decomposition (medium impact, medium effort) **Goal**: Break `backend/main.py` (~2900 lines) into a proper Python package. Steps: 1. Create `backend/investibook/` package with `__init__.py`. 2. Extract route groups into `backend/investibook/api/`: auth, workspaces, chat, sources, files, admin — each a FastAPI `APIRouter`. 3. Extract all SQL query functions into `backend/investibook/db/queries.py`. 4. Move existing `backend/services/` content into `backend/investibook/services/`. 5. Extract Pydantic models into `backend/investibook/models/`. 6. `backend/investibook/main.py` becomes a thin wiring file: creates `FastAPI()`, registers routers, starts up DB. 7. Update `pyproject.toml` entry point from `backend.main:app` to `investibook.main:app`. 8. Update Dockerfile CMD accordingly. **Benefits**: Each domain can be modified independently, tests can import individual modules, the monolith becomes navigable. ### Phase 3 — Expand `agentkit/` as Core Agent Service (medium impact, medium effort) **Goal**: `agentkit/` (the directory keeps this name) is already partially in place with `config/`, `mcp/`, `skills/`, and `plugins/`. This phase adds the remaining subdirectories and wires the backend to use `agentkit/` as its authoritative source for templates, credentials, and env var injection. Existing subdirectories (already in `agentkit/`): | Directory | Status | Role | |-----------|--------|------| | `agentkit/config/` | ✅ exists | Agent runtime config templates (e.g. `opencode.json.template`) | | `agentkit/mcp/` | ✅ exists | MCP server implementations | | `agentkit/skills/` | ✅ exists | Agent skills (builtin + nutshell-skills) | | `agentkit/plugins/` | ✅ exists | OpenCode runtime plugins | | `agentkit/docs/` | ✅ exists | Agent-specific documentation | Subdirectories to add: | Directory | Role | |-----------|------| | `agentkit/template/` | Workspace scaffold templates — `default/` for blank workspaces, `example/` for the demo workspace. Replaces `backend/templates/`. | | `agentkit/global-service/` | Long-running shared local services (e.g. MCP daemons) used by all workspaces. | | `agentkit/credentials/` | Per-workspace user authentication/token storage. The backend writes Clinic, Grafana, DevOps, and other service credentials here on behalf of the user; the agent session reads them at startup. | | `agentkit/env/` | Environment variable management. `global.env` carries shared vars (model keys, common flags) for all workspaces. `workspace-{id}.env` carries per-workspace overrides (per-user MCP tokens, API keys, etc.). The backend renders and writes these files; the agent process sources them when launching a workspace session. | Migration steps: 1. Move `backend/templates/` → `agentkit/template/`. 2. Add `agentkit/credentials/` and `agentkit/env/` to `.gitignore`. 3. Move `backend/data/config/clinic_credentials.json` pattern → `agentkit/credentials/` (runtime, gitignored). Backend writes credentials here after user configures integrations via Settings UI. Agent session reads from `agentkit/credentials/workspace-{id}.json`. 4. Add `agentkit/env/global.env` and `agentkit/env/workspace-{id}.env` as the canonical source for env vars injected into agent subprocesses. Backend renders `workspace-{id}.env` when a workspace session starts (pulling from DB-stored integration config + Secret Manager-backed global keys). 5. Update `backend/main.py` (and later the decomposed package) to read all `agentkit/` paths via a configurable `AGENTKIT_DIR` env var (defaults to `../agentkit/` relative to the backend). 6. Update `backend/scripts/entrypoint.sh` and `Makefile` paths accordingly. **Security note**: `agentkit/credentials/` and `agentkit/env/` must be gitignored and only written at runtime (by the backend process or via Kubernetes Secrets). They are never committed to the repository. ### Phase 4 — Makefile & CI Updates Update all targets after phases 1–3: ```makefile install: cd frontend && pnpm install cd backend && uv sync web: cd frontend && NEXT_PUBLIC_API_URL=http://localhost:8000 pnpm dev api: cd backend && uv run uvicorn investibook.main:app --reload --port 8000 ``` Update GitHub Actions workflows, Dockerfiles, and Kustomize build contexts. --- ## What Does NOT Change The following stay at root level and are unaffected by the restructure: | Stays | Reason | |-------|--------| | `agentkit/` | Core agent service — lives at root, independent of frontend and backend | | `deployment/` | Infra configs reference paths relative to repo root | | `docs/` | Documentation is repo-wide | | `scripts/` | Ops scripts (sync-pod-credentials, etc.) are repo-wide | | `tools/` | Standalone utilities | | `cli/` | Independent Go module | | `README.md`, `CHANGELOG.md`, `AGENTS.md`, `Makefile` | Root-level project artifacts | --- ## Decision: Phased vs. Big-Bang Recommended approach: **phased, one PR per phase**. - Phase 1 (frontend isolation) is the most valuable and can ship independently. - Phase 2 (backend decomposition) is the highest developer-experience gain but carries the most refactor risk — it should have full test coverage before merging. - Phase 3 (expanding `agentkit/`) should follow Phase 1 and can be done in parallel with Phase 2 since it involves moving files, not rewriting logic. Each phase should update `AGENTS.md` to reflect the new canonical paths so AI agents working in the codebase always use correct imports and file references. --- ## Open Questions 1. **`pnpm-workspace.yaml`**: Currently at root and references the frontend packages. After Phase 1, should the pnpm workspace root move to `frontend/`, or should the root `pnpm-workspace.yaml` simply point to `frontend/`? The latter keeps a single `pnpm install` at root working. 2. **Backend package name**: `investibook` (matches product name) vs. `app` (generic). `investibook` is preferred for clarity. 3. **`nutshell-skills/` and `mcp_tools_server/`**: `nutshell-skills/` has already been consolidated under `agentkit/skills/nutshell-skills/`, and `mcp_tools_server/` under `agentkit/mcp/`. The backend will reference them via an `AGENTKIT_DIR` config rather than hard-coded relative paths (Phase 3).