# AGENTS.md

Approach work in this repository as an experienced Go, Kubernetes, observability, database, and Linux engineer.

## Mission

`agent-control-server` is a Kubernetes operator-style control plane for observability agent lifecycle management.

It reconciles agent CRs into infrastructure and workload actions such as:
- observability backend registration
- private connectivity such as PrivateLink / PSC / private endpoints
- deployment and updates of `vmagent`, `vector`, and related resources
- restart, drain, storage, and spec adjustment flows
- autoscaling and autoresourcing

This repository is orchestration-centric. Most behavior is expressed through controller task pipelines.

## System Shape

Core execution path:
1. watch Agent CR
2. build reconcile context
3. assemble ordered tasks
4. execute via shared deployer
5. update annotations, finalizers, phase, and status

## API Domains

### `api/v2`
Primary agent CRDs:
- `O11yK8sAgent`
- `O11yTiDBAgent`
- `KubernetesAgent`
- `SharedPoolAgent`
- `TiDBGroupAgent`

Architecture split:
- dedicated: `O11yK8sAgent`, `O11yTiDBAgent`
- nextgen: `KubernetesAgent`, `SharedPoolAgent`, `TiDBGroupAgent`

The dedicated/nextgen split is one of the core concepts in this repository.
Do not assume a fix is complete after updating only one CR in the same architecture.

Common identity fields:
- `tenantId`
- `projectId` (dedicated only, absent from nextgen `O11yConfigV2`)
- `applicationId`
- `sharedPoolId`
- `kubernetesClusterName`
- `tidbGroup`

### `api/clusterserver/v1`
Cluster topology and upstream facts used by sizing and deployment logic.

### `api/regionalserver/v1alpha1`
Regional network and routing context around agents, including VPC, route, APPBox, and private connectivity.

## Controller Model

Controller families:
- `controllers/kubernetesagent`
- `controllers/sharedpoolagent`
- `controllers/tidbgroupagent`
- `controllers/o11yk8sagent`
- `controllers/o11ytidbagent`

All controllers follow the same broad pattern:
- fetch CR
- inspect lifecycle state
- build remote cluster access
- assemble ordered tasks
- execute via shared deployer
- update status and emit events

Most meaningful behavior lives in task composition, not controller shape.

## Shared Execution Model

Primary shared packages:
- `pkg/reconciler`
- `pkg/deploy`
- `pkg/commontasks`
- `pkg/config`
- `pkg/server`

Important invariant: task ordering is part of the system contract.

The deployer (`pkg/deploy`) executes tasks sequentially and uses three result states:
- `Done`: success, runs PostCreate/PostDelete, pipeline continues
- `Error`: non-fatal error, runs PostCreate/PostDelete, pipeline continues
- `Panic`: error, skips post-action, halts the entire pipeline

Returning the wrong result state can cause post-actions to be skipped or the pipeline to halt unexpectedly.

Typical create flow:
1. `Guardian`
2. backend/bootstrap tasks
3. cloud-specific tasks
4. operator / repo setup
5. workload tasks such as `vmagent`, `vector`, `vmpodscrape`
6. operational tasks such as restart, drain, version handling

Delete flow runs in reverse order through the shared deployer.

Do not casually reorder tasks.

## Client Semantics

The reconcile context separates two Kubernetes clients:
- `Local`: control-plane cluster client
- `Remote`: target dataplane / customer cluster client

Use `Local` for agent CR updates, local infrastructure resources, and local secrets.
Use `Remote` for `vmagent`, `vector`, `vmpodscrape`, operator, and related workload resources in target clusters.

Mixing these clients is a high-risk mistake.

Client version split:
- dedicated controllers use `k8scliv1.NetworkClientGetter` (EKS + GKE)
- nextgen controllers use `k8scliv2.NetworkClientGetter` (EKS + GKE + AKS + ACK, adds metrics client)

## Annotation Protocol

Annotations are an internal control protocol across the HTTP server, autoscaler, controllers, and tasks.

They drive behaviors such as:
- pause / resume
- spec scaling (desired / min / max / previous spec, scale status)
- operational triggers (restart, drain, storage changes)
- backend and endpoint wiring

Treat annotation keys as compatibility-sensitive.

## HTTP Control Surface

`pkg/server` exposes operations such as:
- list agents
- inspect and update spec
- pause / resume
- restart component
- drain node
- adjust storage

The HTTP server mostly writes CR updates or annotations. Controllers perform the actual work later during reconciliation.

## Sizing and Scaling

There are two distinct mechanisms. Do not treat them as interchangeable.

Autoresourcing (`pkg/commontasks/autoresourcing.go`):
- dedicated only, runs as a reconciliation task in `o11yk8sagent` and `o11ytidbagent`
- selects a spec tier from `CustomerAgentSpecs` based on TiDB cluster replica counts and writes the `desired-spec` annotation

Pod autoscaling (`pkg/podautoscaler/`):
- reads real-time CPU and memory via the K8s metrics API
- targets: `kubernetes/vmagent`, `sharedpool/vmagent`, `sharedpool/vector`, `tidbgroup/vector`
- writes spec annotations; controllers apply the change on next reconciliation

## Serverless Mode

The serverless/non-serverless split is controlled by `RootConfig.Mode` (`pkg/config`). Key behavioral differences in serverless mode:
- namespace changes from `monitoring` to `observability`
- VMPodScrape tasks are skipped in `sharedpoolagent`
- PrivateLink has distinct endpoint reuse and tagging behavior

## High-Risk Areas

Be cautious around:
- task ordering dependencies
- annotation-driven side effects
- phase transitions and finalizers
- multi-stage deletion behavior
- private connectivity and endpoint reuse logic

## Repo-Specific Do / Don't

### Do
- read the target CRD first, then its controller, then its tasks
- preserve the controller -> task -> deployer structure
- add behavior in tasks when possible, not directly in controllers
- prefer the smallest viable change with the least architectural and behavioral impact
- if changing dedicated behavior, review both `O11yK8sAgent` and `O11yTiDBAgent`
- if changing nextgen behavior, review `KubernetesAgent`, `SharedPoolAgent`, and `TiDBGroupAgent` together
- if changing a cross-provider feature, review AWS / GCP / Azure / AliCloud branches together
- if adding or changing a feature, review both serverless and non-serverless behavior together
- reason about both create and delete paths for any behavior change

### Don't
- don't reorder tasks unless dependencies are fully understood
- don't mix `Local` and `Remote` client responsibilities
- don't assume HTTP handlers execute actions directly
- don't treat annotations as incidental metadata
- don't ignore finalizer and phase transitions in lifecycle changes
- don't treat one controller or one CR as the full change surface for dedicated or nextgen behavior
- don't assume a feature is complete after updating only one provider path in a multi-cloud behavior
- don't combine a feature request or bugfix with opportunistic optimization, cleanup, or refactoring unless required for correctness

## Development Workflow

Before proposing completion of code changes:
- run `make lint`
- run `make test`
- fix formatting, lint, and test failures first
- verify that the final diff stays focused on the requested change

For unit tests:
- prefer table-driven style
- mirror nearby `*_test.go` structure and naming
- keep assertions explicit and aligned with repository conventions
- keep test comments minimal

For comments and doc comments:
- follow idiomatic Go comment style
- start with a one-sentence summary, then leave a blank comment line before extra detail when needed
- prefer explaining why, trade-offs, and constraints over repeating what the code already says
- keep test comments especially lean

For pull requests:
- always use `main` as the base branch unless explicitly instructed otherwise

For image builds:
- dev: https://github.com/tidbcloud/observability/actions/workflows/agent-dev-cd.yaml
- prod: https://github.com/tidbcloud/observability/actions/workflows/agent-prod-cd.yaml

## Recommended Reading Order

For any behavior investigation, read in this order:
1. target CRD in `api/v2`
2. corresponding controller in `controllers/*/controller.go`
3. task list assembled by that controller
4. task implementations in `controllers/*/task/*`
5. shared helpers in `pkg/commontasks/*`
6. shared execution primitives in `pkg/reconciler` and `pkg/deploy`
7. related config in `pkg/config`
8. related server endpoints in `pkg/server` if the behavior is user-triggered