# Dev Agent Proxy configuration

This overlay routes Agent API and model requests through agent-sidecar. It is a
configuration change only: image tags remain unchanged, and these manifests do not
create AWS resources or contain credential values. Do not sync this overlay until
the image, IAM, Secret and App ID prerequisites below are ready.

## Endpoints and permissions

| Client | Endpoint | Sidecar route |
| --- | --- | --- |
| Bundled business API helpers | `DS_PROXY_URL=http://127.0.0.1:16383` | DevOps, O11y DS/GS, Clinic, Jira and GitHub |
| Codex model provider | `http://127.0.0.1:16383/dashscope/compatible-mode/v1` | DashScope, including `POST /compatible-mode/v1/responses` |
| Ticket Manager CLI | `${DS_PROXY_URL}/devops/prod/ticket-manager` | dev Ticket Manager |
| Lark CLI | `LARKSUITE_CLI_AUTH_PROXY=http://127.0.0.1:16384` | Bot-only signed Auth Proxy |

Both listeners bind loopback inside the existing Agent Pod; neither is exposed
by a Service or Ingress. The ordinary listener's prefixes are stripped before
operation matching. The Lark `/` route preserves the original request path.
The eight routes and their policies come from aisupport's
`services/agent-sidecar/config.example.yaml`, with a TCP internal listener.
The existing model and Responses protocol are preserved; the Agent sends no
model API key, and Sidecar injects `DASHSCOPE_API_KEY`.

Lark initially permits only `GET /open-apis/bot/v3/info`. Other operations need
explicit method/path policy entries plus the application's scopes and resource
access. Business routes retain the example's production endpoints and allowed
callback writes; deploying into `agents-dev` does not turn these into dev upstreams.
The two existing DevOps callback PATCH operations remain enabled.

The workspace skill calls `ticket-manager-cli` with
`--base-url "${DS_PROXY_URL}/devops/prod/ticket-manager"`. This Proxy route forwards
workspace reads and advice creation to
`https://dev.devops.pingcap.com`. Sidecar supplies authentication from its existing
secret store. This nested route requires a Sidecar image with longest-prefix
routing support; the existing `/devops/prod` route continues to use its production
upstream. The Agent needs the CLI image and updated workspace/global skills;
no additional API credentials are injected into the Agent.

## Separate Agent and Sidecar Secrets

Use two AWS Secrets Manager JSON secrets in `us-west-2`, account `552185537300`.
All fields must be strings. Replace the placeholders outside Git.

`dev-agent-codex` remains the Agent's source. ExternalSecret `agent-codex-secrets`
uses `ClusterSecretStore/default` to extract only the following fields into the
Kubernetes Secret `agent-codex-secrets`:

```json
{
  "CODEX_APP_SERVER_TOKEN": "<KEEP_EXISTING_APP_SERVER_TOKEN>",
  "LARK_PROXY_KEY": "<RANDOM_PROXY_SIGNING_KEY>"
}
```

Preserve the existing app-server token so agent-gateway's configured token keeps
matching. Add the proxy signing key to this existing AWS Secret. Agent does not
receive `OPENAI_API_KEY` or other real model/business API credentials.

`dev-agent-codex-sidecar-secrets` is read directly by Sidecar using IRSA and is not
synchronized to a Kubernetes Secret. It contains the upstream credentials:

```json
{
  "DEVOPS_API_TOKEN": "<DEVOPS_API_TOKEN>",
  "DEVOPS_API_KEY": "",
  "CLINIC_API_KEY": "<CLINIC_API_KEY>",
  "JIRA_BASIC_AUTH_RAW": "<EMAIL:API_TOKEN_NOT_BASE64>",
  "GITHUB_PAT": "<GITHUB_PAT_ALSO_USED_BY_O11Y_EXCHANGE>",
  "DASHSCOPE_API_KEY": "<EXISTING_MODEL_API_KEY>",
  "LARK_APP_SECRET": "<LARK_APP_SECRET>",
  "LARK_PROXY_KEY": "<SAME_PROXY_SIGNING_KEY_AS_DEV_AGENT_CODEX>"
}
```

- Both AWS Secrets must contain exactly the same `LARK_PROXY_KEY` value. They are
  managed separately; ExternalSecret does not copy keys between AWS Secrets.
- Copy the existing model key from `dev-agent-codex`'s `OPENAI_API_KEY` into
  Sidecar's `DASHSCOPE_API_KEY`. Do not remove the old field until any remaining
  direct consumers have migrated. Sidecar does not need `CODEX_APP_SERVER_TOKEN`.
- Set at least one of `DEVOPS_API_TOKEN` and `DEVOPS_API_KEY`; leave an unused
  candidate empty. Candidates fall back on unavailable values, not upstream
  authentication errors.
- `JIRA_BASIC_AUTH_RAW` is the literal `email:api-token`, not Base64.
- `GITHUB_PAT` is also the bootstrap credential for O11y token exchange.
- `LARK_PROXY_KEY` is a random signing key, for example 32 random bytes encoded
  as 64 hex characters. CLI and Sidecar use the exact same string without hex
  decoding. `LARK_APP_SECRET` is the separate application credential.
- Do not store derived Tenant Access Tokens or O11y tokens. Sidecar refreshes
  them in memory. No Bedrock route is configured, so no Bedrock key is required.

Keep ExternalSecret's explicit field selection; do not add `dataFrom` or sync
Sidecar's upstream credentials into the Agent's Kubernetes Secret.

ExternalSecret and the Sidecar AWS source refresh every five minutes. Kubernetes
Secret changes do not update environment variables in running containers: restart
the Agent Pod after updating the proxy signing key in both AWS Secrets and waiting
for ExternalSecret to synchronize. The two refresh paths are independent, so a
rotation can cause temporary 401 errors until both sides use the same new key. Do not bypass the
proxy during this interval. App Secret changes are read from the refreshed source
on subsequent TAT exchanges. TATs are never used after expiration.

## Sidecar AWS identity

Create the dedicated role `agent-codex-sidecar-dev` outside this repository:

```text
arn:aws:iam::552185537300:role/agent-codex-sidecar-dev
```

Configure the role's trust policy for the target cluster's OIDC provider, with
`sub` restricted to `system:serviceaccount:agents-dev:agent-codex-sidecar-sa` and
`aud` restricted to `sts.amazonaws.com`. Grant `secretsmanager:GetSecretValue`
only for the actual ARN of `dev-agent-codex-sidecar-secrets`.

If the Secret uses a customer-managed KMS key, supply the corresponding narrowly
scoped decrypt permission and key policy operationally. Do not reuse the Gateway
IAM role.

The Deployment uses `agent-codex-sidecar-sa` with default Kubernetes API token mounting
disabled. The EKS IRSA webhook projects the STS token into `agent-sidecar`, while
`eks.amazonaws.com/skip-containers: agent-codex` excludes the Agent. The IRSA token
must not be mounted into the shared home or projects PVC. See the
[AWS webhook documentation](https://github.com/aws/amazon-eks-pod-identity-webhook).
Kustomize cannot simulate the admission webhook; verify the resulting Pod's
container-specific mounts/environment names during the eventual rollout.

## Deployment prerequisites and migration

1. Publish compatible Sidecar and Agent Codex images first. Sidecar must support
   named TCP listeners and `larkTenantToken`; Agent's CLI must be built with the
   `authsidecar` tag. Update the existing image tag locations before syncing this
   configuration; the tags retained in this change are not evidence of compatibility.
2. Create/populate `dev-agent-codex-sidecar-secrets` and the dedicated IAM role.
   Keep `dev-agent-codex` for Agent, preserving its app-server token and adding the
   matching proxy signing key. The Kubernetes Secret remains `agent-codex-secrets`.
3. Replace `cli_REPLACE_WITH_APP_ID` in both
   `envs/dev/agent-codex/agent-sidecar-config.yaml` and `base/agent-codex/deployment.yaml` with the
   same self-built application's App ID. Default brand is `feishu`; international
   Lark requires both brands to be `lark` and the route origin to be
   `https://open.larksuite.com`.
4. Check ExternalSecret `agent-codex-secrets` has synchronized both required fields.
   Existing ConfigMaps are mounted with `subPath`; configuration changes require
   a Pod restart (Kustomize's changed ConfigMap hash updates the Deployment).
5. For existing PVCs, synchronize the reviewed seed changes from the compatible
   aisupport image/revision, preserving unrelated local modifications:
   - Global access rules: `$CODEX_HOME/AGENTS.md`.
   - Business proxy skills: `$PROJECTS_ROOT_DIR/ticket-manager/.agents/skills/builtin/`.
   - All seven Lark skills and references: `$PROJECTS_ROOT_DIR/ticket-manager/.agents/skills/lark-cli/`.
   New images seed only missing destinations. Do not delete/reseed a project or
   add an automatic overwrite mechanism. Start a new task after updating skills.
6. After the separately authorized rollout, verify `/healthz` from inside the
   Pod and run the bundled proxy configuration check and bot info command. Verify
   the actual admitted Pod gives AWS identity only to Sidecar, without printing
   token files or Secret values. Check a model response and one authorized read
   per needed business route. Existing WebSocket/gRPC Service ports and probes
   remain unchanged.

## Validation

From the app-delivery root:

```bash
kubectl kustomize dev/devops-service/agents/envs/dev
git diff --check
```

Validate the Sidecar ConfigMap content with the current Go config package's YAML
decoder, `Normalize()` and `Validate()`. No live AWS access is required for that
check. Validate the rendered provider TOML and use a simulated upstream with no
client API key to exercise Responses/SSE. The Codex provider configuration follows
the [official configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference).

Connection failures indicate the local listener is unavailable; policy denial is
405, unavailable credentials are 503, and Lark signature failures are 401. Lark
403 indicates a prohibited target/identity/auth header. Report upstream resource
or scope errors separately and do not retry against a direct upstream address.
Listener health alone does not establish credential or resource readiness.

For repeatable CLI compatibility checks, run from `services/agent-sidecar` in the
matching aisupport checkout (both upstreams are simulated and credentials synthetic):

```bash
CODEX_TEST_BINARY=/absolute/path/to/codex \
CODEX_TEST_CONFIG=/absolute/path/to/agents/envs/dev/agent-codex/config.toml \
AGENT_PROXY_TEST_CONFIG=/absolute/path/to/agents/envs/dev/agent-codex/agent-sidecar-config.yaml \
LARK_CLI_TEST_BINARY=/absolute/path/to/authsidecar-lark-cli \
GOWORK=off go test ./internal/services/apiproxy \
  -run 'TestCodexResponsesProxyIntegration|TestLarkCLIIntegration|TestLarkRouteAuthenticatesBeforeRewriting|TestProxyCredentialUnavailable' -count=1
```

The new Codex test is skipped unless `CODEX_TEST_BINARY` is supplied; use the
version pinned by the Agent image. These checks do not establish live IAM, AWS
Secret, application scope or image availability.