# Independent Dedicated API in one dev-full k3d session

## Decision

`dev-full` stores Premium and Dedicated regional metadata in different Kubernetes API servers. The existing k3s API remains the host API and the only scheduler. A session-owned etcd, kube-apiserver, and minimal kube-controller-manager run as Pods on that host and provide a metadata-only Dedicated API.

This is an API and storage boundary, not a second worker cluster. It exists because Premium and Dedicated have incompatible schemas for some identical group/version/kind names. Labels cannot protect a controller from failing while decoding an incompatible object.

## Topology and ownership

```text
cp9 bootstrap / Dedicated services
  |-- host kubeconfig ----------> k3s host API + host etcd
  |                                |-- every Pod, Service, Job and Deployment
  |                                |-- Premium/DPM definitions and objects
  |                                |-- Upbound AWS provider objects
  |                                `-- Dedicated control-plane Pods
  |
  |-- bootstrap/admin kubeconfig -> loopback NodePort -> Dedicated kube-apiserver
  `-- Dedicated service kubeconfigs -> in-cluster Service ----^
                                                   |-- Dedicated regional definitions
                                                   |-- cluster-control metadata
                                                   |-- classic AWS ProviderConfigs/resources
                                                   |-- Dedicated TiCDC metadata
                                                   `-- Dedicated etcd PVC
```

Regional-server, cluster-control-server, cluster-controller, Dedicated TiCDC, and the classic AWS provider are deployed through the host API but use distinct certificate identities and kubeconfigs to control the Dedicated API. Premium services continue using the host API. The secondary API has no scheduler or nodes and rejects Pods, Jobs, Deployments, StatefulSets, and DaemonSets through admission policy.

The exact Dedicated regional and TiCDC definitions are vendored with their upstream commit provenance. The Dedicated infra API retains the classic provider-aws v0.47.2 definitions. Static rendering tests enforce that these definitions and host workloads stay on their intended sides of the boundary.

## Session lifecycle

On a fresh `cp9 up`, CP9:

1. Allocates and persists a loopback host port and fixed internal NodePort.
2. Generates session PKI and component kubeconfigs atomically with private file modes.
3. Creates the host k3d mapping, reads the real k3s server version, and pins matching upstream kube-apiserver/controller-manager images.
4. Deploys Dedicated etcd and the API server onto the host API.
5. Performs authenticated `/readyz`, installs secondary RBAC, admission policy, exact definitions, ProviderConfigs, and component kubeconfig Secret.
6. Waits for every definition to become Established before starting dependent services.
7. Writes non-secret bootstrap evidence containing API identities, definition count/digest, and zero executable secondary workloads.

Resume reuses the persisted port, CA identity, images, member identity, and manifest paths. A pre-feature session with no Dedicated API state fails with a recreate instruction; CP9 does not invent credentials or silently return to shared schemas.

Remote-k3d sessions forward both host and Dedicated loopback endpoints through the existing SSH boundary. The endpoint is never published as a public load balancer.

## Readiness and diagnostics

Ordinary `cp9 status` does not connect to or report the business-internal Dedicated API. Product readiness is observed through the public OpenAPI and UI surfaces.

`cp9 logs service=<dedicated-service>` continues collecting Pod logs from the host API and adds bounded status and event diagnostics queried with the Dedicated admin kubeconfig. The supported service names include regional-server, cluster-control-server, cluster-controller, Dedicated TiCDC, and the classic AWS provider.

If the remote SSH tunnel is no longer alive, only the explicit Dedicated log diagnostics are unavailable. Ordinary status, product proof, and destroy do not repair or depend on that tunnel.

## Recovery and destroy

Dedicated etcd uses a session-owned persistent volume. Restarting the etcd or API Pods preserves secondary objects and API identity. The integration gate proves recovery after restarting both Pods.

`cp9 destroy` does not enter the business-internal Dedicated API. It stops the normal Tilt/host runtime and proves cloud cleanup through session tags, persisted known resource IDs, and the authoritative cleanup workflow. Deleting the host k3d cluster removes the secondary API and etcd PVC with the rest of the session substrate.

Product roots and controller finalizers are owned by the Dedicated OpenAPI/product lifecycle, not by the generic cp9 destroy command. A failed or unproven authoritative cleanup handoff remains fail closed and preserves the runtime for retry.

### Operator recovery caveats

Two provider edge cases were observed in the fresh remote product train and must be handled without bypassing ownership or finalizers:

- EKS deletion can leave a detached VPC-CNI network interface and the EKS-created cluster security group after the EKS cluster itself reaches `NotFound`. These objects have no classic-provider CR. Before deleting either object, verify that the EKS cluster is absent, the ENI is `available` with no attachment, and both objects carry ownership metadata for that exact deleted cluster. Delete only those positively identified dependents with the session runtime role; then allow the managed VPC CR to finalize normally.
- A classic Crossplane managed resource can stop at `cannot determine creation result` with `crossplane.io/external-create-pending`. Never remove its finalizer. First query the external service using the intended name/tags and prove that no external object was created. Only then remove the `crossplane.io/external-create-pending` annotation. Restart or resync the provider if necessary and let the provider perform its ordinary delete/finalizer path. If external existence is ambiguous, preserve the API, provider, and etcd and investigate instead of forcing cleanup.

The product proof must disclose either recovery path when used; it is not equivalent to an automatic clean pass.

## Release proof

The independent acceptance gate is the Kong/OpenAPI product train, not a manifest-only check. It must create Premium and Dedicated concurrently, alternate modifications, delete and recreate each product while continuously checking the other, and finish with zero/gone cleanup evidence.

The proof bundle records:

- both API server and etcd identities;
- definition digests and conflicting-schema placement;
- classic versus Upbound provider placement;
- product API responses and SQL results throughout interleaving;
- absence of cross-plane writes and executable secondary workloads;
- final product, Kubernetes, and cloud-resource absence.

The pre-change baseline remains part of acceptance: Premium passed its sequential train; Dedicated create was accepted but did not become usable; deleting that failed record passed; interleaved isolation was unproven.

## Boundaries

This design does not add a second physical k3d cluster, workload syncer, vcluster, public endpoint, or high-availability control plane. It does not modify upstream service source code. Global/dataflow services that communicate with Dedicated over existing HTTP/gRPC routes remain host-API workloads unless a future feature proves that they directly own secondary Kubernetes metadata.