# 10. Env controllers get AWS credentials from the host instance-profile, not an injected session

Status: Reinstated for remote EC2+k3d (2026-07-11). The 2026-06-29 validation
correctly proved that Upbound `source: None` is not an ambient credential mode,
but v1.21.1 source review and live reconciliation later identified
`source: PodIdentity` as the provider's AWS SDK default-chain path. Local k3d
continues to use `cp9 refresh-creds`. See both updates below.

## Context

A cp9 env's in-cluster controllers (dataplane-manager, cluster-ng, the Crossplane providers)
make AWS calls. Today they read a static AWS **session** from the `aws-creds` secret, injected
at provision time. That session expires in ~1 h with no supported refresh: `cp9 up` does not
rewrite the secret, there is no refresh command, and the controllers cache the credentials at
startup. So any long-running operation (a create reaching ACTIVE is ~33 min) or any long-lived
/ shared env eventually fails with `ExpiredToken`. This directly blocks the warm-pool
(ADR 0009) and shared-env directions, both of which require a long-lived env.

## Decision

Move the in-cluster controllers off the injected session and onto a **non-expiring,
auto-rotating credential source: the EC2 host's instance-profile, consumed via IMDS.**

- The cp9 substrate gives the env's EC2 host an IAM instance-profile carrying the permissions
  the controllers need (directly, or able to assume the existing per-controller sub-roles).
- The k3d controllers use the host IMDS (`169.254.169.254`) for credentials; the AWS SDK
  auto-renews them — **no 1 h cliff and no session injection**.
- **IRSA was rejected:** the cp9 control plane runs in **k3d on the EC2 host, not a real EKS**,
  so there is no OIDC provider to back IRSA.
- Go straight to this root fix; do **not** ship an interim `cp9 refresh-creds` command.

Scope: cp9-internal — host IAM in the substrate plus the controllers' credential configuration.
No upstream business-repo changes.

## Consequences

- Long-running creates and long-lived / shared envs no longer hit `ExpiredToken`; this unblocks
  ADR 0009 (warm pool) and the shared-env direction.
- The injected `aws-creds` session secret is no longer the controllers' credential source and
  can be retired.
- Implementation must confirm: (a) the host instance-profile's permission scope — direct
  permissions vs assuming the per-controller sub-roles; (b) that k3d pods can reach the host
  IMDS (default yes; verify it is not blocked).
- Out of scope: the human-driven `cp9` CLI's own AWS calls still use the operator's SSO
  (`aws sso login`); that is separate from the in-env controllers and unaffected by this ADR.

## Update (2026-06-29): instance-profile/IMDS is infeasible for the upbound provider — superseded by `cp9 refresh-creds`

End-to-end validation of this decision (fresh premium-full env, remote-ec2-k3d) found it cannot
work for the Crossplane **upbound (Terraform-based) provider**, the env's primary AWS controller:

- The provider's `ProviderConfig.spec.credentials.source` enum is `None / Secret / IRSA /
  WebIdentity / PodIdentity / Upbound`. There is **no default-chain / IMDS / instance-profile
  source.** `source: None` does not fall through to the SDK default chain — it yields
  `static credentials are empty`. IRSA / WebIdentity / PodIdentity all need an OIDC issuer or
  EKS Pod Identity, which k3d lacks (the same reason IRSA was rejected above). (`InjectedIdentity`
  exists only on the dedicated **aws.crossplane.io** community provider, not on aws.upbound.io.)
- The host instance-profile and IMDS themselves work: a k3d pod running the standard AWS SDK
  (`aws sts get-caller-identity`) returned the host's workload role via IMDS (hop-limit 3 +
  routing verified). The gap is purely the upbound provider's credential model, not the host or
  the network — a process using the standard SDK in a pod *can* use the instance-profile.
- `source: Secret` works; its temporary creds simply expire (the `ExpiredToken` 1 h cliff this
  ADR set out to kill).

**Revised decision:** keep `source: Secret` and ship the `cp9 refresh-creds` command this ADR
originally rejected — but mint its credentials from the session's source profile + assume-role,
which auto-rotates for the source SSO's lifetime (~8–12 h, not 1 h). The command re-mints and
patches the `aws-creds` secret in every consuming namespace on demand; an agent loop or human
calls it periodically (a thin keeper loop can drive it). The substrate instance-profile / IAM
trusts remain useful for any in-env process that uses the standard SDK (which can reach IMDS),
should a fully non-expiring in-cluster refresher be built later.

## Update (2026-07-11): remote Upbound default-chain path verified

Source review of the exact Upbound AWS provider-family v1.21.1 implementation
showed that `PodIdentity`, not `None`, invokes `config.LoadDefaultConfig` without
the IRSA-only token-file cache. In remote k3d, with no earlier environment,
shared-file, web-identity, or container credential source, that chain falls
through to EC2 IMDS. A real `s3.aws.upbound.io/v1beta2 Bucket` reconciled to
`Ready=True` and deleted successfully through `source: PodIdentity` after the
initiating human SSO session had expired. `None` still fails with empty static
credentials, and tokenless `IRSA` fails because its cache requires
`AWS_WEB_IDENTITY_TOKEN_FILE`.

The remote contract is therefore restored with Upbound `PodIdentity`, classic
Crossplane `InjectedIdentity`, IMDSv2 required, and response hop limit 3 for the
nested EC2 host → k3d node container → pod network. The exact source trace and
live reconciliation are documented in
`docs/20260711-research-upbound-vs-classic-aws-credentials.md`.