# Remote AWS runtime identity ## Contract Human SSO is a bootstrap and operator identity. It may provision or destroy a session, but it is not a credential source for long-running workloads on a `remote-ec2-k3d` host. Remote sessions use one session-owned EC2 Instance Profile. Terraform publishes the typed `remote_ec2_runtime` output, the host launcher attaches that exact profile with IMDSv2 required, and the session context carries the output to both supported Tiltfile paths. Runtime code must consume the AWS SDK default provider chain or the provider's native ambient-identity mode. It must not derive IAM names or materialize access keys from the user's profile. Local k3d intentionally keeps the existing development contract: `aws-creds` and `cp9-aws-credentials` may be materialized from the local operator profile, and `cp9 refresh-creds` updates that local path. On remote EC2 the same command returns `required=false` without minting credentials or writing Kubernetes Secrets. Remote sessions created before this contract are not migrated in place. If a persisted remote EC2 instance has no `InstanceProfileName` / runtime role in its host state, both lifecycle resume and `cp9 refresh-creds` fail closed with a destroy-and-recreate instruction. The host step also rejects a missing or incomplete Terraform runtime output contract before any `RunInstances` call; there is no profile-less fallback launch. The checked-in runtime enforcement lives in [`apply-aws-creds-secret.sh`](../personas/dev-full/runtime/scripts/apply-aws-creds-secret.sh), with read-only evidence captured by [`capture-remote-aws-runtime-identity.sh`](../personas/dev-full/proof/capture-remote-aws-runtime-identity.sh). ## Identity boundaries The remote Instance Profile is the base identity for control-plane processes on the k3d node. Upbound ProviderConfigs use `source: PodIdentity` (the v1.21.1 mode that invokes the AWS SDK default chain), while the classic Dedicated provider uses `InjectedIdentity`; both retain their existing product-role assume chains. Product dataplane identities are not collapsed into the node identity: - Premium and Essential retain dataplane-manager-created roles. - Dedicated retains controller/Crossplane-created roles and Instance Profiles. - Starter retains EKS cluster/node roles and EKS Pod Identity. - Volume Replication inherits the owning product dataplane mechanism; no independent named IAM contract exists today. The remote role currently reuses `cp9-developer-power` under `cp9-developer-boundary`. This node-level permission parity was explicitly accepted for issue #784 so the migration can remove human credential lifetime coupling without changing product authorization semantics in the same change. It is a deliberate, observable exception to the issue's original least-privilege preference, not a claim that the role is already minimal. ## Renewal behavior EC2 exposes automatically rotated temporary role credentials through IMDS. AWS SDK default providers and Crossplane injected providers fetch and refresh those credentials; no Secret replacement or pod restart is part of normal remote operation. The remote manifest filters remove: - access-key environment variables sourced from human credential Secrets; - shared-credentials file variables and mounts; - `aws-creds` and `cp9-aws-credentials` volumes; - Secret-backed ProviderConfig credentials. There is no longer an exception for KMS. The `kms-local` stub and its literal fake credentials are gone; every service now reaches real AWS KMS with the session CMK, so on remote sessions those services are filtered like any other and fall back to the Instance Profile. ## Evidence Normal remote startup writes sanitized evidence under the session evidence directory: - `aws-runtime-identity-.json` records the expected role, STS caller ARN, absence of credential Secrets, and `refresh_required=false`. - `remote-aws-runtime-identity.json` comes from an in-cluster AWS CLI Job. It records the Instance Profile ARN/name and proof pod UID, proves pod-to-IMDS access, checks all control-plane workloads for forbidden Secret wiring, and verifies every installed AWS ProviderConfig uses its API's native ambient source (`PodIdentity` for Upbound, `InjectedIdentity` for classic Crossplane). Only caller ARNs, role ARNs, timestamps, counts, and one-way truncated fingerprints are recorded. Credential values, tokens, Secret contents, and raw IMDS credential documents must never enter logs or evidence. ## Troubleshooting Start with the evidence files and the machine-readable inventory. Safe checks that do not reveal credential material include: ```sh aws sts get-caller-identity kubectl get providerconfigs.aws.upbound.io \ -o custom-columns=NAME:.metadata.name,SOURCE:.spec.credentials.source kubectl get providerconfigs.aws.crossplane.io \ -o custom-columns=NAME:.metadata.name,SOURCE:.spec.credentials.source kubectl get secret -A \ -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name \ --no-headers | grep -E '(^|[[:space:]])(aws-creds|cp9-aws-credentials)$' ``` For a remote session, the STS ARN must have the shape `arn:aws:sts:::assumed-role/-remote-runtime/`, Upbound ProviderConfig sources must be `PodIdentity`, classic Crossplane ProviderConfig sources must be `InjectedIdentity`, and the Secret-name query must return nothing. Do not inspect Secret data or print IMDS credential JSON. If the identity proof Job cannot call STS, verify the host contract and launch settings first: the attached Instance Profile name must match `remote_ec2_runtime.instance_profile_name`, IMDS must be enabled, tokens must be required, and the response hop limit must be 3 for the nested EC2 host → k3d node container → pod network. A profile mismatch fails closed during host creation. ## Cleanup The role and Instance Profile are Terraform session resources tagged with `cp9-owner`, `cp9-session`, `cp9-managed`, and `cp9-ttl`. Normal destroy removes them with the remote host. The tag-scoped reaper and post-cleanup verifier already enumerate IAM roles and Instance Profiles, so either resource is a live survivor and prevents a zero-survivor closeout. A successful destroy reports `final_survivors=0`; acceptance evidence must retain that sanitized closeout.