# Upbound v1.21.1 vs. classic AWS ProviderConfig credentials

## Conclusion

For `aws.upbound.io` provider-family v1.21.1, `credentials.source: None` does
**not** mean “use the AWS SDK default credential chain.” It means that the
common Crossplane credential extractor returns no credential bytes; the
provider then feeds those empty values to a static AWS credential provider.
The resulting `static credentials are empty` live error is therefore the
expected consequence of the v1.21.1 implementation.

The same Upbound API rejects `InjectedIdentity` because that value is absent
from its CRD enum. In this release, the source that reaches the AWS SDK default
chain without also requiring an IRSA token file is `PodIdentity`. On an EKS
cluster that chain normally discovers the EKS Pod Identity container endpoint
([AWS's Pod Identity SDK description](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html));
in the remote k3d environment, where no container-credential endpoint or web
identity variables are present but EC2 IMDS is reachable, the same SDK chain
falls through to the EC2 instance-role provider. Thus `PodIdentity` is the
code-supported remote EC2 instance-profile path; `None` is not. The live
reconciliation described below confirms the source-code inference.

The classic `aws.crossplane.io` provider has a different API and an explicit
`InjectedIdentity` branch. For its AWS SDK v2 controllers that branch calls
`config.LoadDefaultConfig`, so EC2 IMDS is a valid final fallback. This matches
the observed classic-provider and ordinary AWS-SDK pod behavior.

## Source trace

### Why Upbound rejects `InjectedIdentity`

The v1.21.1 Upbound `ProviderCredentials.Source` enum is exactly `None`,
`Secret`, `IRSA`, `WebIdentity`, `PodIdentity`, and `Upbound`; it does not
contain `InjectedIdentity` ([v1.21.1 API type, lines 226–238](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/apis/v1beta1/types.go#L226-L238)).
The generated CRD repeats that enum
([v1.21.1 CRD](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/package/crds/aws.upbound.io_providerconfigs.yaml)).
Kubernetes admission therefore correctly rejects an Upbound ProviderConfig
whose source is `InjectedIdentity`.

This is not a generic Crossplane limitation. Crossplane Runtime defines
`InjectedIdentity` as provider-specific use of the pod's identity and states
that each provider must implement its own handler
([runtime credential-source semantics](https://github.com/crossplane/crossplane-runtime/blob/edccdbc4afa7d805c36a583221b4625f6b2aacce/apis/common/v1/resource.go#L232-L248),
[extractor behavior](https://github.com/crossplane/crossplane-runtime/blob/edccdbc4afa7d805c36a583221b4625f6b2aacce/pkg/resource/providerconfig.go#L90-L106)).
Upbound chose its own AWS-specific source names instead.

### What `None` actually does in Upbound v1.21.1

The behavior is a three-step chain:

1. Upbound special-cases only `IRSA`, `PodIdentity`, `WebIdentity`, and
   `Upbound`. Every other source, including `None`, goes through
   `CommonCredentialExtractor` and then `UseProviderSecret`
   ([provider switch](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/provider_config.go#L95-L131)).
2. Crossplane Runtime defines `None` as “the provider does not require
   credentials” and returns `(nil, nil)` for that source
   ([definition](https://github.com/crossplane/crossplane-runtime/blob/edccdbc4afa7d805c36a583221b4625f6b2aacce/apis/common/v1/resource.go#L232-L241),
   [extractor](https://github.com/crossplane/crossplane-runtime/blob/edccdbc4afa7d805c36a583221b4625f6b2aacce/pkg/resource/providerconfig.go#L90-L106)).
3. Upbound parses those bytes as an INI credential profile and explicitly
   installs an AWS `StaticCredentialsProvider`; it does not leave credential
   resolution to the default chain
   ([static-provider construction](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/provider_config.go#L233-L298)).
   AWS SDK for Go v2 returns `static credentials are empty` when either static
   access-key field is empty
   ([SDK v1.17.47 implementation](https://github.com/aws/aws-sdk-go-v2/blob/01c4d8bb3f39125855bc5afbcb36da832556d52b/credentials/static_provider.go#L12-L53)).

This exactly explains the live `source: None` failure; it is not evidence that
the EC2 profile or IMDS path is broken.

### Which Upbound source can reach EC2 IMDS

In v1.21.1, both `IRSA` and `PodIdentity` call the same `UseDefault` helper,
which calls `config.LoadDefaultConfig`
([source switch](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/provider_config.go#L101-L111),
[helper](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/provider_config.go#L351-L368)).
The pinned AWS SDK default chain selects environment credentials, web
identity, shared/profile credentials, container credentials, and finally EC2
role credentials; its final branch constructs an IMDS-backed
`ec2rolecreds.Provider`
([chain selection](https://github.com/aws/aws-sdk-go-v2/blob/19221e80ee8be9dfaa215451515377c549bd9945/config/resolve_credentials.go#L96-L176),
[EC2 provider construction](https://github.com/aws/aws-sdk-go-v2/blob/19221e80ee8be9dfaa215451515377c549bd9945/config/resolve_credentials.go#L373-L399)).
AWS's credential-chain reference likewise identifies the container provider as
the EKS source and the IMDS provider as the EC2 instance-profile source
([AWS SDK credential-provider reference](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html)).

However, `IRSA` is not usable as a tokenless IMDS alias in this provider
version. After loading the default chain, Upbound's Terraform setup applies an
IRSA-only credential cache whose key hashes `AWS_WEB_IDENTITY_TOKEN_FILE`
([IRSA cache branch](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/creds_cache.go#L146-L192));
an empty filename is an explicit error
([hash helper](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/internal/clients/creds_cache.go#L256-L275)).

`PodIdentity` also loads the default chain, but the cache condition is strictly
`source == IRSA`, so `PodIdentity` bypasses that token-file requirement and
retrieves from the resolved SDK provider directly. The repository ships a
first-party ProviderConfig example containing only `source: PodIdentity`
([v1.21.1 example](https://github.com/crossplane-contrib/provider-upjet-aws/blob/4a5398fcc050393b01d6ebb60683e56b9383b179/examples/providerconfig/v1beta1/pod-identity-providerconfig.yaml)).

Therefore, with no higher-priority AWS credential source injected into the
provider container, `PodIdentity` reaches EC2 IMDS in the remote k3d topology.
The source name describes the intended EKS use case even though the
implementation uses the general default chain.

### How classic `InjectedIdentity` differs

The classic provider v0.47.2 explicitly allows `InjectedIdentity` in its enum
([classic API type](https://github.com/crossplane-contrib/provider-aws/blob/3739ae7881d19d832c98c98b6c07e10b53096cb0/apis/v1beta1/providerconfig_types.go#L51-L58)).
Its provider-config switch handles that value separately and calls
`UsePodServiceAccount` when no additional assume-role mode is configured
([classic switch](https://github.com/crossplane-contrib/provider-aws/blob/3739ae7881d19d832c98c98b6c07e10b53096cb0/pkg/utils/connect/aws/config.go#L104-L153)).
For AWS SDK v2 controllers, `UsePodServiceAccount` obtains a shared
`config.LoadDefaultConfig` result
([classic default-config path](https://github.com/crossplane-contrib/provider-aws/blob/3739ae7881d19d832c98c98b6c07e10b53096cb0/pkg/utils/connect/aws/config.go#L393-L423)).
Consequently, in a pod that can reach the node's IMDS and has no earlier
credential source, classic `InjectedIdentity` resolves the EC2 instance role.

There is one scope caveat: classic v0.47.2 also contains ACK-generated
controllers using AWS SDK v1, and its corresponding injected-identity helper
constructs web-identity credentials from the IRSA environment rather than the
general v2 default chain
([classic v1 helper](https://github.com/crossplane-contrib/provider-aws/blob/3739ae7881d19d832c98c98b6c07e10b53096cb0/pkg/utils/connect/aws/config.go#L665-L705)).
The IMDS conclusion is therefore exact for the v2 controller path demonstrated
live, not a blanket assertion about every classic resource implementation.

## Reconciliation with the cp9 live evidence

- The in-cluster ordinary AWS SDK caller resolved
  `arn:aws:sts::804837308162:assumed-role/cp9-cp9-frank-784-r1-remote-runtime/i-02d8b55c99cde1b3e`, proving the k3d pod-to-IMDS network path and EC2 instance
  profile are functional (`.cp9/cp9-frank-784-r1/tilt/reports/evidence/remote-aws-runtime-identity.json`).
- Upbound `None` failing with `static credentials are empty` is predicted by
  the static-provider path above.
- Upbound `IRSA` first reaches the default chain, but then fails with
  `token file name cannot be empty`, exactly as predicted by its IRSA-only
  token-file cache.
- Upbound `PodIdentity` reconciled a real
  `s3.aws.upbound.io/v1beta2 Bucket` to `Synced=True` and `Ready=True`, then
  deleted it successfully. During that proof the initiating human SSO profile
  was already expired, no human credential environment/file/Secret was present,
  and a direct host check confirmed the test bucket was absent after deletion.
- Classic `InjectedIdentity` succeeding through IMDS is predicted by its
  explicit default-chain branch.

This closes the provider-specific proof gap: the remote contract is Upbound
`PodIdentity`, classic `InjectedIdentity`, and IMDSv2 hop limit 3 for the nested
EC2 host → k3d node container → pod network.