# tools Non-runtime utilities kept out of the `cp9` binary. Each exists because the check or analysis it performs cannot be done from the Go test suite — most need a live AWS account, the render gate needs the manifests as Kustomize actually builds them, and a few assert properties of the repository's own shipped files, which AGENTS.md keeps out of runtime code. | Tool | Answers | |---|---| | `verify-runtime-account-renders.sh` | Do multi-account renders, dynamic Essential v2 pool-id ownership, and the DGS database-before-service bootstrap boundary still hold before manifests reach Kubernetes? | | `github-credential-residue/` | Does the production remote module downloader remove credential material after both success and failure? | | `cost-by-cluster.py` | Where does the account's spend actually go, per VPC / EKS cluster / team? | | `reclaim-candidates.py` | Which resource signals need an ownership and liveness review? | | `cloud-cost-report.py` | Does versioned replay or live provider data fail closed before an Agent writes a report? | ## Cost attribution `cost-by-cluster.py` answers "how much does each control plane cost" in an account where that question has no built-in answer. The obvious mechanisms are unavailable in a member account: activating cost allocation tags requires the payer account, and resource-level billing data is off, so `ce get-cost-and-usage-with-resources` returns nothing but `NoResourceId`. Tagging every resource by hand is possible but slow, and tag activation is not retroactive — it cannot answer anything about last month. So the tool joins the two halves itself. Read-only APIs give a topology — every billable resource reports the VPC it lives in, and an EKS cluster reports its VPC — which yields `resource → VPC → cluster → team` with no tags involved. Cost Explorer, grouped by usage type, gives the money. Because hourly and per-quantity charges have a uniform unit rate, distributing them across that inventory is arithmetic rather than estimation: NAT-gateway hours divided by NAT count is the exact per-gateway cost, EBS scales with each volume's size, and EKS control-plane hours are a flat charge per cluster. ```bash tools/cost-by-cluster.py --profile --start 2026-07-01 --end 2026-07-25 tools/cost-by-cluster.py --profile --months 3 --json audit.json ``` `--months N` audits the last N-1 whole months plus month-to-date. `--metric` accepts `NetAmortizedCost` (default), `AmortizedCost`, `NetUnblendedCost` or `UnblendedCost`. The AWS profile needs `ce:GetCostAndUsage` plus read access to EC2, EKS, ELB, RDS and MSK. ### Why the default is amortized, not unblended A member account that does not buy its own Savings Plans cannot see them in unblended cost. Covered usage appears as `SavingsPlanCoveredUsage` and is cancelled by an equal, negative `SavingsPlanNegation`, so it nets to exactly $0 in this account — the spend is real, it is just billed to the payer account that bought the plan. Auditing on unblended therefore under-reports the account, and does so unevenly: the whole gap lands on EC2 compute, which is also the line that drives per-cluster attribution. In 0414 this hid 20–35% of the account month over month, including instance types that billed $0 unblended against four-figure amortized cost — entire fleets invisible to the audit. The report prints a `savings-plan share` line for any amortized metric, so the gap stays visible rather than being folded silently into the total. It is measured against the matching unblended metric — net against net, gross against gross — so the figure is the plan effect alone and does not quietly absorb private-rate discounts or credits. Keep the distinction in mind when acting on the output: shutting down an SP-covered instance saves the company money, but does not reduce this account's own unblended bill. Use amortized to govern company-wide consumption, unblended to reconcile this account's invoice. ### Reading the output honestly Two properties of the output matter more than the totals. **A shared bucket stays visible.** Bytes-based charges (NAT and cross-AZ data transfer) depend on traffic rather than resource count, and S3, ECR and KMS are not VPC-scoped. The tool refuses to spread these across clusters, because a table that silently apportions shared cost looks precise while misleading whoever reads it. They are reported as `SHARED / NOT ATTRIBUTABLE` instead. Cracking the data-transfer line requires VPC Flow Logs. **Historical periods carry a drift warning.** The inventory is a snapshot of now. Resources created after the period are excluded by creation time, but resources that existed then and have since been deleted cannot be recovered — their cost is spread over whatever survived, which inflates the survivors. The tool quantifies this by comparing billed NAT-gateway hours against the maximum today's NAT inventory could have produced and prints the ratio. A period near 1.0x is trustworthy; a period at 2-3x means most of that month's resources are gone and the per-team numbers are directional only — do not reconcile a team against them. Savings Plans and reserved instances are the trap the default metric now avoids. If you pass an unblended metric explicitly, covered EC2 usage costs almost nothing, so the compute column understates who is actually occupying machines — read instance counts alongside it. ## Reclamation candidates `reclaim-candidates.py` lists resources with signals worth investigating. It is read-only and never deletes; its output is candidate evidence for a human to confirm, not deletion proof. ```bash tools/reclaim-candidates.py --profile tools/reclaim-candidates.py --profile --vpc vpc-aaaa --vpc vpc-bbbb ``` Without `--vpc` it scans the whole account for unattached EBS volumes. With `--vpc` it also audits load balancers and public IPv4 addresses in those VPCs, which is the useful mode when cost attribution points at a specific control plane. Three things it is built to surface: **Unattached EBS volumes** bill in full while attached to nothing. Creation dates matter as much as the total — a cluster of very recent volumes means something is leaking right now, and a regular cadence of identically-named volumes means a scheduled job leaks one per run. Fix the leak before cleaning up, or the cleanup regrows. **Load balancers with no healthy targets** are billed per hour regardless, but that state does not prove abandonment. Kubernetes-owned load balancers and broken services can both report zero healthy targets while still carrying a live contract. Confirm controller ownership, registered targets, DNS and traffic evidence, and the owning service before proposing any reclamation. **Public IPv4 addresses**, reported with what holds them. In a Kubernetes-heavy account these are overwhelmingly held by load balancers — roughly three per internet-facing one, one per availability zone — so public IP spend is largely a function of load balancer count rather than an independent problem. Retire an idle internet-facing load balancer and its addresses go with it. Rates used for the dollar estimates are list prices and are meant for ranking candidates, not for reconciling a bill. ## Multi-cloud report replay `cloud-cost-report.py` is the retained non-runtime contract check and renderer for the multi-cloud cost report. It makes no cloud API calls. It validates the versioned allowlist in `.github/cloud-cost-scopes.json`, normalizes adjacent seven-day billing windows, and renders synthetic or collected evidence. Python owns every published fact, boundary and action; the Agent selects wording under the contract below. ```bash python3 tools/cloud-cost-report.py validate-scopes \ --scopes .github/cloud-cost-scopes.json python3 tools/cloud-cost-report.py replay \ --scopes .github/cloud-cost-scopes.json \ --fixture tools/cloud-cost-report-fixture.json \ --scenario complete \ --output /tmp/cloud-cost-analysis.json python3 tools/cloud-cost-report.py check \ --scopes .github/cloud-cost-scopes.json \ --fixture tools/cloud-cost-report-fixture.json python3 -m venv /tmp/cloud-cost-aws-venv /tmp/cloud-cost-aws-venv/bin/pip install -r tools/cloud-cost-aws-requirements.txt /tmp/cloud-cost-aws-venv/bin/python tools/cloud-cost-aws.py check ``` Both checks run offline without credentials. The collector check protects storage-class summation, VPC-to-EKS association, all-status paginated image reads through the real SDK paginator, tag normalization, and failed reads versus observed empty inventory. Stubbed second-page failures, malformed size evidence and transport timeouts must discard partial repository totals and remain unknown. The report check protects billing service and usage separation, cent closure, eight-week billing history, direct Savings Plans coverage, compatible inventory differences, new candidates outside the billing window, same-account candidate history, unknown image evidence versus retention ranking, full KMS/CA and inventory-difference evidence, grounded Agent selections, and byte-identical report validation. These checks prevent silent cost misclassification and unsupported cleanup claims. The fixture contains no observed account spend. Its `incomplete` scenario retains coverage gaps but suppresses unsupported totals, deltas, resource changes and reclaim conclusions. The workflow uploads the incomplete report, then fails before any Lark setup or write. Synthetic runs never read cloud accounts or publish to Lark. The manifest determines which scopes are enabled. Onboarding requires a reviewed manifest change, the provider-side OIDC inventory/billing reader, and a successful preflight. Runtime text inputs and organization-wide discovery do not onboard accounts. `cloud-cost-aws.py` collects AWS evidence through the AWS CLI and SDK using the same OIDC environment credentials after preflight. It compares the latest complete UTC weeks after the billing delay, follows Cost Explorer pagination, requires contiguous daily billing, and inventories enabled regions. Region + Usage Type supplies regional charges; Service + Usage Type identifies their billing service. Shared usage names such as `TimedStorage-ByteHrs` never establish a service by themselves. Ambiguous service routing stays outside resource attribution; the S3 total uses the S3 service. Deep reads retain selected VPC/resource tags, sampled KMS description categories, CA role identities, and named shared services. Sampled KMS keys already awaiting deletion have a separate window/dependency review gate from enabled keys. S3 reads add recent CloudWatch bytes, version-empty observations and enabled lifecycle action types. Version-empty excludes unfinished multipart uploads and does not authorize deletion. ECR image reads include all statuses and pagination; image-size sums can count shared layers repeatedly and are not deduplicated billable bytes. Image reads reuse one Boto3 client per region across the existing eight workers. This avoids thousands of CLI process startups without sampling repositories or dropping pages. The pinned SDK dependency is in `cloud-cost-aws-requirements.txt`; the workflow installs it in an isolated virtual environment for collection and the offline check. SDK retries and socket waits are bounded. Stage and repository-count progress goes to stderr so collection stalls can be located without logging resource names, image contents or credentials. Lifecycle policies are read for the largest observed repositories in each region. ECR image coverage is tracked separately from repository-list coverage; retention actions rank only repositories with measured images, and missing reads produce an evidence-restoration action. Missing metadata and read failures remain unknown, not zero storage or an empty service. Required billing/ownership failures stop publication; optional deep-read gaps remain visible. `normalize-aws` validates the snapshot, allocates supported charges through current inventory, and keeps unattributable spend separate. Unattached volumes, unassociated addresses and load balancers without healthy targets remain candidates. Resources created after the billing window stay in the queue with an unpriced estimate; ownership tags and source names are routing hints. The report never confirms a deletion or guarantees savings from these signals. Inventory differences use only surfaces covered by both current and previous same-branch snapshots. Without a baseline, changes are unevaluated. `candidate-history.json` carries same-account daily candidate observations across successful runs; the normalizer retains the bounded history window and rejects incompatible identities or dates. Observations show presence on recorded dates, not continuous use or a creating defect. Old snapshots without the new metadata remain valid and explicitly unmeasured. The Agent receives only Python's decision units, with their conclusion, evidence, boundary, decision and confirmation gate. It returns exactly keys `1` through `10`, each set to `action` or `context`. `action` selects the supplied decision sentence; `context` selects that sentence plus its supplied evidence boundary. Free interpretation paragraphs are rejected. The Agent does not write or paraphrase the report. Python materializes all prose, tables and charts from the canonical analysis and validates the selections before publication. The live AWS report follows the seven-section contract derived from the reviewed `0414 Account Cost Breakdown` revision 50. It includes VPC ownership and service components, service/usage breakdowns, bucket families and retention, KMS and CA evidence, ECR image metadata, candidates, resource observations and actions. Compact tables link the run's `report-evidence.md` artifact for the complete queue, sampled KMS IDs, CA identities and comparable inventory differences. `validate-report` requires a byte-identical Python re-render. Only the threat-detection-gated safe job publishes a complete live report through the dedicated Lark bot, then verifies the exact report child and records its hash/readback receipt. The workflow source is `.github/workflows/cloud-cost-report.md`; regenerate its lock after changes. ## Render baseline `render-baseline.sh` is the regression gate for changes that move behaviour between control formats. A refactor that claims to preserve behaviour must produce a byte-identical render, and this proves it across every overlay in the repo in about 15 seconds. The overlay set is discovered at run time; `list` prints it. ```bash tools/render-baseline.sh check # render everything, compare against the baseline tools/render-baseline.sh update # re-record after an intended change tools/render-baseline.sh list # list the overlays it covers ``` `check` exits non-zero when any overlay renders differently, naming each one, printing the digest transition, and giving the `kubectl kustomize` command to inspect it. That speed matters more than it sounds: it is the difference between an agent that can tell immediately whether it broke something and one that cannot. Baselines are digests rather than full manifests. The overlays render to well over a megabyte of YAML, so storing it would bury every intended change in noise; the digest answers the only question the gate asks. When an overlay does change, inspect it with `kubectl kustomize `. The baseline tracks the manifests as they are on `main`, so it needs re-recording whenever a change to `personas/` lands — including one you merely rebased onto. A stale baseline reports a diff that is not yours. Check `git diff --stat -- personas/` before assuming the gate is complaining about your change. A few kustomizations do not render standalone **by design** and are recorded as `NOT-STANDALONE` with the reason: one is a `kind: Component`, which exists to be included by another kustomization, and the rest are content-less anchors for parent overlays. Classification happens *after* a failed render, not by inspecting keys beforehand, so an overlay that genuinely breaks is reported as broken rather than excused as intentional. A genuine render failure is recorded as `FAILED-TO-RENDER` instead, so an overlay that stops rendering shows up as a baseline change rather than disappearing from the report.