AWS bills are hard to reason about for a structural reason: anyone with the right IAM permissions can spend money in seconds, and the bill shows up much later, aggregated by service rather than by the decision that caused it. Cost control in AWS is mostly about closing those two gaps: seeing who spent what for which purpose, and limiting what can be spent before anyone looks.
Seeing where the money goes
Cost Explorer is the starting point. It groups and filters cost by service, account, Region, usage type and tag, and its data is refreshed at least once a day. That delay matters: a misconfigured job that runs for six hours may not be visible until tomorrow. Plan your controls around the fact that you’ll find out late.
To answer “which team, product or experiment spent this?”, you need cost allocation tags. Tag resources with keys like team, service, env and experiment, then activate those keys in the Billing console. Two details trip people up:
- Activation isn’t retroactive. Costs before you activated a tag (and costs for resources that weren’t tagged at the time) show up as “No tag value”.
- Not every cost can be tagged. Data transfer and some shared charges land without a tag, so “untagged” never reaches zero. Aim to explain the large untagged lines, not to eliminate them.
Make tags reliable by applying them at creation (infrastructure as code with default tags, or aws:RequestTag conditions in IAM policies that refuse to create untagged resources) and use tag policies in AWS Organizations to standardise key names and allowed values. Team, team and owner-team as three separate keys make reports useless.
For deeper analysis, export the detailed billing data (Data Exports, formerly the Cost and Usage Report) to S3 and query it with Athena. That’s where you can see per-resource, per-hour line items.
Where unexpected costs usually hide
The surprises on AWS bills tend to come from the same places, and naming them is a good interview signal:
- Things left running. Development instances, GPU boxes from an experiment, an RDS instance created “just to test”.
- Things left behind. EBS volumes and snapshots after instances are terminated, unattached Elastic IPs, old AMIs, load balancers with no targets.
- Data movement. NAT gateway processing charges, cross-AZ traffic between services, internet egress.
- Storage that only grows. Versioned S3 buckets without lifecycle rules, CloudWatch log groups with no retention.
- Public IPv4 addresses, which AWS charges for hourly, whether in use or not.
A quick Python calculation with illustrative prices (check the pricing pages for real ones) shows how the first item dominates:
planned: 2 instances x 32h x $1.00 = $64.00
left running for a month: 2 x 720h x $1.00 = $1,440.00
one idle public IPv4 address for a month: $3.60
500 GB gp3 volume left behind: $40.00/monthThe planned experiment costs $64. The same experiment, forgotten, costs over twenty times that. Most cost control for experiments is about making “forgotten” impossible or short-lived.
Budgets alert; they don’t stop anything
AWS Budgets lets you set a monthly (or other period) amount for an account, a tag or a service, and notifies you by email or SNS when actual or forecasted spend crosses a percentage. Forecast alerts are the useful ones: “forecast will exceed 100%” arrives days before the actual does.
What a budget doesn’t do, on its own, is stop spending. Hitting 100% sends a notification and nothing else. People are often surprised by this after a large bill arrives with an alert email that nobody read on a Saturday.
To make a budget act, configure budget actions. When a threshold is crossed, a budget action can attach an IAM policy (for example, one that denies ec2:RunInstances), apply a service control policy to an account, or stop specific EC2 or RDS instances. Actions can run automatically or wait for approval. They’re still bounded by billing delay: the action fires when the budget notices, which may be hours after the spend began.
Cost Anomaly Detection complements budgets. It learns normal spend per service or tag and alerts when something deviates, which catches “this account’s NAT gateway cost tripled” without you setting a threshold for every service.
Fencing an experiment before it starts
Because detection is slow, the strongest controls limit what can be spent. A pattern that works well for experiments, proofs of concept and sandboxes:
- Use a separate account. Create it in AWS Organizations under a sandbox OU. Its costs are isolated on the bill, it can’t touch production resources, and when the experiment ends you can close the account and everything in it goes away.
- Fence it with an SCP. Deny Regions you don’t use and instance types you don’t need. An SCP applies to every principal in the account, including admins.
- Give it a budget with an action. A forecast alert at 80%, and an action at 100% that applies a deny policy for new compute.
- Give it an expiry. Tag resources with an
expires-ondate and run a scheduled cleanup (a Lambda function or a tool like aws-nuke for whole sandbox accounts) that stops or deletes what’s past due.
The SCP for step 2 might look like this. It’s illustrative: real ones usually exempt global services more carefully, and it wasn’t applied to any account.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OnlyTheseRegions",
"Effect": "Deny",
"NotAction": ["iam:*", "organizations:*", "sts:*", "budgets:*", "ce:*", "support:*"],
"Resource": "*",
"Condition": { "StringNotEquals": { "aws:RequestedRegion": ["eu-west-1"] } }
},
{
"Sid": "SmallInstancesOnly",
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": { "StringNotLike": { "ec2:InstanceType": ["t3.*", "t4g.*", "g5.xlarge"] } }
}
]
}The first statement uses NotAction so that global services, which are served from us-east-1, keep working. Forgetting that exemption is the classic way to lock yourself out of IAM in a sandbox. The second allows one GPU type for the experiment and nothing bigger, so a typo of p4d.24xlarge fails immediately instead of costing hundreds of dollars a day.
Making cost a design input
Visibility and fences handle accidents. The bigger savings come from treating cost as a property of the design, alongside latency and availability:
- Measure unit cost: cost per order, per active user, per GB processed. A total bill that rises with traffic is fine; a cost per order that rises is a regression.
- Right-size with data. Compute Optimizer and CloudWatch utilisation show instances that sit at 5% CPU.
- Commit for the steady baseline only. Savings Plans or Reserved Instances for the capacity you always run; on-demand or Spot for the rest. Committing to peak capacity just moves waste into a contract.
- Use the cheap paths AWS offers: S3 gateway endpoints instead of NAT for S3 traffic, Graviton instance types, lifecycle rules to cheaper storage classes, log retention periods.
In design discussions, the strongest answers name the cost driver of each option (“this adds a NAT gateway per AZ and pays per GB for S3 traffic unless we add an endpoint”) rather than saying “it’s cheaper” in general.
Interview practice: a safe budget for a data science team
A data science team wants to try GPU training on AWS for a month. Leadership is nervous after a previous team left GPU instances running over a holiday and spent $18,000. Design the guardrails.
Answer
I’d give them their own account under a sandbox OU, so their costs are isolated and the account can be closed or cleaned when the trial ends. An SCP on that OU restricts them to one Region and to the GPU instance types the trial needs, which caps the hourly burn rate per instance and stops accidental launches of much larger types. I’d also check the account’s EC2 service quotas for those instance families, since a low vCPU quota is another hard ceiling on concurrent GPUs.
Then a monthly budget for the agreed amount, with forecast alerts at 50% and 80% to the team and their manager, and a budget action at 100% that applies a policy denying new instance launches. Because billing lags, I’d add a behavioural control that doesn’t depend on it: every instance must carry an expires-on tag (enforced by an IAM condition on RunInstances), and a scheduled job stops instances past expiry and every instance at night and on weekends unless tagged to run. Training jobs should use managed options like SageMaker training jobs or Spot with checkpoints, which end on their own when the job finishes.
Finally, I’d make the trial’s success measurable in cost terms (cost per training run) so the decision at the end of the month is based on numbers, not on the bill’s size.