The AWS Well-Architected Framework names six pillars: operational excellence, security, reliability, performance efficiency, cost optimisation and sustainability. Interviewers rarely want them recited. What they want is evidence that you can trade them against each other for a specific system, and that you know which failures your design assumes away. “It’s highly available” isn’t an answer. “It survives the loss of one Availability Zone with no data loss and degrades to read-only if the payment provider is down” is.
Name the failures you’re designing for
Every architecture makes failure assumptions, whether or not anyone writes them down. Making them explicit is the most useful single step in a design review. A typical list, from most to least likely:
- A single instance or container dies. Constant background noise. Any production design should absorb it with no human involvement: multiple instances behind a load balancer, health checks, Auto Scaling replacing failures.
- A dependency is slow or failing. A third-party API, an internal service, a database under load. Often the most common cause of real outages, because the failure spreads through retries and exhausted connection pools.
- A bad deployment or configuration change. Historically one of the largest sources of incidents anywhere. The defence is process: canaries, gradual rollouts, automated rollback.
- An Availability Zone is impaired. Rare but expected over a system’s lifetime. Multi-AZ deployments (app tier in at least two or three AZs, RDS Multi-AZ, NAT gateway per AZ) handle it.
- A Region is impaired. Rarer, and expensive to design for. Many systems reasonably accept this risk; regulated or global ones may not.
- Data corruption or deletion, by a bug, an operator or an attacker. Redundancy doesn’t help here, since replicas faithfully copy the corruption. Only backups and point-in-time recovery do.
For each, state the expected behaviour: absorbed silently, degraded (read-only, delayed processing), or down until restored, with a target time. Then the trade-offs become concrete: surviving a Region failure with no downtime might triple cost and complexity; surviving an AZ failure costs a little more and is usually worth it.
Availability multiplies along the path
A request that has to pass through several components succeeds only if all of them work. If the components fail independently, their availabilities multiply. Redundant copies, on the other hand, fail only if all copies fail at once. Here’s that arithmetic in Python for an ordering API:
# Availability of a request path: serial dependencies multiply; redundant copies fail only together.
MIN_PER_YEAR = 365 * 24 * 60
def serial(*parts):
total = 1.0
for p in parts:
total *= p
return total
def redundant(a, copies):
return 1 - (1 - a) ** copies # assumes copies fail independently
def show(label, a):
print(f"{label:<44} {a*100:8.4f}% ~{(1-a)*MIN_PER_YEAR:6.0f} min/year down")
lb, app, db, payments = 0.9999, 0.999, 0.9995, 0.999
show("LB -> one app AZ -> DB -> payment API", serial(lb, app, db, payments))
show("LB -> app in 3 AZs -> DB -> payment API", serial(lb, redundant(app, 3), db, payments))
show("... and payment API made async (off path)", serial(lb, redundant(app, 3), db))Output from python3 avail.py:
LB -> one app AZ -> DB -> payment API 99.7402% ~ 1365 min/year down
LB -> app in 3 AZs -> DB -> payment API 99.8401% ~ 841 min/year down
... and payment API made async (off path) 99.9400% ~ 315 min/year downThe numbers are illustrative, but the lessons hold:
- Four components that each look good (three nines or better) combine to under 99.75%, about 23 hours of downtime a year.
- Spreading the app tier across three AZs helps, but less than you’d hope, because the database and the payment API are still on the path.
- Taking the payment API off the synchronous path (accept the order, queue the charge, confirm by email) helps more than any redundancy change. The fastest way to improve availability is often to remove a dependency from the request rather than to make it more reliable.
Also be honest about the independence assumption. Three app copies in three AZs aren’t independent if they share a bad deployment, a configuration file or a database connection limit. Correlated failures are why redundancy so often disappoints in practice.
RTO, RPO and the cost of each DR tier
For the rarer failures, especially Region loss and data corruption, two numbers drive the design:
- RTO (recovery time objective): how long the service may be unavailable.
- RPO (recovery point objective): how much recent data you may lose, measured in time.
AWS describes four disaster recovery strategies, in increasing cost and decreasing RTO and RPO:
| Strategy | What runs in the recovery Region | Typical RTO / RPO | Main cost |
|---|---|---|---|
| Backup and restore | Nothing; backups are copied there | Hours / hours | Storage, plus a long, risky restore |
| Pilot light | Data replicated (e.g. database replica); compute off or minimal | Tens of minutes / minutes | Replication, and scaling up under pressure |
| Warm standby | A scaled-down, fully working copy | Minutes / seconds to minutes | Running a second environment all the time |
| Multi-site active-active | Full capacity serving traffic in both | Near zero / near zero | Double infrastructure, plus conflict handling for writes |
The interview-worthy part is what each tier makes you solve. Backup and restore needs regular restore tests, or the RTO is a guess. Pilot light and warm standby need runbooks and automation for failover, and someone who decides when to fail over. Active-active needs the application to handle writes in two Regions: either partition data so each record has one home Region, or accept conflict resolution (DynamoDB global tables, for instance, use last-writer-wins). Picking active-active for a system whose data model assumes a single writer is a common, expensive mistake.
The usual right answer is mixed: multi-AZ for everything, backups with point-in-time recovery for all data, and a DR tier chosen per service from its business impact.
Control planes, data planes and static stability
AWS services have a data plane (the part that does the work: serving S3 objects, routing packets, running instances) and a control plane (the APIs that create and change resources: launching instances, changing DNS records, modifying databases). Data planes are designed to be simpler and more available than control planes.
That leads to a principle AWS calls static stability: a system should keep working during an impairment without needing to make control plane changes. Compare two designs that both survive an AZ failure:
- Run at 50% capacity in each of two AZs and rely on Auto Scaling to launch replacements in the surviving AZ when one fails. Recovery depends on
RunInstancessucceeding quickly, at the same moment every other customer in that Region is doing the same thing. - Run enough capacity in each of three AZs that losing any one leaves enough to serve peak load (each AZ at about 67% of what it could carry). Nothing needs to be launched; the load balancer just stops sending traffic to the impaired AZ.
The second costs more and fails better. The same thinking applies to DR: a failover plan that begins with “create the database in the other Region” depends on control planes and on capacity being available at the worst time. Pre-provisioned standby resources and failover via health-checked DNS records (Route 53 health checks are data plane) are more dependable.
Trading pillars against each other
Most real design decisions trade one pillar for another. Three come up repeatedly:
- Reliability against cost. Multi-AZ NAT gateways, RDS Multi-AZ and spare capacity all cost money every hour to protect against events that happen rarely. Make the decision per service from the cost of downtime.
- Security against operational simplicity. Private subnets, VPC endpoints, per-service IAM roles and KMS keys add moving parts. Usually worth it for production data; often overkill for an internal prototype.
- Performance against consistency. Caching (CloudFront, ElastiCache), read replicas and eventually consistent reads make things faster and make some reads stale. Decide which screens can tolerate it.
Interview practice: availability targets for a checkout service
An e-commerce company wants checkout to be “99.99% available”. The current design runs on EC2 in one AZ with a single RDS instance and calls a fraud API and a payment API synchronously. What would you ask, and what would you change?
Answer
I’d start by clarifying the target. 99.99% is about 53 minutes of downtime a year. Over what window, measured how (successful checkouts, or the endpoint responding), and does it include planned maintenance? I’d also ask what failures it must survive: AZ loss certainly, Region loss probably not at first, and what the business does if the payment provider is down.
Then the arithmetic: the request path has at least four serial dependencies, including two external APIs that likely don’t promise 99.99% themselves. No amount of redundancy on our side gets the product of those above the weakest external link. So the design has to change shape, not just get more copies.
Changes, in order of value: move the app tier to three AZs behind an ALB with capacity to lose one AZ without scaling, and make RDS Multi-AZ (with RDS Proxy or a pooler to shorten failover impact). Take the payment and fraud calls off the synchronous path where the business allows it: accept the order, return “order received”, and process payment through a queue with retries and idempotency keys, emailing the customer if it fails. If fraud checks must be synchronous, set a tight timeout and a fallback policy (accept low-value orders, hold high-value ones for review). Add canary deployments and automatic rollback, since bad deploys are the likeliest cause of missing the target. Finally, define how availability will be measured, alarm on it, and test the AZ failure assumption with a game day, for example with AWS Fault Injection Service.