Cloud Architecture Patterns: Availability Zones, Multi-Region, Landing Zones, and Cost Trade-offs

Cloud resilience is a set of priced decisions. Each extra nine of availability costs money, adds complexity, and usually forces a consistency compromise. This guide works from failure domains and recovery targets to the concrete patterns that meet them: stateless tiers across zones, data replication choices, active-passive and active-active regions, landing zones that limit blast radius, hub-and-spoke networks, event-driven decoupling, identity, cost, and the game days that prove any of it works. AWS, Azure, and Google Cloud terms appear where they help.

Failure domains

A failure domain is a set of resources that can fail together. Redundancy only helps if the copies sit in different domains, so the first design step is to list the domains and decide which ones the system must survive.

The change row is often the most important: a bad config shipped to every region at once defeats all geographic redundancy. Multi-region only helps if changes roll out one region at a time.

Provider models also differ in ways that affect design. An AWS subnet belongs to one Availability Zone, while Azure and GCP subnets span zones within a region. An AWS or Azure virtual network is regional, whereas a GCP VPC is global with regional subnets. AWS zone names can map to different physical zones per account, so correlate failures using zone IDs.

Control plane and data plane

The data plane serves running traffic. The control plane creates, changes, and scales resources, and during a large event it is often what degrades, just as every customer tries to launch replacement capacity. A statically stable design keeps working without any control plane call: capacity already runs in the surviving zones. "Autoscaling will replace the lost zone" is a bet on the control plane at the worst possible time.

RTO, RPO, and availability arithmetic

Recovery Time Objective (RTO) is how long a service may be down. Recovery Point Objective (RPO) is how much recent data may be lost. Both must come from the business, per workflow rather than per system.

Components in series multiply. A request that needs a load balancer at 99.99%, an application tier at 99.95%, and a database at 99.9% gets:

A = 0.9999 * 0.9995 * 0.999 = 0.99840   (99.84%, about 14 hours per year)

The system is less available than its weakest part, and every synchronous dependency makes it worse. Components in parallel, assuming independence, work the other way:

A = 1 - (1 - a)^n
Two independent zones at 99.5%: 1 - 0.005^2 = 0.999975   (99.9975%)

That result is optimistic, because zones share software, deploy pipelines, and regional services. A simple common-cause adjustment assumes that some fraction of unavailability hits all copies at once. If 10% of each zone's unavailability is shared:

U = 0.10 * 0.005 + (0.90 * 0.005)^2
  = 0.0005 + 0.00002025
  = 0.00052025
A = 99.948%

The shared failure term is about 25 times larger than the independent term. Adding a zone barely helps; removing shared dependencies does. These percentages are illustrative, so use your own incident data.

Failover time also caps availability. If you expect four failovers a year, each taking 15 minutes, that is 60 minutes of downtime out of 525,600, or a ceiling of about 99.989% before anything else goes wrong.

Capacity arithmetic for zone loss

To survive losing one of n equal zones, each zone needs to carry its share of peak load after the loss. Total provisioned capacity is therefore n / (n - 1) of peak.

Three zones is the common default: it needs half the spare capacity of two, and quorum systems keep a majority after losing one zone. The same formula applies to regions.

Stateless tiers

A tier is stateless when any instance can serve any request and can be destroyed at any time. Sessions and uploads move to managed stores, and instances are replaced from immutable images rather than patched.

Spread the replicas explicitly, because schedulers do not guarantee it. In Kubernetes:

# Deployment pod template spec
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: myapp

DoNotSchedule preserves balance but leaves pods pending during a zone outage if the surviving zones cannot meet the skew rule. ScheduleAnyway favours running over balance. Add a PodDisruptionBudget too.

Common failure modes:

• Deep health checks that cascade. If a readiness probe tests the database and the database blips, every instance leaves the pool and a partial failure becomes a total one. Keep load balancer checks shallow, and report dependency status separately.
• Retry storms. Retries at each layer multiply. Three layers with three attempts can turn one request into 27. Retry at one layer, with exponential backoff, jitter, and a retry budget.
• Gray failure. A zone that is slow rather than down still passes health checks. Compare latency and errors per zone, and be able to shift traffic away manually, for example with AWS zonal shift.

Data tier options

Light in fibre travels at roughly 200 km per millisecond, so regions 6,000 km apart have a round-trip time of at least 60 ms before any processing. Synchronous replication across that distance adds that time to every commit. No product removes this limit.

Replication is not backup. It faithfully copies a DROP TABLE, a bad migration, or a ransomware encryption. Keep immutable, versioned backups in a separate account with separate credentials, whatever replication you run.

Async RPO can be estimated. If a database commits 200 writes per second and replication lags 5 seconds when the region is lost, around 1,000 committed writes are missing. If the old primary returns, those writes must be reconciled against a new primary that has moved on.

A consensus cluster needs a majority. A cluster split evenly over two sites loses quorum when either side is lost. Practical designs use three regions, or two main regions plus a lightweight witness in a third.

Active-passive and active-active

Most systems should stay single-region with three zones, plus tested cross-region backups. A second region raises cost and operational burden, and failover mistakes cause outages of their own, so require an explicit business case. The standard patterns:

Active-passive

One region takes all writes, so consistency stays simple. The hard parts are the failover sequence and keeping the passive region honest, because it drifts if nothing exercises it. Pilot light also needs capacity during a regional event, when everyone else in that geography wants it too.

DNS is the classic failover mechanism. A low TTL shortens the wait, but some clients cache beyond it, and existing connections stay open. Anycast entry points, such as AWS Global Accelerator, GCP global load balancing, or Azure Front Door, shift traffic without relying on client DNS behaviour.

resource "aws_route53_health_check" "primary" {
  fqdn              = "primary.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/healthz"
  request_interval  = 30
  failure_threshold = 3
}

resource "aws_route53_record" "primary" {
  zone_id         = var.zone_id
  name            = "app.example.com"
  type            = "CNAME"
  ttl             = 60
  records         = ["primary.example.com"]
  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id

  failover_routing_policy {
    type = "PRIMARY"
  }
}

# A matching SECONDARY record points at secondary.example.com

Build an RTO budget from the actual steps rather than guessing. With the settings above:

Automatic data failover needs care. A false positive during a network partition produces two primaries, each accepting writes. Fencing prevents this by making sure the old primary cannot accept writes, for instance by revoking its credentials or using a lease it must renew. Many teams automate traffic shifting but keep database promotion as a human decision, accepting a longer RTO to avoid split-brain.

Failback is often harder than failover: the old primary has diverged and must be rebuilt from the new one. Rehearse this path too.

Active-active and consistency

The CAP theorem says that during a partition a system chooses consistency or availability. PACELC adds the everyday case: without a partition, it still chooses between latency and consistency.

• Region-partitioned (home region). Each user or tenant is assigned one home region that owns its writes. Normal operation has no write conflicts, and losing a region affects only its tenants until they move. This is usually the best active-active pattern; with small independent stacks it is called cell-based architecture.
• Last-writer-wins multi-master. Simple, but it silently discards one of two concurrent writes. It suits data where a lost update is harmless, such as a last-seen timestamp.
• Conflict-free merges. CRDTs converge without coordination. They suit counters and carts, not balances with invariants.
• Consensus writes. Strongly consistent, with quorum latency on every write.

Invariants decide the choice. "Stock must not go below zero" cannot be enforced by two regions accepting writes independently; such operations need a single owner or consensus. Practical designs mix models: a strongly consistent core for orders and money, eventual consistency elsewhere. In two-region active-active, each region must be able to carry 100% of peak.

Landing zones and blast radius

A landing zone is the pre-built organisational foundation that workloads are deployed into: account structure, guardrails, logging, identity, and network. The account boundary is the strongest isolation unit: it separates permissions, quotas, and most billing, so a compromised credential in one account cannot easily reach another.

A typical structure:

Guardrails should stop things that must never happen, and little else. This AWS SCP restricts regions while allowing global services, and protects audit logging:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "cloudfront:*",
        "iam:*",
        "organizations:*",
        "route53:*",
        "sts:*",
        "support:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["eu-west-1", "eu-central-1"]
        }
      }
    },
    {
      "Sid": "ProtectAuditTrail",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:DeleteTrail",
        "cloudtrail:StopLogging",
        "organizations:LeaveOrganization"
      ],
      "Resource": "*"
    }
  ]
}

SCPs do not apply to the management account, one more reason it should hold no workloads. Review the NotAction list for each global service you use, and test in a sandbox OU before attaching widely.

Many small accounts mean a small blast radius, but more cross-account networking, permissions, and quotas to monitor. Automate account creation through a vending pipeline that applies baseline logging, budgets, and roles.

Network topology: hub and spoke

Full-mesh peering grows as n(n-1)/2 connections: 20 networks need 190 peerings. Hub and spoke replaces this with a central router: each spoke attaches once, and egress, inspection, and hybrid connectivity live in the hub. The AWS construct is Transit Gateway; Azure offers VNet peering to a hub or Virtual WAN; GCP offers Network Connectivity Center and Shared VPC.

Plan address space before the first VPC, because overlapping CIDRs block later connectivity. Enforce segmentation with route tables, not only security groups:

resource "aws_ec2_transit_gateway" "hub" {
  description                     = "myproject regional hub"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
}

resource "aws_ec2_transit_gateway_route_table" "spokes" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
}

resource "aws_ec2_transit_gateway_vpc_attachment" "spoke" {
  for_each                                        = var.spokes
  transit_gateway_id                              = aws_ec2_transit_gateway.hub.id
  vpc_id                                          = each.value.vpc_id
  subnet_ids                                      = each.value.attachment_subnet_ids
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

# Associate each attachment with the spokes route table, then:
resource "aws_ec2_transit_gateway_route" "default_to_inspection" {
  destination_cidr_block         = "0.0.0.0/0"
  transit_gateway_attachment_id  = var.inspection_attachment_id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spokes.id
}

Disabling default association and propagation means nothing can talk until a route is added. Spokes associated with one table have no routes to each other, so all traffic goes through inspection. Attach subnets in every zone used, and enable appliance mode on the inspection attachment so that flows stay symmetric through stateful firewalls.

The hub is itself a shared failure domain. A bad route or firewall rule there affects every spoke, so its changes need the strictest process you have. Build one hub per region, with inspection redundant across zones. Hub traffic carries per-gigabyte processing charges, so prefer private endpoints for object storage and other provider services over a central NAT.

Event-driven decoupling

Moving a step behind a queue changes the availability arithmetic: the producer needs only the queue, and the consumer can be down for a while without user impact. Queues (SQS, Service Bus, Pub/Sub), event buses (EventBridge, Event Grid), and partitioned logs (Kafka, Kinesis, Event Hubs) all trade latency for survival.

Delivery is at-least-once in most of these, so consumers must be idempotent. Writing to a database and publishing an event also cannot be atomic. The transactional outbox solves this by storing the event in the same transaction as the data, with a relay that publishes it afterwards. In PostgreSQL:

-- Producer: state change and event commit together
BEGIN;
INSERT INTO orders (id, customer_id, total_cents)
VALUES ($1, $2, $3);
INSERT INTO outbox (id, event_type, payload)
VALUES ($4, 'order.created', $5);
COMMIT;

-- Consumer: skip duplicates inside the same transaction as the side effect
BEGIN;
INSERT INTO processed_events (event_id)
VALUES ($1)
ON CONFLICT (event_id) DO NOTHING;
-- Row count 0 means already processed: roll back and acknowledge.
-- Row count 1 means first delivery: apply the change, then commit.
COMMIT;

Failure modes are different from those of synchronous systems. Backlogs replace errors, so alert on the age of the oldest message, not just queue depth. A backlog drains only at the surplus of consumption over arrival:

backlog = 3,000,000 messages
consume = 2,000 msg/s, arrive = 1,500 msg/s
drain time = 3,000,000 / (2,000 - 1,500) = 6,000 s (100 minutes)

Set visibility timeouts longer than worst-case processing time, or messages get processed twice. Route poison messages to a dead-letter queue with an owner. Rate-limit consumers, because a replayed backlog can overload downstream services. In a regional failover, unconsumed messages stay in the failed region unless replicated, so count them in the RPO.

Identity

A compromised administrator credential crosses every zone and region, so identity is the real perimeter.

• Humans use federation. Single sign-on from a central identity provider, short-lived sessions, and phishing-resistant MFA. No long-lived user keys.
• Workloads use platform identity. AWS IAM roles, Azure managed identities, GCP service accounts with workload identity. Static keys are a last resort.
• CI uses OIDC federation. The pipeline exchanges a signed token for short-lived credentials, so no secret is stored in CI.

Restrict the federated role's trust policy on the token's sub claim, matching exactly, for example repo:myproject/app:ref:refs/heads/main. Without that condition, any pipeline on the identity provider could assume the role.

Identity is also an availability dependency. IAM control planes and central SSO services may sit in a single location, so an outage there can stop you logging in during the incident you need to fix. Keep sealed break-glass credentials for each critical account, with hardware MFA and alarms on use. If failover needs an SSO login that depends on the failed region, the runbook has a hidden circular dependency.

FinOps and egress cost

Resilience adds cost through spare capacity and data movement, and the latter is often missing from early estimates. Rates differ by provider and region, so model volumes and then apply the current price sheet.

With three zones and even load balancing, about two-thirds of service-to-service calls cross a zone boundary. Kubernetes can favour local endpoints with spec.trafficDistribution on a Service, though strict locality can overload a zone with fewer replicas. A log replicated to three brokers across zones carries each byte over zone links at least twice.

replication_volume_month = write_gb_per_day * 30 * (regions - 1)
transfer_cost_month      = replication_volume_month * inter_region_rate
standby_cost_month       = standby_instances * instance_rate * hours_per_month

• Enforce tags such as owner and cost-center at creation.
• Track unit cost per order, tenant, or thousand requests.
• Buy commitments (Savings Plans, Azure reservations, GCP committed use discounts) only for the steady baseline, and check that they fit failover capacity plans.

Failure testing and game days

An untested failover is a hypothesis. Testing finds the missing IAM permission and the expired certificate in the standby region. AWS Fault Injection Service and Azure Chaos Studio inject faults with guardrails; Chaos Toolkit and LitmusChaos are open-source options.

A game day needs a hypothesis, a scope, abort conditions, named roles, and a rollback. Start with announced tests in staging, then move towards production.

Track every finding to closure, and record measured RTO and RPO over time. Fix the system, not just the runbook.

Verifying it works

Use these during an approved drill, with one engineer running the probe while another injects the fault.

# User's view: one line per second, keep this running through the test
while true; do
  curl -s -o /dev/null -m 3 -w "$(date +%T) %{http_code} %{time_total}\n" \
    https://app.example.com/healthz
  sleep 1
done

# Trigger a Multi-AZ database failover
aws rds reboot-db-instance --db-instance-identifier myproject-prod --force-failover

# Start a prepared fault injection experiment
aws fis start-experiment --experiment-template-id EXT123456789

# Confirm remaining targets are healthy
aws elbv2 describe-target-health --target-group-arn "$TARGET_GROUP_ARN"

# Replication lag on a PostgreSQL replica
psql -h replica.example.com -d myproject \
  -c "SELECT now() - pg_last_xact_replay_timestamp() AS replica_lag;"

# What are resolvers returning right now?
dig +noall +answer app.example.com

# Pod spread per node and zone
kubectl get pods -l app=myapp -o wide
kubectl get nodes -L topology.kubernetes.io/zone

Checklist for sign-off:

• Each workflow has a written RTO and RPO, approved by its business owner.
• Removing any one zone leaves enough running capacity for peak, with no control plane call required.
• Measured failover time is recorded from a drill within the last quarter, and is inside the RTO budget.
• Backups live in a separate account, are immutable, and a restore has been timed.
• Changes reach one zone or region at a time, with automated halt on bad metrics.
• A policy test confirms that denied actions fail.
• No two networks overlap, and non-prod has no route to prod. Verify from a host, not from a diagram.
• Break-glass login has been tested and the alarm fired.