Mastering Multi-Day AZ Evacuation Drills with AWS ARC Zonal Shift

Running a multi-day Availability Zone (AZ) evacuation drill using AWS ARC Zonal Shift is a smart way to see if your application can take a hit and keep going. Sure, deploying across multiple AZs is what we all aim for, but actually proving that your setup can handle stress over time is another story. Many disaster recovery tests only shuffle traffic around and check if targets respond. These quick tests often overlook problems that creep up when systems are under pressure for longer periods, like:

• Auto Scaling Policies: Policies that seem fine for short spikes might be inadequate for prolonged N-1 operations. They need to maintain performance for days, not just minutes.
• Deployment Pipelines: If your pipelines aren't checking AZ health, they might deploy into impaired zones, causing failures that become obvious only when tested over longer periods.
• Stale DNS or Cached Endpoints: DNS records that don’t refresh fast enough can lead to connectivity problems, noticeable when a zone stays down for a while.
• Time-Dependent Processes: Things like certificate, credential, and log rotations can falter if an AZ is out for a long time, messing up scheduled processes.
• Long-Lived Database Connections: If connections are pinned to a down AZ, they may fail to reconnect, which can be a big problem for the apps relying on them.
• Recovery Procedures: Moving back to normal after a long shift is different from a quick rollback. You need solid recovery plans for that.

By diverting all traffic away from a single AZ for 48 to 72 hours using ARC Zonal Shift, you’re doing more than just a test. You’re seeing if your architecture can handle full loads with one less zone, which tells you a lot about capacity, database reliability, and how clients reconnect.

Why Financial Services are Leading the Charge

Financial services are leading the way here, mainly because of regulatory requirements. They’re running these multi-day drills as part of their resilience programs, which demand actual proof of recovery capabilities, not just theory. This push for real-world testing is moving them away from just having runbooks to showing they can handle real scenarios.

With ARC Zonal Shift, traffic is shifted at the infrastructure level, requiring no changes to application code. This works across AWS services like Application Load Balancer, Network Load Balancer, Amazon EC2 Auto Scaling groups, and Amazon EKS clusters.

Planning and Executing a Multi-Day AZ Evacuation Drill

For such drills, the architecture usually involves an AWS multi-tier digital platform:

• Traffic Ingress: An ALB fronts ECS across 3 AZs with cross-zone load balancing active. This setup helps the remaining AZs manage the traffic when one is evacuated.
• Compute: ECS (Fargate) stateless tasks and EKS services spread across 3 AZ subnets. Stateless tasks make redistribution and scaling easier during an AZ evacuation.
• Database: Amazon RDS for PostgreSQL and Amazon Aurora PostgreSQL with multi-AZ configurations keep your data accessible if an AZ goes down.

In this scenario, the drill involves evacuating AZ A, which hosts the RDS primary and Aurora writer. Thanks to Aurora's replication across six storage nodes independent of compute instances, only the compute instance needs to failover, ensuring resilience.

Executing the Zonal Shift

The ARC Zonal Shift involves coordinated steps for Amazon Route 53 and Elastic Load Balancers:

• DNS Removal: Remove the load balancer's IP in the affected AZ from DNS, stopping new client queries from resolving to that endpoint.
• Traffic Blocking: Healthy AZs stop routing requests to the impaired AZ, ensuring all traffic redirects away, even with cross-zone load balancing enabled.

For Amazon EKS clusters, ARC additionally cordons nodes in the impacted AZ and adjusts EndpointSlice resources to direct service traffic to healthy AZs, ensuring Kubernetes workloads remain stable.

Steps for Amazon ECS

For ECS services behind an ALB, stopping new traffic to targets in the evacuated AZ is the goal. To carry out the evacuation:

1. Ensure ARC Zonal Shift is enabled.
2. Start the zonal shift on the load balancer.
3. Update network configurations to restrict new task placement to healthy AZs.
4. Pre-scale services to manage an AZ’s capacity loss before the drill. This often means increasing the desired count of tasks in the healthy AZs, so they can deal with more load.
5. Monitor task distribution and reset network configurations post-drill. Make sure tasks are evenly spread across the remaining AZs to avoid overloading one.

Steps for Amazon EKS

For EKS, ARC takes care of both infrastructure and Kubernetes networking:

1. Activate zonal shift on the EKS cluster.
2. Initiate the zonal shift on the load balancer and EKS cluster.
3. Use kubectl get endpointslices to check EndpointSlice updates and confirm that pods in the shifted AZ aren't getting traffic.
4. Apply topologySpreadConstraints with maxSkew: 1 for pods. This keeps pods evenly spread across AZs, avoiding any single AZ becoming a bottleneck.

Keep in mind that ARC zonal shift doesn't manage outbound connections from pods to external dependencies like Amazon RDS. Make sure these connections can handle zone failures.

Considerations for Multi-Day Shifts

Running a zonal shift for days brings its own set of challenges:

• Expiry Management: Zonal shifts have time limits. Keep track and extend them if needed, or traffic might accidentally revert to the shifted AZ, causing issues.
• Scaling Drift: Over time, Auto Scaling events in healthy AZs can create capacity imbalances. Monitor and cap scaling to avoid overloading the returning AZ. I find setting scaling limits helps keep things balanced.
• Connection Pool Cycling: After more than 24 hours, most client connections will have cycled, testing DNS TTL adherence across your client fleet. It’s crucial for ensuring smooth reconnections to new endpoints.
• Operational Confidence: Teams get valuable experience working in reduced AZ environments, which can be a lifesaver during real AZ troubles.
• Safe Recovery: Carefully plan the restoration after days at N-1 capacity. Restore vital services first for a smooth return to normal operations.

What This Does Not Solve

Even with thorough multi-day AZ evacuation drills, some issues remain:

• Cost Implications: Pre-scaling services for N-1 capacity can cost a lot, so weigh the expenses against the resilience boost.
• Control Plane Limitations: Real AZ failures might degrade control plane operations, impacting manual interventions. Automate where possible to mitigate this.
• Complex Implementations: Complex architectures can make setup tricky. Careful planning and testing are key to pulling it off successfully.

If you're curious about AWS capabilities, check out our other posts on Deploying Oracle Database on Amazon EVS and AWS MCP Update: No More Session Affinity.