Automating AWS DRS Agent Lifecycle with SSM and EventBridge

AWS recently rolled out an automation option for handling AWS Elastic Disaster Recovery (AWS DRS) agents' entire lifecycle. This setup handles installation, healing, and offboarding phases of these agents, which are vital for keeping disaster recovery operations running smoothly without spending too much money. It's especially handy for managing large EC2 fleets under AWS DRS, where doing everything manually can be a real hassle and prone to mistakes.

Automating AWS DRS Agent Management

Honestly, automating the AWS DRS agent lifecycle solves quite a few headaches that enterprise IT teams deal with regularly. Agents can stall after network changes, and there's the issue of descoped servers still replicating, which can rack up unnecessary costs. When an agent stalls, it can mess with the recovery point objective (RPO), potentially causing more data loss than your team can afford during a disruption. Regular checks in the AWS DRS console before scheduled disaster recovery drills might miss any problems that crop up in between, leaving your system exposed.

This automation aims to tackle these issues head-on. It ensures agents are healthy and billing stops when servers are no longer relevant. Using AWS Systems Manager (SSM) Automation documents triggered by Amazon EventBridge or scheduled events, it only needs a single EC2 tag to determine what's needed for each instance. That cuts out the need for AWS Lambda functions, making life a bit easier and the system simpler to maintain.

Getting Started with AWS DRS Automation

To set this solution up, you'll need access to an AWS account with admin rights and two AWS Regions: one for your EC2 instances and another for disaster recovery. You must have AWS Elastic Disaster Recovery ready in both, with network connections between them. Also, you'll need specific IAM permissions to roll out CloudFormation stacks and register SSM Automation documents.

The automation uses three main SSM Automation documents:

• DRS-Agent-Onboard: This runs every 30 minutes via State Manager for instances tagged with DR=yes. It installs or repairs the agent, adds the AWSDRS:AllowLaunchingIntoThisInstance tag, and checks AWS DRS registration. This keeps the instance prepped for failback and corrects any deviations from the desired setup.
• DRS-Agent-Heal: This kicks in via Amazon EventBridge upon detecting a replication-stalled event. It specifically targets the STALLED state to avoid nuisance actions during recovery transitions. It differentiates between agent and infrastructure issues, restarting or reinstalling agents when required, and escalates infrastructure problems through Amazon SNS.
• DRS-Agent-Offboard: Activated when a DR tag changes or gets removed. It includes checks like a tag re-check, a terminated-instance guard, and an optional approval gate. Once cleared, it stops replication, disconnects and deletes the source server, uninstalls the agent, and removes tags.

These documents mainly function in the source Region but will make cross-Region calls to the disaster recovery Region when needed by AWS DRS, like during registration checks or teardown operations.

Deploying the Automation System

Setting up this automation involves a few steps. First, make sure your EC2 instances have instance profiles with the necessary managed policies: AWSElasticDisasterRecoveryAgentInstallationPolicy and AmazonSSMManagedInstanceCore. Ensure you have an existing Amazon SNS topic for notifications and, if you want, an S3 bucket for audit reports. Double-check that the AWS CLI is installed and configured, and have Git, Bash, and yq available to run the deployment scripts.

The needed SSM Automation documents and deployment scripts are available in the aws-samples GitHub repository. These include CloudFormation templates to set up the required IAM roles and EventBridge rules. Once deployed, automation runs based on each EC2 instance's DR tag value.

Deciphering the DR Tag System

The DR tag is crucial to this automation, setting each instance's state:

• yes: The agent is installed and maintained, with a failback tag applied. This ensures the agent is healthy and ready for action if disaster strikes.
• pause: The agent stays installed but suppresses alerts and blocks offboarding. Handy for halting replication temporarily without uninstalling the agent.
• no or deleted: The instance is removed from scope, stopping replication and uninstalling the agent. This prevents unnecessary billing and resource waste on out-of-scope instances.
• Misconfigured: Any other tag value results in the automation skipping the instance and logging it as misconfigured. It's a safety measure to avoid acting on unrecognized tags.

The automation standardizes the tag value, ensuring consistent evaluation. For instance, DR=Yes is treated the same as DR=yes, so differences in capitalization or whitespace don't throw off the process.

Limitations of the Solution

While the automation cuts down on manual tasks, it doesn't solve everything. Non-agent issues like bandwidth throttling or undersized servers still need someone to step in. Also, since it depends on Amazon EventBridge and SSM, any problems there could affect how well it runs. From personal experience, it's wise to have backups and monitoring for these services.

Also, any changes or deprecations by AWS in these services could mean you have to update the automation scripts. Staying on top of AWS announcements and updates is something I'd recommend.

Example: Tagging an Instance for Protection

Let's say you want to protect an instance. You add a DR tag with a value of yes. This action sets off the DRS-Agent-Onboard document, checking that the replication agent is in place and in good shape. If it's missing or unhealthy, the automation takes care of the installation or repairs and adds the necessary failback tags. It does this check every 30 minutes, so any issues are quickly looked into.

On the other hand, if you want to remove an instance from protection, change the DR tag to no or delete it. This triggers the DRS-Agent-Offboard document, which goes through checks before stopping replication, uninstalling the agent, and clearing the tags, making sure the instance is cleanly removed from the disaster recovery setup.

If your team is managing a big field of EC2 instances with AWS DRS, automating the agent lifecycle can streamline operations and cut back on human error. Using AWS SSM and EventBridge, this solution lets you manage the lifecycle process without needing custom Lambda functions, making it a solid choice for many setups.

In critical environments like a newsroom CMS where uptime is crucial, this kind of automation ensures agents are ready to jump in and prevent data loss without you having to oversee every move. However, don't forget to pair this automation with robust monitoring and alert systems to catch any edge cases it might miss.