Agoda Migrates 72-Shard SQL Cache to DragonflyDB for Speed and Scale

Agoda recently made a big move in its tech setup by shifting its primary hotel Price Cache from a complicated 72-shard Microsoft SQL Server configuration to DragonflyDB, an in-memory datastore. Why? They needed to handle more read and write operations as demands grew. Plus, they wanted to streamline the scaling process and boost performance metrics. Switching to DragonflyDB seemed necessary to manage large data volumes more efficiently and to cut down on the operational headaches that came with their previous system.

What Changed

Before the switch, the Price Cache was handling about 1.5 TB of fast-changing pricing data, with roughly 300,000 reads and a whopping 1.5 million writes every second. Now, with DragonflyDB in play, Agoda's seeing an eightfold improvement in P99 read latency. The system processes about 300,000 requests per second, averaging around 8 milliseconds for P99 latency. That's a huge drop in read latency, and it makes a noticeable difference in user experience. This shift can be credited to DragonflyDB's efficient in-memory data management and its knack for handling high throughput with lesser overhead.

Why It Matters

If your organization deals with extensive datasets and has loads of concurrent read and write actions, picking the right database technology is crucial. Agoda's earlier setup involved application-level shard routing across 72 SQL Server shards, presenting various scalability hurdles. Scaling up meant adding hardware in pre-defined increments, manual shard remapping, and data migration. Not only was this process clunky, but it was also expensive. As their workload grew, they found themselves upgrading hardware frequently, like when capacity was doubled in early 2024—only for limits to be reached again in less than a year. The manual scaling process increased the chance of errors and needed a hefty amount of human resources to keep everything in check.

Clarkson Chang, Agoda's lead engineer, pointed out that sticking with SQL Server wasn't a feasible long-term plan. This is a common snag with scaling traditional SQL deployments, where hardware and manual partitioning become serious bottlenecks. Moving to DragonflyDB wasn’t just a quick fix for performance issues; it laid the groundwork for future growth without the same operational burdens.

DragonflyDB's Role

DragonflyDB was tested against Agoda’s specific workload rather than just relying on published benchmarks. Its shared-nothing, multithreaded architecture, Redis compatibility, cluster-based scaling, and built-in key expiration fit well with Agoda's Price Cache, which heavily uses MGET and SET operations. Agoda used tools like memtier_benchmark to emulate production-like conditions with a 1:6 read-to-write ratio and average 10-key MGET operations. They needed to ensure DragonflyDB could handle their specific needs, rather than trusting generic performance metrics that might not hold up in real-world conditions.

At first, Agoda rolled out a 1 TB DragonflyDB instance. But when the dataset started nearing the 90% memory-safety threshold due to natural growth, they expanded to a three-shard-per-cluster setup. This allowed DragonflyDB to manage the entire 1.5 TB dataset, handling roughly 1.6 million writes per second at around 10 milliseconds P99 latency. This architecture carried the current data volume and provided the flexibility to scale further as needed, free from the constraints they'd faced with SQL Server.

A standout feature of DragonflyDB's architecture that made this transition smoother is its shared-nothing architecture. Unlike traditional databases that might use a central node or coordinator, each node in a shared-nothing setup operates independently. This makes scaling more efficient and improves fault tolerance, as there isn't a single point of failure. The system can keep running smoothly even if individual nodes hit issues.

Steps for Implementation

If you're thinking of a similar transition, here's a playbook based on Agoda's journey:

• Benchmark Real Workloads: Evaluate the new tech against your workload before jumping in. Use tools like memtier_benchmark to simulate realistic conditions and ensure the solution can handle your demands. Avoid unexpected post-migration performance issues by making this step a priority.
• Incremental Migration: Start small with a single instance for hot data, then gradually scale up. Keep an eye on memory usage to keep it within safe limits. This way, you reduce risk and can make adjustments based on actual performance.
• Dual Reads for Parity Checks: Use dual reads to verify data consistency across the old and new systems. Instead of comparing all the data, Agoda checked supplier counts and price-data lengths, achieving over 99.9% parity. This approach offers a reliable way to ensure the new system is working correctly without the overhead of full data comparisons.
• Controlled Traffic Migration: Use A/B testing to slowly transition customer traffic to the new system, similar to how Agoda spread this out over several weeks. This method minimizes disruptions and ensures any issues are resolved before a full switchover.
• Decentralized Failure Detection: Adopt a decentralized method for detecting failures. At Agoda, applications independently monitor cache-hit ratios to detect and manage failures, entering a failover mode if they spot major disparities. This enhances system resilience, letting applications respond quickly without needing a central coordinator.

Limitations and Considerations

While DragonflyDB offers big perks in performance and scalability, there are some trade-offs and things to watch out for:

• Memory Constraints: Being an in-memory datastore, DragonflyDB requires ample RAM to handle your dataset, which can get expensive at larger scales. Weigh the cost of increased memory needs against the performance benefits.
• Complexity of Migration: Switching from a traditional SQL setup to a distributed in-memory datastore can be complex and risky, especially concerning data consistency and system reliability. Careful planning and testing are key to mitigating these risks.
• Operational Overhead: While operational tasks like managing stale data and failovers are reduced, the system still needs monitoring and management to maintain its performance and availability. This includes regular checks on memory use and performance metrics to make sure the system continues to run efficiently.
• Potential for Overhead in Key Expiration: DragonflyDB's built-in key expiration is great for managing volatile data, but it needs careful configuration to prevent unintended data loss or excessive management overhead.
• Cluster Management: Handling multiple shards per cluster can introduce complexity, especially if the system needs to dynamically scale based on workload changes. Adequate monitoring tools and alerting systems are crucial for effective management.

Agoda’s migration shows how in-memory datastores like DragonflyDB can tackle scaling challenges related to traditional database systems. But every solution has its own set of hurdles. It's essential to weigh the trade-offs and ensure they align with your specific workload needs for a successful migration. Organizations need to examine their current and future data needs, think about costs, and plan a structured migration process to really reap the rewards of such a change.

For more insights into database and scaling solutions, check out our articles on Cloudflare's Cache Transcoding and Unified KV Store. These resources offer additional viewpoints on how other organizations are dealing with similar data management and scalability challenges.