Grab's Storage Migration: Lower Latency and Cost with Aerospike

Grab, a leading super app in Southeast Asia, has recently completed a significant migration of its Counter Service storage backend from a wide-column database to Aerospike. This change has resulted in a 50% reduction in production p99 read latency and substantial cost savings. Here's what this migration involved and why it might matter for your own engineering projects.

What Changed: From Wide-Column to Aerospike

Originally, Grab's Counter Service used a wide-column data model similar to what Apache Cassandra offers, which involved storing 15-minute, hourly, and daily counts as separate rows. Each incoming event required three parallel reads and a batch write, leading to four network round trips. The service supports Grab's anti-fraud platform, handling tens of thousands of queries per second and up to a billion requests per day.

In the new setup, Grab transitioned to Aerospike, a high-performance database designed for real-time analytics and scalable storage. The migration was not a simple replacement; it involved a complete redesign of the data model and a staged traffic migration strategy to minimize downtime. The original design's reliance on clustering columns for efficient row retrieval within a time range was replaced with Aerospike's usage of a timestamp-ordered map within a single record, capitalizing on Aerospike's atomic map increment operations. This change significantly reduced the record count and, consequently, the storage footprint.

Why It Matters: Latency and Cost Improvements

The migration brought about two primary benefits: improved latency and reduced costs. The p99 read latency was cut by approximately 50%. Additionally, the on-disk data footprint was reduced from about 3 TB to 1 TB, and the cost per node dropped by 45% to 50%. For a service handling such a high volume of requests, these improvements are substantial. The redesign also meant fewer network round trips due to the batch API's ability to group multiple record operations into requests, further enhancing the system's efficiency.

For engineers managing high-load systems, this case study is a compelling example of the potential benefits of revisiting and optimizing your data storage solutions. The reduction in latency can enhance user experience significantly, while cost savings can free up resources for other critical initiatives.

Implementing the Migration: Key Steps and Strategies

Grab's migration strategy was methodical and multifaceted:

• Storage Facade: Before switching to Aerospike, Grab decoupled storage access from the service's business logic. This was achieved through a storage facade that exposed multiple implementations: legacy, Aerospike, and mock. This layer of abstraction allowed for flexible testing and gradual transitions between different storage systems without impacting the core business logic.
• Traffic Migration: Traffic was migrated in stages using shadow reads, gradually increasing from 5% to 100%. This approach allowed Grab to validate data parity and ensure system reliability before fully committing to Aerospike. The gradual increase in shadow reads was crucial for monitoring performance and correctness at each stage of the migration.
• Data Model Redesign: Instead of maintaining a row per time bucket, Grab consolidated all buckets for a counter into a single Aerospike record containing a timestamp-ordered map. This reduced record count by more than an order of magnitude, addressing the capacity concerns associated with Aerospike's 64-byte primary index metadata overhead per record. By leveraging Aerospike's map data type, Grab efficiently managed the storage of multiple time-windowed counts within a single record.
• Batch Processing: The read path was shifted from server-side filtering of paginated rows to client-side filtering of records containing maps. Aerospike's batch API was used to reduce network requests, aligning well with Grab's multi-key access pattern. This change allowed the system to handle multiple key operations in a single network request, improving overall read efficiency.

Rolling Out Aerospike in Your Own Systems

If you're considering a similar migration, here are some practical steps based on Grab's approach:

• Assess Your Current Architecture: Determine if your current database model is meeting performance and cost objectives. Consider alternatives like Aerospike for high-volume, low-latency requirements. Evaluate the trade-offs of different data models and storage backends in the context of your specific use cases.
• Decouple Business Logic from Storage: Create a storage facade to separate business logic from the underlying storage implementation. This will enable smoother transitions and flexibility to swap storage backends. In my experience, this separation can also facilitate testing and debugging, as it isolates storage-related issues from business logic concerns.
• Use Staged Migrations: Implement shadow reads and gradual traffic migration to ensure that the new system is performing as expected before fully committing. This staged approach allows for identifying issues early and making necessary adjustments with minimal risk to production systems.
• Optimize Data Models: Redesign your data model to minimize overhead and leverage the strengths of the new database system. This might involve consolidating records or using more efficient data structures. Consider the specific features and limitations of the new system to maximize performance and storage efficiency.
• Leverage Batch Operations: Use batch processing to reduce network overhead and improve performance, particularly if your service pattern involves accessing multiple records simultaneously. In my experience, batch operations can significantly reduce latency and improve throughput in high-concurrency environments.

Challenges and Limitations: What to Watch Out For

While the migration was successful, Grab encountered several challenges that are worth noting:

• Client API Adjustments: The initial Aerospike Rust client had a synchronous API and DNS handling issues during cluster replacement. Grab worked with maintainers to resolve these, eventually adopting an asynchronous client. If you're using Aerospike, be prepared to handle similar client-side challenges. Ensuring that your client libraries are well-supported and fit your architectural needs is crucial for a smooth migration.
• Capacity Considerations: Aerospike's primary index metadata overhead could be significant at high record counts. Ensure your data model is optimized to manage this efficiently. Consider the trade-offs between record granularity and storage overhead, and plan accordingly to avoid capacity issues.
• Data Integrity and Downtime Risks: While Grab completed the migration without downtime or data integrity issues, these risks are inherent in any large-scale migration and should be carefully managed. Conduct thorough testing and validation at each stage of the migration to mitigate these risks.

In my experience, the key takeaway from Grab's migration is the importance of a well-planned transition strategy that includes gradual traffic shifts, comprehensive testing, and collaboration with technology partners to address any unforeseen challenges. Such a strategy not only minimizes risk but also maximizes the potential for a successful migration outcome.