Implementing Health Checks in Node.js SaaS: Readiness and Liveness

Maintaining the health and reliability of your Node.js SaaS applications is crucial. A recent article on Dev talks about the importance of implementing effective health checks through readiness and liveness endpoints. Here's why these checks matter and how to implement them effectively.

The Basics of Health Checks

A health check in a Node.js SaaS application typically involves two primary endpoints: liveness and readiness. The liveness endpoint answers the question, "Is the application process alive and able to serve work?" In contrast, the readiness endpoint determines if the application is ready to handle incoming requests by verifying the state of its dependencies like databases or external APIs.

Liveness Endpoint

The liveness check should be shallow. The main goal here is to ensure that the application process itself is running, without checking the state of downstream dependencies. This is crucial because restarting a process due to a temporary unavailability of a database can lead to unnecessary "restart storms," where the application continuously restarts, compounding the problem.

app.get('/livez', (req, res) => {
  res.status(200).send('OK');
});

In this example, the handler simply returns a success status if the event loop can execute the function, indicating the process is alive. The focus here is on the health of the process itself, not its ability to connect to other services or databases.

Readiness Endpoint

The readiness check is more in-depth. It involves checking if the application’s dependencies, like Postgres and Redis, are in a state that supports normal traffic. Each dependency should have a short, explicit deadline for these checks to avoid unnecessary delays, ensuring that the application does not become bogged down while waiting for responses.

app.get('/readyz', async (req, res) => {
  try {
    await checkPostgres();
    await checkRedis();
    res.status(200).send('Ready');
  } catch (error) {
    res.status(503).send('Service Unavailable');
  }
});

Here, the readiness check verifies the state of databases and returns a non-success status if they fail, ensuring the application does not receive work it can't handle effectively. This approach prevents the system from accepting new requests if it cannot guarantee that those requests can be processed correctly. Special attention should be paid to distinguishing between essential and non-essential dependencies in these checks. For instance, if a third-party API is not critical for a particular request path, the service can be reported as degraded rather than completely unready.

Why Health Checks Matter

Implementing these checks is crucial for maintaining application stability. They help in preventing "green is not ready" scenarios, where an application might appear operational but isn't fully prepared to handle real-world traffic due to dependency issues. This separation of concerns—liveness for process health and readiness for dependency health—allows for more precise incident management and recovery strategies.

Practical Implementation Steps

Here’s how you can start implementing these checks in your Node.js SaaS:

• Define Your Endpoints: Implement shallow liveness checks and more comprehensive readiness checks as shown in the examples above. Ensure that each endpoint is clearly documented and that their purposes are well understood by your team.
• External Monitoring: Use an external service to poll these endpoints. This ensures that your application isn't silently failing without notice. Consider integrating external services like Better Uptime or Healthchecks.io for monitoring these endpoints. This external perspective is crucial as it confirms whether the service is reachable from outside your network.
• Log and Metric Management: Emit stable metrics for readiness and structured logs for detailed incident reconstruction. This is particularly useful when experimenting with different tenant cohorts in a SaaS model. By keeping logs structured and consistent, you can easily correlate different events and reconstruct incidents effectively.

Trade-offs and Limitations

While these health checks are essential, they come with certain trade-offs. Implementing deep dependency checks in readiness endpoints can increase latency if not properly managed. It's also important to handle external monitoring gracefully to avoid overwhelming your service with health check traffic.

Moreover, these checks shouldn't be used as exhaustive diagnostic tools. They are meant to provide a quick assessment of the application's health status. For detailed diagnostics, rely on structured logs and metrics. It's also worth noting that while these checks can indicate when something is wrong, they do not inherently provide the means to fix the problem.

What I'd Do on Monday

Given these insights, my approach on Monday would involve:

• Reviewing the current implementation of health checks in our Node.js SaaS apps to ensure they align with best practices. This includes verifying that our liveness and readiness endpoints are correctly implemented and that they are being effectively monitored.
• Establishing external monitoring for our readiness and liveness endpoints if not already in place. This is crucial for ensuring that we can detect outages or performance issues from an outside perspective.
• Ensuring our logging and metrics systems are robust enough to support effective incident reconstruction. This involves checking that our logs are structured and comprehensive, and that our metrics are accurately reflecting the state of our application and its dependencies.
• Documenting the incident response process, including roles and responsibilities, to ensure that the team is prepared to act swiftly in the event of an issue. This includes defining what constitutes a critical incident and how it should be escalated.

This approach will not only improve the reliability of our applications but also provide a clear path for incident management and recovery. By having a comprehensive view of our application's health and its dependencies, we can respond more effectively to incidents and ensure a stable experience for our users.