Implementing Health Check Endpoints in Node.js with Express

Understanding Health Check Endpoints

In modern infrastructure, maintaining the health of your services is crucial. For Node.js applications running on Docker, implementing health check endpoints using Express is key to ensuring that your application remains responsive and stable. The source article Express Production Health Check Endpoints in Node.js Docker provides a comprehensive overview of setting up liveness and readiness endpoints, which are crucial for monitoring and managing the health of your application.

What Changed?

The article emphasizes the importance of two specific endpoints: /health/live and /health/ready. The liveness endpoint checks if the Node.js process can respond, ensuring the application isn’t restarted unnecessarily if a dependency fails. The readiness endpoint, on the other hand, determines if an instance is ready to accept traffic, which is crucial before routing live traffic to the service. This separation ensures that the health of the application is checked in a more granular way, preventing unnecessary disruptions.

Why It Matters

For engineers maintaining production systems, these health checks are essential. They allow for better orchestration and resource management within Docker and Kubernetes environments. By implementing these checks, you can ensure that your application only receives traffic when it is fully ready and capable of handling requests, reducing downtime and improving reliability. This is particularly important for applications that rely on multiple dependencies, as it helps identify and isolate issues without interrupting the entire service.

Implementing Health Check Endpoints

To add these endpoints to your Express app, you can follow these practical steps:

const express = require('express');
const app = express();

// Liveness Endpoint
app.get('/health/live', (req, res) => {
  res.status(200).send('OK');
});

// Readiness Endpoint
app.get('/health/ready', (req, res) => {
  const isReady = checkIfAppIsReady(); // Implement your own readiness logic
  if (isReady) {
    res.status(200).send('Ready');
  } else {
    res.status(503).send('Service Unavailable');
  }
});

function checkIfAppIsReady() {
  // Logic to determine if the app is ready
  return true; // or false based on your conditions
}

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

These endpoints should be kept simple. The liveness endpoint checks if the event loop is responsive, while the readiness endpoint should include checks such as configuration loading and startup completion. It is crucial to ensure the readiness endpoint fails before a worker drains, preventing new batches from arriving unexpectedly. This setup ensures that your application can handle traffic effectively and reduces the risk of overloading or failing under pressure.

Managing 5xx Logs

Logging is another critical aspect of maintaining a healthy application. The source article suggests retaining logs of 5xx errors and other significant events while avoiding the storage of verbose success events. This approach helps in diagnosing issues without overwhelming your logging system. The strategy involves storing metadata for failed transitions, dropping repetitive successes, and focusing on significant event logs. Retaining evidence should be in proportion to its diagnostic value, as this helps in identifying patterns and potential issues efficiently.

Consider the following logging strategy:

• Log every 5xx error with details such as timestamp, severity, service, environment, and pipeline run ID.
• Avoid logging sensitive information like tokens or payment data.
• Sample and aggregate routine successes to reduce log volume.
• Use a stable run ID for logs and traces, and avoid using high-cardinality identifiers in metric labels.

Practical Steps for Log Management

To effectively manage logs, you can implement the following strategies:

• Use structured logging to make your logs easier to query and analyze. This can be done by ensuring logs are in a consistent format, which helps in quickly identifying issues.
• Employ log sampling to retain only significant event logs, which can be particularly useful in high-volume environments. Outcome-aware sampling can be beneficial, where failures are retained while routine successes are sampled.
• Ensure your log storage solution can handle the cardinality of identifiers like run IDs and trace identifiers without performance degradation. This involves careful planning of storage needs based on anticipated log volume and retention requirements.
• Attribute costs to each run by attaching a stable run ID to logs and traces, allowing for accurate usage accounting and cost management.

Limitations and Considerations

While these health checks and logging strategies are beneficial, they come with limitations:

• Health checks do not cover external dependencies such as DNS and certificates. These should be monitored separately to ensure a comprehensive view of application health.
• Sampling logs can lead to missing some context in the event of a failure. Ensure you have a balance between performance and the need for detailed logs. A clear policy on what to retain and for how long can mitigate this risk.
• Regulatory requirements might necessitate retaining more detailed logs than usual for compliance reasons. In such cases, a dedicated audit logging system should be considered.

In cases where a complete audit trail is required, consider implementing a dedicated logging and audit system separate from your diagnostic logs. This ensures compliance and provides a detailed history for investigation and dispute resolution. It's important to understand that while health checks and logging strategies enhance application reliability, they do not replace the need for a robust monitoring and alerting system that covers all aspects of application performance.

Worked Example: Calculating Log Storage Needs

To illustrate the concepts discussed, consider a scenario with a Node.js application handling 8 million offers, each emitting four 700-byte events. This results in 22.4 GB of data per run. If we retain logs for 30 runs, the total storage required before indexing or replication would be 672 GB. By retaining only one terminal event per offer, the storage requirement drops to 5.6 GB. Sampling 10,000 successful events at four events each adds only 28 MB. This example highlights the importance of strategic log retention and sampling in managing storage costs effectively.

In my experience, it's vital to regularly review your logging strategy to ensure it aligns with your application's evolving needs. As your application scales, so too will the volume of logs, and without an effective strategy, you may encounter performance bottlenecks or increased costs.

Conclusion

Implementing health check endpoints in your Node.js application using Express can significantly enhance your application's reliability and maintainability. By also adopting a thoughtful approach to logging, focusing on significant events, and sampling routine successes, you can keep your application performant while retaining the necessary diagnostic information. This approach not only aids in maintaining application health but also ensures efficient resource utilization and cost management.

For more insights, you can explore posts like Separating Node.js Probes: Liveness, Readiness, and Startup and Integrating React Frontend Errors with Node.js API Logs to understand better how to manage dependencies and handle errors effectively in your applications.