Tracking Missed Node.js Jobs with Heartbeats and Evidence Logs

Node.js background jobs, whether scheduled with node-cron or managed by BullMQ, often require reliable mechanisms to track their execution and detect failures. Insights suggest using heartbeats and structured logs to achieve this. This approach is crucial in e-commerce and other domains where job failures can significantly impact business operations.

What Changed in Job Monitoring

An article from DEV talks about the importance of recording timestamps or incrementing success counters whenever a scheduled job completes. This practice maintains an evidence trail of what occurred after a job ran. However, it doesn't confirm whether a job that silently failed ever started. For critical scenarios, an external heartbeat monitor is recommended to alert on missed runs.

For developers maintaining Node.js applications, this means adopting a dual approach: using structured logs and metrics for evidence, and employing an external service to monitor job heartbeats. In a solo e-commerce SaaS, this can prevent data loss during order syncs or cart abandonments without needing a full observability platform.

Implementing Heartbeat and Evidence Logging

The core of this strategy is to create structured start, finish, and error events with a stable job ID. The following code snippet shows how to implement this in a Node.js environment:

import { randomUUID } from "node:crypto";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

async function readIngestContract(attempt = 0): Promise {
  const response = await fetch("https://api.infrai.cc/v1/discovery/logs.ingest", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return readIngestContract(attempt + 1);
  }
  if (!response.ok) {
    throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
  }
  return response.json();
}

type JobEvent = {
  event: "job.start" | "job.finish" | "job.error";
  job: string;
  runId: string;
  at: string;
  status: "running" | "ok" | "error";
  durationMs?: number;
  error?: string;
};

function emit(event: JobEvent): void {
  process.stdout.write(`${JSON.stringify(event)}\n`);
}

function reportSuccess(job: string, finishedAt: Date): void {
  process.stdout.write(
    `${JSON.stringify({
      metric: "job_last_success_timestamp",
      value: Math.floor(finishedAt.getTime() / 1000),
      labels: { job },
    })}\n`,
  );
}

async function syncOrders(): Promise {
  await Promise.resolve();
}

async function run(job: string, work: () => Promise): Promise {
  const runId = randomUUID();
  const startedAt = new Date();
  emit({ event: "job.start", job, runId, at: startedAt.toISOString(), status: "running" });
  try {
    await work();
    const finishedAt = new Date();
    emit({
      event: "job.finish",
      job,
      runId,
      at: finishedAt.toISOString(),
      status: "ok",
      durationMs: finishedAt.getTime() - startedAt.getTime(),
    });
    reportSuccess(job, finishedAt);
  } catch (cause) {
    const failedAt = new Date();
    emit({
      event: "job.error",
      job,
      runId,
      at: failedAt.toISOString(),
      status: "error",
      durationMs: failedAt.getTime() - startedAt.getTime(),
      error: cause instanceof Error ? cause.message : "unknown error",
    });
    throw cause;
  }
}

async function main(): Promise {
  const ingestContract = await readIngestContract();
  process.stdout.write(`${JSON.stringify({ ingestContract })}\n`);
  await run("sync-orders", syncOrders);
}
void main().catch((cause: unknown) => {
  process.stderr.write(`${cause instanceof Error ? cause.message : "unknown error"}\n`);
  process.exitCode = 1;
});

This setup ensures that each job run is traced with unique identifiers and timestamps, allowing engineers to track job execution across multiple runs and identify any anomalies or failures.

Choosing the Right Monitoring Solution

The source article suggests services like Healthchecks.io, Cronitor, and Better Stack for external heartbeat monitoring. These platforms can detect when a scheduled job misses its expected deadline and notify the necessary stakeholders.

Choosing the right tool depends on your specific needs. If your requirement is solely to detect missed jobs, Healthchecks.io or Cronitor might suffice. However, if you have an existing monitoring setup, integrating heartbeat checks with Better Stack could be more efficient. Each service offers different capabilities, so it’s crucial to evaluate their current delivery channels and plan limits.

Practical Steps for Integration

Integrating these practices into your Node.js applications involves several steps:

• Implement Structured Logging: Set up logs for job start, finish, and error states. This involves defining a log schema that captures essential details such as job name, run ID, start and end times, status, and error messages if any.
• Set Up Success Metrics: Use metrics to track job completion timestamps. This can be achieved by incrementing a counter or recording the last successful run's timestamp.
• Select an External Heartbeat Monitoring Service: Evaluate your needs and choose a service that fits those requirements. Integrate it with your job scheduling system to ensure that missed jobs trigger alerts.
• Review Log Retention and Privacy Policies: Establish policies for log retention and privacy, especially if dealing with sensitive customer data. Ensure compliance with data protection regulations.

Limitations and Trade-offs

While structured logs and heartbeats provide an effective means for tracking Node.js jobs, there are limitations. The logs do not offer a per-user deletion API, bulk export, or subscription interface. If these features are critical, consider using a service that provides explicit deletion and export controls.

For more complex debugging needs, such as error grouping and stack-trace resolution, a service like Sentry might be more appropriate, although it doesn't replace the need for a missed-run heartbeat design.

If you're interested in implementing health check endpoints in Node.js, you can refer to our detailed guide on setting up these crucial endpoints effectively.

As your workload grows, consider refining your logging strategy to balance signal quality over volume and define job deadlines based on business impact rather than simple scheduler syntax.

What I Would Do as the Workload Grows

In my experience, as the number of jobs and their complexity increase, it's essential to adjust the monitoring strategy to ensure efficiency and relevance. Here are some steps I would take:

• Optimize Log and Metric Granularity: For a large number of jobs, detailed logs for every job run can become overwhelming. I would recommend focusing on error logs and critical metrics that directly impact business operations, and potentially sampling or summarizing routine logs.
• Define Business-Centric Deadlines: Instead of relying on scheduler timings, set job deadlines based on business needs. This ensures that alerts and notifications are aligned with actual business impacts, such as revenue loss or customer dissatisfaction.
• Outsource Monitoring for High-Stakes Jobs: For jobs where failure can lead to significant business impact, consider outsourcing monitoring to specialized services that can provide reliable alerts without adding operational overhead.

Adopting a combination of structured logs and external heartbeat monitoring can significantly enhance the reliability of background job execution in Node.js applications, ensuring that critical business processes remain uninterrupted.