Monitor your batch processing and heavy jobs with precision
Batch jobs are the hidden workhorses of IT infrastructure. Whether importing thousands of orders, synchronizing databases between multiple systems, generating nightly reports, or processing accumulated files, these processes are often scheduled to run outside business hours when no one is watching them.
This invisibility poses a major problem: when a batch job fails, the consequences don't appear immediately. It may take hours or even days to notice that the previous night's import never ran. By then, the damage is done: outdated data, inventory errors, missing invoices, desynchronized systems.
Heartbeat monitoring is the ideal solution for batch jobs. Instead of monitoring the server or service, we monitor the job's vital signs: did it start? Is it progressing? Did it finish successfully? If any signal is missing, the alert is triggered immediately.
Unlike interactive applications that fail visibly when there's a problem, batch jobs present unique challenges:
Un monitoring efficace des jobs batch couvre quatre aspects clés : démarrage, progression, terminaison et résultat.
Integrating heartbeat into a batch job follows a simple pattern with three key moments:
Here is a complete implementation example for a typical batch job in PHP:
$order) {
try {
processOrder($order);
$processed++;
} catch (Exception $e) {
logError($order->id, $e->getMessage());
$errors++;
}
// Progress ping every 1000 items
if (($index + 1) % 1000 === 0) {
$progress = round(($index + 1) / $total * 100);
file_get_contents($heartbeatUrl . "?progress={$progress}");
}
}
// Success or partial failure
$duration = round(microtime(true) - $startTime);
if ($errors === 0) {
file_get_contents($heartbeatUrl . "?status=success&processed={$processed}&duration={$duration}");
} else {
file_get_contents($heartbeatUrl . "/fail?processed={$processed}&errors={$errors}&duration={$duration}");
}
} catch (Exception $e) {
// Critical error
file_get_contents($heartbeatUrl . "/fail?error=" . urlencode($e->getMessage()));
exit(1);
}This example covers the three key scenarios: normal operation with success ping, partial failures with tracking, and critical errors with immediate alert.
Proper alert setup is crucial to avoid false alarms while catching real problems:
Import files received via SFTP every night. The heartbeat ensures the import ran, the file was found, and the data was correctly integrated. A single grace period matching the typical import duration is sufficient.
Synchronize data between master database and replicas. Heartbeat monitoring detects not only sync failures but also abnormally long durations indicating potential network or performance issues.
Generate and send daily or weekly reports. The heartbeat ensures reports were generated on time. With metadata, you can verify expected file size and correct destination.
Set a 4-hour timeout (maximum observed duration + 33% margin). If the batch takes longer than its historical maximum, that itself is an anomaly worth investigating.
Add progress pings every 100,000 or 500,000 items. This lets you track advancement in real-time and detect stalls before the global timeout triggers.
Define a clear policy: below X% errors = success, above = fail. Send the fail ping with error count. You can also create two heartbeats with different thresholds (warning vs critical).
Create two distinct heartbeats: one for scheduled executions with strict timeout, one for manual executions with wider timeout. Use a parameter to select the appropriate URL.
Yes, create one heartbeat per independent batch. For parallel batches that are part of the same workflow, consider an orchestrator that sends the final ping once all components complete.
Set up a local proxy that receives internal pings and forwards them to MoniTao. Alternatively, write a local log file that another monitored process reads and transmits.
Batch jobs are the invisible gears of your business. When they run correctly, nobody notices them. When they fail silently, the consequences can be disastrous: desynchronized data, missing reports, failed imports, frustrated customers.
Heartbeat monitoring with MoniTao transforms every batch job into a supervised process. With a few lines of code, you gain complete visibility: did it start? Is it progressing? Did it complete successfully? If any of these questions has a concerning answer, you're alerted immediately, not after discovering the problem manually.
Start free, no credit card required.