Data Import Monitoring

Make sure your imports and ETL run correctly.

Data imports are often business-critical. A failed sync can have significant impacts.

MoniTao Heartbeat monitors your import scripts and measures their execution time.

Import challenges

Imports can fail for many reasons:

  • Source file not available or corrupted
  • Database connection error
  • Timeout during large imports
  • Silent failure with incomplete data

The solution: Heartbeat

MoniTao Heartbeat allows you to monitor your imports and be alerted in case of problems.

Integration example

Here is a Python example with heartbeat integration:

# Python example with heartbeat
import requests

HEARTBEAT_TOKEN = "YOUR_TOKEN"
API_SECRET = "YOUR_SECRET"
BASE_URL = "https://monitao.com/api/heartbeat"

def import_data():
    # Signal start
    requests.post(f"{BASE_URL}/start/{HEARTBEAT_TOKEN}",
                  headers={"Authorization": f"Bearer {API_SECRET}"})

    try:
        # Your import logic here
        process_csv_file()

        # Signal success
        requests.post(f"{BASE_URL}/ping/{HEARTBEAT_TOKEN}",
                      headers={"Authorization": f"Bearer {API_SECRET}"})
    except Exception as e:
        # Signal failure
        requests.post(f"{BASE_URL}/fail/{HEARTBEAT_TOKEN}",
                      headers={"Authorization": f"Bearer {API_SECRET}"})
        raise e

Use Cases

  • Daily e-commerce order import
  • CRM-ERP synchronization
  • Supplier product feed retrieval

Configuration

  1. Create a heartbeat job with your import interval
  2. Use start/ping/fail based on import result
  3. Configure alerts appropriate to criticality

Frequently Asked Questions

How to detect an import taking too long?

Use /start at the beginning and /ping at the end. MoniTao will calculate the duration.

Can I be alerted if import exceeds a certain time?

Alerts on excessive duration are planned for a future version.

How to handle imports that can partially fail?

Send /fail if critical errors occur, /ping otherwise.

Can I send the number of imported rows?

Currently heartbeat doesn't accept additional data.

Ready to Sleep Soundly?

Start free, no credit card required.