Cron/Job Dead Man's Switch Monitor

Cron/Job Dead Man's Switch Monitor

Lyrinox JobWatch is a self-hosted dead man's switch for scheduled work. It runs as a single Go binary, listens for heartbeat calls from jobs, keeps local JSON state, exposes a status page, returns machine-readable status, emits Prometheus-style metrics, and can trigger webhook or local command alerts when a job misses its expected window.

Use it for work where silence is a failure signal:

• Database and file backups.
• Billing reconciliation.
• Subscription renewal jobs.
• Import and export pipelines.
• Feed pulls and partner syncs.
• Report generation.
• Cleanup and retention tasks.
• Certificate renewal checks.
• Scheduled worker queues.
• CI maintenance jobs.

JobWatch is designed for small production stacks that need real monitoring without adding a large hosted dependency for every scheduled task.

What ships in version 1.0.0

Version 1.0.0 includes:

• lyrinox-jobwatch init for generating a starter JSON config.
• lyrinox-jobwatch serve for running the heartbeat receiver, status page, API, metrics, and alert loop.
• lyrinox-jobwatch check for one-shot health checks from CI, cron, or systemd.
• lyrinox-jobwatch jobs for printing configured heartbeat URLs.
• Lyrinox Market license validation and activation before operational commands run.
• Token-protected heartbeat endpoints.
• Per-job interval and grace windows.
• Persistent local JSON state.
• Duration warnings for jobs that finish too quickly or too slowly.
• Optional from and message fields on heartbeat calls.
• Human-readable status page.
• JSON status API.
• Prometheus-style metrics.
• Webhook alerts.
• Local command alerts.
• Alert cooldowns.
• Optional recovery alerts.

Install

Download the Linux amd64 package from your Lyrinox Market library after purchase. Licensed downloads include the binary plus a lyrinox/ folder containing the license key, product client key, product metadata, and package signature.

Extract the package:

tar -xzf cron-job-dead-man-s-switch-monitor-1.0.0.tar.gz
cd cron-job-dead-man-s-switch-monitor-1.0.0

Place the binary somewhere stable on the host that will run the monitor:

sudo install -m 0755 product/lyrinox-jobwatch-v1.0.0-linux-amd64 /usr/local/bin/lyrinox-jobwatch
sudo mkdir -p /etc/lyrinox-jobwatch /var/lib/lyrinox-jobwatch
sudo cp -R lyrinox /etc/lyrinox-jobwatch/
sudo chmod 0750 /etc/lyrinox-jobwatch

Check the installed version:

lyrinox-jobwatch version

Expected output:

1.0.0

Command reference

JobWatch uses subcommands:

lyrinox-jobwatch init  -config jobwatch.json
lyrinox-jobwatch serve -config jobwatch.json -addr :8088
lyrinox-jobwatch check -config jobwatch.json
lyrinox-jobwatch jobs  -config jobwatch.json
lyrinox-jobwatch version

Quick start

From the extracted package directory, create a config:

sudo lyrinox-jobwatch init \
  -config /etc/lyrinox-jobwatch/jobwatch.json \
  -package-dir /etc/lyrinox-jobwatch

init reads the packaged license files automatically and writes:

• Product ID.
• License key.
• Product client key.
• Market URL.
• Activation domain.
• A long random token for the starter job.

Then edit the job list and set:

• The expected job interval.
• A grace window.
• A writable state file path.
• Alert settings if needed.

Run the server:

lyrinox-jobwatch serve -config /etc/lyrinox-jobwatch/jobwatch.json -addr :8088

On first start, JobWatch activates the license for the configured activation_domain, checks the tier limits, and then starts the heartbeat receiver. If the license cannot be validated, the server does not start.

Open the status page:

http://localhost:8088/status

Print heartbeat URLs:

lyrinox-jobwatch jobs -config /etc/lyrinox-jobwatch/jobwatch.json

How JobWatch decides status

Each job has:

• interval_seconds: how often the job is expected to check in.
• grace_seconds: extra time allowed after the interval.
• A stored last_check_in_at timestamp.

JobWatch computes:

next_expected_by = last_check_in_at + interval_seconds + grace_seconds

The job state is:

The overall report is failed if any job is failed, warning if no jobs are failed but at least one job is warning, and ok otherwise.

Configuration file

JobWatch uses JSON. This makes the config easy to review, generate, and store in private infrastructure repositories.

Starter config:

{
  "listen_addr": ":8088",
  "base_url": "http://localhost:8088",
  "state_file": "/var/lib/lyrinox-jobwatch/state.json",
  "alert_cooldown_seconds": 900,
  "license": {
    "market_base_url": "https://market.lyrinox.com",
    "product_id": 123,
    "license_key": "paste-license-key",
    "product_client_key": "paste-product-client-key",
    "activation_domain": "jobwatch.ops.example.com"
  },
  "jobs": [
    {
      "name": "nightly-backup",
      "token": "replace-with-a-long-random-token",
      "interval_seconds": 86400,
      "grace_seconds": 1800,
      "description": "Nightly database and upload backup",
      "failure_instructions": "Check backup logs, disk space, and last successful restore test.",
      "expected_from": "backup01",
      "minimum_duration_ms": 100,
      "maximum_duration_ms": 900000,
      "alert_webhook_url": "",
      "alert_command": "",
      "alert_on_recovered": true
    }
  ]
}

Top-level fields

License fields

JobWatch is a paid product. The licensed download package includes lyrinox/lyrinox-license-key.txt, lyrinox/lyrinox-product-client-key.txt, and lyrinox/lyrinox-license.json. Copy those values into the config or provide them through environment variables.

Operational commands validate the license before they run:

• serve activates the license for activation_domain, then starts the monitor.
• check validates license status before printing health.
• jobs validates license status before printing heartbeat URLs.

If the subscription is inactive, revoked, expired, over its activation limit, or not valid for this product, JobWatch exits before serving heartbeat routes.

For most buyers, the easiest path is to leave the license files in /etc/lyrinox-jobwatch/lyrinox and let init -package-dir /etc/lyrinox-jobwatch copy the required values into the config. Environment variables are available for containerized deployments or secret managers.

License tiers

JobWatch uses Lyrinox Market activation limits and a local job-count limit to make tiers meaningful:

Choose Solo for one operator monitoring a small stack. Choose Team when several people operate shared infrastructure, when you need multiple JobWatch installations, or when one monitor needs to track more than five scheduled jobs.

If a config exceeds the license's job limit, JobWatch exits with an error such as:

Solo license allows 5 jobs; config has 6 jobs

Job fields

Heartbeat endpoint

Each job reports success with:

GET /ping/:job_name?token=:token

Example:

curl -fsS "http://localhost:8088/ping/nightly-backup?token=$JOBWATCH_TOKEN"

Supported query parameters:

Rejected heartbeats return HTTP 403 with heartbeat rejected.

Accepted heartbeats return a JSON job report.

Adding JobWatch to cron jobs

Call the heartbeat only after the real job succeeds.

#!/bin/sh
set -eu

start_ms=$(date +%s%3N)
/usr/local/bin/run-nightly-backup
duration_ms=$(( $(date +%s%3N) - start_ms ))

curl -fsS "http://localhost:8088/ping/nightly-backup?token=$JOBWATCH_TOKEN&from=$(hostname)&duration_ms=$duration_ms"

Do not put the heartbeat before the job command. A heartbeat before the job only proves that the scheduler started, not that the work completed.

For older systems where date +%s%3N is unavailable, omit duration_ms or calculate duration another way.

Safer shell pattern

This pattern only calls JobWatch after success and keeps the job exit code meaningful:

#!/bin/sh
set -eu

JOB_NAME="nightly-backup"
JOBWATCH_URL="http://localhost:8088/ping/$JOB_NAME"

start=$(date +%s)
/usr/local/bin/run-nightly-backup
elapsed_seconds=$(( $(date +%s) - start ))
duration_ms=$(( elapsed_seconds * 1000 ))

curl -fsS "$JOBWATCH_URL?token=$JOBWATCH_TOKEN&from=$(hostname)&duration_ms=$duration_ms"

If the backup fails, the script exits before the heartbeat.

systemd timer pattern

For a job run by a systemd service or timer, put the heartbeat in ExecStartPost so it only runs after a successful ExecStart.

[Service]
Type=oneshot
EnvironmentFile=/etc/lyrinox-jobwatch/nightly-backup.env
ExecStart=/usr/local/bin/run-nightly-backup
ExecStartPost=/usr/bin/curl -fsS "http://127.0.0.1:8088/ping/nightly-backup?token=${JOBWATCH_TOKEN}&from=%H"

For more detailed duration tracking, wrap the job in a shell script and call the heartbeat from the script after success.

Status page

The human-readable status page is available at:

GET /
GET /status

It shows:

• Current overall status.
• Count of ok, warning, and failed jobs.
• Each configured job.
• Last job message.
• Next expected heartbeat time.
• Job description.
• Failure runbook text.

The status page is intentionally local and compact. Put it behind a VPN, firewall, or authenticated reverse proxy if it is exposed outside the host.

JSON status API

Machine-readable status is available at:

GET /api/status

Example:

curl -fsS http://localhost:8088/api/status

Response shape:

{
  "status": "ok",
  "checked_at": "2026-08-01T18:00:00Z",
  "ok": 1,
  "warning": 0,
  "failed": 0,
  "jobs": [
    {
      "name": "nightly-backup",
      "status": "ok",
      "message": "heartbeat accepted",
      "last_check_in_at": "2026-08-01T17:45:00Z",
      "next_expected_by": "2026-08-02T18:15:00Z",
      "description": "Nightly database and upload backup",
      "failure_instructions": "Check backup logs, disk space, and last successful restore test.",
      "last_duration_ms": 42000,
      "last_from": "backup01"
    }
  ]
}

Use this endpoint for dashboards, scripted checks, and external monitoring systems that can poll HTTP JSON.

One-shot checks

Run:

lyrinox-jobwatch check -config /etc/lyrinox-jobwatch/jobwatch.json

The command prints a compact text report and exits non-zero when any job is failed.

Use JSON output when another tool needs to parse the report:

lyrinox-jobwatch check -config /etc/lyrinox-jobwatch/jobwatch.json -json

Good uses for check:

• CI smoke checks after deployment.
• A second monitoring system that runs commands.
• A systemd health timer.
• A cron entry that pages when JobWatch itself reports failures.

Metrics endpoint

Prometheus-style metrics are available at:

GET /metrics

Metric names:

lyrinox_jobwatch_jobs{status="ok"}
lyrinox_jobwatch_jobs{status="warning"}
lyrinox_jobwatch_jobs{status="failed"}
lyrinox_jobwatch_job_failed{name="job-name"}

Example output:

lyrinox_jobwatch_jobs{status="ok"} 7
lyrinox_jobwatch_jobs{status="warning"} 1
lyrinox_jobwatch_jobs{status="failed"} 0
lyrinox_jobwatch_job_failed{name="nightly-backup"} 0

The per-job failed metric is 1 when that job is failed and 0 otherwise.

Alerts

JobWatch can alert in two ways:

• HTTP webhook.
• Local command.

Both are configured per job.

{
  "name": "nightly-backup",
  "token": "long-random-token",
  "interval_seconds": 86400,
  "grace_seconds": 1800,
  "alert_webhook_url": "https://alerts.example.test/jobwatch",
  "alert_command": "/usr/local/bin/jobwatch-alert",
  "alert_on_recovered": true
}

The alert loop runs while serve is active and evaluates jobs every 30 seconds.

alert_cooldown_seconds controls repeated alerts. If it is set to 900, JobWatch will not send another alert for the same job until at least 15 minutes have passed.

Webhook payload

Webhook alerts send the job report as JSON. The payload includes the job name, status, message, timestamps, description, runbook text, last duration, and last reporting source when available.

Your webhook should return a successful HTTP response. Version 1.0.0 sends the webhook with Content-Type: application/json.

Local alert command

alert_command runs a local executable with three arguments:

alert_command job_name status message

Example script:

#!/bin/sh
set -eu

job_name="$1"
status="$2"
message="$3"

logger -t lyrinox-jobwatch "JobWatch alert: $job_name $status $message"

Make the script executable:

sudo install -m 0755 jobwatch-alert /usr/local/bin/jobwatch-alert

Keep alert scripts fast. If a command blocks for a long time, it can delay alert processing.

Running as a service

Create a dedicated user:

sudo useradd --system --home /var/lib/lyrinox-jobwatch --shell /usr/sbin/nologin lyrinox-jobwatch
sudo mkdir -p /etc/lyrinox-jobwatch /var/lib/lyrinox-jobwatch
sudo chown lyrinox-jobwatch:lyrinox-jobwatch /var/lib/lyrinox-jobwatch
sudo chmod 0750 /var/lib/lyrinox-jobwatch
sudo chmod 0750 /etc/lyrinox-jobwatch

Set config permissions:

sudo chown root:lyrinox-jobwatch /etc/lyrinox-jobwatch/jobwatch.json
sudo chmod 0640 /etc/lyrinox-jobwatch/jobwatch.json

Example unit:

[Unit]
Description=Lyrinox JobWatch
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=lyrinox-jobwatch
Group=lyrinox-jobwatch
ExecStart=/usr/local/bin/lyrinox-jobwatch serve -config /etc/lyrinox-jobwatch/jobwatch.json -addr 127.0.0.1:8088
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/var/lib/lyrinox-jobwatch

[Install]
WantedBy=multi-user.target

Install and start:

sudo systemctl daemon-reload
sudo systemctl enable --now lyrinox-jobwatch
sudo systemctl status lyrinox-jobwatch

View logs:

journalctl -u lyrinox-jobwatch -f

Reverse proxy notes

For a single-host setup, keep JobWatch bound to loopback:

lyrinox-jobwatch serve -config /etc/lyrinox-jobwatch/jobwatch.json -addr 127.0.0.1:8088

Then expose only the routes you need through a reverse proxy.

Recommended policy:

• Allow private jobs to call /ping/:job_name.
• Restrict /status, /api/status, and /metrics to trusted users, VPNs, or monitoring systems.
• Use HTTPS when any heartbeat crosses a network boundary.
• Do not put heartbeat tokens in public documentation, public repositories, browser screenshots, or shared support tickets.

Token management

Heartbeat tokens are bearer secrets. Anyone with the token can mark that job as checked in.

Use these practices:

• Generate a different token for every job.
• Use long random tokens.
• Store tokens in environment files or secret managers, not inline in world-readable scripts.
• Rotate a token by changing the config and updating the job's environment.
• Restart JobWatch after config changes.
• Treat leaked tokens as compromised.

Example token generation:

openssl rand -hex 32

Choosing intervals and grace windows

Start with the job's real schedule:

Use a longer grace window when:

• The job start time naturally drifts.
• The job often waits on remote systems.
• The scheduler has variable queue time.
• You prefer fewer alerts over faster detection.

Use a shorter grace window when:

• Missing the job has customer impact.
• The job schedule is strict.
• Another system already handles retries.

Duration warnings

minimum_duration_ms and maximum_duration_ms let JobWatch flag unusual successful runs.

Examples:

• A backup that finishes in 50 ms might not have backed up anything.
• A reconciliation job that usually takes 2 minutes but reports 45 minutes may be stuck or overloaded.
• An import that runs too quickly may have received an empty feed.

Duration warnings do not mean the heartbeat was rejected. The heartbeat is stored, but the job status becomes warning until a later healthy heartbeat replaces it.

State file behavior

JobWatch stores runtime state in the configured JSON file.

The state includes:

• Last check-in time.
• Last message.
• Last duration.
• Last source.
• Success and failure counters.
• Last alert time.

Back up the state file if historical continuity matters. If the state file is removed, configured jobs return to warning until they receive a new heartbeat.

The process user must be able to create and write the state file.

Configuration validation

At startup, JobWatch rejects:

• Missing config file.
• Invalid JSON.
• Empty jobs.
• Jobs without names.
• Duplicate job names.
• Jobs without tokens.
• Non-positive interval_seconds.
• Negative grace_seconds.

Run a one-shot check after editing the config:

lyrinox-jobwatch check -config /etc/lyrinox-jobwatch/jobwatch.json

If the config is invalid, the command exits with an error before evaluating job state.

Common deployment patterns

One monitor on the same host as the jobs

Run JobWatch on 127.0.0.1:8088. Cron jobs call localhost. This is the simplest and safest pattern.

One monitor for several hosts

Run JobWatch on a private network address or behind a reverse proxy. Each remote job calls its own tokenized heartbeat URL over HTTPS or a trusted private network.

Set from=$(hostname) in every heartbeat so reports show where the heartbeat came from.

Monitor inside a private ops network

Run JobWatch behind VPN-only access. Let workers call /ping/..., let operators view /status, and let Prometheus or a compatible scraper read /metrics.

Troubleshooting

The service starts, but all jobs are warning

No heartbeat has been received yet. Run one monitored job or send a manual test heartbeat:

curl -fsS "http://localhost:8088/ping/nightly-backup?token=$JOBWATCH_TOKEN&from=test"

A heartbeat returns 403

Check:

• The job name in the URL.
• The token value.
• Shell quoting around the URL.
• Whether the config file loaded by the running service is the config you edited.

A job is failed even though cron ran

Confirm the heartbeat runs after the actual job command. If cron starts the job but the job fails before the heartbeat, JobWatch is behaving correctly.

Also check:

• The job took longer than interval_seconds + grace_seconds.
• The host could reach the JobWatch server.
• The curl command failed and cron swallowed the output.
• The token changed but the job script still has the old token.

A job is warning after a successful heartbeat

Look for duration bounds:

• minimum_duration_ms
• maximum_duration_ms

If the job duration is normal, widen the bounds. If the duration is not normal, investigate the job output.

Alerts are not sending

Check:

• alert_webhook_url or alert_command is set for that job.
• The job is actually failed.
• alert_cooldown_seconds has elapsed.
• The service user can execute the local alert command.
• The host can reach the webhook URL.
• The webhook endpoint accepts JSON POST requests.

The state file cannot be written

Check the service user and path:

sudo -u lyrinox-jobwatch test -w /var/lib/lyrinox-jobwatch

Make sure state_file points inside a directory writable by the service user.

/metrics works but Prometheus has no data

Check the scrape target, network path, firewall, and reverse proxy rules. The endpoint returns plain text and does not require a special client.

Operational checklist

Before relying on JobWatch in production:

• Install the binary in a stable path.
• Create a dedicated service user.
• Store config in /etc/lyrinox-jobwatch.
• Keep the packaged lyrinox/ license files somewhere private.
• Store state in /var/lib/lyrinox-jobwatch.
• Run lyrinox-jobwatch init -package-dir ... so license fields are prefilled.
• Use one long random token per job. init generates one for the starter job.
• Put heartbeat calls after successful job completion.
• Test each heartbeat manually once.
• Check /status.
• Check /api/status.
• Check /metrics.
• Trigger one controlled failure to confirm alerting.
• Save runbook text in failure_instructions.
• Keep the config private.

Upgrading

Before upgrading:

1. Download the new version from your Lyrinox Market library when available. 2. Stop the service. 3. Back up the config and state file. 4. Replace the binary. 5. Run lyrinox-jobwatch version. 6. Run lyrinox-jobwatch check -config /etc/lyrinox-jobwatch/jobwatch.json. 7. Start the service. 8. Confirm /status, /api/status, and /metrics.

Version 1.0.0 uses a JSON state file and JSON config. Preserve both files across binary upgrades.

Limits in version 1.0.0

Version 1.0.0 is intentionally small. It does not include:

• Built-in user accounts.
• Built-in TLS termination.
• A database server dependency.
• Built-in notification providers beyond generic webhooks and local commands.
• Multi-tenant account separation.
• A hosted control plane.

Use your reverse proxy, VPN, firewall, or monitoring stack for access control and network policy.

Support scope

Lyrinox support covers:

• The official lyrinox-jobwatch binary distributed through Lyrinox Market.
• Command behavior documented on this page.
• JSON configuration loading and validation.
• Heartbeat endpoint behavior.
• Status API and metrics output.
• Alert webhook and local command invocation behavior.
• Packaging and download issues.

Support does not cover custom job scripts, third-party alert receivers, reverse proxy administration, operating system hardening, or recovery of data that was never backed up.

Related pages

• Cron/Job Dead Man's Switch Monitor product page
• License management for sellers
• Understanding software license types