Cron looks like the simplest tool in operations: a time expression and a command. Most scheduled-job incidents come from the gap between "runs at 02:00" and "runs correctly exactly once."
If a job sometimes takes longer than its interval, two copies run at once and fight over the same data. Use a lock: flock -n /tmp/job.lock cmd on a single host, or a database row or lease when jobs run on several machines.
A job scheduled for 02:30 local time will be skipped on the night clocks jump forward and may run twice when they fall back. Schedule in UTC unless the business really needs local time, and if it does, avoid the 01:00 to 03:00 window.
Assume the job will be retried or run twice. Process "records not yet processed" instead of "records from the last 24 hours," and mark progress as you go. Then a rerun after a crash is safe.
Cron does not tell anyone when a job stops running. Add a dead-man's switch: the job pings a monitor on success, and the monitor alerts if no ping arrives within the expected window. Also log start, end, duration, and row counts.
Cron runs with a minimal environment: a short PATH, no shell profile, a different working directory. Use absolute paths, set variables explicitly, and test the exact command with env -i.
If thousands of servers all run a job at minute 0, the shared database suffers. Add random jitter to the start time.
Kubernetes CronJobs (set concurrencyPolicy: Forbid), systemd timers (Persistent=true catches up missed runs), and workflow engines like Airflow or Temporal add retries and visibility.
Ops stories like these make good interview examples. I work on real-time interview help tool Linkjob AI.