Job scheduler

A job scheduler runs tasks at the right time, like “send this reminder at 9am” or “rebuild this report hourly”, across a pool of worker machines. Jobs are stored with their next run time, and workers repeatedly claim jobs that are due. The core difficulty is making sure each job runs once even though workers crash and networks hiccup, and the standard tool is a lease: a worker owns a job only for a limited time, and if it dies without finishing, the lease expires and another worker picks the job up. Since that can still cause an occasional second run, jobs are written to be idempotent, meaning running them twice has the same effect as once.