Skip to content

What Is a Job Scheduler?

The School Timetable

A school runs on a timetable. Maths is at nine, lunch is at twelve, and the science lab can only start once the teacher has set out the kit. The bell rings on time whether or not anyone is looking at a clock.

A job scheduler is a timetable for computer jobs. It knows what to run, when to run it, what must finish first, and who to tell when something goes wrong.

PyScheduler is a small, from-scratch scheduler in the style of Autosys and Control-M, built to be read and learned from. It runs on our own libraries: pynet (event loop and processes), pyweb (the HTTP API) and pydb (the journalled store).

Try It

The repository has a small nightly ETL chain in examples/etl.yaml: a calendar, a pool, a flaky job that is retried, an on-failure clean-up and previous_business_day run dates. From the repository root:

pysched validate examples/etl.yaml
pysched serve examples/etl.yaml        # leave this running
pysched trigger extract --date 2026-10-16
pysched trigger fetch-rates --date 2026-10-16
pysched status --date 2026-10-16

Or, without starting anything, ask what the example would do:

from pathlib import Path

from pyscheduler.config import load_config

config = load_config(Path("examples/etl.yaml"), env={})
for name, job in config.jobs.items():
    print(name, [dep.job for dep in job.depends_on])
# extract []
# fetch-rates []
# transform ['extract']
# load ['transform', 'fetch-rates']
# cleanup ['load']

Learning Path

Read these in order; each page starts with an everyday picture and ends with something you can run.

  1. The Scheduler Engine (The School Timetable): What the engine does all day: ring the bell, check, start, write down.
  2. YAML (A Recipe Card With Indents): How config files are written and read.
  3. Config Files (The Recipe Book): Turning the YAML into a checked, read-only config.
  4. Templates and Values (Form Letters and Labels): {run_date} placeholders and ${SECRET} variables.
  5. Cron Expressions (The Alarm Clock With Many Dials): When a job fires.
  6. Calendars and Run Dates (School Days and Homework): Business days, holidays and which day a run is for.
  7. Time Zones and DST (The Clocks Change, and the Alarm Still Rings Once): Firing exactly once when the clocks change.
  8. Dependencies (You Can't Ice the Cake Before It's Baked): Waiting for other jobs, and on-failure clean-ups.
  9. Runs and States (The Traffic Lights of a Run): Queued, running, succeeded, failed, blocked...
  10. Retries and Backoff (Knock Again, but Wait a Bit Longer Each Time): Trying again after a failure.
  11. Pools and Limits (Only Two Kids on the Trampoline): How many jobs may run at once.
  12. The Minder (A Babysitter Who Writes a Note When the Job's Done): The small program that runs each attempt.
  13. Clocks and Launchers (A Practice Clock and a Starting Whistle): Real and fake clocks; how attempts are started.
  14. Group Commit (One Trip to the Post Box for Many Letters): Saving many changes in one trip to the disk.
  15. Concurrency (One Waiter, Two Hundred Tables): Proving 200 jobs started together finish fast without the loop stalling.
  16. Recovery and Catch-Up (Waking Up After a Nap and Reading the Babysitter's Notes): Restarting after a crash without losing a run.
  17. Validation (The Teacher Who Checks Your Work Before the Test): Every mistake in a config, found before it runs.
  18. Operator Actions (The Train Controller's Desk): Hold, kill, rerun, force and friends.
  19. Forecast (The Weather Report for Jobs): What will run tomorrow, worked out in advance.
  20. The HTTP API (The Front Desk With Wristbands): Talking to the scheduler over HTTP, with tokens.
  21. The Service and the CLI (Opening the Shop and the Remote Control): pysched serve and the pysched commands.
  22. Notifiers (The Ways to Reach Someone): Log lines, commands and messages that carry an alert to a person.
  23. Routing Alerts (The Post Office With a Pigeonhole per Notifier): Topics, * and #, retries, dead letters and replay.
  24. Alerts (The Smoke Alarm and the Teacher Who Notices You Never Arrived): Every alert kind, must_start_by, max_runtime, min_runtime, dedupe and a job's alerts:.
  25. File Watchers (Waiting at the Letterbox Until the Parcel Stops Rustling): Waiting for a file to arrive and sit still, {files}, give-up, and why no slot is used.
  26. The Web UI (The School Noticeboard): The board, run pages, roles and the seeded demo.
  27. Remote Agents (The Delivery Drivers): pysched-agent workers, agent and agent_tags, lost attempts, remote logs and the two-agent demo.
  28. Heartbeats (The Walkie-Talkie Check-In): How the scheduler knows an agent is still there: agent_grace and lost_after.
  29. Signed Messages (The Wax Seal and the Date Stamp): HMAC seals, fresh dates, one-time nonces, and what the broker still leaves open.
  30. Live Updates (The Radio With Numbered Messages): Server-sent events with ids, replay and reset, and the live log tail.
  31. Graph Layout (The Seating Plan): Layers, barycentres and dashed cross-date edges.
  32. Timelines (The Train Timetable With Ghost Trains): Lanes, bars, ghost bars and the day the clocks change.

Install

pip install fromscratch-pyscheduler

Or get the whole series at once with pip install fromscratch-pystack. To read the code or run the tests, clone it from GitHub.