Job Schedulers

APScheduler schedules your Python code to run later, once or periodically. It runs inside your existing application, not as a service. Give each job a trigger: date runs it once, interval at fixed intervals, and cron at set times of day. Jobs live in memory by default. When they must survive restarts and crashes, add a persistent job store.

schedule is an in-process scheduler for periodic jobs, with no extra processes and no dependencies. Write schedule.every(10).minutes.do(job), then call schedule.run_pending() in a loop. Its docs call it a simple solution for simple scheduling problems, and say to look elsewhere when jobs must persist between restarts or run concurrently.

Airflow is a platform for orchestrating batch workflows. Its docs say workflows with a clear start and end that run on a schedule are a great fit. It comes with a wide range of built-in operators for integrating other technologies. It's a set of services: a minimal install runs a scheduler, a processor that parses your workflow files, and an API server with the UI. It also needs a metadata database, usually PostgreSQL or MySQL. Write tasks with the TaskFlow API: decorate plain Python functions, and Airflow creates the tasks, wires their dependencies, and passes data between them.

Prefect turns your Python functions into data pipelines, with no DSLs or complex config files. Put @flow on your script's entrypoint and @task on each function it calls. Prefect tracks each task's state, so a failed run can resume from its point of failure. With the open-source server running, call .serve() on your flow with a cron schedule: it starts a process that runs the flow on that schedule. Its docs call serving simple to reason about for flows on a machine you control.

Dagster is a data orchestrator built for data engineers, with lineage and observability built in. You declare data assets like tables, datasets, and ML models as Python functions. Dagster runs them at the right time to keep them up to date. If you're just starting out, its docs strongly recommend assets rather than ops. Run assets on a cron schedule.

In an orchestrator, make every task safe to run twice. Airflow can retry a failed task, so its docs say a task should produce the same outcome on every re-run. Prefect tasks are retryable units of work too.