Airflow

Airflow is the open-source platform for programmatically authoring, scheduling, and monitoring complex data workflows (DAGs).

Airflow is your mission control for workflow orchestration, initially developed at Airbnb in 2014. It operates on the principle of 'configuration as code,' defining workflows as Directed Acyclic Graphs (DAGs) written in Python. This approach ensures pipelines are maintainable, versionable, and testable. The platform's core components (Scheduler, Executor, Web UI) handle scheduling, distributed execution across workers, and real-time monitoring. It provides robust, plug-and-play operators for integration with services like AWS, GCP, and Azure, making it highly scalable for batch-oriented ETL, ELT, and MLOps pipelines.

https://airflow.apache.org/