automation

Airflow: Automate Workflows

Name: Airflow: Automate Workflows
Author: apache

Streamline tasks with Apache Airflow, a workflow automation platform for startups, with 46k+ GitHub stars

45,794 stars17,229 forksPythonQuality 8/10Updated 6/13/2026100% free · open source

What it does

Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows to build scalable data pipelines across ML, analytics, and integration projects.

Install / run

pip install apache-airflow

When to use it

•When you need to manage complex data workflows across multiple systems and teams
•To automate and schedule recurring data tasks, such as data ingestion and processing
•For building and managing machine learning pipelines that require data preparation, training, and deployment

Quick start

1Create a new Airflow project by running `airflow db init` to initialize the database
2Define a DAG (Direct Acyclic Graph) in a Python file, e.g., `my_dag.py`, using the `DefaultArgs` and `DAG` classes from the Airflow library
3Configure the Airflow scheduler to run your DAG by setting the `start_date` and `schedule_interval` parameters
4Run the Airflow web server with `airflow webserver --port 8080` to monitor and manage your workflows
5Trigger a DAG run manually with `airflow trigger_dag my_dag` or schedule it to run automatically

Ready-to-paste prompt

Run a sample DAG with `airflow trigger_dag example_bash_operator` to see how Airflow can execute a series of bash commands

Heads up: Airflow requires a database backend to store its metadata, so you need to configure a database connection, such as PostgreSQL or MySQL, before running the `airflow db init` command

Saves to your device