Data Pipelines with Apache Airflow, Second Edition (MEAP 09)

Simplify, streamline, and scale your data operations with data pipelines built on Apache Airflow.

Apache Airflow provides a batteries-included platform for designing, implementing, and monitoring data pipelines. Building pipelines on Airflow eliminates the need for patchwork stacks and homegrown processes, adding security and consistency to the process. Now in its second edition, Data Pipelines with Apache Airflow teaches you to harness this powerful platform to simplify and automate your data pipelines, reduce operational overhead, and seamlessly integrate all the technologies in your stack.

In Data Pipelines with Apache Airflow, Second Edition you'll learn how to:

Master the core concepts of Airflow architecture and workflow design
Schedule data pipelines using the Dataset API and time tables, including complex irregular schedules
Develop custom Airflow components for your specific needs
Implement comprehensive testing strategies for your pipelines
Apply industry best practices for building and maintaining Airflow workflows
Deploy and operate Airflow in production environments
Orchestrate workflows in container-native environments
Build and deploy Machine Learning and Generative AI models using Airflow


Data Pipelines with Apache Airflow has empowered thousands of data engineers to build more successful data platforms. This new second edition has been fully revised to cover the latest features of Apache Airflow, including the Taskflow API, deferrable operators, and Large Language Model integration. Filled with real-world scenarios and examples, you'll be carefully guided from Airflow novice to expert.