Building an analytics workflow using Apache Airflow
4 likes2,645 views
This document discusses using Apache Airflow to build an analytics workflow. It begins with an overview of Airflow and how it can be used to author workflows through Python code. Examples are shown of using Airflow to copy files between S3 buckets. The document then covers setting up a highly available Airflow cluster, implementing continuous integration/deployment, and monitoring workflows. It emphasizes that Google Cloud Composer can simplify deploying and managing Airflow clusters on Google Kubernetes Engine and integrating with other Google Cloud services.
![Concept: Directed acyclic graph, operator, task, etc
custom_param_per_dag = {'sg': { ... }, 'eu': { ... }, 'us': { ... }}
for region, v in custom_param_per_dag.items():
dag = DAG('shipment_{}'.format(region), ...)
t1 = PostgresToS3Operator(task_id='db_to_s3', ...)
t2 = S3CopyObjectOperator(task_id='s3_to_s3', ...)
t1 >> t2
globals()[dag] = dag
10](https://image.slidesharecdn.com/copyofpyconapac2019-190223070453/85/Building-an-analytics-workflow-using-Apache-Airflow-10-320.jpg)
![template
t1 = PostgresToS3Operator(
task_id='db_to_s3',
sql="SELECT * FROM shipment WHERE region = '{{ params.region }}'
AND ship_date = '{{ execution_date.strftime("%Y-%m-%d") }}'",
bucket=default_args['source_bucket'],
object_key='{{ params.region }}/{{
execution_date.strftime("%Y%m%d%H%M%S") }}.csv',
params={'region':region},
dag=dag) 11](https://image.slidesharecdn.com/copyofpyconapac2019-190223070453/85/Building-an-analytics-workflow-using-Apache-Airflow-11-320.jpg)
Building an analytics workflow using Apache Airflow
- 1. Building an Analytics Workflow using Apache Airflow Yohei Onishi PyCon APAC 2019, Feb. 23-24 2019
- 2. Presenter Profile ● Yohei Onishi ● Twitter: legoboku, Github: yohei1126 ● Data Engineer at a Japanese retail company ● Based in Singapore since Oct. 2018 ● Apache Airflow Contributor 2
- 3. Session overview ● Expected audiences: Data engineers ○ who are working on building a pipleline ○ who are looking for a better workflow solution ● Goal: Provide the following so they can use Airflow ○ Airflow overview and how to author workflow ○ Server configuration and CI/CD in my usecase ○ Recommendations for new users (GCP Cloud Composer) 3
- 4. Data pipeline data source collect ETL analytics data consumer micro services enterprise systems IoT devices object storage message queue micro services enterprise systems BI tool 4
- 5. Our requirements for ETL worflow ● Already built a data lake on AWS S3 to store structured / unstructured data ● Want to build a batch based analytics platform ● Requirements ○ Workflow generation by code (Python) rather than GUI ○ OSS: avoid vendor lock in ○ Scalable: batch data processing and workflow ○ Simple and easily extensible ○ Workflow visualization 5
- 6. Another workflow engine: Apache Nifi 6
- 7. Airflow overview ● Brief history ○ Open sourced by Airbnb and Apache top project ○ Cloud Composer: managed Airflow on GCP ● Characteristics ○ Dynamic workflow generation by Python code ○ Easily extensible so you can fit it to your usecase ○ Scalable by using a message queue to orchestrate arbitrary number of workers 7
- 8. Example: Copy a file from s3 bucket to another export records as CSV Singapore region US region EU region transfer it to a regional bucket 8 local region
- 9. DEMO: UI and source code sample code: https://github.com/yohei1126/pycon-apac-2019-airflow-sample 9
- 10. Concept: Directed acyclic graph, operator, task, etc custom_param_per_dag = {'sg': { ... }, 'eu': { ... }, 'us': { ... }} for region, v in custom_param_per_dag.items(): dag = DAG('shipment_{}'.format(region), ...) t1 = PostgresToS3Operator(task_id='db_to_s3', ...) t2 = S3CopyObjectOperator(task_id='s3_to_s3', ...) t1 >> t2 globals()[dag] = dag 10
- 11. template t1 = PostgresToS3Operator( task_id='db_to_s3', sql="SELECT * FROM shipment WHERE region = '{{ params.region }}' AND ship_date = '{{ execution_date.strftime("%Y-%m-%d") }}'", bucket=default_args['source_bucket'], object_key='{{ params.region }}/{{ execution_date.strftime("%Y%m%d%H%M%S") }}.csv', params={'region':region}, dag=dag) 11
- 12. Operator class PostgresToS3Operator(BaseOperator): template_fields = ('sql', 'bucket', 'object_key') def __init__(self, ..., *args, **kwargs): super(PostgresToS3Operator, self).__init__(*args, **kwargs) ... def execute(self, context): ... 12
- 13. HA Airflow cluster executor (1..N) worker node (1) executor (1..N) worker node (2) executor (1..N) worker node (1) ... scheduler master node (1) web server master node (2) web server LB admin Airflow metadata DBCelery result backend message broker 13 http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/
- 14. CI/CD pipeline AWS SNS AWS SQS Github repo raise / merge a PR Airflow worker polling run Ansible script git pull test deployment 14
- 15. Monitoring Airflow worker (EC2) AWS CloudWatch notify an error if DAG fails using Airflow slack webhook notify an error if a CloudWatch Alarm is triggered slack webhook 15
- 16. GCP Cloud Composer ● Fully managed Airflow cluster provided by GCP ○ Fully managed ○ Built in integrated with the other GCP services ● To focus on business logic, you should build Airflow cluster using GCP composer 16
- 17. Create a cluster using CLI $ gcloud composer environments create ENVIRONMENT_NAME --location LOCATION OTHER_ARGUMENTS ● New Airflow cluster will be deployed as Kubenetes cluster on GKE ● We usually specify the following options as OTHER_ARGUMENTS ○ infra: instance type, disk size, VPC network, etc. ○ software configuration: Python version, Airflow version, etc. 17
- 18. Deploy your source code to the cluster $ gcloud composer environments storage dags import --environment my-environment --location us-central1 --source test-dags/quickstart.py ● This will upload your source code to cluster specific GCS bucket. ○ You can also directly upload your file to the bucket ● Then the file will be automatically deployed 18
- 19. monitoring cluster using Stackdriver 19
- 20. Demo: GCP Cloud Composer ● Create an environment ● Stackdriver logging ● GKE as backend 20
- 21. Summary ● Data Engineers have to build reliable and scalable data pipeline to accelate data analytics activities ● Airflow is great tool to author and monitor workflow ● HA Airflow cluster is required for high availablity ● GCP Cloud Compose enables us to build a cluster easily and focus on business logic 21
- 22. References ● Apache Airflow ● GCP Cloud Composer ● Airflow: a workflow management platform ● ETL best practices in Airflow 1.8 ● Data Science for Startups: Data Pipelines ● Airflow: Tips, Tricks, and Pitfalls 22