Powering machine learning workflows with Apache Airflow and Python
3 likes1,461 views
This document provides an overview of using Apache Airflow to power machine learning workflows with Python. It discusses Airflow concepts like DAGs, operators, relationships and visualizations. It also covers installing Airflow, common issues experienced like debugging and versioning, and using Airflow for machine learning tasks like model building and hyperparameter tuning. Examples of Airflow pipelines for data ingestion and machine learning are demonstrated. The presenter's background and the BBC Datalab team are briefly introduced.
![scheduling processes k8s jobs
apiVersion: batch/v1
kind: Job
metadata:
generateName: backup-
namespace: sample
spec:
template:
spec:
containers:
- name: backup
image: alpine:3.9
command: ["sh", "-c", "backup.sh"]
backoffLimit: 3](https://image.slidesharecdn.com/2019-11-16machinelearningworkflowswithapacheairflow-191117171208/85/Powering-machine-learning-workflows-with-Apache-Airflow-and-Python-21-320.jpg)
Powering machine learning workflows with Apache Airflow and Python
- 1. Powering machine learning workflows with Apache Airflow and Python @tati_alchueyr OctopusCon Python Edition Kharkiv, 16 November 2019
- 2. мне приятно быть здесь с тобой большое Вам спасибо Image credit: Turkish Airlines
- 3. tati_alchueyr.__doc__ ● Brazilian living in London since 2014 ● Senior Data Engineer at the BBC Datalab team ● Graduated in Computer Engineering at Unicamp ● Passionate software developer for 16 years ● Experience in the private and public sectors ● Developed software for Medicine, Media and Education ● Loves Open Source ● Loves Brazilian Jiu Jitsu ● Proud mother of Amanda Software InVesalius 3
- 4. Distance from Rio de Janeiro to London: 9,272 km
- 5. Distance from London to Kharkiv: 2,899 km
- 6. help(bbc) ● British Broadcasting Corporation ● Values ○ Independent, impartial and honest ○ Audiences are at the heart of everything we do ○ We take pride in delivering quality and value for money ○ Creativity is the lifeblood of our organisation ○ We respect each other and celebrate our diversity so that everyone can give their best ● Purpose ○ Inform ○ Educate ○ Entertain New Broadcasting House London, UK
- 7. bbc.stats() ➢ BBC TV reaches 91% UK adult population ➢ BBC News reaches 426 million global audience weekly Reference 1: BBC Reference 2: BBC Image Credit: BBC
- 8. bbc. .mission “Bring the BBC’s data together accessible through a common platform, along with flexible and scalable tools to support machine learning to enable content enrichment and deeper personalisation”
- 9. Some of the Datalab team members (15 August 2019) bbc. .mission
- 10. Have you ever built or used a machine learning model? no yes ?
- 11. machine learning application personalisation Image credit: BBC
- 12. machine learning application content creation Image credit: BBCMade by the Machine: when AI met the archive (BBC 4)
- 13. machine learning training & prediction supervised
- 14. machine learning tools jupyter notebooks Image credit: Jupyter.org BBC Datalab ML course
- 15. machine learning tools python scripts
- 16. machine learning tools (remote) servers
- 17. machine learning tools containers Image credit: RedHat
- 18. have you ever scheduled processes? no yes ?
- 20. scheduling processes cron jobs 0 2 * * * /bin/sh backup.sh
- 21. scheduling processes k8s jobs apiVersion: batch/v1 kind: Job metadata: generateName: backup- namespace: sample spec: template: spec: containers: - name: backup image: alpine:3.9 command: ["sh", "-c", "backup.sh"] backoffLimit: 3
- 22. scheduling workflows cron jobs Several cron jobs running... It seems a critical job didn’t run last night... Didn’t it run? Did it fail? Why could it have failed? Original image credit: XKCD
- 24. how much do you know about Apache Airflow? none basic mid high
- 25. airflow Image credit: House Plans
- 26. air(bnb)flow Airflow release blog post by Airbnb https://github.com/apache/airflow Image credit: Airbnb
- 27. airflow why ● Handle complex relationships between jobs ● Handle all the jobs centrally with a well defined user interface ● Error reporting and alerting ● Viewing and analyzing job run times ● Security (protecting credentials of databases)
- 28. airflow why not ● In many cases, cron jobs are the simplest and most effective tool ● Airflow is a complex tool made of several components ○ Learning curve ○ Infrastructure management cost
- 29. airflow concepts (i) DAG ● All workflows are considered to be DAGs ○ DAG: Direct Acyclic Graph nodes direct edge
- 30. DAG not DAG Job A Job B Job C
- 31. DAG not DAG Job A Job B Job C Job D
- 32. DAG not DAG Job A Job B Job C Job D
- 33. DAG not DAG Job A Job B Job C Job D
- 34. DAG not DAG Job A Job B Job C Job D
- 35. airflow concepts (ii) DAG properties ● DAGs (usually) have: ○ schedule ○ start time ○ unique name (ID) ○ nodes: jobs (instances of Operators) ○ edges: dependencies between the nodes
- 36. airflow concepts (iii) operators ● Operators define the task or job ○ BashOperator: execute shell commands/scripts ○ PythonOperator: execute Python code ○ BranchPythonOperator: execute a code if condition ○ SlackOperator ○ (...) ○ Custom operators
- 37. demo Apache Airflow pipeline (example with a Python operator)
- 38. airflow concepts (iv) relationships ● Edges define dependencies ○ When some tasks need to execute one after another Image credit: Airbnb
- 39. airflow concepts (v) connections ● Connections encrypt credentials ○ The jobs do not need to worry about securing credentials Image credit: Airbnb
- 40. airflow concepts (vi) visualisation Image credit: Airbnb
- 41. airflow concepts (vi) visualisation Image credit: Airbnb
- 42. demo Apache Airflow pipeline (example of data ingestion at the BBC)
- 43. inside airflow
- 45. airflow managed service GCP Cloud Composer Image credit: Google
- 46. scars of experience Image credit: XKCD
- 47. scars of experience installing python packages
- 48. scars of experience installing python packages ● When using a Python Operator, the job is run within the worker ● Therefore, by default, Python dependencies are installed globally to the workers ● In other words, application deployments can break your Airflow environment
- 49. scars of experience installing python packages ● Isolate the execution from the scheduling, when reasonable ● To debug native operators means to debug Airflow itself ● Alternatives to isolate them: ○ PythonVirtualenvOperator ○ DockerOperator ○ KubernetesPodOperator ○ GceInstanceStartOperator Interest reading: Medium
- 50. scars of experience debugging There was a breaking change in an Airflow plugin, the scheduler couldn’t process the DAG
- 51. scars of experience debugging The DAG in the worker instances was deleted but its metadata was no longer available in the scheduler
- 52. scars of experience debugging ● Error messages are not always obvious ○ Understand what is happening in the system ○ The webserver and scheduler are independent processes
- 53. scars of experience versioning can be tricky
- 54. scars of experience versioning can be tricky
- 55. scars of experience versioning can be tricky ● Log the version of the Dag Operator and Plugins when they are run ● When catchup is enabled, new jobs will be added to previous executions
- 56. scars of experience using xcom between jobs
- 57. scars of experience using xcom between jobs
- 58. scars of experience using xcom between jobs ● Alternatives ○ By default, the return value of the operator execute method is stored in XCom ○ XCom values are stored in the Airflow metadata DB ○ Avoid using XCom ○ Store the state in data stores (databases, object stores, etc)
- 59. scars of experience scheduling duration
- 60. scars of experience breaking changes ● Minor versions of Airflow can introduce breaking changes ○ Example: named parameter in S3Hook (1.8 -> 1.9) ■ aws_conn_id ■ s3_conn_id Reference: Airflow development mailing list
- 61. where did all the magic of machine learning workflows go? Image credit: XKCD
- 62. have you ever built machine learning pipelines? no yes ?
- 63. Reference: TFX
- 64. Interest reading: Medium airflow machine learning specifics ● Machine learning jobs are similar to usual jobs ● Factors which can affect the operator choice: ○ is the model built using the same Python version? ○ how much CPU and memory does your model need? ○ how can you make Airflow use your existing infrastructure ○ how many concurrent workers do you need? ■ Limitation on scaling celery executors ■ Kubernetes executors in early stage
- 65. demo Apache Airflow pipeline (example of model building pipeline)
- 66. sample DAG model building pipeline
- 67. sample DAG model building pipeline
- 68. sample DAG model building pipeline content
- 69. sample DAG model building pipeline user data
- 70. sample DAG model building pipeline model
- 71. demo Apache Airflow pipeline (example of hyperparameter tuning)
- 72. sample DAG hyperparameter tuning
- 73. getting involved Image credit: XKCD
- 74. airflow install $ pip install apache-airflow
- 77. airflow issue tracker jira https://issues.apache.org/jira/browse?jql=project=AIRFLOW
- 79. is Airflow the right tool for you? Image credit: XKCD
- 81. дуже тобі дякую Спасибо Image credit: Wikipedia Commons @tati_alchueyr