Open core summit: Observability for data pipelines with OpenLineage
3 likes1,267 views
This document discusses Open Lineage and the Marquez project for collecting metadata and data lineage information from data pipelines. It describes how Open Lineage defines a standard model and protocol for instrumentation to collect metadata on jobs, datasets, and runs in a consistent way. This metadata can then provide context on the data source, schema, owners, usage, and changes. The document outlines how Marquez implements the Open Lineage standard by defining entities, relationships, and facets to store this metadata and enable use cases like data governance, discovery, and debugging. It also positions Marquez as a centralized but modular framework to integrate various data platforms and extensions like Datakin's lineage analysis tools.
![01 Job
v1
{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[],
...
}
LINEAGE
JOBDATASET
Marquez: Metadata collection](https://image.slidesharecdn.com/opencoresummit-observabilityfordatapipelineswithopenlineage-201218231602/85/Open-core-summit-Observability-for-data-pipelines-with-OpenLineage-28-320.jpg)
![{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[],
...
}
JOBDATASET
Marquez: Metadata collection
02 Job
v2
{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[{
"namespace":"datascience",
"name":"room_bookings_aggsâ
}],
...
}
LINEAGE
LINEAGE
01 Job
v1](https://image.slidesharecdn.com/opencoresummit-observabilityfordatapipelineswithopenlineage-201218231602/85/Open-core-summit-Observability-for-data-pipelines-with-OpenLineage-29-320.jpg)
Open core summit: Observability for data pipelines with OpenLineage
- 1. Observability for data pipelines with Open Lineage Julien Le Dem CTO & Co-Founder Datakin @J_
- 2. AGENDA Open Lineage and Marquez Why metadata? Community
- 4. Need to create a healthy data ecosystem
- 5. Team interdependencies Team A Team B Team C
- 6. DATA â What is the data source? â What is the schema? â Who is the owner? â How often is it updated? â Where is it coming from? â Who is using the data? â What has changed? Today: Limited context
- 7. Maslowâs Data hierarchy of needs New Business Opportunities Business optimization Data Quality Data Freshness Data Availability
- 8. Open Lineage
- 9. Problem Today: â Duplication of effort: Each project has to instrument all jobs â Integrations are external and can break with new versions â Effort of integration is shared â Integration can be pushed in each project: no need to play catch up With Open Lineage
- 10. Purpose - Open standard for metadata and lineage collection - Instrument jobs as they are running - DeďŹne a generic model of job/dataset/runs entities - Consistent naming strategies for jobs and datasets - DeďŹne speciďŹc facets that can enrich those entities
- 11. Projects involved in Open Lineage (so far)
- 12. Open Lineage scope Not in scope BackendIntegrations Metadata and lineage collection standard Warehouse Schedulers ... Kafka topic Graph db HTTP client Consumers Kafka client GraphDB client ...
- 13. Core Model
- 14. Consistent naming: - Jobs: Example: scheduler.job.task - Datasets: Example: instance.schema.table Core Model
- 15. Facets Facets are atomic pieces of metadata identiďŹed by a unique name that can be attached to the core entities. PreďŹxes in facet names allow the deďŹnition of Custom facets that can be promoted to the spec at a later point.
- 16. Facet examples Dataset: - Stats - Schema - Version - Column level lineage Job: - Source code - Dependencies - params - Source control - Query plan - Query profile Run: - Schedule time - Batch id
- 17. Protocol - Asynchronous events - unique id for identifying a run and correlate events - ConďŹgurable backend - Kafka - Http - ...
- 18. Lifecycle - Create unique run id - Run start event - Send plan/proďŹle info - Run complete event - Send output Dataset version updates
- 19. Join the conversation Github: https://github.com/OpenLineage Slack: OpenLineage.slack.com Email: https://groups.google.com/g/openlineage
- 23. Metadata: Ingest Storage Compute StreamingBatch/ETL â Data Platform built around Marquez â Integrations â Ingest â Storage â Compute Flink Airflow Kafka Iceberg / S3 BI
- 24. Marquez: Data model Job Dataset Job Version Run * 1 * 1 * 1 1* 1* Source 1 * â MYSQL â POSTGRESQL â REDSHIFT â SNOWFLAKE â KAFKA â S3 â ICEBERG â DELTALAKE â BATCH â STREAM â SERVICE Dataset Version
- 25. v1 v4Dataset v2 v4 v4 Job v1 Dataset v4 Job v2 Marquez: Data model â Debugging â What job version(s) produced and consumed dataset version X? â BackďŹlling â Full / incremental processing Design beneďŹts
- 26. â Centralized metadata management â Sources â Datasets â Jobs â Modular framework â Data governance â Data lineage â Data discovery + exploration Metadata Service Marquez: Design Marquez Core Lineage Search REST API ETL Batch Stream
- 27. Extensions datakin Lineage analysis Lineage collectionAPIs Integrations Client - side Metadata Core DB Graph Storage Marquez UI Listener Core API
- 29. { "type":"BATCH", "name":"room_bookings_7_daysâ "inputs":[{ "namespace":"datascience", "name":"room_bookingsâ }], "outputs":[], ... } JOBDATASET Marquez: Metadata collection 02 Job v2 { "type":"BATCH", "name":"room_bookings_7_daysâ "inputs":[{ "namespace":"datascience", "name":"room_bookingsâ }], "outputs":[{ "namespace":"datascience", "name":"room_bookings_aggsâ }], ... } LINEAGE LINEAGE 01 Job v1
- 30. Marquez API â Open Lineage and Marquez standardize metadata collection â Job runs â parameters â version â inputs / outputs â Datakin enables â Understanding operational dependencies â Impact analysis â Troubleshooting: What has changed since the last time it worked? Datakin leverages Marquez metadata Datakin Lineage analysis Graph Integrations
- 31. Community
- 33. Neutral â Not controlled by a company â Community driven Community â Build trust â Grow adoption â Everybody is on an equal footing Governance â Decision mechanisms â Becoming a maintainer â Code of Conduct Part of the LF AI & Data foundation
- 35. Thank You