Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh

Rakesh Suresh
Jainik Vora
Transforming Data Processing with
Kubernetes: Journey Towards a
Self-Serve Data Mesh
Nov 6, 2023

© 2023 Intuit Inc. All rights reserved. 2
Speaker Introduction
Jainik Vora
Sr. Staff Software Engineer
Rakesh Suresh
Sr. Staff Software Engineer
jainiksvora
rakeshsuresh

3
© 2023 Intuit Inc. All rights reserved.
Agenda
Intuit
About Intuit and its mission
Data Mesh
What is Data Mesh and the problems it addresses
Data Lake & Data Mesh
How Intuit implements data mesh with a real example
Intuit’s Data Mesh Concepts
Foundational concepts deﬁned for Data Mesh
Self-Serve Data Processing on Kubernetes
Architecture of Batch and Stream Processing Platform

©2022 Intuit Inc. All rights reserved. 4
100%
services on
Modern SaaS
65B
machine learning
predictions per day
24k
ﬁnancial institutions
[+50 crypto]
$560B
money moved
3.6B
requests during
peak season (no
customer failures)
Data
Integration
Fintech
Infrastructure
Identity
AI
Infrastructure
Modern Dev
Experience
AI-driven expert platform
Intuit is leading the way in building an AI-native development platform using cloud native open source
technology. We’re committed to building tools that scale and giving back to the open source community.

©2022 Intuit Inc. All rights reserved.
We believe in open source
and open collaboration
bit.ly/intuit-oss
Created, open-sourced,
used, and maintained
by Intuit
Recipient of the
End User Award
in 2019 & 2022
End user of Cloud
Native and mobile
open source tech

Data Mesh

What is Data Mesh?

A data mesh is a decentralized data
architecture that organizes data by a
speciﬁc business domain.

Instead of data acting as a
by-product of a process, it becomes
the product, where data producers
act as data product owners.

Why Data Mesh?
Improve value
of Data
Smart Product
Experiences
using Data
Power AI
Power Generative AI Applications
like Intuit Assist
Reduce time
to discover &
access Data
Serve variety
of Data
Personas

Data Mesh Principles
Lorem ipsum
congue
Data Mesh
Domain
Driven
Ownership
Data
Product
Data Access
Self Serve
Infrastructure

Data Lake & Data
Mesh

Small Business Owner has unpaid
Invoices
The small business owner logins to quickbooks and
realizes there are unpaid invoices from customers.
Options provided by the system:
◆ (A) The system reminds the owner about unpaid invoices
◆ (B) The system also offers an add-on feature that
automatically sends invoice reminders to customers
Small Biz & Quickbooks

Invoice Business Technical Requirement
❖ Notiﬁcation: Track &
Remind Business
Owner and their
Customer
❖ Get unpaid invoices
by Business
❖ Get unpaid invoices
for each Customer
grouped by
Business

Traditional Data Lake Architecture

Traditional Data Lake Architecture for Invoice

Invoice is our Data Domain

Lets ask some questions on Invoice Data..
◆ How do I ﬁnd Invoice data for my use
case?
◆ Who is the domain expert for Invoice
data?
◆ What is the schema of the Invoice data?
◆ Where is Invoice data located for
consumption?
◆ How can I get access to Invoice data
and who can approve?
◆ Is there derived data from Invoice? How
do I derive data from Invoice?

Data Mesh
Concepts for Intuit

Organization & Discovery of Data
How do I ﬁnd Invoice data for my use case?
Data Map
Organization of data using domain, sub-domain and
bounded context

Organization & Discovery of Data
How do I ﬁnd Invoice data for my use case?
Data Map
Organization of data using domain, sub-domain and
bounded context
Data Product
Foundational unit of data map, organized by data map

Ownership of Data
Who is the domain expert for Invoice data?
Data Product
Consolidates essential information to enable data
consumers
Data Steward
Deﬁnes the data product and responsible for its contract

Data Contract
What is the schema of the Invoice data?
Semantic Model
Consolidates essential modeling and schema
information enabling data consumers understand the
data
SLA
Deﬁnes the data product and responsible for its contract
like data quality, data freshness etc…

Data Ports
Where is Invoice data located for consumption?
Data Assets
Provides location and medium through which data can
be consumed
Tag
Additional context for optimal discovery

Access & Governance
How can I get access to Invoice data and who can approve?
Access Control List
Track explicit read and write access control
Access approved by Data Steward

Data Concepts - Data Processing & Lineage
Is there derived data from Invoice? How do I derive data from Invoice?

Self-Serve
Data Processing on
Kubernetes

Scope of Data Processing At Intuit
Scale
2000+ users
100,000+ pipelines (batch
and streaming)
Variety of Users
Data Engineers
Data Scientists
Machine Learning Engineers
Data Analysts
Variety of Use Cases
Type
◆ Batch
◆ Streaming
Categories
◆ Model Training & Feature Gen
◆ Derivation & Enrichments
◆ Data Movement

Self Serve Data Processing as Paved Path
Operate & Monitor
◆ Log forwarding & Metrics
reporting
◆ Alert & Notiﬁcation
◆ DR Failover & Failback
Provision & Deploy
◆ Infrastructure provisioning
◆ Deployment of processing
artifacts
◆ Data Map Registration
◆ Lineage
Author & Deﬁne
◆ Authoring tools geared towards
user persona and expertise
◆ Access to input and output
◆ Scheduling & Orchestration

Batch Processing Platform Architecture

Stream Processing Platform Architecture

Kubernetes Power Data Processing
Intuit Kubernetes Service
◆ Core Infrastructure Layer
◆ Runs
– Control plane APIs
– Processing jobs
Argo Workﬂow & Events
◆ Scheduling & Orchestration
◆ Deployment Workﬂow

Learn more
Data Mesh
◆ Data Mesh Principles & Logical Architecture by Zhamak Dehghani
◆ Intuit’s Data Mesh Strategy
◆ Intuit’s Data Mesh Concepts
Data Processing
◆ How Intuit Built Stream Processing Platform with Flink
◆ Large Scale Batch Processing with Argo Workﬂow and Events

Q&A

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh

More Related Content

Similar to Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh (20)

More from DoKC (20)

Recently uploaded (20)

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh