It’s recommended to have access to a Databricks premium workspace to follow along with the code examples at the end of the chapter. It’s also recommended to have Databricks workspace permissions to create an all-purpose cluster and a DLT pipeline using a cluster policy. Users will create and attach a notebook to a cluster and execute the notebook cells. All code samples can be downloaded from this chapter’s GitHub repository, located at https://github.com/PacktPublishing/Building-Modern-Data-Applications-Using-Databricks-Lakehouse/tree/main/chapter01. This chapter will create and run a new DLT pipeline using the Core product edition. As a result, the pipeline is estimated to consume around 5–10 Databricks Units (DBUs).

During the early 1980s, the data warehouse was a great tool for processing structured data. Combined with the right indexing methods, data warehouses allowed us to serve business intelligence (BI) reports at blazing speeds. However, after the turn of the century, data warehouses could not keep up with newer data formats such as JSON, as well as new data modalities such as audio and video. Simply put, data warehouses struggled to process semi-structured and unstructured data that most businesses used. Additionally, data warehouses struggled to scale to millions or billions of rows, common in the new information era of the early 2000s. Overnight, batch data processing jobs soon ran into BI reports scheduled to refresh during the early morning business hours.

At the same time, cloud computing became a popular choice among organizations because it provided enterprises with an elastic computing capacity that could quickly grow or shrink, based on the current...

Spark Structured Streaming provides near-real-time stream processing with fault tolerance, and exactly-once processing guarantees through the use of a DataFrame API that is near-identical to batch processing in Spark. As a result of a common DataFrame API, data engineering teams can convert existing batch Spark workloads to streaming with minimal effort. However, as the volume of data increases and the number of ingestion sources and data pipelines naturally grows over time, data engineering teams face the burden of augmenting existing data pipelines to keep up with new data transformations or changing business logic. In addition, Spark Streaming comes with additional configuration maintenance such as updating checkpoint locations, managing watermarks and triggers, and even backfilling tables when a significant data change or data correction occurs. Advanced data engineering teams may even be expected to build data validation and...

DLT is a declarative framework that aims to simplify the development and maintenance operations of a data pipeline by abstracting away a lot of the boilerplate complexities. For example, rather than declaring how to transform, enrich, and validate data, data engineers can declare what transformations to apply to newly arriving data. Furthermore, DLT provides support to enforce data quality, preventing a data lake from becoming a data swamp. DLT gives data teams the ability to choose how to handle poor-quality data, whether that means printing a warning message to the system logs, dropping invalid data, or failing a data pipeline run altogether. Lastly, DLT automatically handles the mundane data engineering tasks of maintaining optimized data file sizes of the underlying tables, as well as cleaning up obsolete data files that are no longer present in the Delta transaction log (Optimize and Vacuum operations are covered later in the A quick Delta Lake primer...

The DLT framework relies heavily on the Delta Lake format to incrementally process data at every step of the way. For example, streaming tables and materialized views defined in a DLT pipeline are backed by a Delta table. Features that make Delta Lake an ideal storage format for a streaming pipeline include support for Atomicity, Consistency, Isolation, and Durability (ACID) transactions so that concurrent data modifications such as inserts, updates, and deletions can be incrementally applied to a streaming table. Plus, Delta Lake features scalable metadata handling, allowing Delta Lake to easily scale to petabytes and beyond. If there is incorrect data computation, Delta Lake offers time travel – the ability to restore a copy of a table to a previous snapshot. Lastly, Delta Lake inherently tracks audit information in each table’s transaction log. Provenance information such as what type of operation modified the table, by what cluster...

The DLT framework automatically manages task orchestration, cluster creation, and exception handling, allowing data engineers to focus on defining transformations, data enrichment, and data validation logic. Data engineers will define a data pipeline using one or more dataset types. Under the hood, the DLT system will determine how to keep these datasets up to date. A data pipeline using the DLT framework is made up of the streaming tables, materialized views, and views dataset types, which we’ll discuss in detail in the following sections. We’ll also briefly discuss how to visualize the pipeline, view its triggering method, and look at the entire pipeline data flow from a bird’s-eye view. We’ll also briefly understand the different types of Databricks compute and runtime, and Unity Catalog. Let’s go ahead and get started.

Streaming tables

Streaming tables leverage the benefits of Delta Lake and Spark Structured Streaming...

Delta Lake is a big data file protocol built around a multi-version transaction log that provides features such as ACID transactions, schema enforcement, time travel, data file management, and other performance features on top of existing data files in a lakehouse.

Originally, big data architectures had many concurrent processes that both read and modified data, leading to data corruption and even data loss. As previously mentioned, a two-pronged Lambda architecture was created, providing a layer of isolation between processes that applied streaming updates to data and downstream processes that needed a consistent snapshot of the data, such as BI workloads that generated daily reports or refreshed dashboards. However, these Lambda architectures duplicated data to support these batch and streaming workloads, leading to inconsistent data changes that needed to be reconciled at the end of each business day.

Fortunately, the Delta Lake format provides a...

In this section, we’ll use a NYC taxi sample dataset to declare a data pipeline, using the DLT framework, and apply a basic transformation to enrich the data.

Important note

To get the most value out of this section, it’s recommended to have Databricks workspace permissions to create an all-purpose cluster and a DLT pipeline, using a cluster policy. In this section, you will attach a notebook to a cluster, execute notebook cells, as well as create and run a new DLT pipeline.

Let’s start by creating a new all-purpose cluster. Navigate to the Databricks Compute UI by selecting the Compute button from the sidebar navigation on the left side.

Figure 1.7 – Navigate to the Compute UI from the left-hand sidebar

Click the button titled Create compute at the top right. Next, provide a name for the cluster. For this exercise, the cluster can be a small...

In this chapter, we examined how and why the data industry has settled on a lakehouse architecture, which aims to merge the scalability of ETL processing and the fast data warehousing speeds for BI workloads under a single, unified architecture. We learned how real-time data processing is essential to uncovering value from the latest data as soon as it arrives, but real-time data pipelines can halt the productivity of data engineering teams as complexity grows over time. Finally, we learned the core concepts of the Delta Live Tables framework and how, with just a few lines of PySpark code and function decorators, we can quickly declare a real-time data pipeline that is capable of incrementally processing data with high throughput and low latency.

In the next chapter, we’ll take a deep dive into the advanced settings of Delta Live Tables pipelines and how the framework will optimize the underlying datasets for us. Then, we’ll look at more advanced data transformations...