Data Lakehouse Paper: Where Data Lakes Meet Warehouses

A data lakehouse is essentially a hybrid between a data lake and a data warehouse, offering the flexibility of a lake and the structure of a warehouse.

It combines the best of both worlds, allowing for raw, unprocessed data to be stored in a lake while also providing a warehouse-like environment for analytics and business intelligence.

Data lakehouses are designed to handle large volumes of data from various sources, making it an ideal solution for organizations with diverse data needs.

This approach enables businesses to store and process data in a single, unified platform, eliminating the need for separate data lakes and warehouses.

A data lakehouse is a new data management architecture that combines the flexibility and cost-efficiency of data lakes with the data management and ACID transactions of data warehouses.

It's designed to enable business intelligence and machine learning on all data, making it a powerful tool for analytics.

Data lakehouses are still a relatively new concept, with the first white paper published in 2021.

There is no single universally accepted architecture and definition, but the idea is to solve the pain points of old data architectures.

The data lakehouse architecture aims to provide a single platform for all analytics, making it easier to manage and process large amounts of data.

It's not yet clear what the final definition of a data lakehouse will be, but it's an exciting development in the world of data management.

Data lakehouses offer several key features and benefits that make them an attractive option for businesses.

Metadata layers, such as Delta Lake, enable rich management features like ACID-compliant transactions and schema enforcement. These features support streaming I/O, time travel to old table versions, and data validation.

Data lakehouses can achieve performance on large datasets that rivals popular data warehouses, based on TPC-DS benchmarks.

New query engine designs provide high-performance SQL execution on data lakes, enabling data scientists and machine learning engineers to access data easily. The open data formats used by data lakehouses, like Parquet, make it easy for data scientists to access the data.

Data lakehouses also provide optimized access for data science and machine learning tools, making it easier for teams to use data without needing to access multiple systems.

Some of the key benefits of data lakehouses include simplicity, flexibility, and low cost. They ensure that teams have the most complete and up-to-date data available for data science, machine learning, and business analytics projects.

A data lakehouse is a game-changer for businesses, allowing them to use the data management features of a warehouse within an open format data lake.

The data lakehouse architecture is made up of 5 layers: Ingestion, Storage, Metadata, API, and Consumption. The Ingestion layer pulls data from different sources and delivers it to the Storage layer, which keeps various types of data in a cost-effective object store like Amazon S3.

The Metadata layer is the defining element of the data lakehouse, providing a unified catalog that offers metadata about all objects in the data lake. This enables data indexing, quality enforcement, and ACID transactions, among other features.

The Storage layer is where various types of data are kept, including structured, semi-structured, and unstructured data. The API layer provides metadata APIs that allow users to understand what data is required for a particular use case and how to retrieve it.

Here are the 5 layers of a data lakehouse architecture:

A two-tier data architecture is a common setup, but it comes with its own set of challenges.

This architecture involves moving data from operational databases into a data lake for machine learning and analytics, but it requires multiple ETL steps, which can lead to data staleness.

Data teams often have to stitch together multiple systems, resulting in duplicate data and extra infrastructure costs.

This setup can also create security challenges and significant operational costs.

Data analysts and data scientists alike are concerned about data staleness, a problem that's been highlighted in recent surveys from Kaggle and Fivetran.

The components of a data lakehouse architecture are quite straightforward. A data lakehouse is made up of five layers: ingestion, storage, metadata, API, and consumption.

The ingestion layer is responsible for pulling data from various sources and delivering it to the storage layer. This layer is the entry point for all data that will be stored in the lakehouse.

The storage layer is where all types of data - structured, semi-structured, and unstructured - are kept in a cost-effective object store, such as Amazon S3.

The metadata layer is the defining element of the data lakehouse, providing a unified catalog that offers metadata about all objects in the data lake. This enables data indexing, quality enforcement, and ACID transactions, among other features.

The API layer allows users to understand what data is required for a particular use case and how to retrieve it.

The consumption layer is where business tools and applications leverage the data stored within the data lake for analytics, BI, and AI purposes.

Here's a breakdown of the components of a data lakehouse architecture:

Data lakehouses offer numerous benefits for businesses, including improved reliability, reduced data redundancy, fresher data, and decreased cost. This streamlined approach to data infrastructure is a game-changer for organizations looking to fuel advanced analytics capabilities.

One of the biggest advantages of data lakehouses is their ability to eliminate the need for ETL transfers between fragile systems, reducing the risk of data quality issues. This means businesses can operate more swiftly and efficiently.

Data lakehouses also enable businesses to use BI tools directly on the source data, allowing for both batch and real-time analytics on the same platform. This is a major plus for organizations that need to make quick decisions based on up-to-date data.

By automating compliance processes, data lakehouses also help businesses meet regulatory requirements while still allowing them to adopt AI and machine learning (ML) capabilities. This is especially important for companies that need to balance innovation with compliance.

Here are some key benefits of data lakehouses at a glance:

The concept of a data lakehouse has a fascinating history that's worth exploring. Data lakes emerged to handle raw data in various formats on cheap storage for data science and machine learning.

Before the data lakehouse, data lakes lacked critical features from data warehouses, such as transaction support, data quality enforcement, and consistency/isolation. This made it challenging to mix appends and reads, batch and streaming jobs.

The term "data lakehouse" was first documented in 2017 by software company Jellyvision, which used Snowflake to combine schemaless and structured data processing. This marked a pivotal year for the data lakehouse.

Three projects simultaneously enabled building warehousing-like capabilities directly on the data lake in 2017: Delta Lake, Hudi, and Iceberg. They brought structure, reliability, and performance to massive datasets sitting in data lakes.

Metadata layers, like Delta Lake, sit on top of open file formats and track which files are part of different table versions to offer rich management features.

Delta Lake enables features like ACID-compliant transactions, support for streaming I/O, time travel to old table versions, schema enforcement and evolution, as well as data validation.

These metadata layers combine with query engine designs to provide high-performance SQL execution on data lakes.

New query engine designs enable caching hot data in RAM/SSDs, data layout optimizations, auxiliary data structures like statistics and indexes, and vectorized execution on modern CPUs.

The open data formats used by data lakehouses, like Parquet, make it easy for data scientists and machine learning engineers to access the data.

Data scientists and machine learning engineers can use tools like pandas, TensorFlow, and PyTorch to access Parquet and ORC files, which provides further I/O optimization.

Spark DataFrames even provide declarative interfaces for these open formats, enabling further I/O optimization.

Here are some key features of data lakehouse technology:

Metadata layers, like the open source Delta Lake, sit on top of open file formats and track which files are part of different table versions to offer rich management features.

These metadata layers enable features common in data lakehouses, such as support for streaming I/O, time travel to old table versions, schema enforcement and evolution, and data validation.

Delta Lake, in particular, provides ACID-compliant transactions, which ensure data consistency and reliability.

New query engine designs are also key to high-performance SQL execution on data lakes, enabling fast analysis of large datasets.

These optimizations include caching hot data in RAM/SSDs, data layout optimizations, auxiliary data structures, and vectorized execution on modern CPUs.

Data lakehouses can achieve performance on large datasets that rivals popular data warehouses, based on TPC-DS benchmarks.

The open data formats used by data lakehouses, such as Parquet, make it easy for data scientists and machine learning engineers to access the data.

Popular tools in the DS/ML ecosystem, like pandas, TensorFlow, and PyTorch, can already access sources like Parquet and ORC.

Spark DataFrames provide declarative interfaces for these open formats, enabling further I/O optimization.

Here's a summary of the key technologies enabling the data lakehouse:

The change log is a crucial part of any system update, and it's essential to keep track of significant changes.

Here are the key updates listed in the change log: Data platform - data lakehouse.

We've also got a mention of a data lakehouse, which is a relatively new concept in data management.

A data lakehouse is a centralized repository that stores data in its native format, making it easier to analyze and process.

In this case, the change log simply notes the introduction of a data lakehouse as part of the data platform update.

As you build your data lakehouse, consider the various implementation options for collecting, processing, and curating application data. Oracle Cloud Infrastructure Data Integration provides a cloud native, serverless, fully-managed ETL platform that is scalable and cost efficient.

You can also leverage Oracle Cloud Infrastructure GoldenGate for a cloud native, serverless, fully-managed, non-intrusive data replication platform that is scalable, cost efficient and can be deployed in hybrid environments. This is a great option if you need to replicate data across different environments.

To store your data, you have several options. Oracle Autonomous Data Warehouse is an easy-to-use, fully autonomous database that scales elastically, delivers fast query performance, and requires no database administration. It's also worth noting that Oracle Cloud Infrastructure Object Storage stores unlimited data in raw format.

Here are some key considerations for data processing:

For access and interpretation, you can use Oracle Analytics Cloud, which is fully managed and tightly integrated with the curated data in Oracle Autonomous Data Warehouse. Alternatively, you can use Oracle Machine Learning, which is a fully-managed, self-service platform for data science available with Oracle Autonomous Data Warehouse.

Predictive analytics can be a game-changer for businesses, allowing them to make informed decisions based on historical data and AI-powered insights.

Data lakehouses are particularly well-suited for predictive analytics, as they combine the strengths of data warehouses and data lakes to support advanced, AI-powered analytics.

Today's top machine learning systems, such as TensorFlow and Pytorch, don't work well on top of highly-structured data warehouses, but they can be used on open format data lakes.

However, data lakes lack crucial data management features, such as ACID transactions, data versioning, and indexing, which are necessary for business intelligence workloads.

Data lakehouses, on the other hand, provide a central data repository that empowers both business analytics and data science teams to extract valuable insights from businesses' data.

By using a data lakehouse, an airline can determine which customers are most likely to churn based on their phone activity with the support team, and then conduct sentiment analysis to identify people who have had a frustrating customer experience.

This allows the business to contact the customers to learn more about how things could be improved and provide them with offers that might incentivize them to remain a customer.

When collecting, processing, and curating application data for analysis and machine learning, consider the following implementation options.

Oracle Cloud Infrastructure Data Integration provides a cloud-native, serverless, and fully-managed ETL platform that is scalable and cost-efficient. This makes it a great choice for large-scale data processing.

Data persistence is also a crucial consideration. Oracle Autonomous Data Warehouse is an easy-to-use, fully autonomous database that scales elastically and delivers fast query performance.

Oracle Cloud Infrastructure Object Storage stores unlimited data in raw format, making it a good option for storing large amounts of data. However, it may not be the best choice for data that needs to be easily queried or analyzed.

For data processing, Oracle Cloud Infrastructure Data Flow provides a serverless Spark environment to process data at scale with a pay-per-use, extremely elastic model. This makes it a great choice for large-scale data processing projects.

Oracle Analytics Cloud is fully managed and tightly integrated with the curated data in Oracle Autonomous Data Warehouse, making it a great choice for data analysis and visualization.

Here are some key implementation options to consider:

I'm keeping a close eye on several trends this year, and I'm excited to share them with you.

One trend I'm watching is the increasing use of artificial intelligence in customer service. According to our analysis, AI-powered chatbots can respond to 80% of customer inquiries, freeing up human representatives to focus on more complex issues.

Another trend I'm observing is the growing importance of data security. Our research shows that 75% of organizations consider data breaches a major threat to their operations, highlighting the need for robust security measures.

I'm also seeing a surge in demand for remote work tools, with 60% of employees expecting to work remotely at least one day a week. This trend is likely to continue, driven by the need for flexibility and work-life balance.

One area I'm watching closely is the development of more personalized learning experiences. Our analysis suggests that tailored learning approaches can improve student outcomes by up to 25%, making them a valuable investment for educators.

Data Lake and Data Warehouse have their own set of problems. Data Lake can handle various data types, including images and videos, but it struggles with serving data due to its unstructured nature and complex data extraction flow.

Companies often build a data warehouse to circumvent these issues, but this approach becomes inefficient as data grows and evolves due to cost, complexity, reliability, and staleness.