Medallion Architecture

Bronze Layer - Medallion Architecturemedallion architectureRaw sources fan in through the ingestion engines and land as append-only Delta tables — never modified after… Gold Layer: Medallion Architecture in Databricksmedallion architectureGold tables fan out through purpose-built serving surfaces (materialized views, vector search, SQL endpoints,… Medallion Architecture with Partitioningmedallion architectureDataset containing three columns: category , order_id , and sales_amount . Silver Layer: The Conformed Data Foundationmedallion architectureThe Silver layer is the cleansed, deduplicated, and conformed data tier of the Medallion Architecture.

Storing Data in the Bronze Layer

Medallion Architecture Best Practices

Maintain the data in its original format (e.g., JSON, Parquet, Excel) to ensure traceability and auditability.
Storing data as-is ensures data completeness and prevents potential data loss from premature transformations.
Provides flexibility for data adjustments and future transformations without the need for re-ingestion.

Benefits of Keeping Original Format:

Traceability and Auditability: Original data offers a clear audit trail for data integrity checks.
Flexibility: Allows for modifications to data processing logic without re-ingesting data.
Data Completeness: Preserves all variations and details from source systems.

Cost Savings

Reduced Storage Costs: Storing data incrementally (Bronze, Silver, Gold) minimizes storage costs by reducing the need to retain multiple versions or raw data copies. Only essential transformations are stored in the Gold layer, conserving storage space.
Optimized Resource Usage: Data processing tasks can be tailored to each layer, allowing for efficient resource allocation. For instance, resource-intensive tasks can be confined to specific layers, reducing cloud costs for computing and storage.
Lower Maintenance Costs: By standardizing data processing and quality checks across layers, maintenance overhead is reduced. This decreases the need for frequent data reconciliations and error correction, cutting down operational expenses.

Overview of Layers

Bronze Layer (Raw Data): Stores unprocessed data from various sources in its original state, preserving data fidelity. This serves as a single source of truth, allowing for reprocessing and auditing when needed.
Silver Layer (Cleansed Data): Data from the Bronze layer is cleaned, validated, and transformed. This intermediate layer improves data quality and ensures that only reliable data moves downstream, reducing the need for extensive processing in later stages.
Gold Layer (Curated Data): Contains refined, aggregated data tailored to specific business needs, such as analytics and reporting. This layer is aligned with business logic and optimized for high-performance queries and dashboards.

Key Benefits

Data Quality: Incremental processing through each layer enhances data reliability, reducing the need for costly data cleansing operations later in the workflow and enabling more accurate analytics and machine learning applications.
Scalability: The structured approach efficiently handles large data volumes. As the data grows, the architecture scales without requiring significant infrastructure investment, lowering overall data management costs.
Flexibility: Different teams can work on separate layers simultaneously, enabling parallel processing and faster data pipelines. This reduces development time, lowering operational costs associated with waiting on dependencies and rework.
Enhanced Data Governance: By having distinct layers, the architecture supports robust data governance practices, including access controls, data lineage tracking, and compliance measures. This approach reduces costs associated with regulatory compliance and minimizes the risk of expensive data breaches.

In summary, the Medallion Architecture not only enhances data quality, scalability, and governance but also results in considerable cost savings by optimizing data storage, processing, and resource allocation across the entire data pipeline.

Example Code for Data Ingestion in Python

# Ingesting raw data into the Bronze layer
import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BronzeLayerIngestion").getOrCreate()

# Read data from different formats
json_df    = spark.read.json("/path/to/source/data.json")
parquet_df = spark.read.parquet("/path/to/source/data.parquet")
excel_df   = spark.read.format("com.crealytics.spark.excel") \\
    .option("useHeader", "true") \\
    .load("/path/to/source/data.xlsx")

# Write data to Bronze layer
json_df.write.format("delta").mode("append").save("/path/to/bronze/json")
parquet_df.write.format("delta").mode("append").save("/path/to/bronze/parquet")
excel_df.write.format("delta").mode("append").save("/path/to/bronze/excel")

SQL Query for Verifying Bronze Data

-- Verify data ingestion in the Bronze layer
SELECT COUNT(*) AS record_count, source_format
FROM bronze_layer_table
GROUP BY source_format;

Expected Output

| record_count | source_format |
|--------------|---------------|
|    5000      |   JSON        |
|   12000      |   Parquet     |
|     800      |   Excel       |