Data Preparation

Cleaning data in PySpark: missing values, data-cleaning patterns and pivot tables.

Data preparation is the work between raw ingestion and analysis: fixing nulls, standardising types and values, removing duplicates and reshaping tables. The pages here show how to do each step in PySpark at scale.

PySpark Handling Missing Data

PySpark is the Python API for Apache Spark, an open-source, distributed computing system designed for processing large-scale data. PySpark enables Python…

Pivot Table Overview

A pivot table is a tool used for summarizing data, allowing you to group and aggregate information based on categorical columns. In the context of PySpark, a…

PySpark Data Cleaning

Data cleaning is the work that turns a Bronze layer (raw, append-only) into a Silver layer (validated, conformed, deduplicated) in a Medallion architecture…

1. Overview

In a Medallion architecture, raw data lands in a Bronze layer exactly as received. Preparation produces the Silver layer: records with correct types, consistent codes, no exact or business-key duplicates, and a deliberate policy for missing values. Reshaping, such as pivoting rows of (date, product, amount) into one column per product, usually comes last because it depends on clean category values.

All three pages use the DataFrame API, so the logic runs distributed without Python UDFs where possible. Built-in functions (na.fill, dropDuplicates, pivot) are executed in the JVM and optimised by Catalyst, which is much faster than row-by-row Python.

2. Which Technique When

Problem	Technique	Page
Nulls in numeric or categorical columns	dropna, fillna, Imputer	Handling Missing Data
Bad types, whitespace, duplicates, invalid rows	cast, trim, dropDuplicates, quarantine	Data Cleaning
Long table needs to become wide for reporting	groupBy().pivot().agg()	Pivot Table

A practical order is: clean types and values first, then decide on missing values (because casting can create new nulls), then deduplicate, then reshape. See also PySpark.

2. Common Pitfalls

Filling nulls with zero by default, which silently changes averages and counts; choose a fill strategy per column.
Calling pivot() without the list of values, which forces an extra pass to discover distinct values and can explode the column count.
Using dropDuplicates() with no subset when duplicates differ only in ingestion timestamp; deduplicate on business keys and keep the latest row with a window function.
Python UDFs for simple string fixes that built-in functions already provide.