How PySpark works: lazy evaluation, SQL functions and Parquet, worked examples, and interview questions.
PySpark Coding Exercises 1. Count the Number of Occurrences of Each Word 2. Filter Data Based on a Condition 3. Join Two DataFrames 4. Group By and Aggregate…
Lazy evaluation is a key concept in PySpark (and Spark in general) that refers to the deferred execution of operations until an action is triggered. This…
What is PySpark? PySpark is the Python API for Apache Spark, an open-source distributed computing system. It enables scalable, big data processing through…
PySpark SQL Functions 1. Count the Number of Occurrences of Each Word 2. Filter Data Based on a Condition 3. Join Two DataFrames 4. Group By and Aggregate…
PySpark is the Python API for Apache Spark, an open-source distributed computing system. It enables scalable, big data processing through parallel computing…
PySpark is the Python API for Apache Spark. Your Python program runs on the driver; DataFrame operations such as filter, select, join and groupBy do not process data immediately but add steps to a logical plan. This is lazy evaluation. When an action such as count(), show(), collect() or a write is called, the Catalyst optimizer rewrites the plan (pushing filters down to the Parquet reader, pruning unused columns, choosing join strategies), splits it into stages at shuffle boundaries, and runs one task per partition on the executors. Built-in functions from pyspark.sql.functions run inside the JVM; Python UDFs require shipping rows to Python worker processes and are much slower, so prefer built-ins and use pandas UDFs when custom Python is unavoidable.
| Page | Covers | Read when |
|---|---|---|
| PySpark and Databricks Deep Dive 101 | Architecture, DataFrames, Databricks workflow | You are new to Spark and want the big picture |
| Lazy Evaluation in PySpark | Transformations vs actions, DAGs, caching | Jobs run at surprising times or recompute work |
| PySpark Spark SQL | SQL functions, Parquet reads and writes | You need the function reference and file I/O patterns |
| PySpark SQL Exercises | Word count, filters, joins, aggregations | You want runnable examples to adapt |
| PySpark Questions and Answers | Interview-style concepts | Reviewing before an interview |
A good order is the deep dive first, then lazy evaluation, because understanding when work actually happens explains most performance surprises. The SQL functions and exercises pages are references to come back to while writing code. Data cleaning and streaming topics live in the sibling sections Data Prep and Streaming and I/O.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("demo").getOrCreate()
orders = spark.read.parquet("/data/orders") # nothing read yet
daily = (orders.filter(F.col("status") == "SHIPPED") # transformations: lazy
.groupBy(F.to_date("ts").alias("day"))
.agg(F.sum("amount").alias("revenue")))
daily.explain() # show the optimised plan
daily.write.mode("overwrite").parquet("/data/daily") # action: job runs here