Structured Streaming and fixing small-file problems on S3.
This section covers how PySpark moves data in and out of a cluster continuously: reading unbounded streams with Structured Streaming, and dealing with the most common side effect of streaming and frequent batch writes on object storage, the small-files problem.
Real-Time Data Processing: PySpark Streaming enables the processing of live data streams, allowing you to handle continuous data input, like logs, sensor…
Structured Streaming treats a stream as an unbounded table. You write the same DataFrame code you would for batch data, and Spark runs it incrementally: on every trigger it reads new records from the source, updates any aggregation state, and writes results to a sink. A checkpoint location stores offsets and state, which, combined with an idempotent or transactional sink such as Delta Lake, gives end-to-end exactly-once results. Event-time windows and watermarks bound how long Spark keeps state for late data.
The cost of low latency is file count. A query that triggers every 10 seconds into Parquet on S3 writes thousands of small files per day per partition. Readers then spend more time listing and opening files than reading data. The compaction page shows how to rewrite many small files into fewer large ones (target sizes in the 128 MB to 1 GB range), and on Delta Lake the same job is done by OPTIMIZE and auto-compaction.
| Question | Page |
|---|---|
| How do I read from Kafka or a folder of files continuously, use windows, watermarks, triggers and checkpoints? | PySpark Data Streaming |
| My S3 table has millions of tiny files and queries are slow. How do I merge them safely? | PySpark S3 Small Files Compaction |
Read them in order: streaming first, because it explains why small files appear, then compaction as the operational fix. If your tables are Delta or Iceberg, prefer the table format's built-in compaction over hand-written rewrite jobs, since those commit atomically and do not break concurrent readers. Related: PySpark and Databricks.