Real-time IoT streaming pipeline — Bronze/Silver/Gold medallion on PySpark Structured Streaming

Context

IoT sensor-telemetry workloads have a shape that batch pipelines handle badly: events arrive continuously, validation has to happen per-record, and downstream aggregations need windowing rather than full-dataset scans. The build is a proof that the lakehouse medallion architecture extends cleanly to the streaming case.

Problem

A streaming pipeline that mixes ingestion, enrichment, and aggregation into one notebook is impossible to debug. Per-record validation gets tangled with cross-event windowing; cross-source joins fight schema drift; the whole graph becomes hard to inspect when something silently goes wrong.

Approach

A Bronze / Silver / Gold medallion architecture applied to the streaming case. Bronze ingests raw JSON sensor records as an append-only, schema-aware stream. Silver enriches each event with metadata and runs per-record quality validation. Gold materializes 1-minute windowed aggregations using stream-stream joins for cross-source enrichment. Four independently-runnable terminals (generator + the three medallion layers) so each stage can be debugged in isolation.

Solution

• Data generator emitting JSON sensor records to simulate live IoT emissions for local development.
• Bronze layer with raw JSON-stream ingestion, append-only schema-aware writes.
• Silver layer with metadata enrichment and per-record quality validation.
• Gold layer producing 1-minute windowed aggregations with stream-stream joins across the enriched event streams.
• validate_pipeline.py validation harness covering the three layers independently.
• Pinned Python dependencies in requirements.txt for reproducibility.
• Setup documentation for Apache Spark 3.x, Java 11 or 17, and Hadoop winutils for Windows with PowerShell-friendly env-var configuration.
• Four-terminal run pattern (generator, Bronze, Silver, Gold) so each stage is independently debuggable.

Outcome

Above. The medallion-streaming pairing is the methodology contribution: the same Bronze / Silver / Gold pattern used in modern lakehouse deployments, applied here as a working proof on a realistic IoT telemetry workload. Reusable as a reference for any live-enrichment + windowed-aggregation pipeline.