Hadoop MapReduce batch analytics — HDFS Streaming pipeline in Python

Context

Modern lakehouse work doesn’t replace the need to know on-prem big-data tooling: buyers in regulated or air-gapped environments still ask whether the team can ship to a Hadoop cluster they already operate. The pipeline is the proof that the foundational MapReduce pattern is intact and runnable end-to-end.

Problem

A MapReduce job authored in Java forces a JVM dependency on every node and slows iteration. A Streaming job in Python (stdin/stdout piping) keeps the cluster Java-only on the runtime side and lets the analytics logic live in plain scripts that any data engineer can read.

Approach

A three-stage Hadoop Streaming pipeline: mapper emits per-record tuples, combiner does local-node aggregation before the network shuffle, reducer produces final cross-key aggregations. Pure stdin/stdout piping, no Hadoop-Java dependencies. Same scripts run on a pseudo-distributed single-node install for local development and on a full multi-node cluster for production-style runs.

Solution

• mapper.py parses tab-separated movie records (UID, title, genres, year, rating) and emits per-record key-value tuples.
• combiner.py runs local-node aggregation before the network shuffle: the textbook MapReduce performance optimization that reduces shuffle volume.
• reducer.py produces final aggregations across genre, rating, and year dimensions.
• Optional years.txt filter so the same pipeline can be re-run scoped to a subset of years without recompiling logic.
• Hadoop-Streaming-compatible: pure stdin/stdout piping, runnable through any Hadoop 3.x cluster.
• Cluster-portable: same scripts on a pseudo-distributed single-node install or a full multi-node cluster.
• Final reducer output captured in results.txt for inspection and downstream consumption.
• README covering prerequisites (Java 8 or 11, Hadoop 3.x, Python 3.x per node) and the exact hadoop jar hadoop-streaming-*.jar invocation.

Outcome

Above. Paired with the PySpark streaming pipeline, the combo demonstrates full-stack distributed-processing range (on-prem legacy MapReduce alongside modern lakehouse streaming) for buyers whose environment spans both worlds.