Your Spark job is a mailroom

Shrey Patel and Jay Patel, Coconut Labs · Published 2026-08-05 · Last updated 2026-08-08 · 9 min · Systems

There is a sentence every data engineer says within their first month: "the job ran in four stages, and stage two did the heavy shuffle." We say it, the dashboard agrees, the pipeline ships. Most of us go years without asking what a stage actually is, or why the boundary sits exactly where it sits.

I want to hand you the picture that made it obvious for me. It involves clerks, paper slips, and a mailroom. Once you see it, stage boundaries, skew, and about half of the Spark tuning guide stop being trivia and become one picture you can reason about.

One clerk, fifty million names

Forget computers. A census bureau wants one table: every surname in the country and how many people carry it. The input is fifty million printed names, and you have one very diligent clerk. At one name per second, that clerk needs about a year and a half without sleep. The work is linear and the pile is enormous, so you hire a thousand ordinary clerks instead, hand each a chunk of pages, and you are done counting by dinner.

Except you are not done. At five o'clock you have a thousand desks, each holding a private pile of tallies. Nobody has the answer. "How many Patels" currently has a thousand partial answers and zero final ones. Getting from private piles to one true table is a different kind of work from the counting, and that second kind of work is the whole subject of distributed computing.

The three beats

Map is the part you can do alone. Each clerk reads their own pages and writes one slip per sighting: patel, 1. No clerk talks to another. That silence is not laziness, it is the design. Work that needs no coordination scales to a thousand desks for free, so the system pushes everything it can into this beat.

The shuffle is the mail. To finish the count for one name, every slip carrying that name must reach one counter. Not most of them. All of them. So slips move from the desks where they were written to the counters that own them. Who decides which counter owns "patel"? Nobody does. Arithmetic does: hash the name, take it modulo the number of counters, and every clerk in the building computes the same destination without a single phone call. A thousand strangers agree forever because they all run the same little formula.

Reduce is the counting-house. Each counter groups its mail by name and adds. Also private work, also perfectly parallel. The only social act in the entire system was the mail.

Why your job has stages

Spark's planner reads your code and sorts every operation into those beats. A filter, a per-row calculation, a column derivation: desk work. A groupBy, a join, a repartition: mailroom work, because rows that share a key must physically meet. The planner fuses every unbroken run of desk work into one pass over the data. That fused run is a stage.

So the boundary between stage one and stage two is not an implementation detail. It is the moment your job stops being private work and becomes a postal system. In the Spark UI, "Shuffle Write" is slips being filed by desks and "Shuffle Read" is counters fetching their mail. When the query plan says Exchange, that is the mailroom door.

Where the money goes

Three properties of the mailroom explain most real-world Spark pain.

It multiplies. Any desk may hold mail for any counter, so the number of mail routes is desks times counters. A modest job with ten thousand map tasks and two hundred partitions is running two million little mail streams. Data volume matters, but route count is what makes shuffles expensive at scale.

It waits. No counter's total is final until the last desk delivers. One slow clerk, one overloaded machine, one unlucky chunk, and the entire building stands around. Under a barrier, your job runs at the speed of its slowest worker.

It clumps. Hashing spreads names evenly. It says nothing about mail volume, because popularity lives in the data, not the algorithm. If one key covers a tenth of your rows, one counter drowns while its neighbors go home early. That is skew, and it is a mailroom problem wearing a memory-error costume.

What to do about it

Every standard fix is a postal strategy, which is why they finally stick once you see the room.

Shrink the mail before shipping. A clerk who saw "patel" three hundred times can mail one slip that says patel, 300. Spark does this automatically for many aggregations. When a query plan reads partial aggregate, exchange, final aggregate, that is pre-summing at the desk, then the mail, then the counting-house.

Skip the mailroom entirely. If one side of a join is small, photocopy it to every desk and no rows move at all. That is a broadcast join, and it is the single cheapest fix in the book when it applies.

Split the whale. For a drowning counter, give the popular key a handful of temporary suffixes so several counters share it, then merge the few subtotals at the end. That is salting. Adaptive execution can do a version of this for you, because a stage boundary is the one place the engine holds real, observed sizes instead of estimates.

Choose the counter count deliberately. Too few partitions and each counter's pile overflows to disk. Too many and you pay route overhead on millions of near-empty envelopes. There is no magic number, but there is a magic question: what does this setting do to the mailroom?

The takeaway

You do not tune Spark. You tune a postal system. Narrow work is free, wide work pays postage, and the postage is charged in routes, barriers, and clumps. The next time a job crawls, skip the config forums for ten minutes and ask the three mailroom questions instead: how much mail, how late is the slowest desk, and who is drowning. The answer is usually standing right there, holding a very large pile of envelopes.