Unit 01 of 02
Distributed Computing
Building the basic abstractions for splitting a job across several machines on a fictional cluster and counting the cost of each one: the driver's plan, the executor's share and the definition of a task, with a single-worker baseline giving a makespan of 2250 and a total work of 2200; the case where block placement and replication make a read local, and the number of blocks stored as the replication factor grows; which step of the map and reduce abstraction stays local and which one touches the network; splitting rows into partitions and lazy evaluation eliminating computation that is never used; the number of tasks a resource manager makes wait while balancing the number of workers against concurrent tasks, and the fixed cost of fifty units for handing work to each worker.
Published Updated
Lessons
- 01 Cluster Computing Fundamentals Splitting a job across several machines with the driver, executor and task abstractions, and measuring the split with two separate numbers: the 2200 rows coming out of the pipeline give a makespan of 2250 and a total work of 2200 on a single worker, but when split across four workers by the subscriber key the makespan drops to 611, when split by the region key two of the four workers stay idle, the skew is 3.32 instead of 1.02 and the makespan can only fall to 1876; because the total work of the three setups is exactly the same 2200, a speedup claim that does not state which one was measured is incomplete. 14 min read
- 02 Distributed File Systems Splitting data into blocks, placing them on the cluster, and measuring whether the worker that processes a block already holds it: when 2200 rows are split into 22 blocks of 100 rows, with a single copy and assignment by block order only 5 of the 22 blocks are read locally, the makespan is 1850 and the total work is 5600; at a replication factor of 2 the local blocks rise to 17 and the makespan falls to 1150, at 3 they rise to 22 and the makespan falls to 650 but the stored blocks grow from 22 to 66, and a fourth copy adds nothing to reading; locality-aware assignment reads 22 of 22 blocks locally with a single copy, but because the placement is unbalanced the makespan stays at 750 instead of 650. 14 min read
- 03 MapReduce Model Splitting a computation into a step that works independently per row and a step that sums per key, and measuring that only one of the two steps touches the network: the map processes 2200 rows locally and moves 0 rows, the makespan is 611 and the total work stays at 2200, while the reduce step that produces the same result moves 1738 rows from the subscriber layout to the region key and raises the total work to 7414 and the makespan to 1914; when every worker reduces its own share before moving it, the moved rows are only 15 of 20 intermediate rows, the total work falls to 2245 and the makespan to 622, and the five region totals presented by the three setups remain exactly the same. 14 min read
- 04 Distributed Datasets A distributed dataset consisting of partitions and a plan that has not run yet, and counting the computation that lazy evaluation eliminates: while eleven partitions are spread over four workers as three, three, three and two, the eager setup opens 44 partitions, reads 6556 rows and writes 3727 intermediate rows for a result of five rows, the lazy setup gives the same result by opening a single partition and reading 200 rows, leaving 2000 rows never read; in a result of a hundred rows the rows read triple while the makespan stays at 250 because three partitions run on three separate workers; in the count action that asks for the whole result the lazy setup reads 2200 rows and the difference comes only from the unused branch and from the intermediate rows not written; because building the plan reads no rows, the type error in the plan appears only at the first action and after 200 rows have been read. 15 min read
- 05 Resource Managers Counting that the number of tasks and the number of concurrent tasks are two separate knobs in scheduling a job on a cluster, and where the fixed cost of fifty units per task goes: with one task in one slot the makespan is 2250 and the fixed cost 50, while sixteen tasks in sixteen slots bring the makespan down to 215 but raise the fixed cost to 800; increasing concurrency does not change the fixed cost at all, whereas increasing the number of tasks raises it by fifty units per task; splitting the job into more tasks at the same concurrency raises the makespan from 611 to 950; going from eight tasks to sixteen the makespan gain stays at 121 and loses to the 400 units of added fixed cost; and when two jobs run on the same cluster the priority and the shared policy change the order of finishing but do not change the 600 units of fixed cost paid. 15 min read