Skip to content
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing 3 min

Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica · NSDI 2012

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

The paper behind Spark: datasets that remember their recipe, so lost memory is recomputed, not copied. Kill a worker and watch the repair.

Read the original paper

Systems & Distributed · Yellow Belt · Pro · 3 min read

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Early big data tools such as MapReduce were great for a single pass over huge files. But many useful jobs go over the same data again and again: machine learning that refines a model, graph algorithms that repeat until they settle, or an analyst who asks one question after another. Re-reading from disk each time is slow.

The obvious fix is to keep the data in the memory of the cluster. But memory is lost when a machine dies, and a cluster has many machines. The 2012 paper on resilient distributed datasets, or RDDs, shows how to keep data in memory and survive failures without copying everything. It is the idea behind Apache Spark.

An RDD is a read-only, partitioned collection of records that can only be built by transformations such as map and filter. Because Spark remembers each dataset's recipe, called its lineage, a lost partition is rebuilt by replaying the recipe on just that piece of data. No replication of the data is needed.

A dataset that remembers its recipe

The paper defines an RDD as a read-only, partitioned collection of records. RDDs can only be created through deterministic operations on data in stable storage or on other RDDs. These operations are called transformations. Examples are map, filter and join.

An RDD does not need to be fully stored at all times. It holds enough information about how it was derived from other datasets, its lineage, to compute its partitions again from stable storage. Nothing is actually computed until an action, such as counting or saving the result, asks for an answer.

A lineage graph. Each box is one partition of a dataset. The program is a chain of transformations, and Spark remembers the chain instead of copying the data.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Why coarse steps make recovery cheap
  • How much faster, and what it costs

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.