Imagine ten identical servers that all want to be "the leader". If two of them think they are in charge at once, they can corrupt the data. Something has to decide, and that something must keep working even when machines fail.
Google's answer, described by Mike Burrows in 2006, was Chubby: a small, highly available service that hands out locks and stores tiny files. GFS uses it to pick its master, and Bigtable uses it in several ways. Chubby is a good paper for beginners because it admits its mistakes. Half of it explains how real engineers used the service differently than the designers expected.
Chubby is a lock service that looks like a tiny file system. Clients keep a session alive with heartbeats called KeepAlives. A lock is held only while the session is alive. Fencing numbers called sequencers stop a slow, confused old lock holder from doing damage.
A lock service, not a library
Why run a separate service for locks? Because almost every team needs the same two things: a way to elect one leader, and a place to publish small facts such as "the primary is on host 17." Writing a correct consensus protocol inside each product is hard. Calling one shared service is easy.
Chubby gives each team a file-system-like tree of names. A file can hold a few bytes of data, and any file can also act as a lock. A leader election then takes four steps. Every candidate tries to lock the same file. One wins. The winner writes its own address into the file. Everyone else reads the file to find the leader.
The paper says Chubby is meant for coarse-grained locks, ones held for hours or days, like "who is the primary right now?" Rare, long-lived locks put little load on the server. Teams that need quick, fine-grained locks are expected to build those on top.
Inside, each Chubby "cell" is five replicas, and three must be running for the cell to work. One replica is the master and answers all requests. The others keep copies. If the master dies, the replicas elect a new one. Clients find the master through DNS and by asking any replica.
