Big systems made of many machines need a few small shared facts. Who is the leader right now? Which servers are alive? What is the current configuration? Getting these answers wrong is dangerous, and getting them right in a distributed system is surprisingly hard.
A team at Yahoo built ZooKeeper to give every application the same trustworthy answers. Their 2010 paper describes a service for coordinating processes of distributed applications. It does not try to be a lock server or a queue. It offers a few small, fast building blocks, from which clients make leader election, locks, group membership and more.
ZooKeeper keeps a tiny tree of data nodes, replicated across an ensemble of servers. Its operations never block on other clients (wait-free), writes are strictly ordered, and each client's requests run in order. With ephemeral nodes that vanish when a client dies, and watches that report changes, you can build locks, leader election and membership out of the pieces.
A tiny file system for facts
The paper describes ZooKeeper as a set of znodes in a hierarchical name space. Znodes are named like files, with paths such as /app1/p_1. Each znode holds a little data, and a znode can have children.
Two ideas make this more than a file system. First, an ephemeral znode exists only as long as the session of the client that created it. If the client crashes, the service deletes the znode for it. Second, when creating a znode, a client can set a sequential flag, and ZooKeeper appends an increasing counter to the name, so names are unique and show arrival order.
Clients can also set a watch when reading. The watch is a one time notification that fires when the data changes. It lets a client wait efficiently without asking over and over.
Think of a shared whiteboard in a control room. Anyone can read it. Names are written in order. An operator's name is erased automatically when they leave the room. And you can ask to be tapped on the shoulder when a particular line changes.
