A company like Google runs a huge number of programs on a huge number of machines. Someone, or something, must decide which program runs where, restart it when a machine dies, and make sure important work is not crowded out by less important work. Doing it by hand is impossible.
Borg is the system Google built for this. Its 2015 paper describes how it has run Google's applications and services for years. Borg's ideas, jobs made of tasks, priorities, a replicated controller and an agent on each machine, shaped the open source system Kubernetes that many teams use today.
Borg lets users submit jobs, and it runs the tasks of each job on a shared pool of machines called a cell. A replicated Borgmaster keeps the state, a scheduler decides where each task goes, and a Borglet on every machine runs the tasks. Priorities and quotas keep important work first, and mixing all kinds of work in one cell saves a lot of machines.
Jobs, tasks and cells
Users submit jobs. A job has a name, an owner and a number of tasks, each of which is one copy of the program. Jobs can state constraints, such as a particular kind of processor, and tasks state how much CPU and memory they need.
The paper describes two main kinds of work. Long-running services should never go down and answer short requests. Batch jobs take from a few seconds to a few days and care less about brief hiccups. Borg marks the higher priority work as production (prod) and the rest as non-production.
The Borgmaster is logically one process, but it runs as five replicas, so no single failure stops the cell. One is elected master, and its state is recorded in a Paxos based store. The paper says electing a new master after a failure typically takes about 10 seconds.
