Skip to content
Large-scale Cluster Management at Google with Borg 4 min

Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, John Wilkes · EuroSys 2015

Large-scale Cluster Management at Google with Borg

How Google runs everything on shared machines: jobs, tasks, priorities and preemption. Place tasks yourself and compare scheduling policies.

Read the original paper

Systems & Distributed · Yellow Belt · Pro · 4 min read

Large-scale Cluster Management at Google with Borg

A company like Google runs a huge number of programs on a huge number of machines. Someone, or something, must decide which program runs where, restart it when a machine dies, and make sure important work is not crowded out by less important work. Doing it by hand is impossible.

Borg is the system Google built for this. Its 2015 paper describes how it has run Google's applications and services for years. Borg's ideas, jobs made of tasks, priorities, a replicated controller and an agent on each machine, shaped the open source system Kubernetes that many teams use today.

Borg lets users submit jobs, and it runs the tasks of each job on a shared pool of machines called a cell. A replicated Borgmaster keeps the state, a scheduler decides where each task goes, and a Borglet on every machine runs the tasks. Priorities and quotas keep important work first, and mixing all kinds of work in one cell saves a lot of machines.

Jobs, tasks and cells

Users submit jobs. A job has a name, an owner and a number of tasks, each of which is one copy of the program. Jobs can state constraints, such as a particular kind of processor, and tasks state how much CPU and memory they need.

The paper describes two main kinds of work. Long-running services should never go down and answer short requests. Batch jobs take from a few seconds to a few days and care less about brief hiccups. Borg marks the higher priority work as production (prod) and the rest as non-production.

A Borg cell is a set of machines, a logically centralized controller called the Borgmaster, and an agent called the Borglet on every machine. The paper says the median cell has about 10,000 machines.

The Borgmaster is logically one process, but it runs as five replicas, so no single failure stops the cell. One is elected master, and its state is recorded in a Paxos based store. The paper says electing a new master after a failure typically takes about 10 seconds.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Where does each task go?
  • Why share one big pool?

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.