Skip to content
The Anatomy of a Large-Scale Hypertextual Web Search Engine 4 min

Sergey Brin, Lawrence Page · WWW 1998

The Anatomy of a Large-Scale Hypertextual Web Search Engine

Google's original paper: PageRank as a random surfer, anchor text, and the crawl, index and search pipeline. Edit a tiny web and watch ranks flow.

Read the original paper

Algorithms · White Belt · Pro · 4 min read

The Anatomy of a Large-Scale Hypertextual Web Search Engine

In 1994 one of the first search engines for the Web, the World Wide Web Worm, indexed about 110,000 pages. By late 1997 the leading engines claimed to index between 2 million and 100 million, and the authors expected the Web to pass a billion documents by the year 2000. At that size, matching keywords stops being enough. A query for a common word finds tens of thousands of pages, and most of them are junk.

Sergey Brin and Larry Page, then PhD students at Stanford, described their answer in a 1998 paper: a prototype called Google. It made two big bets. Use the links between pages as a signal of quality, and build the whole system around the physical limits of disks and networks.

Google ranked pages by PageRank, the idea that a page matters if important pages link to it, and by the text of the links that point to it. Behind that sat a pipeline of crawlers, a compressed repository and an inverted index built to avoid slow disk seeks.

Think of a link from page A to page B as a vote for B. A page with many votes looks worth reading. But not all votes are equal. A link from a respected page counts for more than a link from an unknown one. The paper says a page has a high PageRank if many pages point to it, or if a few highly ranked pages do.

The authors give a simple story to make this concrete: the random surfer. Someone starts on a random page and keeps clicking links, never pressing back. Now and then they get bored and jump to a random page. The probability that the surfer is on a page at any moment is its PageRank.

A tiny web. Bigger circles have a higher PageRank. A random surfer clicks links and only occasionally jumps to a random page, so pages that many pages point to are visited most often. Page C collects links from four pages.

The boredom is controlled by a damping factor, d, which the paper usually sets to 0.85. With probability d the surfer follows a link, and with probability 1 minus d they jump anywhere. Written as a formula, a page's score is the jump share plus its share of every ranking that flows in through links:

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Anchor text: what other pages say about you
  • The architecture: built around disk seeks

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.