Skip to content
LoRA: Low-Rank Adaptation of Large Language Models 4 min

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen · ICLR 2022

LoRA: Low-Rank Adaptation of Large Language Models

Fine-tune a giant model by training a tiny add-on. How LoRA's low-rank update shrinks trainable parameters by 10,000 times with no slowdown.

Read the original paper

AI & ML · Yellow Belt · Pro · 4 min read

LoRA: Low-Rank Adaptation of Large Language Models

You have a huge pre-trained language model, and you want it to write like your support team, or follow your company's style, or read legal documents. The traditional move is to fine-tune it: keep training and update every one of its billions of numbers. That works, but it is heavy. Fine-tuning a 175-billion-parameter model produces a second model with 175 billion parameters. Do that for ten tasks and you are storing and serving ten giant models.

In 2021, researchers at Microsoft suggested a much lighter way. Leave the big model completely frozen. Train only a tiny add-on next to each large matrix. The method, LoRA (Low-Rank Adaptation), is now one of the most widely used ways to adapt open language models.

When you adapt a model, the change to its weights is surprisingly simple. LoRA writes that change as the product of two thin matrices and trains only those. Compared with fully fine-tuning GPT-3 175B, the paper reports about 10,000 times fewer trainable parameters and about 3 times less GPU memory, with equal or better quality.

The idea: the update is low rank

A layer of a transformer multiplies its input by a big weight matrix W, with shape d by d. Full fine-tuning learns an update to every entry of W. That update is also a d by d matrix, so it has as many numbers as the original.

The authors build on earlier work showing that large models have a low "intrinsic dimension": they can be adapted by moving in only a few directions. They hypothesize that the weight update during adaptation has a low intrinsic rank. In plain words, the change can be described with far fewer numbers than the full matrix suggests.

So LoRA writes the update as the product of two thin matrices: B with shape d by r and A with shape r by d, where the rank r is tiny, such as 4 or 8:

W=W0+BAW = W_0 + B A
LoRA keeps the big pretrained matrix frozen and adds a small trainable path next to it. The path squeezes the input down to r numbers and back up, so it has far fewer parameters than the full matrix.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Try it: count the savings
  • No slowdown when serving

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.