You have a huge pre-trained language model, and you want it to write like your support team, or follow your company's style, or read legal documents. The traditional move is to fine-tune it: keep training and update every one of its billions of numbers. That works, but it is heavy. Fine-tuning a 175-billion-parameter model produces a second model with 175 billion parameters. Do that for ten tasks and you are storing and serving ten giant models.
In 2021, researchers at Microsoft suggested a much lighter way. Leave the big model completely frozen. Train only a tiny add-on next to each large matrix. The method, LoRA (Low-Rank Adaptation), is now one of the most widely used ways to adapt open language models.
When you adapt a model, the change to its weights is surprisingly simple. LoRA writes that change as the product of two thin matrices and trains only those. Compared with fully fine-tuning GPT-3 175B, the paper reports about 10,000 times fewer trainable parameters and about 3 times less GPU memory, with equal or better quality.
The idea: the update is low rank
A layer of a transformer multiplies its input by a big weight matrix W, with shape d by d. Full fine-tuning learns an update to every entry of W. That update is also a d by d matrix, so it has as many numbers as the original.
The authors build on earlier work showing that large models have a low "intrinsic dimension": they can be adapted by moving in only a few directions. They hypothesize that the weight update during adaptation has a low intrinsic rank. In plain words, the change can be described with far fewer numbers than the full matrix suggests.
So LoRA writes the update as the product of two thin matrices: B with shape d by r and A with shape r by d, where the rank r is tiny, such as 4 or 8:
