Serving Many Fine-Tunes: LoRA Adapters on a Shared Base Model

The cost problem. Fine-tuning produces a full copy of the model: a 7B-parameter model in 16-bit is about 14 GB. Serving 100 customers' fine-tunes as 100 full models would need 1.4 TB of GPU memory, almost all of it duplicated base weights.

What LoRA changes. Low-Rank Adaptation freezes the base weights and learns, for selected layers, two small matrices A (d x r) and B (r x d) whose product is added to the layer's output. With rank r of 8-64, an adapter is typically 10-200 MB: under 1% of the model. Training also needs far less memory.

Serving idea. Keep one copy of the base model on the GPU. Store adapters in cheap storage (object storage, CPU RAM, local disk) and load the needed one into a small GPU adapter cache on demand. A request names its adapter id; the server fetches it if absent, evicting the least recently used.

Batching across adapters. The expensive part, the base-model matrix multiplications, is identical for all requests, so requests for *different* adapters can share one batch. The small per-adapter product x * A * B is then computed per request with specialised kernels (the approach of systems like S-LoRA and Punica). This keeps GPU utilisation high even when each customer sends little traffic.

Merge vs keep separate. For a single hot adapter, you can merge it into the base weights (W + BA) and serve at full speed with zero overhead, but then that GPU serves only that adapter. Unmerged adapters add a small overhead per token and give you multi-tenancy. The decision follows traffic: merge for a dedicated heavy customer, share for the long tail.

Operational concerns. Cold start: the first request for an adapter waits for it to load (hundreds of milliseconds to seconds), so pre-warm popular ones and route a customer's traffic to the same replica for cache hits. Version adapters, pin each one to the base-model version it was trained on (an adapter does not work on a different base), and enforce isolation so one tenant can never select another's adapter.

Capacity and cost. Budget GPU memory as base weights + KV cache + adapter cache. Charge or limit per tenant on tokens, not on adapters stored. Because adapters are tiny, the marginal cost of one more customer is mostly storage plus the traffic they actually generate.