LoRA: fine-tuning a large model without retraining all of it
LoRA is a cheap way to fine-tune an AI model: the original model is frozen and only small add-on pieces are trained. The result is a lightweight file you can plug in or remove whenever you like.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2021 |
| Introduced by | Edward J. Hu et al. |
How it works
LoRA lets you customize an AI model by adding a few small adjustable pieces instead of retraining the whole thing.
Classic fine-tuning updates every parameter in a model, which needs as much memory as training and leaves you with a full copy to store. LoRA, short for Low-Rank Adaptation, starts from a different assumption: the useful change is “simple” and can be written as the product of two narrow matrices, A and B. The original weights are frozen and only A and B are trained, usually inside the attention layers. Two settings matter most: the rank r, which sets the size of the matrices, and the scaling factor lora_alpha.
The method was introduced in June 2021 by Edward J. Hu and co-authors in the original arXiv paper. On GPT-3 175B, they report 10,000 times fewer trainable parameters and three times less GPU memory than full fine-tuning, with equal or better quality in their experiments.
A concrete example
A translation agency wants a model that follows the glossary and tone of each of its three main clients. Rather than storing three copies of a model weighing tens of gigabytes, it keeps a single base model and trains three LoRA adapters, each far smaller than the model itself, their size depending on the chosen rank. When a document comes in, the server loads the matching client’s adapter.
Why it matters
LoRA opened up fine-tuning of open-weight models to people without a data center. Combined with quantization in QLoRA, it allowed a 65-billion-parameter model to be fine-tuned on a single 48 GB GPU, according to its authors. The technique also spread to image generation, where people share style adapters.
Common myths
- “A LoRA is a model.” It is an add-on: without the exact base model, the adapter is useless.
- “Higher rank is always better.” It increases adaptation capacity, but also the risk of overfitting and the file size. The right value depends on your data.
- “LoRA adds reliable knowledge.” Like any fine-tuning, it mostly changes behavior; for facts that change, RAG is usually a better fit.
Related terms: fine-tuning, quantization, parameters and VRAM.
Related terms
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
- Model parameters · The internal numbers a model tunes during training; how many it has indicates its size, not directly how good it is.
- Quantization · A technique that stores a model's weights with fewer bits (8, 4 or even fewer) so it needs far less memory to run.
- Open-weights model · An AI model whose trained parameters (the weights) can be downloaded, so you can run it on your own hardware.
- VRAM (video memory) · The memory built into a graphics card, where an AI model must fit to run fast; it is often what limits what you can run locally.
- GPU (graphics processing unit) · A chip built to run thousands of calculations in parallel, which became the engine for training and running AI models.
Try it
Full glossary
Frequently asked questions
What does low-rank actually mean?
Instead of editing a large weight matrix directly, LoRA learns the change as the product of two narrow matrices. The rank r sets how wide they are: smaller means fewer trainable parameters but a less expressive adaptation.
Does a LoRA adapter work with any model?
No. It is trained for one specific base model and only applies to it, or to versions with exactly the same architecture and starting weights.
Does LoRA slow down responses?
Not necessarily. Both the original paper and the PEFT library documentation note that adapter weights can be merged into the base model, which adds no inference latency.
What is the difference between LoRA and QLoRA?
QLoRA, published in 2023, loads the base model quantized to 4 bits and trains LoRA adapters on top. That cuts the memory needed to fine-tune very large models even further.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.