For Developers/Glossary/LoRA (Low-Rank Adaptation)
Training

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning method that freezes a model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters by up to 10,000x (shown on GPT-3 175B in the original paper) with quality close to full fine-tuning on many tasks.

LoRA (Low-Rank Adaptation), introduced by Microsoft researchers in 2021, avoids updating all of a model's weights. It freezes the original weight matrices and adds a pair of small matrices (A and B) alongside selected ones, usually the attention projections and often the MLP layers as well. Only A and B are trained. Because their inner dimension, the rank, is small (values such as 8 or 16 are common), the trainable parameter count is a tiny fraction of the full model's.

Practical efficiency gains

On GPT-3 175B, the original paper reported 10,000x fewer trainable parameters, 3x lower GPU memory use in training, and task checkpoints of about 35MB instead of 350GB. Together AI, which uses LoRA as its default fine-tuning method, says LoRA trains 0.1% to 1% of the parameters that full fine-tuning would. The frozen base model still has to fit in memory, though: a 70B model needs about 140GB for its weights at 16-bit precision, so plain LoRA at that size needs multiple GPUs or a quantized base.

QLoRA: further reducing hardware requirements

QLoRA (2023) stores the frozen base weights in 4-bit precision while training 16-bit LoRA adapters. Its authors fine-tuned a 65B model on a single 48GB GPU while matching 16-bit fine-tuning results on their benchmarks. Both LoRA and QLoRA are supported in Hugging Face's PEFT library.

Where LoRA and fine-tuning are offered today

  • Open-weight models: Open-weight models such as Gemma 4 or Qwen 3.8 27B can be LoRA-tuned on your own hardware, subject to their licenses. Together AI's managed fine-tuning covers models including DeepSeek V4 Flash, Qwen 3.5 to 3.8, Kimi K2.6 and K2.7 Code, GLM-5.1 to 5.3, Llama 4, gpt-oss and Gemma 4; large MoE models there are LoRA-only, and since Together discontinued serverless LoRA inference, tuned adapters run on dedicated endpoints. Thinking Machines' Tinker is a LoRA-based training API for open-weight models from 1B to over 1T parameters.
  • Google: Gemini Enterprise Agent Platform (formerly Vertex AI) offers adapter-based supervised fine-tuning for Gemini 3.5 Flash, Gemini 3.5 Flash-Lite and Gemini 3.1 Flash-Lite, plus older Gemini 2.5 models. Gemini 4 Argon and Gemini 3.8 Flash are not on the list.
  • OpenAI: OpenAI is winding down its fine-tuning platform. It is closed to new users, eligible existing customers can create jobs until January 6, 2027, and the remaining tunable models, GPT-4.1 and GPT-4.1 mini, shut down on January 14, 2027. GPT-6 models do not support fine-tuning.
  • Anthropic and Mistral: Anthropic does not offer fine-tuning of current Claude models; the only fine-tunable Claude model, Claude 3 Haiku on Amazon Bedrock, reached end of life there on September 10, 2026. Mistral's documentation marks its fine-tuning API as deprecated and no longer actively supported.

When LoRA is appropriate

LoRA works best when the base model already has the relevant capabilities and you are adapting behavior for a specific task, format, or domain. Common use cases include instruction tuning, style transfer, domain-specific terminology, and task-specific adaptation. LoRA typically underperforms when injecting knowledge the base model lacks or when substantially changing fundamental model behavior. In those cases, continued pre-training or full fine-tuning may be necessary; Together AI suggests switching to full fine-tuning when LoRA results plateau below target even after raising the rank.

Serving and merging adapters

An adapter can be merged into the base weights, which removes any extra inference latency but gives up cheap swapping. Kept separate, many adapters can share one copy of the base model, and serving engines such as vLLM can apply a different adapter to each request. Several adapters can also be merged into one model with weighting, though results vary and need testing. This flexibility has made LoRA the default method for open-weight fine-tuning.

Related terms

Models relevant to LoRA (Low-Rank Adaptation)