Parameter-Efficient Fine-Tuning with LoRA Using Unsloth and Hugging Face

Fine-tuning large language models has traditionally required prohibitive amounts of GPU memory. A full fine-tune of a 7B parameter model needs roughly 60GB of VRAM for optimizer states, gradients, and activations — beyond the reach of consumer hardware. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), reduce this to under 16GB while preserving most of the quality gains. This article covers how LoRA works, why Unsloth accelerates it further, and how to build a complete fine-tuning pipeline with Hugging Face.

How LoRA Works

LoRA (Hu et al., 2021) is based on the observation that the weight updates during fine-tuning have low intrinsic rank. Instead of updating the full weight matrix W ∈ R^(d×k), LoRA freezes W and injects two small trainable matrices A ∈ R^(d×r) and B ∈ R^(r×k), where r is the rank (typically 8-64, orders of magnitude smaller than d and k). The forward pass becomes h = Wx + BAx. Only A and B are updated during training — typically 0.1-1% of the total parameters. This reduces memory from storing full gradient and optimizer states for all parameters to storing them only for the tiny LoRA adapters. After training, the LoRA weights can be merged into the original weights (W’ = W + BA) for inference with zero latency overhead, or kept separate for modular swapping between tasks.

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                     # Rank — higher = more capacity, more memory
    lora_alpha=32,            # Scaling factor — typically 2x rank
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,        # Prevents overfitting on small datasets
    bias="none",              # Don't train bias terms
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()  # Trainable: ~0.5% of all params

Unsloth: Optimized LoRA Training

Unsloth is an open-source library that optimizes the LoRA training loop through custom CUDA kernels and memory-efficient attention. It achieves 2x faster training and 50% less memory usage compared to standard Hugging Face + PEFT implementations. Unsloth achieves this through four key optimizations. First, it uses a custom linear layer implementation that avoids materializing the full LoRA weight matrices during the forward pass — the LoRA computation is fused into the base matrix multiplication. Second, it applies 4-bit NormalFloat quantization (NF4) with double quantization, reducing the base model to 4-bit while storing quantization constants in 8-bit. Third, it implements Flash Attention 2 for the attention computation, reducing memory from O(n²) to linear in sequence length. Fourth, it enables gradient checkpointing by default with a custom implementation that stores only the minimal activations needed for the backward pass.

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-bnb-4bit",
    max_seq_length=2048,
    dtype=torch.bfloat16,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    use_gradient_checkpointing="unsloth",
    random_state=42,
)

print(f"Memory after LoRA: {model.get_memory_footprint() / 1e9:.2f} GB")

With Unsloth, a 7B parameter model can be fine-tuned on a single RTX 3090 (24GB) with a batch size of 4 and sequence length of 2048 — tasks that would require an A100 (80GB) with standard PEFT. The memory savings come primarily from the 4-bit quantization (16GB → 4GB for a 7B model) and the fused LoRA kernels (avoiding the ~8GB overhead of separate LoRA weight materialization).

Building the Training Pipeline with Hugging Face TRL

The Transformers Reinforcement Learning (TRL) library provides the SFTTrainer class for supervised fine-tuning, optimized for PEFT methods. The trainer handles dataset formatting, padding, attention masking, and gradient accumulation automatically. For instruction-tuning datasets (like OpenAssistant or Alpaca), the data is formatted as conversation turns with a template. The trainer applies the chat template through the tokenizer and masks the loss on prompt tokens so only the response tokens contribute to training.

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

dataset = load_dataset("yahma/alpaca-cleaned", split="train")
dataset = dataset.select(range(1000))  # Use a subset for fast prototyping

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="./lora_finetuned",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,
        learning_rate=2e-4,
        logging_steps=10,
        save_steps=50,
        optim="adamw_8bit",       # 8-bit Adam reduces optimizer memory
        dataset_text_field="text", # Column containing formatted prompts
        max_seq_length=2048,
        packing=True,              # Pack multiple short sequences
    ),
)

trainer.train()
model.save_pretrained("./lora_adapter")  # Saves only LoRA weights (~16MB)
tokenizer.save_pretrained("./lora_adapter")

Merging and Inference

After training, the LoRA adapter can be merged into the base model for deployment. Merging adds the LoRA weights into the original weight matrices, producing a single model with zero inference overhead. The merged model is indistinguishable from a full fine-tune in terms of inference speed and memory. Unsloth’s merge_and_unload() is 2x faster than the Hugging Face equivalent because it fuses the dequantization and LoRA merge into a single CUDA kernel. The merged model can be further quantized (GPTQ, AWQ) for production serving. For multi-task setups, keep adapters separate and load them dynamically — a single base model with 10 LoRA adapters takes the same memory as one model.

# Merge LoRA weights into base model
model = model.merge_and_unload()

# Save merged model
model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")

# Inference with the merged model
from transformers import pipeline
pipe = pipeline("text-generation", model="./merged_model", tokenizer=tokenizer)
result = pipe("Explain LoRA in one sentence:", max_new_tokens=64)
print(result[0]["generated_text"])

LoRA hyperparameters worth tuning: rank r (8-64, higher for more diverse tasks), alpha (typically 2x rank, higher for stronger adaptation), target modules (Q,K,V,O, gate, up, down projections — all linear layers in the transformer), and dropout (0-0.1, higher for very small datasets). The scaling factor alpha/r controls how much the LoRA update affects the output — start with alpha=2*r and adjust based on validation loss. Unsloth’s zero dropout default works well for most datasets because PEFT has inherent regularization through the low-rank bottleneck.