XIO

xArchive

Stored page text, served from the archive.

Parameter-efficient fine-tuning · Hugging Face

Archived: 2026-11-16 01:45:36 UTC View live page ↗
https://huggingface.co/docs/transformers/peft

Stored text only -- layout, images, and scripts from the original page are not preserved.

Hugging Face Search models, datasets, users... Ctrl K Models Datasets Spaces Buckets Docs Enterprise Pricing Website Tasks HuggingChat Collections Languages Organizations Community Blog Posts Daily Papers Hardware Learn Discord Forum GitHub Solutions Team & Enterprise Hugging Face PRO Enterprise Support Inference Providers Inference Endpoints Storage Buckets Log In Sign Up Transformers documentation Parameter-efficient fine-tuning Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Tokenizers Trackio Transformers Transformers.js Xet smolagents timm Search documentation main v5.23.0 v5.22.2 v5.21.1 v5.20.2 v5.19.0 v5.17.0 v5.15.1 v5.14.0 v5.13.1 v5.12.0 v5.11.0 v5.10.4 v5.9.0 v5.8.1 v5.7.0 v5.6.2 v5.5.4 v5.4.0 v5.3.0 v5.2.0 v5.1.0 v5.0.0 v4.57.6 v4.56.2 v4.55.4 v4.53.3 v4.52.3 v4.51.3 v4.50.0 v4.49.0 v4.48.2 v4.47.1 v4.46.3 v4.45.2 v4.44.2 v4.43.4 v4.42.4 v4.41.2 v4.40.2 v4.39.3 v4.38.2 v4.37.2 v4.36.1 v4.35.2 v4.34.1 v4.33.3 v4.32.1 v4.31.0 v4.30.0 v4.29.1 v4.28.1 v4.27.2 v4.26.1 v4.25.1 v4.24.0 v4.23.1 v4.22.2 v4.21.3 v4.20.1 v4.19.4 v4.18.0 v4.17.0 v4.16.2 v4.15.0 v4.14.1 v4.13.0 v4.12.5 v4.11.3 v4.10.1 v4.9.2 v4.8.2 v4.7.0 v4.6.0 v4.5.1 v4.4.2 v4.3.3 v4.2.2 v4.1.1 v4.0.1 v3.5.1 v3.4.0 v3.3.1 v3.2.0 v3.1.0 v3.0.2 v2.11.0 v2.10.0 v2.9.1 v2.8.0 v2.7.0 v2.6.0 v2.5.1 v2.4.1 v2.3.0 v2.2.2 v2.1.1 v2.0.0 v1.2.0 v1.1.0 v1.0.0 doc-builder-html AR EN ES FR ID JA KO PT RU TR VI ZH Get started Transformers Installation Quickstart Base classes Models Preprocessors Inference Pipeline API Generate API Optimization Chat with models Serving Training Get started Customization Parameter-efficient fine-tuning Performance Distributed training Hardware Quantization Ecosystem integrations Resources API Join the Hugging Face community and get access to the augmented documentation experience Collaborate on models, datasets and Spaces Faster examples with accelerated inference Switch between documentation themes Sign Up to get started Copy page Parameter-efficient fine-tuning Parameter-efficient fine-tuning (PEFT) methods only fine-tune a small number of extra model parameters (adapters) on top of a pretrained model. Because only adapter parameters are updated, the optimizer tracks far fewer gradients and states, reducing memory usage significantly. Adapters are lightweight, making them convenient to share, store, and load.

Transformers integrates directly with the PEFT library through PeftAdapterMixin , added to all PreTrainedModel classes. You can load, add, train, switch, and delete adapters without wrapping your model in a separate PeftModel . All non-prompt-learning PEFT methods are supported (LoRA, IA3, AdaLoRA). Prompt-based methods like prompt tuning and prefix tuning require using the PEFT library directly.

Install PEFT to get started. The integration requires peft >= 0.20.0 . Copied pip install -U peft Add an adapter Create a PEFT config, like LoraConfig for example, and attach it to a model with add_adapter() . Copied from peft import LoraConfig, TaskType from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b") lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1, ) model.add_adapter(lora_config, adapter_name="my_adapter") Fully fine-tuning specific layers To train additional modules alongside an adapter (for example, the language model head), specify them in modules_to_save .

modules_to_save specifies layers that are fully fine-tuned alongside the adapter, so all of their parameters are updated. This is useful when certain layers need updates, for example the language model head ( lm_head ), when adapting a causal LM for sequence classification. Copied lora_config = LoraConfig( modules_to_save=["lm_head"], ... ) model.add_adapter(lora_config) Choosing which layers to adapt For common architectures (Llama, Gemma, Qwen, etc.), PEFT has predefined default targets (like q_proj and v_proj ), so you don’t need to specify target_modules .

If you want to target different layers, or the model doesn’t have predefined targets, pass target_modules explicitly as a list of module names or a regex pattern. Copied lora_config = LoraConfig( target_modules=["q_proj", "k_proj"], ... ) model.add_adapter(lora_config) Training Pass the model with an attached adapter to Trainer and call train() . Trainer only updates the adapter parameters (those with requires_grad=True ) because the base model is frozen.

Copied from transformers import Trainer, TrainingArguments training_args = TrainingArguments( output_dir="./output", num_train_epochs=3, per_device_train_batch_size=4, ) trainer = Trainer( model=model, args=training_args, train_dataset=dataset, ) trainer.train() During training, Trainer checkpoints contain only the adapter weights ( adapter_model.safetensors ) and configuration ( adapter_config.json ), keeping checkpoints small. The base model isn’t included. After training, save the final adapter with save_pretrained() . Copied model.save_pretrained("./my_adapter") Resuming from a checkpoint Trainer automatically detects adapter checkpoints when resuming.

Trainer scans the checkpoint directory for subdirectories containing adapter weights and reloads each adapter with the correct trainable state. Copied trainer.train(resume_from_checkpoint="./output/checkpoint-1000") Distributed training PEFT adapters work with distributed training out of the box. For ZeRO-3, Trainer passes exclude_frozen_parameters=True when saving checkpoints with a PEFT model. Frozen base model weights are skipped.

Only the trainable adapter parameters are saved, reducing checkpoint size and save time. For FSDP, Trainer updates the FSDP auto-wrap policy to correctly handle LoRA layers. For QLoRA (quantized base model + LoRA), Trainer also adjusts the mixed precision policy to match the quantization storage dtype. You can load PEFT adapters onto a tensor parallel base model and let PEFT handle sharding the adapter weights.

Loading an adapter requires peft >= 0.21.1 . See Tensor parallelism for training with TP. Loading an adapter To load an adapter, the Hub repository or local directory must contain an adapter_config.json file and the adapter weights. from_pretrained load_adapter from_pretrained() automatically detects adapters.

When it finds an adapter_config.json , it reads the base_model_name_or_path field to load the correct base model, then loads the adapter on top. Copied from transformers import AutoModelForCausalLM # Automatically loads the base model and attaches the adapter model = AutoModelForCausalLM.from_pretrained("klcsp/gemma7b-lora-alpaca-11-v1") To load an adapter onto an existing model, use load_adapter() . Copied from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("google/gemma-7b") model.load_adapter("klcsp/gemma7b-lora-alpaca-11-v1") For large models, load a quantized version in 8-bit or 4-bit precision with bitsandbytes to save memory. Add device_map="auto" to distribute the model across available hardware.

Copied from transformers import AutoModelForCausalLM, BitsAndBytesConfig model = AutoModelForCausalLM.from_pretrained( "klcsp/gemma7b-lora-alpaca-11-v1", quantization_config=BitsAndBytesConfig(load_in_8bit=True), device_map="auto", ) Managing multiple adapters A model can hold multiple adapters at once. Add adapters with unique names, and switch between them as needed. Copied from peft import LoraConfig model.add_adapter(LoraConfig(r=8, lora_alpha=32), adapter_name="adapter_1") model.add_adapter(LoraConfig(r=16, lora_alpha=64), adapter_name="adapter_2") Use set_adapter() to activate a specific adapter. The other adapters are disabled but remain in memory.

Copied model.set_adapter("adapter_2") enable_adapters() enables all attached adapters, and disable_adapters() disables all of them. Copied # Disable all adapters for base model inference model.disable_adapters() # Re-enable all adapters model.enable_adapters() Use active_adapters() to see which adapters are currently active. Copied model.active_adapters() # ["adapter_1"] Remove adapters you no longer need with delete_adapter() to free memory. Copied model.delete_adapter("adapter_1") Hotswapping adapters Loading a new adapter each time you serve a request allocates new memory.

If the model is compiled with torch.compile , each new adapter triggers recompilation. Hotswapping replaces adapter weights in-place, avoiding both issues. Only LoRA adapters are supported. Pass hotswap=True when loading a LoRA adapter to swap its weights into an existing adapter slot.

Set adapter_name to the name of the adapter to replace ( "default" is the default adapter name). Copied model = AutoModel.from_pretrained(...) # Load the first adapter normally model.load_adapter(adapter_path_1) # Generate outputs with adapter 1 ... # Hotswap the second adapter in-place model.load_adapter(adapter_path_2, hotswap=True, adapter_name="default") # Generate outputs with adapter 2 torch.compile For compiled models, call enable_peft_hotswap() before loading the first adapter and before compiling. Copied model = AutoModel.from_pretrained(...) max_rank = ...

# highest rank among all LoRAs you'll load model.enable_peft_hotswap(target_rank=max_rank) model.load_adapter(adapter_path_1, adapter_name="default") model = torch.compile(model, ...) output_1 = model(...) # Hotswap without recompilation model.load_adapter(adapter_path_2, adapter_name="default") output_2 = model(...) The target_rank argument sets the maximum rank among all LoRA adapters you’ll load. If you have adapters with rank 8 and rank 16, pass target_rank=16 . The default is 128. After calling enable_peft_hotswap , all subsequent load_adapter calls hotswap by default.

Pass hotswap=False explicitly to disable hotswapping. Recompilation may still occur if the hotswapped adapter targets more layers than the initial adapter. Load the adapter that targets the most layers first to avoid recompilation. Wrap your code in with torch._dynamo.config.patch(error_on_recompile=True) to detect unexpected recompilation.

If you detect recompilation despite following the steps above, open an issue with PEFT with a reproducible example. Next steps The PEFT documentation covers the full range of PEFT methods and options. The PEFT hotswapping reference details limitations and edge cases. A blog post benchmarks how torch.compile with hotswapping improves runtime.

Update on GitHub ← Trainer features Gradient accumulation → Parameter-efficient fine-tuning Add an adapter Fully fine-tuning specific layers Choosing which layers to adapt Training Resuming from a checkpoint Distributed training Loading an adapter Managing multiple adapters Hotswapping adapters torch.compile Next steps