Glossary3 sections1 min read
What is Fine-tuning?
Fine-tuning continues training a pretrained model on your own data so it adapts to a task, style or domain. Full fine-tuning updates every weight; parameter-efficient methods such as LoRA update only small adapters.
Why it matters when renting a GPU
Full fine-tuning stores weights, gradients and optimizer states, so it needs far more memory than inference.
Choose the method first, then size the GPU: LoRA often fits on one GPU where full fine-tuning needs several.
Worked example
A common estimate for full fine-tuning with Adam in mixed precision is 16 bytes per parameter (2 for weights, 2 for gradients, 12 for FP32 master weights and two optimizer moments). A 7B model therefore needs about 7 × 16 = 112 GB before activations, more than one 80 GB H100.
Frequently asked questions
- Is fine-tuning the same as training?
- It is training that starts from pretrained weights, with far less data and compute than pretraining.