Turkey (Turkish)
Germany (German)
Worldwide (English)
The fix depends on where OOM happens: loading OOM and backward OOM are not the same problem.
Halkali merkez neighborhood fatih st ozgur apt no 46 , Kucukcekmece , Istanbul , 34303 , TR
Fix CUDA OOM during load, forward, backward and optimizer steps using batch/context tuning, checkpointing, QLoRA, fragmentation checks and offload.
The fix depends on where OOM happens: loading OOM and backward OOM are not the same problem.
Gradient checkpointing reduces activation memory, while reducing batch/context directly lowers activation footprint.
High reserved-but-unallocated memory can indicate allocator fragmentation, but allocator environment variables should not replace root-cause analysis.
If OOM occurs in `from_pretrained()`, focus on weight loading, quantization and device mapping. Forward OOM points to activations/batch/context; backward adds gradient pressure; optimizer-step OOM often points to optimizer state and peak allocations.
Record total VRAM, other processes, PyTorch allocated/reserved memory and batch/context settings. Capture `nvidia-smi` plus `torch.cuda.memory_summary()` before and after the first step. Non-deterministic OOMs often involve background processes or validation/data-loader spikes.
nvidia-smipython - <<'PY'
import torch
print(torch.cuda.memory_summary())
PYps aux --sort=-rss | head -20Reduce batch to 1 while keeping context fixed. If OOM persists, reduce context stepwise. Changing both at once hides which variable actually caused the pressure.
Gradient accumulation simulates a large effective batch using small micro-batches. It reduces batch-related activation pressure, but cannot fix a model whose weights alone do not fit on the GPU.
When activations dominate memory, checkpointing is effective. Transformers documents that it discards some activations and recomputes them during backward, trading speed for memory.
If base weights are the problem, 4-bit QLoRA is one of the strongest levers. bitsandbytes reduces weight footprint, while 8-bit optimizers can shrink optimizer states. PEFT recommends `prepare_model_for_kbit_training()` for quantized training.
If reserved-but-unallocated is tiny, allocator tuning is unlikely to help; the GPU is genuinely full. If it is large, fragmentation is worth investigating. Close unrelated processes and measure validation peaks before changing allocator settings.
If training is stable but evaluation OOMs, inspect eval batch size, generation length, `predict_with_generate`, logits accumulation and validation generation. Diffusion validation can create a separate peak by constructing pipelines.
If single-GPU optimizations are insufficient, DeepSpeed ZeRO-3/FSDP can shard parameters across GPUs. CPU/NVMe offload saves VRAM but can substantially slow training due to data movement.
| Phase | First suspect | First action |
|---|---|---|
| Model load | Weights / dtype | 4-bit / device map |
| Forward | Batch / context / activations | Batch=1, reduce context |
| Backward | Activations + gradients | Checkpointing |
| Optimizer step | Optimizer state | 8-bit optimizer / sharding |
| Eval | Generation/logits peak | Reduce eval batch/generation |
Do not blindly upgrade packages in a working training environment. Record GPU, driver, CUDA/PyTorch runtime, Transformers, Accelerate, PEFT, TRL, bitsandbytes/Diffusers, model revision and dataset fingerprint for every run. Reproduce minimally before changing production training.
Usually not the root cause. It releases unused cached blocks, but cannot fix a true peak-memory requirement that exceeds capacity.
Weights, long context, validation peaks or optimizer state can exceed VRAM even with batch 1.
Evaluate model size, precision, context, batch, LoRA/QLoRA or full fine-tuning and multi-GPU needs together instead of choosing by GPU name alone.