Use sharded-state checkpoints during training with FSDP, then switch to full-state for final export
When fine-tuning a large model with PyTorch FSDP, save intermediate checkpoints using the sharded state-dict format for faster saving/resuming, then at the end switch the FSDP plugin to export the entire model state dict for a standard final checkpoint.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The source describes a key pain point: FULL_STATE_DICT checkpoints require gathering the whole model on rank 0 (on CPU) before saving, which is slow and can lead to communication timeouts. Using SHARDED_STATE_DICT for intermediate checkpoints avoids that gather step while still allowing resumption in FSDP-compatible format. The final checkpoint still needs a full, non-sharded state dict for downstream use.
What changed
Checkpointing configuration and export behavior: use a sharded state dict type (SHARDED_STATE_DICT) for intermediate saves, and then explicitly set the state dict type to FULL_STATE_DICT only when saving the final model.
Where it helps
- FSDP fine-tuning where frequent checkpointing is needed
- Large-model training jobs that require fast resume from intermediate checkpoints
- Workflows that ultimately require a standard full-model state dict at the end of training
- Multi-node or multi-GPU setups where full-gather checkpointing is problematic
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach requires FSDP-aware checkpoint formats during training; FULL_STATE_DICT is still described as costly (it gathers the full model on rank 0) and is therefore intended only for the final export, not frequent intermediate saves.
Sources
- Fine-tuning Llama 2 70B using PyTorch FSDP ↗
Hugging Face · primary · Published 2023-09-13 · Checked 2026-11-10
