← AI Techniques Radar
Training & fine-tuning

Use sharded-state checkpoints during training with FSDP, then switch to full-state for final export

When fine-tuning a large model with PyTorch FSDP, save intermediate checkpoints using the sharded state-dict format for faster saving/resuming, then at the end switch the FSDP plugin to export the entire model state dict for a standard final checkpoint.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

The source describes a key pain point: FULL_STATE_DICT checkpoints require gathering the whole model on rank 0 (on CPU) before saving, which is slow and can lead to communication timeouts. Using SHARDED_STATE_DICT for intermediate checkpoints avoids that gather step while still allowing resumption in FSDP-compatible format. The final checkpoint still needs a full, non-sharded state dict for downstream use.

What changed

Checkpointing configuration and export behavior: use a sharded state dict type (SHARDED_STATE_DICT) for intermediate saves, and then explicitly set the state dict type to FULL_STATE_DICT only when saving the final model.

Where it helps

  • FSDP fine-tuning where frequent checkpointing is needed
  • Large-model training jobs that require fast resume from intermediate checkpoints
  • Workflows that ultimately require a standard full-model state dict at the end of training
  • Multi-node or multi-GPU setups where full-gather checkpointing is problematic

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The approach requires FSDP-aware checkpoint formats during training; FULL_STATE_DICT is still described as costly (it gathers the full model on rank 0) and is therefore intended only for the final export, not frequent intermediate saves.

Sources