FFN-Perturbed Self-Distillation (FPD) for Multimodal Unlearning
Use self-distillation from a teacher’s original output distribution as a zero-shift baseline, then selectively perturb feed-forward network (FFN) activations linked to private information across text and vision to create new supervision targets that remove private content while limiting distribution shift.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Unlearning methods that directly use constructed supervision targets can force a trade-off where either forgetting is incomplete or utility degrades due to large changes from the original output distribution. The technique explicitly reduces private-information content in the supervision signal while keeping the supervision close to the teacher’s original outputs.
What changed
Instead of optimizing unlearning toward labels/output distributions that may embed too much private information or cause large shifts, the method (1) anchors supervision to the teacher’s original output distribution as a reference with no initial shift, and (2) selectively perturbs FFN activations associated with private information across both modalities before forming the targets.
Where it helps
- Multimodal model unlearning to reduce memorization of sensitive content
- Privacy-preserving refinement of multimodal LLMs without heavily changing output behavior
- Target-construction for unlearning where supervision targets control forgetting-utility trade-offs
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The method relies on identifying/selecting FFN activations associated with private information in both textual and visual modalities; the text does not specify how that selection is obtained.
Sources
- Rethinking Supervision Targets for Multimodal LLM Unlearning: Less Private Information, Reduced Distribution Shift ↗
arXiv · primary · Checked 2026-11-10
