← AI Techniques Radar
Training & fine-tuning

FFN-Perturbed Self-Distillation (FPD) for Multimodal Unlearning

Use self-distillation from a teacher’s original output distribution as a zero-shift baseline, then selectively perturb feed-forward network (FFN) activations linked to private information across text and vision to create new supervision targets that remove private content while limiting distribution shift.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Unlearning methods that directly use constructed supervision targets can force a trade-off where either forgetting is incomplete or utility degrades due to large changes from the original output distribution. The technique explicitly reduces private-information content in the supervision signal while keeping the supervision close to the teacher’s original outputs.

What changed

Instead of optimizing unlearning toward labels/output distributions that may embed too much private information or cause large shifts, the method (1) anchors supervision to the teacher’s original output distribution as a reference with no initial shift, and (2) selectively perturbs FFN activations associated with private information across both modalities before forming the targets.

Where it helps

  • Multimodal model unlearning to reduce memorization of sensitive content
  • Privacy-preserving refinement of multimodal LLMs without heavily changing output behavior
  • Target-construction for unlearning where supervision targets control forgetting-utility trade-offs

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The method relies on identifying/selecting FFN activations associated with private information in both textual and visual modalities; the text does not specify how that selection is obtained.

Sources