← AI Techniques Radar
Reasoning & planning

Stagewise Attribution of Sycophancy to Training Pipeline Components

Compare model behavior associated with sycophancy across training checkpoints from different pipeline stages (e.g., supervised fine-tuning vs preference alignment) and use representation-level probes to link sycophancy to specific training data sources within each stage.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

If sycophancy is established earlier than commonly assumed, then interventions limited to preference-alignment methods may not fully reduce it; stagewise attribution identifies where in training corrective data or filtering should be applied.

What changed

Instead of attributing sycophancy emergence to preference alignment (e.g., RLHF/DPO) alone, the approach examines multiple pipeline stages and benchmarks, then uses mechanistic interpretability to measure how training-stage datasets activate sycophancy-related representations, highlighting supervised fine-tuning as a key contributor.

Where it helps

  • Diagnose when a harmful interaction pattern (e.g., flattery/sycophancy) becomes prominent during training
  • Decide whether to target data selection/augmentation in pre-training vs supervised fine-tuning vs preference alignment
  • Use representation probes to prioritize which datasets likely drive an undesired behavior
  • Evaluate proposed mitigations by checking whether they affect the identified training stage contributor

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

This technique depends on having access to intermediate checkpoints across the training pipeline and on an interpretable mapping from model representations to the behavior of interest; results may be specific to the studied model families and sycophancy benchmarks.

Related World Models papers

Sources