Foundation Model
A large AI model trained on broad data at scale and designed to be adapted to many downstream tasks — GPT-4, ESM-2, and AlphaFold are all examples in different scientific domains.
What it means
A foundation model is a large AI model trained on vast amounts of data in a self-supervised way, designed to be adapted (fine-tuned) to many different downstream tasks rather than trained for a single purpose from scratch. The term was coined at Stanford in 2021 to describe a paradigm shift in how AI models are built: instead of training a new model for each task, researchers train one large, general model and adapt it.
Examples in science
Foundation models exist across scientific domains:
- Language: GPT-4, Claude, Gemini — trained on broad text corpora, adapted for scientific writing, code generation, literature synthesis
- Proteins: ESM-2, ESM-3 (Meta) — trained on hundreds of millions of protein sequences; adapted for structure prediction, property prediction, and design
- Molecules: ChemBERTa, MolBERT — trained on large molecular datasets; adapted for property prediction
- Biomedical text: BioMedLM, PubMedBERT — trained on biomedical literature; adapted for clinical NLP tasks
- Earth observation: foundation models trained on satellite imagery; adapted for land use classification, crop monitoring
Why it matters
The foundation model paradigm changes the research workflow. Before, training a model for a new task required collecting task-specific labeled data, defining an architecture, and training from scratch — feasible only for well-resourced groups. With foundation models:
- A lab with a small dataset of experimental measurements can fine-tune a pre-trained model rather than training from scratch
- The pre-trained representations capture general structure (protein sequence patterns, molecular graph features) that transfers across tasks
- The cost and data requirements for a new task drop substantially
Trade-offs to understand
Foundation models are not universally better than task-specific models:
- Distribution shift: A foundation model trained on broad data may not capture the specific distribution of your task (e.g., a protein language model trained on all proteins may perform worse on a narrow family of membrane proteins than a model specifically trained on them)
- Interpretability: Larger models are harder to interpret — it’s not always clear what representations the model has learned or why it makes specific predictions
- Resource requirements: Training foundation models requires enormous compute; fine-tuning still requires GPU resources and some labeled data