Named entity recognition (NER)

Named entity recognition (NER) is an NLP task that automatically identifies spans of text referring to specific named entities and classifies them into predefined categories: people, organizations, locations, dates, monetary amounts, and domain-specific types like gene names, protein names, chemical compounds, or medical conditions.

For example, in the sentence “BRCA1 mutations were associated with breast cancer risk in the NURSE’s Health Study cohort,” a NER model trained on biomedical text would tag: BRCA1 → gene, breast cancer → disease, NURSE's Health Study → study/organization.

Why researchers use NER:

• Literature mining: extracting all mentions of a specific drug, disease, gene, or technique across thousands of papers without reading each one
• Building knowledge graphs: populating relationship databases (drug-target interactions, gene-disease associations) by extracting entities and their co-occurrences from literature
• Survey and interview analysis: identifying mentions of specific themes, places, or organizations across large qualitative datasets
• Data harmonization: standardizing free-text fields in datasets where the same entity appears under multiple names

How it’s done:

Older approaches used hand-crafted rules or statistical sequence models (CRF). Modern NER uses transformer-based models (BERT and its biomedical variants like BioBERT, PubMedBERT) fine-tuned on annotated corpora. For social science text (interviews, news, social media), general-purpose models work reasonably well; for specialized domains (clinical text, chemistry), domain-adapted models significantly outperform them.

Tools: spaCy (general purpose, with NER pipelines), scispaCy (scientific text), Hugging Face fine-tuned models (PubMedBERT, BioBERT), ATLAS.ti AI assistant

Related terms: Zero-Shot Classification, Topic Modeling

Related guide: Social Science