Named entity recognition finds spans of text that refer to real-world entities and classifies them into types such as PERSON, ORGANIZATION, LOCATION, DATE, MONEY, or domain-specific types like DRUG, GENE, PRODUCT or ACCOUNT_NUMBER. It is a building block for information extraction, search indexing, knowledge-graph population, PII redaction, and entity linking. Technically it is sequence labeling: one decision per token, with strong dependencies between neighboring decisions.
Part of Natural Language Processing. Builds on Tokenization; contrast with document-level Text Classification.
To turn span extraction into per-token classification, each token receives a tag. In BIO (IOB2), B-X starts an entity of type X, I-X continues it and O is outside any entity, so K entity types give 2K + 1 labels. BILOU (also called BIOES) adds explicit last and single-token tags, which some models find easier to learn. Standard BIO cannot represent nested entities ("Bank of [America]LOC" inside an ORG) or overlapping spans; those need span-based or hypergraph models.
A linear-chain conditional random field models P(y | x) for a whole tag sequence, scoring each position's tag from features (emissions) plus a learned score for each tag-to-tag transition, so it can learn that I-PER never follows O. Decoding uses the Viterbi algorithm in O(n·T2) for n tokens and T tags. BiLSTM-CRF replaced hand-written features with learned word and character representations. Today the default is a pretrained transformer encoder with a token-classification head: contextual embeddings capture that "Apple" in "joined Apple" is an organization. A CRF layer on top sometimes helps consistency, especially with small data.
Annotated datasets label words, but transformers see subwords. Use the tokenizer's word_ids() mapping to give the first subword of each word its word's label and set the others to -100 so the loss ignores them (alternatively, propagate I-X to continuation pieces). At inference, aggregate back to words and use character offsets to recover exact spans in the original text.
# pip install spacy transformers torch
# python -m spacy download en_core_web_sm
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Tim Cook joined Apple in March 1998 and later moved to Cupertino.")
for ent in doc.ents:
print(f"{ent.text:15s} {ent.label_:8s} chars {ent.start_char}-{ent.end_char}")
# Rule-based additions for patterns a statistical model handles poorly
ruler = nlp.add_pipe("entity_ruler", before="ner")
ruler.add_patterns([{"label": "TICKER", "pattern": [{"TEXT": {"REGEX": r"^\$[A-Z]{1,5}$"}}]}])
print([(e.text, e.label_) for e in nlp("Shares of $AAPL rose.").ents])
# Transformer token classification with word-level aggregation
from transformers import pipeline
ner = pipeline("token-classification", model="dslim/bert-base-NER",
aggregation_strategy="simple")
for e in ner("Angela Merkel met executives from Siemens in Munich."):
print(e["word"], e["entity_group"], round(float(e["score"]), 3), e["start"], e["end"])
# Label alignment when fine-tuning your own model
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-cased")
words = ["Kubernetes", "scales", "pods"]
labels = [1, 0, 0] # 1 = B-PROD, 0 = O
enc = tok(words, is_split_into_words=True)
aligned, prev = [], None
for wid in enc.word_ids():
aligned.append(-100 if wid is None or wid == prev else labels[wid])
prev = wid
print(list(zip(enc.tokens(), aligned)))
For domain entities, fine-tune with AutoModelForTokenClassification and DataCollatorForTokenClassification, or train a spaCy pipeline from a config. A few hundred to a few thousand carefully annotated sentences per entity type is a common starting range; annotation tools such as Prodigy, Label Studio or doccano speed this up.
Token accuracy is inflated by the many O tokens. The standard (CoNLL) metric is exact-match span F1: a predicted entity counts only if both its boundaries and its type match the gold span. Report precision, recall and F1 per type and micro-averaged; the seqeval library implements this. For error analysis, also count partial-boundary matches and type confusions separately, since "right span, wrong type" and "off by one token" call for different fixes. Evaluate on text from the target domain; models trained on news often drop sharply on clinical notes, legal documents or social media.
Instruction-tuned LLMs can extract entities zero-shot from a type description, which is valuable for new or rare entity types and for bootstrapping training data. Ask for structured output (JSON with entity text, type and character offsets) and verify that each returned span actually occurs in the input, since generative models can paraphrase or hallucinate spans. For high-volume pipelines, a common pattern is to label data with an LLM, have humans review a sample, and fine-tune a small encoder that runs at a fraction of the cost. Compact generalist models such as GLiNER take entity type names as input and offer a middle ground.
| Need | Approach |
|---|---|
| Standard types on general text, fast | spaCy or a pretrained BERT NER model |
| Structured identifiers (emails, IBANs, SKUs) | Regex / entity ruler, with validation checksums |
| Custom domain types with labeled data | Fine-tuned transformer token classifier |
| Custom types, no labels yet | LLM or GLiNER zero-shot, then distill |
| PII redaction | Ensemble of model + rules, tuned for recall |
Take the sentence "Barack Obama was born in Honolulu, Hawaii, and studied at Columbia University." The gold annotation has four entities: Barack Obama (PER), Honolulu (LOC), Hawaii (LOC) and Columbia University (ORG). A model predicts four spans: Barack Obama (PER), Honolulu (LOC), Hawaii (ORG) and Columbia (ORG).
| Scoring scheme | Correct | Precision | Recall | F1 |
|---|---|---|---|---|
| Exact span + type (CoNLL) | 2 | 2/4 = 0.50 | 2/4 = 0.50 | 0.50 |
| Exact span, type ignored | 3 (Hawaii counts) | 0.75 | 0.75 | 0.75 |
| Overlapping span + correct type | 3 (Columbia counts) | 0.75 | 0.75 | 0.75 |
Under strict scoring, a boundary error counts twice in spirit: the truncated Columbia is a false positive and the full Columbia University is a false negative. Relaxed schemes, such as those defined for the MUC and SemEval 2013 evaluations, separate boundary errors from type errors, which is useful for diagnosis even when the strict number is the one you report. A confusion breakdown by error kind (missed, spurious, wrong type, wrong boundary) usually tells you whether to fix the annotation guidelines, the tokenization or the model.
NER answers "which spans are entities, and of what type". Most applications then need entity linking: mapping each span to a unique record in a knowledge base such as Wikidata or an internal customer table. Linking has two stages. Candidate generation looks up the surface form in an alias table (names, abbreviations, misspellings) to produce a short list. Disambiguation ranks those candidates using the surrounding context, often by comparing an embedding of the mention context with an embedding of each candidate's description, plus a prior for how often the alias refers to each candidate. Spans with no good candidate are flagged as new entities (NIL) rather than forced onto the nearest record.
Downstream, linked entities populate knowledge graphs, drive faceted search and support relation extraction ("X acquired Y"). For PII redaction, linking is usually unnecessary, but recall is paramount; combine a model with pattern rules, tune the threshold for recall, and audit a sample of the redacted output regularly. For retrieval systems, extracting entities from documents and queries is a cheap way to add exact-match filters on top of vector search.