Find the people and organisations named in an article
List the proper names mentioned in a text and tell what kind of entity each one is.
AI, but a small model on your own machines
négligeableFastYour data never leaves
The verdict
N2, because the hard half of the need — saying what a name is — is written nowhere in the text, and a model of a few tens of megabytes running on your machine settles it. That is not a retreat from frugality, it is where frugality leads: the pipeline is local, free, deterministic, and it does not send a list of people’s names anywhere. Level N0 keeps its full use, and it does two things the model does not. It answers `unknown` when the sentence proves nothing, where the pipeline always claims something. And it counts what it let pass — the one-word names at the start of a sentence — instead of keeping quiet about them. One decision is owned along the way: “et” is not allowed inside a name, which cuts “Marks and Spencer” into two pieces, because allowing it merged “Alex Ferguson et Acme” into a single entity — inventing one entity out of two is worse than returning one name in two pieces.
An entity recognition pipeline loaded locally
AI, but a small model on your own machines
Small self-hosted specialised modelN2négligeableFast
Python
"""
Type the names with a small pre-trained recogniser, on your own machine.
Rung N2. This is the level that answers the half rung N0 cannot: what a name
*is*, when the sentence does not say. The model is a few tens of megabytes, it
runs on a processor in milliseconds, and nothing leaves the machine — which
matters more here than elsewhere, because what is being extracted is a list of
people.
What you own on this rung is not the model. It is the cap, the batch, the map
from its label set to yours, and the answer to « what does the code do with a
label we do not map ». The French pipeline of spaCy has four labels — person,
organisation, place, and a fourth that means « something else » — where this
entry has three meanings. `MISC` is therefore dropped rather than guessed, and
counted so that the caller sees how much was dropped.
The limit to keep in mind is the shape of the answer, not its quality: the
pipeline returns a label for every name it finds, and no score alongside it.
There is no threshold to raise and no « unknown » to fall back on. Rung N0
answers `unknown` when the sentence proves nothing; here, something is always
claimed.
"""
from __future__ import annotations
# The pipeline name is an example: pick the one trained for your language, and
# read its licence — the French pipelines of spaCy are published under
# LGPL-LR, which is not the licence of spaCy itself.
MODEL = "fr_core_news_sm"
# A document longer than this is cut, not refused: a recogniser that raises on
# a long article returns nothing at all.
MAX_CHARACTERS = 20000
# Their label set is theirs, and wider than ours. What is not mapped is
# dropped on purpose: an unmapped label that silently became a type would be a
# surprise in a list of people.
LABELS = {"PER": "person", "PERSON": "person", "ORG": "company", "LOC": "place",
"GPE": "place"}
class SpacyRecogniser:
"""The real recogniser: a pipeline loaded once, then fed in batches."""
def __init__(self, model: str = MODEL):
import spacy # a large local install, and a model to download
self._nlp = spacy.load(model, disable=["lemmatizer", "textcat"])
def predict(self, texts: list[str]) -> list[dict]:
return [{"entities": [{"text": entity.text, "label": entity.label_,
"start": entity.start_char, "end": entity.end_char}
for entity in document.ents]}
for document in self._nlp.pipe(texts)]
class RecognitionUnavailable(Exception):
"""The recogniser could not be loaded, or answered something unusable."""
def extract_names(texts, recogniser=None) -> list[dict]:
"""
For each text, the names the model found, with its labels mapped to ours.
`recogniser` is injected so this function can be tested without the model.
In production it defaults to the real pipeline.
"""
recogniser = recogniser or SpacyRecogniser()
cut = [text[:MAX_CHARACTERS] if isinstance(text, str) else "" for text in texts]
try:
answers = recogniser.predict(cut)
except Exception as error: # noqa: BLE001 - a local model failing is still a failure
raise RecognitionUnavailable(str(error)) from error
if not isinstance(answers, list) or len(answers) != len(cut):
raise RecognitionUnavailable("the recogniser did not answer once per text")
reports = []
for text, answer in zip(cut, answers):
names, unmapped = [], []
for entity in (answer or {}).get("entities", []):
label = str(entity.get("label", ""))
found = str(entity.get("text", ""))
if label not in LABELS:
unmapped.append(label)
elif found and found in text:
names.append({"text": found, "type": LABELS[label], "evidence": label,
"start": int(entity.get("start", text.index(found))),
"end": int(entity.get("end", text.index(found) + len(found)))})
reports.append({"names": names, "unmapped": unmapped,
"characters_read": len(text)})
return reportsJavaScript
/**
* Type the names with a small pre-trained recogniser, on your own machine.
*
* Rung N2. This is the level that answers the half rung N0 cannot: what a name
* *is*, when the sentence does not say. The model is a few tens of megabytes,
* it runs on a processor in milliseconds, and nothing leaves the machine —
* which matters more here than elsewhere, because what is being extracted is a
* list of people.
*
* What you own on this rung is not the model. It is the cap, the batch, the
* map from its label set to yours, and the answer to « what does the code do
* with a label we do not map ». The multilingual pipelines carry four labels —
* person, organisation, place, and a fourth that means « something else » —
* where this entry has three meanings. `MISC` is therefore dropped rather than
* guessed, and counted so that the caller sees how much was dropped.
*
* The limit to keep in mind is the shape of the answer, not its quality: the
* pipeline returns a label for every name it finds. Rung N0 answers `unknown`
* when the sentence proves nothing; here, something is always claimed.
*/
// The model name is an example: pick one trained for your language, and read
// its licence, which is not the licence of the runtime that loads it.
export const MODEL = 'Xenova/bert-base-multilingual-cased-ner-hrl';
// A document longer than this is cut, not refused: a recogniser that throws on
// a long article returns nothing at all.
export const MAX_CHARACTERS = 20000;
// Their label set is theirs, and wider than ours. What is not mapped is
// dropped on purpose: an unmapped label that silently became a type would be a
// surprise in a list of people.
export const LABELS = {
PER: 'person', PERSON: 'person', ORG: 'company', LOC: 'place', GPE: 'place',
};
/** The real recogniser: a pipeline loaded once, then fed in batches. */
export async function transformersRecogniser(model = MODEL) {
const { pipeline } = await import('@huggingface/transformers'); // a large local install
const run = await pipeline('token-classification', model, { aggregation_strategy: 'simple' });
return {
async predict(texts) {
const answers = await run(texts);
return answers.map((entities) => ({
entities: entities.map((entity) => ({
text: entity.word,
label: entity.entity_group,
start: entity.start,
end: entity.end,
})),
}));
},
};
}
export class RecognitionUnavailable extends Error {}
/**
* For each text, the names the model found, with its labels mapped to ours.
*
* `recogniser` is injected so this function can be tested without the model.
* In production it defaults to the real pipeline.
*/
export async function extractNames(texts, recogniser) {
recogniser ??= await transformersRecogniser();
const cut = texts.map((text) => (typeof text === 'string' ? text.slice(0, MAX_CHARACTERS) : ''));
let answers;
try {
answers = await recogniser.predict(cut);
} catch (error) {
throw new RecognitionUnavailable(String(error));
}
if (!Array.isArray(answers) || answers.length !== cut.length) {
throw new RecognitionUnavailable('the recogniser did not answer once per text');
}
return cut.map((text, index) => {
const names = [];
const unmapped = [];
for (const entity of (answers[index] ?? {}).entities ?? []) {
const label = String(entity.label ?? '');
const found = String(entity.text ?? '');
if (!(label in LABELS)) {
unmapped.push(label);
} else if (found && text.includes(found)) {
names.push({
text: found,
type: LABELS[label],
evidence: label,
start: Number(entity.start ?? text.indexOf(found)),
end: Number(entity.end ?? text.indexOf(found) + found.length),
});
}
}
return { names, unmapped, characters_read: text.length };
});
}Try it
This code cannot run in a browser. The outputs below are computed when the site is built, by running the real code.
The labels the model returns are simulated by the local double the tests use. What is computed here is what the code does with them: what it maps, what it drops, and what it refuses.
A press paragraph, with its markers
Given
The contract binds the company Lumiere Ltd to Jean de La Fontaine and to Mrs Marie Martin, of Lyon.
What it returns
3 name(s) typed by the model
In detail
Name Type returned Model label Jean de La Fontaine person PER Marie Martin person PER Lyon place LOC Without a model, level N0 answers: « Lumiere Ltd » company, « Jean de La Fontaine » unknown, « Marie Martin » person, « Lyon » unknown.
The same paragraph, with no marker at all
Given
The contract binds Lumiere to Jean de La Fontaine and to Marie Martin, of Lyon.
What it returns
3 name(s) typed by the model
In detail
Name Type returned Model label Jean de La Fontaine person PER Marie Martin person PER Lyon place LOC Without a model, level N0 answers: « Lumiere » unknown, « Jean de La Fontaine » unknown, « Marie Martin » unknown, « Lyon » unknown.
A label this entry cannot map
Given
The Goncourt prize was awarded to Marie Martin in Paris.
What it returns
2 name(s) typed by the model · 1 label(s) dropped for want of a mapping
In detail
Name Type returned Model label Marie Martin person PER Paris place LOC Without a model, level N0 answers: « The Goncourt » unknown, « Marie Martin » unknown, « Paris » unknown.
The model returns a name the text does not contain
Given
The parcel leaves Lyon on Wednesday.
What it returns
1 name(s) typed by the model
In detail
Name Type returned Model label Lyon place LOC Without a model, level N0 answers: « Lyon » unknown, « Wednesday » unknown.
The model gets the type wrong, and nothing lets you doubt it
Given
The parcel leaves Lyon on Wednesday.
What it returns
1 name(s) typed by the model
In detail
Name Type returned Model label Lyon company ORG Without a model, level N0 answers: « Lyon » unknown, « Wednesday » unknown.
Here it breaksThe model returns “Lyon” as a company, and the report shows it as one: there is no score, no threshold, and no “unknown” to fall back on, because the pipeline returns none. What the code can check — that the name really is in the text, that the label can be mapped — is checked; what it cannot check is whether the label is right. Level N0 answers “unknown” on that same name, which is exactly what the sentence proves: the note under the table shows it on every case.
The model worker dies
Given
the same text, but the model does not answer
What it returns
The model did not answer
The code throws a named error rather than returning an empty list, which would read as “no name in this text”.
Without a model, level N0 answers: « Lyon » unknown, « Wednesday » unknown.
What this means for you
- Your data stays put
- Always the same result
- Measured on a sample, not case by case
- Depends on a library
- Moderate footprint
What this adds to your regulatory scope
- No specific scope added: the model runs on your machine and the text does not leave it
- The model’s licence is to be read apart from the library’s: spaCy’s French pipelines are published under LGPL-LR, where spaCy itself is MIT
Breaking point
The pipeline returns a label for every name it finds, and nothing beside it: no score, so no threshold to raise and no `unknown` to fall back on — “Lyon” returned as a company enters the list as a company. Witness in the same test: on that same name, level N0 answers `unknown`.
When to move up a level
There is no level above this one worth taking here. What this level lacks — knowing that it does not know — is not something a paid call provides.
The four options
- N0The capitals, and a type only where the sentence proves itFree · Instant
- N1Not worth it here
- RecommendedN2An entity recognition pipeline loaded locallynégligeable · Fast
- N3Not worth it here
The other options
N0The capitals, and a type only where the sentence proves itFree · Instant
The capitals, and a type only where the sentence proves it
Rule and classic algorithmN0FreeInstant
Python
"""
The proper names an article carries, typed only where something proves the type.
Rung N0. In French as in English a proper name is capitalised, so finding the
candidates is a scan. Saying what each one *is* — a person, a company, a place
— is the hard half, and nothing in the letters themselves says it: « Boulanger »
is a family name and a chain of shops, « Orange » is a company and a fruit,
« Lyon » is a city and a surname.
So this rung types a name only when the sentence proves it. « M. » or « Mme »
before it proves a person; a legal form such as « SARL » or « Ltd » after it,
or « la société » before it, proves a company. Everything else comes back as
`unknown`, which is what the sentence actually supports. Guessing from the
shape of the word is how « Boulanger a livré le colis » becomes a bakery.
Two limits are counted rather than hidden. A single capitalised word opening a
sentence is not returned, because most of them are just the first word of the
sentence; `skipped_at_sentence_start` says how many were passed over. And the
markers below are French and English ones: they are a declared list, they are
meant to be extended, and a text that carries none of them comes back entirely
`unknown`.
"""
from __future__ import annotations
import re
# What proves a person, before the name.
HONORIFICS = {"m", "m.", "mme", "mlle", "dr", "dr.", "me", "pr", "prof", "prof.",
"mr", "mr.", "mrs", "mrs.", "ms", "ms.", "miss", "sir"}
# What proves a company, after the name.
LEGAL_FORMS = {"sarl", "sas", "sasu", "sa", "snc", "sci", "eurl", "scop", "gie",
"gmbh", "ag", "ltd", "ltd.", "inc", "inc.", "llc", "plc", "bv", "nv", "spa"}
# What proves a company, before the name. Two words at most, lowercase.
ORG_LEADS = {("la", "société"), ("l", "entreprise"), ("le", "groupe"), ("la", "marque"),
("the", "company"), ("the", "firm"), ("the", "group"), ("l", "enseigne")}
# Lowercase words a name may contain without being cut: « Jean de La Fontaine »,
# « Ludwig van Beethoven », « Banque de France ». « et » and « and » are
# deliberately absent, although « Marks and Spencer » wants them: keeping them
# merges « Alex Ferguson et Acme » into a single name, and inventing one entity
# out of two is worse than returning one name in two pieces.
PARTICLES = {"de", "du", "des", "da", "della", "van", "von", "der", "den", "ten",
"of", "la", "le", "les", "el", "al", "d", "l", "o"}
# Words that open a sentence and are never part of a name. A capitalised run
# that starts a sentence with one of them starts one word later: « Après
# Renault » is Renault, « Selon Le Monde » is Le Monde. The list is declared
# rather than inferred, like HONORIFICS and LEGAL_FORMS, because a rung that
# guessed which openers are names would be guessing exactly what it says it
# will not — and it is meant to be extended.
#
# Articles and particles are deliberately absent from it: « Le Monde » opens a
# sentence with « Le », and dropping that word would lose the newspaper.
SENTENCE_OPENERS = {
"après", "avant", "selon", "depuis", "chez", "pour", "contre", "malgré",
"dès", "lors", "entre", "face", "sans", "sous", "sur", "dans", "avec",
"par", "vers", "pendant", "durant", "outre", "parmi", "quant", "comme",
"alors", "ainsi", "aussi", "cependant", "pourtant", "toutefois", "enfin",
"ensuite", "puis", "donc", "mais", "car", "quand", "lorsque",
"ce", "cet", "cette", "ces", "son", "sa", "ses", "leur", "leurs",
"notre", "votre", "nos", "vos", "mon", "ma", "mes",
"il", "elle", "ils", "elles", "on", "nous", "vous",
"after", "before", "since", "according", "despite", "during", "among",
"between", "through", "under", "over", "about", "against", "within",
"without", "across", "around", "because", "although", "however",
"therefore", "meanwhile", "instead", "finally", "then", "when", "while",
"where", "this", "these", "that", "those", "their", "his", "her", "its",
"our", "your", "but", "and", "for", "nor", "yet", "with", "from",
}
# A word, with neither full stop nor apostrophe inside it. « M. » is therefore
# the token « M » followed by a separator, which keeps an honorific out of the
# name, and « L'enseigne » is two tokens, which keeps the article out of it.
WORD = re.compile(r"[^\W\d_][\w-]*", re.UNICODE)
# What two words of one name may be separated by: a space, or the apostrophe
# French elides on — « Jean d'Artagnan » is one name.
JOINS = re.compile(r"[ \t\r\n\f\v\u00a0\u202f]+|['’]")
SENTENCE_END = re.compile(r"[.!?…:;\n\r]\s*\Z")
def extract_names(text) -> dict:
"""
The proper names of `text`, each with what the sentence proves about it.
`type` is « person », « company » or « unknown », and `evidence` says what
proved it. A name with no evidence is never typed by its spelling.
`skipped_at_sentence_start` counts the capitalised words this rung set
aside because they open a sentence: a word alone, which it cannot tell from
a name, and the opener in front of a name — « Après » in « Après Renault ».
What this rung has no way to recognise at all is what a name is *of*: a
product reference like « A350 » in « Airbus a livré son premier A350 » is
capitalised, follows no opener and carries no evidence, so it comes back as
an `unknown` name. That is a rung above.
"""
if not isinstance(text, str):
return {"names": [], "skipped_at_sentence_start": 0,
"reason": f"expected text, not {type(text).__name__}"}
tokens = [(m.group(), m.start(), m.end()) for m in WORD.finditer(text)]
names, skipped, index = [], 0, 0
while index < len(tokens):
length = _run_length(tokens, index, text)
if length == 0:
index += 1
continue
start, end = tokens[index][1], tokens[index + length - 1][2]
opens = index == 0 or SENTENCE_END.search(text[tokens[index - 1][2]:start])
if opens and tokens[index][0].lower() in SENTENCE_OPENERS:
# « Après Renault, … » — the name starts one word later, and the
# word set aside is counted rather than swallowed.
skipped += 1
index += 1
continue
kind, evidence = _typed(tokens, index, length)
if length > 1 and tokens[index][0].lower() in HONORIFICS:
start = tokens[index + 1][1] # « Mme » is not part of the name
alone = tokens[index][0].lower()
ordinary = alone in PARTICLES or alone in HONORIFICS or len(alone) == 1
if length == 1 and opens and kind == "unknown" and not ordinary:
# « Lumière a livré le colis. » — most sentence openers are not
# names, and this rung has no way to tell which ones are. An
# opener that is plainly a determiner is not even counted.
skipped += 1
elif not (length == 1 and ordinary):
names.append({"text": text[start:end], "type": kind, "evidence": evidence,
"start": start, "end": end})
index += length
return {"names": names, "skipped_at_sentence_start": skipped, "reason": None}
def _run_length(tokens, index: int, text: str) -> int:
"""How many tokens from `index` belong to one capitalised run."""
if not tokens[index][0][:1].isupper():
return 0
length, last = 1, index
while last + 1 < len(tokens):
word = tokens[last + 1][0]
if not JOINS.fullmatch(text[tokens[last][2]:tokens[last + 1][1]]):
break
if word[:1].isupper():
last += 1
elif word.lower() in PARTICLES and last + 2 < len(tokens) \
and tokens[last + 2][0][:1].isupper():
last += 2 # a particle only counts between two capitalised words
else:
break
length = last - index + 1
return length
def _typed(tokens, index: int, length: int):
"""What the sentence proves about this run, and the word that proves it."""
before = [tokens[i][0].lower() for i in range(max(0, index - 2), index)]
run = [tokens[i][0].lower() for i in range(index, index + length)]
after = tokens[index + length][0].lower() if index + length < len(tokens) else ""
if before and before[-1] in HONORIFICS:
return "person", before[-1]
if length > 1 and run[0] in HONORIFICS:
return "person", run[0] # « Sir Alex », written without a full stop
if run[-1] in LEGAL_FORMS and length > 1:
return "company", run[-1]
if after in LEGAL_FORMS:
return "company", after
if len(before) == 2 and tuple(before) in ORG_LEADS:
# « l'entreprise » is two tokens; it is written back as one word.
return "company", ("'" if len(before[0]) == 1 else " ").join(before)
return "unknown", NoneJavaScript
/**
* The proper names an article carries, typed only where something proves the type.
*
* Rung N0. In French as in English a proper name is capitalised, so finding
* the candidates is a scan. Saying what each one *is* — a person, a company, a
* place — is the hard half, and nothing in the letters themselves says it:
* « Boulanger » is a family name and a chain of shops, « Orange » is a company
* and a fruit, « Lyon » is a city and a surname.
*
* So this rung types a name only when the sentence proves it. « M. » or
* « Mme » before it proves a person; a legal form such as « SARL » or « Ltd »
* after it, or « la société » before it, proves a company. Everything else
* comes back as `unknown`, which is what the sentence actually supports.
* Guessing from the shape of the word is how « Boulanger a livré le colis »
* becomes a bakery.
*
* Two limits are counted rather than hidden. A single capitalised word opening
* a sentence is not returned, because most of them are just the first word of
* the sentence; `skipped_at_sentence_start` says how many were passed over.
* And the markers below are French and English ones: they are a declared list,
* they are meant to be extended, and a text that carries none of them comes
* back entirely `unknown`.
*/
// What proves a person, before the name.
export const HONORIFICS = new Set(['m', 'm.', 'mme', 'mlle', 'dr', 'dr.', 'me', 'pr',
'prof', 'prof.', 'mr', 'mr.', 'mrs', 'mrs.', 'ms', 'ms.', 'miss', 'sir']);
// What proves a company, after the name.
export const LEGAL_FORMS = new Set(['sarl', 'sas', 'sasu', 'sa', 'snc', 'sci', 'eurl',
'scop', 'gie', 'gmbh', 'ag', 'ltd', 'ltd.', 'inc', 'inc.', 'llc', 'plc', 'bv', 'nv', 'spa']);
// What proves a company, before the name. Two words at most, lowercase.
const ORG_LEADS = new Set(['la société', 'l entreprise', 'le groupe', 'la marque',
'the company', 'the firm', 'the group', 'l enseigne']);
// Lowercase words a name may contain without being cut: « Jean de La Fontaine »,
// « Ludwig van Beethoven », « Banque de France ». « et » and « and » are
// deliberately absent, although « Marks and Spencer » wants them: keeping them
// merges « Alex Ferguson et Acme » into a single name, and inventing one entity
// out of two is worse than returning one name in two pieces.
export const PARTICLES = new Set(['de', 'du', 'des', 'da', 'della', 'van', 'von',
'der', 'den', 'ten', 'of', 'la', 'le', 'les', 'el', 'al', 'd', 'l', 'o']);
// A word, with neither full stop nor apostrophe inside it. « M. » is therefore
// the token « M » followed by a separator, which keeps an honorific out of the
// name, and « L'enseigne » is two tokens, which keeps the article out of it.
const WORD = /\p{L}[\p{L}\p{N}_-]*/gu;
// Words that open a sentence and are never part of a name. A capitalised run
// that starts a sentence with one of them starts one word later: « Après
// Renault » is Renault, « Selon Le Monde » is Le Monde. The list is declared
// rather than inferred, like HONORIFICS and LEGAL_FORMS, because a rung that
// guessed which openers are names would be guessing exactly what it says it
// will not — and it is meant to be extended.
//
// Articles and particles are deliberately absent from it: « Le Monde » opens a
// sentence with « Le », and dropping that word would lose the newspaper.
export const SENTENCE_OPENERS = new Set([
'après', 'avant', 'selon', 'depuis', 'chez', 'pour', 'contre', 'malgré',
'dès', 'lors', 'entre', 'face', 'sans', 'sous', 'sur', 'dans', 'avec',
'par', 'vers', 'pendant', 'durant', 'outre', 'parmi', 'quant', 'comme',
'alors', 'ainsi', 'aussi', 'cependant', 'pourtant', 'toutefois', 'enfin',
'ensuite', 'puis', 'donc', 'mais', 'car', 'quand', 'lorsque',
'ce', 'cet', 'cette', 'ces', 'son', 'sa', 'ses', 'leur', 'leurs',
'notre', 'votre', 'nos', 'vos', 'mon', 'ma', 'mes',
'il', 'elle', 'ils', 'elles', 'on', 'nous', 'vous',
'after', 'before', 'since', 'according', 'despite', 'during', 'among',
'between', 'through', 'under', 'over', 'about', 'against', 'within',
'without', 'across', 'around', 'because', 'although', 'however',
'therefore', 'meanwhile', 'instead', 'finally', 'then', 'when', 'while',
'where', 'this', 'these', 'that', 'those', 'their', 'his', 'her', 'its',
'our', 'your', 'but', 'and', 'for', 'nor', 'yet', 'with', 'from',
]);
// What two words of one name may be separated by: a space, or the apostrophe
// French elides on — « Jean d'Artagnan » is one name.
const JOINS = /^([ \t\r\n\f\v ]+|['’])$/;
const SENTENCE_END = /[.!?…:;\n\r]\s*$/;
const UPPER = /^\p{Lu}/u;
/**
* The proper names of `text`, each with what the sentence proves about it.
*
* `type` is « person », « company » or « unknown », and `evidence` says what
* proved it. A name with no evidence is never typed by its spelling.
*
* `skipped_at_sentence_start` counts the capitalised words this rung set aside
* because they open a sentence: a word alone, which it cannot tell from a
* name, and the opener in front of a name — « Après » in « Après Renault ».
*
* What this rung has no way to recognise at all is what a name is *of*: a
* product reference like « A350 » in « Airbus a livré son premier A350 » is
* capitalised, follows no opener and carries no evidence, so it comes back as
* an `unknown` name. That is a rung above.
*/
export function extractNames(text) {
if (typeof text !== 'string') {
return { names: [], skipped_at_sentence_start: 0, reason: `expected text, not ${typeof text}` };
}
const tokens = [...text.matchAll(WORD)]
.map((m) => [m[0], m.index, m.index + m[0].length]);
const names = [];
let skipped = 0;
let index = 0;
while (index < tokens.length) {
const length = runLength(tokens, index, text);
if (length === 0) {
index += 1;
continue;
}
let start = tokens[index][1];
const end = tokens[index + length - 1][2];
const opens = index === 0 || SENTENCE_END.test(text.slice(tokens[index - 1][2], start));
if (opens && SENTENCE_OPENERS.has(tokens[index][0].toLowerCase())) {
// « Après Renault, … » — the name starts one word later, and the word
// set aside is counted rather than swallowed.
skipped += 1;
index += 1;
continue;
}
const [kind, evidence] = typed(tokens, index, length);
if (length > 1 && HONORIFICS.has(tokens[index][0].toLowerCase())) {
start = tokens[index + 1][1]; // « Mme » is not part of the name
}
const alone = tokens[index][0].toLowerCase();
const ordinary = PARTICLES.has(alone) || HONORIFICS.has(alone) || alone.length === 1;
if (length === 1 && opens && kind === 'unknown' && !ordinary) {
// « Lumière a livré le colis. » — most sentence openers are not names,
// and this rung has no way to tell which ones are. An opener that is
// plainly a determiner is not even counted.
skipped += 1;
} else if (!(length === 1 && ordinary)) {
names.push({ text: text.slice(start, end), type: kind, evidence, start, end });
}
index += length;
}
return { names, skipped_at_sentence_start: skipped, reason: null };
}
/** How many tokens from `index` belong to one capitalised run. */
function runLength(tokens, index, text) {
if (!UPPER.test(tokens[index][0])) return 0;
let length = 1;
let last = index;
while (last + 1 < tokens.length) {
const word = tokens[last + 1][0];
if (!JOINS.test(text.slice(tokens[last][2], tokens[last + 1][1]))) break;
if (UPPER.test(word)) {
last += 1;
} else if (PARTICLES.has(word.toLowerCase()) && last + 2 < tokens.length
&& UPPER.test(tokens[last + 2][0])) {
last += 2; // a particle only counts between two capitalised words
} else {
break;
}
length = last - index + 1;
}
return length;
}
/** What the sentence proves about this run, and the word that proves it. */
function typed(tokens, index, length) {
const before = tokens.slice(Math.max(0, index - 2), index).map((t) => t[0].toLowerCase());
const run = tokens.slice(index, index + length).map((t) => t[0].toLowerCase());
const after = index + length < tokens.length ? tokens[index + length][0].toLowerCase() : '';
if (before.length && HONORIFICS.has(before.at(-1))) return ['person', before.at(-1)];
if (length > 1 && HONORIFICS.has(run[0])) return ['person', run[0]];
if (LEGAL_FORMS.has(run.at(-1)) && length > 1) return ['company', run.at(-1)];
if (LEGAL_FORMS.has(after)) return ['company', after];
if (before.length === 2 && ORG_LEADS.has(before.join(' '))) {
// « l'entreprise » is two tokens; it is written back as one word.
return ['company', before.join(before[0].length === 1 ? "'" : ' ')];
}
return ['unknown', null];
}What this means for you
- Your data stays put
- Always the same result
- Unit-testable, case by case
- No provider in the loop
- Negligible footprint
What this adds to your regulatory scope
- No specific scope added: the text is read inside your own process and is sent to nobody
- A list of people’s names is personal data processing, whichever level produced it
Breaking point
With no marker, the sentence proves nothing: in “Le colis a été livré par Boulanger”, the name comes back `unknown`, and it is a family name as much as a shop sign. Witness in the same test: the same sentence with “la société” in front gives a company, and with “M.” in front, a person.
When to move up a level
Your texts do not carry those markers. Press prose names people and companies without ever writing “la société”, and that is the ordinary case: the share of `unknown` in the report will tell you before you notice.
N1Not worth it here
Not worth it here
Lightweight classic modelN1
The classic model for this problem is a conditional random field over word features — sklearn-crfsuite does it — and it needs a corpus annotated entity by entity in the language handled. That is precisely what you do not have: annotating a few thousand sentences costs more than everything else on this page. The pipelines of the level above exist because that annotation was done once, publicly, and ships with them.
N3Not worth it here
Not worth it here
General-purpose LLM APIN3
Sending a whole article to a third party to get the list of people it names, per document and over an archive counted in thousands, is the case where cost and regulatory scope point the same way. And the result is no more checkable than at level N2: a general-purpose model will happily add names that are not in the text, where the local pipeline returns spans of the document.
Further reading
- spaCy — les pipelines français, leurs étiquettes et leur licence
- Transformers.js — faire tourner un modèle de reconnaissance d’entités dans Node
- sklearn-crfsuite — le champ aléatoire conditionnel, et le corpus annoté qu’il réclame
- Masquer les données personnelles dans un échange — la fiche voisine, quand le but est de les cacher et non de les lister
Where the figures come from
- spaCy, modèles français (consulté) : `fr_core_news_sm` 3.8.0, licence LGPL-LR, quatre étiquettes d’entités — LOC, MISC, ORG, PER
- spaCy sur PyPI (consulté) : version 3.8.16, publiée le 24 août 2026, licence MIT — celle de la bibliothèque, pas celle des modèles
- @huggingface/transformers, consulté sur le registre npm : version 4.3.0, publiée le 16 septembre 2026, licence Apache 2.0 ; c’est par lui qu’un pipeline de reconnaissance d’entités tourne dans Node
- compromise sur son dépôt (consulté) : « a considerate, common-sense baseline for english grammar » ; les variantes d’autres langues y sont décrites comme des travaux en cours, et `fr-compromise` en est à la version 0.3.1
- sklearn-crfsuite sur PyPI (consulté) : version 0.5.0, publiée le 18 juin 2024, licence MIT
Metadata
General information, kept up to date as best we can. It is not legal advice. Have a professional review your situation.
Share, or paste into a README
The badge, for a README
[](https://isitoverkill.dev/en/fiches/extract-people-and-companies-from-an-article)