Want to see the practical retrieval examples?
Jump to the ten real English retrieval examples.
Our earlier German MIRACL benchmark showed that linguistic analysis can substantially improve lexical search in a morphologically complex language. The obvious follow-up question was harder: does richer linguistic analysis still matter in English, where stemming is mature, inexpensive, and often assumed to be good enough?
We tested three Elasticsearch 9.4.4 analyzer configurations on the public English MIRACL development benchmark:
Elasticsearch Standard: standard tokenization and lowercasing, with no English-specific stemming or lemmatization.
Elasticsearch English: standard tokenization, possessive normalization, lowercasing, English stopword removal, and Porter stemming.
Bitext English: sentence-aware tokenization, possessive normalization, contextual handling of named entities, lemmatization, lowercasing, and English stopword removal.
Bitext English builds upon the components of Elasticsearch English, the default English analyzer, by:
adding more intelligent case handling, including preserving case through lexical analysis, using sentence-aware tokenization to distinguish sentence-initial capitalization from meaningful capitalization, and deferring lowercasing until after lexical and entity decisions have been made;
replacing Porter stemming with dictionary-based lemmatization, while preserving original word forms alongside their lemmas to retain exact-match evidence;
adding contextual handling of named-entity terms, so words that are part of names can be protected from inappropriate ordinary-word normalization while the same forms can still be lemmatized when used as regular vocabulary.
The answer was clear. Compared with Elasticsearch Standard, Bitext English improved the top-10 retrieval metrics by an average of more than 16%:
nDCG@10 by 14.8%.
Recall@10 by 15.5%.
Precision@10 by 19.6%.
F1@10 by 18.5%.
MRR@10 by 13.0%.
And compared with Elasticsearch English, Bitext improved the top-10 metrics by an average of more than 7%:
nDCG@10 by 7.6%.
Recall@10 by 6.3%.
Precision@10 by 7.7%.
F1@10 by 7.5%.
MRR@10 by 7.0%.
For users, the practical result is straightforward: more known-relevant passages appear near the top of the lexical ranking, where they are most likely to be selected, reranked, fused with vector results, or passed to a RAG system or AI agent.
German makes the value of linguistics relatively easy to see. Inflection is richer, compounds are pervasive, and irregular word forms often look very different from one another.
English is often thought to be different. Search systems have used English stemmers for decades, and English morphology is often treated as simple enough that more precise lexical analysis is unnecessary.
But English is not morphology-free. A suffix stemmer does not reliably connect irregular forms such as:
sell and sold;
come and came;
build and built;
make and made.
Stemming can also go too far, and this is probably the main issue with stemming and accuracy: it merges lexical items that are semantically unrelated, as in:
United is stemmed to unit, merging United with unrelated words like units.
Customs is stemmed to custom, merging Customs with unrelated words like customer or customize.
Stemming is also applied to proper names and titles, generating stems that mix unrelated words with brand names, as in:
Booking becoming related to book.
Progressive becoming related to progress.
Named entities add more problems when they contain more than one word. A typical example is General Motors, which is stemmed to gener motor, merging General with unrelated words like generation or generous.
A useful analyzer should not treat every capitalized word as an entity, since many are first words in sentences; but it should not blindly reduce every entity component to an ordinary-word stem either. This benchmark tests whether a linguistically informed analyzer can handle those distinctions well enough to improve actual retrieval.
The evaluation uses the public English development split of MIRACL, a multilingual monolingual-retrieval benchmark built from Wikipedia passages and human relevance assessments.
The final benchmark contains:
32,893,221 English Wikipedia passages.
799 English queries.
8,350 original MIRACL relevance judgments.
All three configurations use the same corpus, queries, judgments, Elasticsearch version, BM25 parameters, index shape, retrieval depth, and metric implementation. The only intended difference is text analysis.
We focus on the first ten results because that is the part of the ranking most relevant to interactive search, reranking, hybrid fusion, RAG candidate selection, and agentic retrieval.
Methodological note. MIRACL, like other pooled information-retrieval benchmarks, does not judge every retrieved query-passage pair. Unjudged passages receive zero relevance credit in the canonical evaluation while remaining at their returned ranks.
Every system uses:
Elasticsearch 9.4.4.
BM25 with the default values of k1 = 1.2 and b = 0.75.
One primary shard and no replicas.
Identical indexed title and passage fields.
The same 799 query set.
The same TREC evaluation code and query population.
No vector model, reranker, LLM, query expansion system, or learned retrieval component is involved. This isolates the effect of lexical text analysis.
Approach 1: Elasticsearch Standard. Elasticsearch’s default standard analyzer performs Unicode-aware tokenization and lowercasing, but no English stemming or lemmatization. This preserves surface distinctions, but misses many useful morphological relationships.
Approach 2: Elasticsearch English. Elasticsearch’s built-in English analyzer adds:
English possessive normalization.
Lowercasing.
English stopword removal.
Porter stemming.
This improves lexical recall over standard analysis, but Porter stemming is algorithmic and suffix-oriented. It produces stems rather than dictionary lemmas, so it can miss irregular relationships and sometimes collapse terms too aggressively.
Approach 3: Bitext English. Bitext English builds on the same general pipeline, while replacing and reordering several steps:
Sentence-aware tokenization adds the context needed for better case and named-entity handling.
English possessive normalization is retained.
Lowercasing is deferred until after lexical and named-entity decisions have been made.
Porter stemming is replaced with dictionary-based lemmatization, while preserving the original word form for stronger exact matching.
Contextual named-entity handling protects terms from inappropriate ordinary-word normalization.
Duplicate removal removes redundant output when original forms and lemmas converge.
English stopword removal is retained.
The sentence-aware logic matters because capitalization is ambiguous. At the start of a sentence, Children may simply be the plural of child. In Save the Children, the same form is part of a name and should remain protected. Bitext uses sentence position and local entity context to make that distinction without requiring a full named-entity extraction model at query time.
| Configuration | nDCG@10 | Recall@10 | Precision@10 | F1@10 | MRR@10 |
|---|---|---|---|---|---|
| Approach 1 – Elasticsearch Standard | 0.250616 | 0.321233 | 0.089362 | 0.129218 | 0.304496 |
| Approach 2 – Elasticsearch English | 0.267361 | 0.348826 | 0.099249 | 0.142433 | 0.321796 |
| Approach 3 – Bitext Linguistic Analysis | 0.287705 | 0.370885 | 0.106884 | 0.153105 | 0.344212 |
Meaning of the metrics:
nDCG@10 measures ranking quality in the first ten results and rewards systems that place known-relevant passages higher.
Recall@10 measures how much of the known-relevant material appears on the first result page.
Precision@10 measures the proportion of the first ten results that is known relevant.
F1@10 balances first-page precision and recall.
MRR@10 measures how early the first known-relevant passage appears.
Bitext produced the best result on every reported top-10 metric.
| Comparison | nDCG@10 | Recall@10 | Precision@10 | F1@10 | MRR@10 |
|---|---|---|---|---|---|
| Bitext English vs Elasticsearch Standard | +14.8% | +15.5% | +19.6% | +18.5% | +13.0% |
| Bitext English vs Elasticsearch English | +7.6% | +6.3% | +7.7% | +7.5% | +7.0% |
The Elasticsearch English configuration clearly improves over Elasticsearch Standard analysis. That confirms that English morphology matters. Bitext English then improves again over the stemmer by replacing approximate stems with dictionary lemmas and adding contextual entity treatment.
The examples show several recurring advantages of using linguistic knowledge. In the annex below, we include real examples of these issues and their solutions. The main issues are:
Avoiding over-stemming. A stemmer may improve recall by conflating terms, but the conflation is not always linguistically sound. For example, the Porter stemmer reduces majority to major. Bitext preserves the complete word majority, keeping its specific lexical identity instead of merging it with a broader adjective and noun.
Handling irregular morphology. Bitext connects dictionary forms that suffix stemming misses:
meaning / meant → mean
sell / sold → sell
make / made → make
come / came → come
These are ordinary English relationships, not rare linguistic curiosities. When the query and passage use different irregular forms, a stemmer may leave them disconnected.
Preserving names and titles. Named entities are often built from ordinary words. General and Motors are morphologically analyzable in isolation, but in General Motors they are parts of a brand name. The sentence-aware Bitext analyzer preserves the brand name terms in context instead of reducing them to stems such as gener and motor.
Organizations, products, policies, locations, and technical names often contain words that an ordinary stemmer would transform.
A search analyzer has to make decisions about morphology, possessives, and names before BM25, vector fusion, reranking, or generation can do anything with the text. If those decisions discard useful relationships or introduce noisy conflations, every downstream stage starts from weaker evidence.
A stronger lexical layer has impact at different levels:
Lexical search: more relevant passages appear near the top of the result page.
Hybrid retrieval: the sparse channel contributes better candidates and better exact-term evidence before fusion with embeddings.
RAG: rerankers and language models receive stronger passages to ground the answer.
AI agents: exact names, titles, terminology, and irregular word forms are less likely to prevent retrieval of the required source.
To avoid potential confusion: this benchmark does not claim that linguistic analysis replaces vector retrieval, reranking, or LLM reasoning. It shows that those systems benefit from a stronger lexical foundation. Named entities are particularly important to that complementarity. Semantic models are useful for conceptual similarity; lexical retrieval remains valuable for exact identities, rare names, titles, products, organizations, and terminology. A hybrid system benefits when its lexical analyzer preserves those distinctions instead of turning them into generic stems.
English is often treated as the language where generic stemming is enough. This benchmark suggests otherwise. Across 32.9 million passages and 799 MIRACL queries, Bitext’s linguistically informed English analyzer produced the strongest top-ranked lexical retrieval results:
14.8% higher nDCG@10 than the default analyzer.
15.5% higher Recall@10.
19.6% higher Precision@10.
18.5% higher F1@10.
13.0% higher MRR@10.
The gains come from treating language as language: normalizing possessives, connecting irregular forms through dictionary lemmas, preserving useful surface terms, and using context to avoid damaging names and titles.
For lexical search, that means better first-page results. For hybrid and RAG systems, it means a stronger sparse channel and better candidates before expensive downstream processing begins.
If you would like to evaluate Bitext English analysis in an Elasticsearch, OpenSearch, Solr, hybrid-search, or RAG deployment, contact us and we’ll send the data.
The aggregate benchmark shows that the Bitext configuration delivered the strongest English top-10 results. The following examples make the practical differences easier to see. In every example, the ranks refer to the same passage marked relevant in the original MIRACL judgments.
English query: What is the meaning of public affairs?
Original English Passage: Public Affairs Council. In the 1950s, the concept of corporate public affairs was only beginning to come into vogue, and at the time meant mostly legislature watching and corporate community involvement. Today, the definition of “public affairs” is much broader, encompassing political involvement, lobbying, corporate community involvement, issues management, grassroots advocacy, and public relations.
The Language Issue. The query asks for the meaning of a term, while the defining passage says what the term meant. The noun/gerund form and the irregular past-tense form belong to the same lexical family, but they do not share a suffix pattern that a light stemmer can reliably normalize.
The Result. Bitext ranks the judged-relevant passage at position 5. The Elasticsearch English analyzer does not return the same passage in the top 100.
The How. Bitext maps both meaning in the query and meant in the passage to the lemma mean. It also maps plural affairs to affair while retaining the original surface form. Elasticsearch obtains mean from the query’s meaning, but does not obtain the same term from the passage’s irregular meant.
English query: How many copies did Salem’s Lot sell?
Original English Passage: ‘Salem’s Lot. In 2005, Centipede Press released a deluxe limited edition of “‘Salem’s Lot” with black and white photographs by Jerry Uelsmann and two short stories. The book was limited to 315 copies. An unsigned hardcover edition limited to 600 copies was later released. Both the signed and unsigned editions were sold out.
The Language Issue. The query uses the base verb sell, while the passage uses the irregular past form sold. It also repeats the plural noun copies. Suffix stemming does not reliably connect sell and sold.
The Result. Bitext ranks the judged-relevant passage at position 1. The Elasticsearch English analyzer ranks the same passage at position 20.
The How. Bitext maps sold in the passage to the lemma sell and maps copies to copy while preserving the original forms. The possessive in Salem’s is normalized before lexical analysis, but the decisive additional query-to-passage link is the irregular sell / sold correspondence.
English query: Can you grow tobacco in Tennessee?
Original English Passage: Types of tobacco. Prior to the American Civil War, most tobacco grown in the US was fire-cured dark-leaf. In the United States, it is grown in northern middle Tennessee, western Kentucky and in Virginia. Fire-cured tobacco grown in Kentucky and Tennessee is used in some chewing tobaccos, moist snuff, some cigarettes and as a condiment leaf in pipe tobacco blends.
The Language Issue. The query uses the base verb grow, while the passage repeatedly uses the irregular past participle grown. Removing or modifying an ending cannot derive one form from the other.
The Result. Bitext ranks the judged-relevant passage at position 3. The Elasticsearch English analyzer ranks the same passage at position 23.
The How. Bitext maps every relevant occurrence of grown in the passage to the lemma grow, which appears directly in the query. The passage then provides the location evidence through repeated mentions of Tennessee. Elasticsearch passage analysis does not produce grow from grown.
English query: When was the Berlin Wall built?
Original English Passage: History of Berlin. On August 13, 1961 the Communist East German government started to build a wall, physically separating West Berlin from East Berlin and the rest of East Germany. The wall was built overnight with no warning.
The Language Issue. The query contains the irregular past-participle form built. The passage uses both the base form build and the participle built. A suffix-oriented stemmer leaves these as separate forms.
The Result. Bitext ranks the judged-relevant passage at position 4. The Elasticsearch English analyzer ranks the same passage at position 16.
The How. Bitext maps built to the dictionary lemma build while retaining the surface form. The query therefore matches both started to build a wall and the wall was built. Elasticsearch keeps built separate from build, so it cannot accumulate the same shared lexical evidence.
English query: What material are stents made out of?
Original English Passage: Drug-eluting stent. The stent platform itself is an expandable framework, generally with an elaborate mesh-like design to allow expansion, flexibility, and in some cases the ability to make/enlarge side openings for side vessels. As of 2009, materials that had been explored included magnesium, polylactic acid, polycarbonate polymers, and salicylic acid polymers.
The Language Issue. The query uses made, while the passage uses make in make/enlarge. It also alternates between singular and plural forms such as stent / stents and material / materials. The irregular verb pair is not recoverable through suffix stripping.
The Result. Bitext ranks the judged-relevant passage at position 6. The Elasticsearch English analyzer ranks the same passage at position 22.
The How. Bitext maps made to make, stents to stent, and materials to material, while retaining the original surface terms. This gives the query direct lexical links to make/enlarge, the repeated mentions of stents, and the sentence listing the materials explored.
English query: What is the majority race in California?
Original English Passage: Race and ethnicity in the United States. White Americans are the majority in 49 of the 50 states, with Hawaii as the exception. Non-Hispanic Whites are the majority in 46 states; Hawaii, New Mexico, California, and Texas are the exceptions. These five jurisdictions have minority majorities.
The Language Issue. The word majority is a complete lexical item with a meaning distinct from major. Elasticsearch’s English stemmer reduces majority to the stem major, discarding the exact term even though the query and passage use it repeatedly.
The Result. Bitext ranks the judged-relevant passage at position 3. The Elasticsearch English analyzer ranks the same passage at position 23.
The How. Bitext preserves majority as the searchable term in both the query and passage, while Elasticsearch represents it as major. That keeps the exact lexical item separate from the broader word major instead of forcing both into one stem.
English query: How long has Days of Our Lives been on?
Original English Passage: Days of Our Lives. Days of Our Lives is an American daytime soap opera broadcast on the NBC television network. It is one of the longest-running scripted television programs in the world, airing nearly every weekday since November 8, 1965.
The Language Issue. The title Days of Our Lives is made from ordinary English words whose lowercase forms are morphologically analyzable. In this context, however, Days and Lives are components of a named work. Generic stemming reduces them to dai and live, losing the specific title surfaces.
The Result. Bitext ranks the judged-relevant passage at position 2. The Elasticsearch English analyzer ranks the same passage at position 24.
The How. The sentence-aware Bitext analyzer preserves the title terms Days and Lives in the query’s entity context. Elasticsearch emits the stems dai and live. The passage repeats the exact series title and states that it has aired nearly every weekday since November 8, 1965.
English query: When was Ukrainization policy first implemented in the Ukraine?
Original English Passage: Ukrainization. Until the early-1930s, Ukrainian culture enjoyed a widespread revival due to Bolshevik policies known as the policy of Korenization. In these years a Ukrainization program was implemented throughout the republic.
The Language Issue. The query distinguishes the policy name Ukrainization from the country Ukraine. Elasticsearch’s stemmer reduces both to ukrain, also collapsing related nationality forms. That removes a useful distinction between the thing being asked about and its geographical context.
The Result. Bitext ranks the judged-relevant passage at position 5. The Elasticsearch English analyzer ranks the same passage at position 22.
The How. Bitext preserves the exact term Ukrainization and keeps it distinct from the query’s Ukraine, whereas Elasticsearch reduces both to ukrain. Bitext also maps implemented to the lemma implement.
English query: Does Christianity come from Judaism?
Original English Passage: Origins of Judaism. For centuries, the traditional understanding has been that Judaism came before Christianity and that Christianity separated from Judaism some time after the destruction of the Second Temple in 70 CE.
The Language Issue. The query uses come, while the passage uses the irregular past form came. At the same time, Christianity is the name of a religion, not merely the adjective or noun Christian. Elasticsearch stems Christianity to christian and does not connect came to come.
The Result. Bitext ranks the judged-relevant passage at position 6. The Elasticsearch English analyzer ranks the same passage at position 38.
The How. Bitext maps came in the passage and come in the query to the shared lemma come, while preserving Christianity and Judaism as complete lexical terms. The passage directly discusses the traditional view that Judaism came before Christianity.
English query: When was Disney’s animated Cinderella first released?
Original English Passage: Cinderella (1950 film). Cinderella is a 1950 American animated musical fantasy film produced by Walt Disney and originally released by RKO Radio Pictures. Based on the fairy tale “Cinderella” by Charles Perrault, it is the twelfth Disney animated feature film.
The Language Issue. The possessive ending in Disney’s must be removed before lexical and entity lookup, but the proper name itself should remain intact. The query and passage also use inflected verb forms such as animated and released. Elasticsearch handles the possessive but then Porter-stems the name and verbs to disnei, anim, and releas.
The Result. Bitext ranks the judged-relevant passage at position 3. The Elasticsearch English analyzer ranks the same passage at position 14.
The How. Bitext first normalizes Disney’s to Disney, then preserves the proper name rather than stemming it. It retains the surface forms animated and released while adding the lemmas animate and release. The ranking gain is best understood as cumulative precise lexical evidence, not possessive stripping alone.
Vector search, also known as semantic search, has transformed enterprise search in the past few…
In a search benchmark for German, Bitext Linguistic Analysis SDK returned more relevant results and…
Search systems have relied on stemming for decades. The reason is simple: stemming is fast,…
Most teams working with Elasticsearch, OpenSearch or RAG pipelines focus on ranking, embeddings or model…
Some RAG issues have a simpler fix than people think: better text normalization. One common…
The Experiment We tested this idea using the Leipzig English News corpora from the Wortschatz…