NER

Some of your RAG-related issues have an easy & quick solution: lemmatization

.bitext-example-box p { margin: 0 0 10px; font-size: 16px; color: #333333; line-height: 1.6; }

.bitext-example-box p:last-child { margin-bottom: 0; }

.bitext-highlight { display: inline-block; background: #fdeaea; color: #b71c1c; font-weight: 700; padding: 2px 6px; border-radius: 4px; }

.bitext-benefits { background: #fafafa; border: 1px solid #e6e6e6; padding: 14px 16px; margin: 18px 0 22px; border-radius: 6px; }

.bitext-benefits ul { margin: 0; padding-left: 20px; }

.bitext-benefits li { margin: 6px 0; font-size: 16px; color: #333333; line-height: 1.6; }

Some RAG issues have a simpler fix than people think: better text normalization.

One common culprit is stemming. Stemming is a blunt, error-prone approach: it strips word endings mechanically, without properly accounting for morphology, part of speech, or context. That can and will often collapse unrelated words into the same stem just because they look similar on the surface.

The result is noisy normalization.

For example, in English, according to the widely used Porter stemmer:

“organization” is wrongly linked to “organ”

“news” is wrongly associated to “new”

“united” is wrongly connected to “unit”

In languages with more complex morphologies like Spanish, German, French, Italian and others, these problems get worse.

Since stemming is performed at the beginning of the text analysis process, these errors affect every task that follows. The noise does not stay contained. It flows downstream into indexing, retrieval, and search, which means some of the “RAG problems” teams run into actually begin much earlier in the pipeline.


Why lemmatization is different

Lemmatization avoids these noisy associations. Instead of chopping words mechanically, lemmatization maps inflected forms to their correct dictionary form, typically using morphological analysis and part-of-speech information.

That makes it much better at normalizing real linguistic variation while avoiding many of the false matches that stemming introduces.

In the above examples:

“organization” is correctly linked to “organizations”

“news” is not associated to “new”; they are independent, unrelated words

“united” is properly connected to “unite”

Also, lemmatization is a fully deterministic, consistent and reliable process.

  • fewer false positives
  • cleaner indexing
  • better retrieval quality
  • more robust multilingual search

And since retrieval quality is critical for RAG, improving normalization upstream can have an outsized impact downstream.


The real source of some RAG issues

A lot of teams treat retrieval issues as if they were generation issues.

Often, they are not.

Sometimes the problem starts with stemming.

For a deeper understanding of how normalization impacts search relevance, check out this post on lemmatization vs stemming.

 

admin

Recent Posts

Bitext Linguistic Analyzer Improves Elasticsearch’s English Search Quality

Our earlier German MIRACL benchmark showed that linguistic analysis can substantially improve lexical search in…

16 hours ago

Hybrid Search Is the New Baseline: Better Lexical Search Is the Next Advantage

Vector search, also known as semantic search, has transformed enterprise search in the past few…

2 months ago

Bitext Linguistic Analyzer beats every Elasticsearch Configuration on German Search Quality

In a search benchmark for German, Bitext Linguistic Analysis SDK returned more relevant results and…

2 months ago

Stemming kills AI Accuracy: Why German Search Needs Lemmatization

Search systems have relied on stemming for decades. The reason is simple: stemming is fast,…

3 months ago

Some of your RAG-related issues have an easy & quick solution: decompounding

Most teams working with Elasticsearch, OpenSearch or RAG pipelines focus on ranking, embeddings or model…

5 months ago

The Hidden Signal in Millions of News Articles That Reveals How Global Narratives Form

The Experiment We tested this idea using the Leipzig English News corpora from the Wortschatz…

7 months ago