Stemming kills AI Accuracy: Why German Search Needs Lemmatization

Stemming kills AI Accuracy: Why German Search Needs Lemmatization

Search systems have relied on stemming for decades. The reason is simple: stemming is fast, easy to integrate, and widely available in engines such as Elasticsearch, OpenSearch, Solr, and other Lucene-based platforms.

The goal is reasonable. If a user searches for one form of a word, the search engine should also find closely related forms of the same word. In English, a search for document should match documents. In German, a search for Kind should match Kinder; and a search for Buch should match Bücher.

The problem is that stemming only cuts strings through mechanical rules, it does not understand language, does not understand the morphological rules that affect every word.

For modern search, hybrid retrieval, semantic search, RAG, and agents, this distinction matters. These systems depend on source evidence: the documents, passages, entities, terms, and phrases that must be retrieved (because they appear in user query) before ranking, generation, or any other action can happen. If the retrieval layer connects the wrong words, the AI layer receives noisy evidence. If the retrieval layer misses the right words, the AI layer receives incomplete evidence. In other words: bad normalization (stemming) becomes bad retrieval, and bad retrieval becomes bad AI output.

What Stemming Is Supposed to Do

Stemming tries to reduce word forms to a shorter common representation.

In a good case, this works in German like:

This is useful. A search for Dokument should probably match Dokumente. The arrow “→” means that connecting the two words will help improve the retrieval results.

The same applies to a German noun like Kind:

Again, this is useful. These forms are related according to their morphology. They belong to the same noun lemma. This is the part of stemming that works.

Where Stemming Goes Wrong

The problem is that a stemmer does not know whether two forms are correctly related. It only sees word shape. So it may connect words that look similar but are not related. A good example in German is as follows. For these word groups:

A stemmer will collapse all of these words under the same stem “lag”:

This creates false relations between unrelated words like “Lage/n” and “Lager/n”.

For search, this is not a small technical detail. It means a system can retrieve evidence about Lager when the user’s intent was Lage, simply because both words were mechanically reduced to the same stem. That is false source evidence. This problem is more prominent in languages like German, but t’s also present in English, where stem “custom” will connect unrelated words like “customer”, “customize” and “customs”

Measuring the Impact of Noisy Stemming in German

If stemming can add relevant word connections but it can also introduce noise, it’s important to measure how much noise it introduces in the search process. To quantify the problem, we looked at stem groups rather than isolated word pairs. We started from a and filtered it to 9,124 unique alphabetic German word forms. We then stemmed all of those forms with a standard German Snowball stemmer and grouped together the forms that produced the same stem, which gave us 6,113 stem groups.

The key cases are the multi-form groups: there were 1,759 groups where two or more different word forms share the same stem. These are the cases where stemming is actively creating a connection between forms.

We then lemmatized the forms in those stem groups and grouped the results by lemma and part of speech. This allowed us to distinguish between two very different types of stem groups.

A multi-form stem group was classified as safe when the forms in the group could be explained as variants of the same lemma+POS pair.

This is the behavior we want: the stemmer connects inflected forms of the same lexical item.

A multi-form stem group was classified as unsafe when the forms in the group could not all be explained by one shared lemma+POS analysis.

This is the problematic behavior: the stemmer connects words that look similar, but are not forms of the same lexical item.

The test produced the following result:

The main finding is that 34.4% of the analyzed German multi-form stem groups were unsafe: 605 out of 1,759.

This result shows how often stemming creates linguistically unsafe word relations that can then affect lexical search, the lexical side of hybrid retrieval, RAG grounding, and agent context selection.

Why This Matters for Search and RAG

A stemmer’s mistakes are not isolated. They happen before retrieval, which means they affect everything downstream. A wrong stem relation can affect indexing, lexical retrieval, and the lexical side of hybrid retrieval. In systems where those candidates feed RAG or agents, the noise can then flow into grounding, context selection, and generated answers.

If unrelated words are connected under the same stem, the system retrieves noisy evidence. If noisy evidence enters the candidate set, ranking and semantic search are more prone to failing. If the wrong passage is retrieved, a RAG system may generate a fluent answer poorly grounded in the wrong source. And, very important, the model is not the problem here, since the evidence retrieval layer has already damaged the input.

This is why early linguistic lemmatization matters. This as an upstream source-evidence problem: if stemming damages the evidence before ranking or generation, the error flows downstream into embeddings, RAG, agents, and generated answers. The next question is: what alternatives can we use to avoid introducing these noisy relationships?

Stemming vs. Lemmatization

Lemmatization, a standard procedure in linguistic analysis, can offer a safe alternative to stemming. Stemming and lemmatization are sometimes considered equivalent, but they are significantly different:

• Stemming says that a group of words look similar because they have some characters in common, so stemming reduces them to the same string or stem. This procedure connects these words to create noise. For example:
• Words: «Lage, Lager, lag, lagen» are collapsed under stem «lag»
• Lemmatization says that these word forms belong to three different lexical item, different roots or lemmas, defining strong and safe relations without noise.
• Lemma : Lage/NOUN; Forms: Lage, Lagen (situation; position; location)
• Lemma : Lager/NOUN; Forms: Lager, Lagern (warehouse; camp; stock)
• Lemma : liegen/VERB; Forms: lag, lagen, liegen (lay; be located)

A stemmer cannot make this distinction. A lemmatizer using linguistic analysis can.

The Claim

Stemming is fast and it’s not precise enough for modern AI search. For keyword search, stemming errors were already a relevance problem. For hybrid search, RAG, and agents, they become more serious because the retrieval layer feeds downstream AI components. Noisy retrieval becomes noisy grounding. Noisy grounding becomes unreliable generation. In short, standard stemming creates false lexical relations that impact AI search.

The problem is more prominent for languages with rich morphologies:

Lemmatization by Bitext Linguistic Analyzer

Bitext linguistic analysis exposes structured linguistic information about words in +30 languages: lemma, POS, inflection paradigm, morphological rules; together with other useful sources of information like compound structure and word formation in general.

The essential improvement with respect to stemming is that Bitext Lemmatization connects forms with the same lemma and POS, instead of collapsing words by character similarity. As a result Bitext can distinguish:

and preserve safe relations like:

while rejecting unsafe relations like:

That gives retrieval systems clean evidence.

Conclusion

German stemming creates useful connections, but it also creates many false ones. For current end-user demands in the AI era, this is a significant obstacle for accuracy.

The problem is structural: stemming works mechanically from word surface characters, not from linguistic analysis. In our test for German, more than one third of multi-form stem groups created false lexical relations. This has an impact:

• For traditional search and hybrid, that means poorer relevance
• For RAG, it means weaker grounding
• For agents, it means worse context

The better solution is to fix the evidence layer before retrieval: use linguistic analysis to preserve valid morphological relations and reject false ones.

If you’d like to learn more or test this approach in your Elasticsearch or OpenSearch setup, feel free to
contact us here.