Why Language Detection is Difficult

Language Detection Basics

Language detection usually operates on the basis of giving some text to answer the question of "What language is this text in?". This is usually done in the context of other language processing that works better when the language of a text is known.

I'm coming from the context of building a web search engine where the language information is very useful to correctly sort and filter text resulting in better search results.

The usual implementation of a language detector is a function that takes some text as input and turns it into a list of languages and confidence scores. Pick the highest scoring language from that and you've answered the question "What language is this text in?", done, end of blogpost …

… if that was the solution I'd be perfectly happy with the existing libraries.

There will be no solution, because I'm still searching for one, this is a hard problem.

Language Detection Complexity by Example

Let's use some examples to explore why language detection is difficult.

But to start easy:

The quick brown fox jumps over the lazy dog.

Is pretty obviously English to a human.

To a well working language detector as described in the previous section the output will be English paired with a high confidence score and some other languages that share similarities with English that score lower.

Multiple Languages, easy Mode

Now what language is this in?

The quick brown fox jumps over the lazy dog. Aber dann kommt ein deutscher Satz.

The question doesn't really make sense here as we should have asked for languages, plural.

To a human this is clearly a mix of German and English.

But the detector as described above will get confused as it is only designed to handle monolingual text. Both the confidence scores for English and German will be much lower and the result will largely depend on the implementation details. Maybe it will say its English, maybe German, maybe the scores for German and English fall so low that the text will be classified as Dutch. This is a huge problem.

So let's assume there is a new detection function that doesn't return one confidence list for the whole text, but manages to somehow slice the two sentences apart and returns a list of text intervals, each with its own scoring. All good now?

This is important since some texts are multilingual and switch between languages, maybe because of an incomplete translation (intentional or not), maybe because it made sense to switch languages because of the topic, maybe there is no reason at all and the language still switches around.

I don't know of any library that has implemented this in an efficient way without an "experimental"-tag unfortunately. Recommendations welcome!

Language switch for a single Word

From the English Wikipedia page "Radical 174", what language is this?

Radical 174 or radical blue (靑部/青部) meaning "green" or "blue" or "black" (see Distinguishing blue from green in Chinese) is one of …

It is English, both to the human and the language detector.

But what about the "靑部/青部" part? Those are not English and would be quite out of place when looking at a list of words in the English corpus. And in the context of search engines such misdetections can really mess up the search results, because the English part of the pipeline will not process this correctly.

This is of course easier when there is a script change, without a script change the line between a language switch and a loanword becomes very blurry.

Single Word Detection

What language is this in?

radio

Of course it is English, but is is also a valid word in German, French, Swedish, Croatian, … a lot of languages.

This mainly happens for names and loanwords, but also for false friends that happen to be spelled exactly the same way.

The output of the language detection software will again largely depend on its implementation details and could be any random language with a hopefully low confidence score.

But how did the Human come to the "Of course it is English" conclusion? Context! This whole post is in English so the natural conclusion is that radio is also English. The language detector doesn't know that but one should be able to give it a hint.

The correct answer without any context would be: I don't know.

This is important for things like metadata where there is not enough text for reliable language detection, that are still in the context of a larger text but not part of it. It is also important that the detector still works when there is no further context.

How Language detection is usually implemented

Usually language detectors are implemented as hybrid multi-stage detectors:

• Script detection: Knowing the script a text is written narrows down the number of possible languages quite a lot.
• Alphabet detection: Sometimes it is possible to narrow down the language by the used letters, for example when German umlauts are present.
• N-gram detection: To detect common patterns in languages without full dictionaries.

The N-gram Problem

Script and Alphabet detection are pretty straight forward, but the N-gram detection involves a bit of statistics and machine learning, meaning one needs training data to generate it.

The training data for libraries that implement N-gram language detection usually comes from the Wortschatz Project by the University of Leipzig which comes from sources like Wikipedia and there we run into an chicken and egg problem …

… Remember the examples about language switching? The internet is full of those because switching languages is something that humans do all the time.

This means that the Corpora are never truly single language. English words have made it everywhere and into corpora of all languages and in reverse almost every language out there has made it into the English corpora. The same is true for other languages, but it seems to be most extreme for English.

One example of a language switch out of nowhere that is the Wikipedia article about Thomas Chauke in Xitsonga. The first paragraph of the section titled "Ntirho wa vunanga" is in English and seems to be a writing or translation artefact since the paragraph below it is almost the same text again but in Xitsonga.

N-gram detectors trained on those datasets will inherit this mixing of languages, resulting in misdetections with high confidence.

A solution?

The only way to get large lists of words that are either free of unannounced language switches I've found so far are curated lists. Or more plainly speaking: Dictionaries.

The largest freely licensed dictionaries are Wiktionary and Wikidata Lexemes. Unfortunately the Wiktionary is not machine readable and the Wikidata Lexemes have a clear bias towards European languages.

Wikidata: Lexicographical data / How to help

The Wishlist

So what would a language detector that is search engine ready look like?

• It should be as efficient as possible, search engines process a lot of data.
• Probably uses the multistage architecture described above.
• Can detect language switches, no matter how short or long.
  • Compromises on this are made with the fact in mind that the language detection is followed by language specific post processing
• Supports and verifies clues about the language (i.e. from page metadata) and language switches.
• Has an "I have no idea" response
• If using N-grams, derives these from a curated corpus.

That would be my wishlist, anyway.

I hope I've shined some light on the Complexity behind the question of "What language is … ?". If you know if any of those problems have been solved I'm very open for recommendations.

Thank you for reading!