Enhance Japanese language detection

[ManyTheFish]

Pull Request

This PR is a prototype and can be tested by downloading the dedicated docker image:

$ docker pull getmeili/meilisearch:prototype-better-language-detection-0

Context

Some Languages are harder to detect than others, this miss-detection leads to bad tokenization making some words or even documents completely unsearchable. Japanese is the main Language affected and can be detected as Chinese which has a completely different way of tokenization.

A first iteration has been implemented for v1.1.0 but is an insufficient enhancement to make Japanese work. This first implementation was detecting the Language during the indexing to avoid bad detections during the search.
Unfortunately, some documents (shorter ones) can be wrongly detected as Chinese running bad tokenization for these documents and making possible the detection of Chinese during the search because it has been detected during the indexing.

For instance, a Japanese document {"id": 1, "name": "東京スカパラダイスオーケストラ"} is detected as Japanese during indexing, during the search the query 東京 will be detected as Japanese because only Japanese documents have been detected during indexing despite the fact that v1.0.2 would detect it as Chinese.
However if in the dataset there is at least one document containing a field with only Kanjis like:
A document with only 1 field containing only Kanjis:

{
 "id":4,
 "name": "東京特許許可局"
}

A document with 1 field containing only Kanjis and 1 field containing several Japanese characters:

{
 "id":105,
 "name": "東京特許許可局",
 "desc": "日経平均株価は26日 に約8カ月ぶりに2万4000円の心理的な節目を上回った。株高を支える材料のひとつは、自民党総裁選で3選を決めた安倍晋三首相の経済政策への期待だ。恩恵が見込まれるとされる人材サービスや建設株の一角が買われている。ただ思惑が先行して資金が集まっている面 は否めない。実際に政策効果を取り込む企業はどこか、なお未知数だ。"
}

Then, in both cases, the field name will be detected as Chinese during indexing allowing the search to detect Chinese in queries. Therefore, the query 東京 will be detected as Chinese and only the two last documents will be retrieved by Meilisearch.

Technical Approach

The current PR partially fixes these issues by:

1. Adding a check over potential miss-detections and rerunning the extraction of the document forcing the tokenization over the main Languages detected in it.

1. run a first extraction allowing the tokenizer to detect any Language in any Script
2. generate a distribution of tokens by Script and Languages (script_language)
3. if for a Script we have a token distribution of one of the Language that is under the threshold, then we rerun the extraction forbidding the tokenizer to detect the marginal Languages
4. the tokenizer will fall back on the other available Languages to tokenize the text. For example, if the Chinese were marginally detected compared to the Japanese on the CJ script, then the second extraction will force Japanese tokenization for CJ text in the document. however, the text on another script like Latin will not be impacted by this restriction.

2. Adding a filtering threshold during the search over Languages that have been marginally detected in documents

Limits

This PR introduces 2 arbitrary thresholds:

1. during the indexing, a Language is considered miss-detected if the number of detected tokens of this Language is under 10% of the tokens detected in the same Script (Japanese and Chinese are 2 different Languages sharing the "same" script "CJK").
2. during the search, a Language is considered marginal if less than 5% of documents are detected as this Language.

This PR only partially fixes these issues:

• ✅ the query 東京 now find Japanese documents if less than 5% of documents are detected as Chinese.
• ✅ the document with the id 105 containing the Japanese field desc but the miss-detected field name is now completely detected and tokenized as Japanese and is found with the query 東京.
• ❌ the document with the id 4 no longer breaks the search Language detection but continues to be detected as a Chinese document and can't be found during the search.

Related issue

Fixes #3565

Possible future enhancements

• Change or contribute to the Library used to detect the Language
  • the related issue on Whatlang: Can distinguish between Simplified Chinese and Japanese Kanji? greyblake/whatlang-rs#122

[github-actions]

Uffizzi Preview deployment-18362 was deleted.

[ManyTheFish]

Hey @dureuill,
I'd like your POV on this PR, thank you if you have the time to look at it

[dureuill]

Hello @ManyTheFish, thanks for thinking of me to review your PR!

Before I dive into the code, a question: did we consider the option of having some documents left as "language-ambiguous" and tokenized as many time as necessary in each of the candidate languages? Similarly, for search queries that could match several languages, tokenize them in all of these languages and return results for all the resulting queries?

Such a change could cause structural change and so be a large undertaking. Still, if it has been discussed somewhere, I'd love to be pointed to that discussion.

Meanwhile, the change proposed here appears like a nice heuristic to prevent the misdetection of languages using frequency.

I'm not 100% sure of what the scope of this sentence is:

rerunning the extraction of the document forcing the tokenization over the main Languages detected

can you maybe rephrase a bit?

[ManyTheFish]

Hello @dureuill, I added a section in Technical Approach detailing the indexing process and giving an example at the end. Let me know if you need more! 😄

[dureuill]

Thank you Many it is indeed clearer with the added explanation!

[dureuill]

I have a couple of suggestions and questions.

I'm OK with the approach :-)

[dureuill]

Approving. There remain unresolved inline comments above that you can take or leave :-).

[ManyTheFish]

@dureuill, I added a small documentation for buffers don't hesitate to bors merge if everything is good for you!

[dureuill]

perfect thank you ❤️

[dureuill]

bors merge

[bors]

Build succeeded:

• Run Clippy
• Run Rustfmt
• Run tests in debug
• Tests on macos-12
• Tests on ubuntu-18.04
• Tests on windows-2022