Bug with Japanese Kanji support

[curquiza]

Discussed in #2391

^{Originally posted by toomozoo May 13, 2022}
Thank you for supporting Japanese in the 0.27.0 version!
I have tried several Kanji searches and did not get the results I wanted.
Do I need to do any settings to search in Japanese?

% brew install meilisearch
% meilisearch -V
meilisearch-http 0.27.0

% irb
require 'json'
require 'meilisearch'

MeiliSearch::VERSION
=> "0.18.3"

client = MeiliSearch::Client.new('http://127.0.0.1:7700')
json = JSON.parse('[{"id": "1","name": "東京バナナ"},{"id": "2","name": "東京 ポテチ"}]')
client.index('test').add_documents(json)

# NG
client.index('test').search("東")
=> {"hits"=>[], "nbHits"=>0, "exhaustiveNbHits"=>false, "query"=>"東", "limit"=>20, "offset"=>0, "processingTimeMs"=>0}

# NG
client.index('test').search("東京")
=> {"hits"=>[], "nbHits"=>0, "exhaustiveNbHits"=>false, "query"=>"東京", "limit"=>20, "offset"=>0, "processingTimeMs"=>0}

# OK
client.index('test').search("バ")
=> {"hits"=>[{"id"=>"1", "name"=>"東京バナナ"}], "nbHits"=>1, "exhaustiveNbHits"=>false, "query"=>"バ", "limit"=>20, "offset"=>0, "processingTimeMs"=>0}
```</div>

---

## TODO

- [ ] Implement changes in [Milli](https://github.com/meilisearch/milli/): https://github.com/meilisearch/meilisearch/issues/3357
- [ ] Release a Milli version containing these changes
- [ ] Bump this new Milli version in Meilisearch and merge it into `main`

[curquiza]

For people following this issue: we are focusing on our v0.28.0 sprint (see the Milestones), and we will investigate once the prioritized tasks are done

[miiton]

Yes, this is a problem with "Kanji-only strings" and I know whatlang-rs needs to do something about it.
However, I don't have an idea to completely distinguish between Kanji and Simplified Chinese.

[miiton]

I write about what I have found out.

• Why do these problems occur?
  • Indexing phase
    1. When we try to index 東京のお寿司 into Meilisearch.
    2. Since this contains "Hiragana", whatlang judges it as "Japanese".
    3. It is separated into ["東京", "の", "お", "寿司"] by Lindera, and indexed.
  • Search phase
    1. When we search by keyword as 東京.
    2. Since this contains "Kanji only", whatlang judges it as "Mandarin(Chinese)".
    3. It is normalized to 东京. And this does not match 東京.
• I can think of several solutions.
  • Patch to whatlang.
    • Related: Can distinguish between Simplified Chinese and Japanese Kanji? greyblake/whatlang-rs#122
  • Stop normalizing Chinese.
  • Add parameters like Algolia.
    • indexLanguages
    • queryLanguages

I hope it will be useful to teams.

[ManyTheFish]

Hello @miiton, you're right!
The issue comes from the normalization process.
Currently, we are discussing the redesign of the Chinese normalization to be phonologically oriented.
Because Chinese Languages and Japanese shares some characters making it difficult to detect the Language when the provided string is short (like a query), we could think of unifying the normalization for these Languages to avoid this kind of issue. 🤔 We just need to find an accurate phonological representation of Japanese like the Hanyu Pinyin for Mandarin.

I've created a dedicated discussion on the product repository to centralize feedback about Japanese support.
I will feed this discussion with the current behavior, a link to this issue, and the possible enhancement we can do in charabia and its dependencies like whatlang or Lindera.

I've read your issue on whatlang, It's an interesting enhancement by the way.

[miiton]

@ManyTheFish

Thank you, I'll watch that discussion 😊

[curquiza]

#3357 needs to be done to fix this.

Due to the huge works we had on during the previous sprint (+ Hacktoberfest) we will not be able to integrate the new charabia version into milli and then Meilisearch for v0.30.0. Will be integrated in the next version (v1)

Sorry for the inconvenience anyone!

[curquiza]

Will be fixed for v1.1.0, not enough time during v1.0 sprint, sorry!
Fix is already on the way, see: meilisearch/milli#749

[curquiza]

Hello everyone following this issue 👋

We have just released the first RC (release candidate) of Meilisearch containing this new fix!
You can test it by using

• the release assets
• or the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.1.0-rc.0

If you still have the bug, please let us know!
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on 3rd April 2023

⚠️ RC (release candidates) are not recommended for production

[Rhilip]

#3508
see My test case, Japanese also cannot be search properly with Chinese document, will v1.1.0 fix this bug ?

[curquiza]

Hello @Rhilip sorry for the delay on the discussion you linked.
Can you test the RC and let us know if the bug you have in #3508 is still there?