[WIP] Updating CLDR data #941

gavishpoddar · 2021-07-03T21:48:24Z

Updating CLDR data to 39.0.0.
Fixing CLDR downlaod error.
Updating CLDR Data URL : https://github.com/unicode-cldr/cldr-dates-full (archived) -> https://github.com/unicode-org/cldr-json.

TODO :

Fixing tests

Fixes issue #940

gavishpoddar · 2021-07-03T22:01:30Z

Many tests seem to be wrong just for example in tests/test_languages.py:805 for language Zulu

Current data translates son 23 umasingana 1996 to sunday 23 january 1996
but according to Google Translate it's isonto 23 Januwari 1996 for sunday 23 january 1996

Additionally languages like as are poorly translated.

This PR fixes those issues but currently, the tests are not updated.

@noviluni, please suggest should I update the tests accordingly.

A review will be helpful.

Thanks

Note: This PR breaks 39 tests.

noviluni · 2021-07-04T07:41:34Z

Hi @gavishpoddar,
I created a "guide" to handle this (CLDR updates), but we never started doing it. It would be nice if you read it to see if you missed anything: #826

My initial idea was to update version by version, but it's OK if we update directly to the last version as you did. After that we will need to check file by file to see if we are removing things that could generate "breaking changes" (and possibly adding them to our own data), but before starting the review I would like to understand why you removed the "version".

It is really important to point to a specific version and not directly to master to easily understand which version are we pointing and to be able to update easily in the future (master could be "incomplete" or "wrong"). In the past we didn't have a way to know it, so we didn't know which version we were using and how outdated we were, so I would like you to reconsider adding again the cldr_version and the repo.git.co(cldr_version) statements. We need to keep this. If it doesn't work because they are tags instead of branches, etc. maybe you need to change the step, but as I mentioned we need to point to a specific version.

thanks! :)

gavishpoddar

Hi, I am trying to fix update the CLDR data which is breaking the multiple tests so I am trying to highlight a few changes I have made along with the reasoning.

Please check and suggest.

gavishpoddar · 2021-07-06T19:40:19Z

tests/test_languages.py

@@ -802,7 +802,7 @@ def setUp(self):

        # zu
        param('zu', "3 mashi 2007 ulwesibili 10:08", "3 march 2007 tuesday 10:08"),
-        param('zu', "son 23 umasingana 1996", "sunday 23 january 1996"),
+        param('zu', "isonto 23 Januwari 1996", "sunday 23 january 1996"),


This was incorrectly translated verified via Google Translation

gavishpoddar · 2021-07-06T19:43:34Z

tests/test_languages.py

@@ -573,7 +573,7 @@ def setUp(self):
        param('mn', "12 9-р сар 2019 пүрэв", "12 september 2019 thursday"),

        # mr
-        param('mr', "16 फेब्रुवारी 1908 गुरु 02:03 मउ", "16 february 1908 thursday 02:03 pm"),
+        param('mr', "16 फेब्रुवारी 1908 गुरु 02:03", "16 february 1908 thursday 02:03"),


CLDR 39 has removed मउ (pm) -> pm. Either way, both of them are wrong.

gavishpoddar · 2021-07-07T09:56:05Z

tests/test_languages.py

@@ -210,7 +210,7 @@ def setUp(self):

        # as
        param('as', '17 জানুৱাৰী 1885', '17 january 1885'),
-        param('as', 'বৃহষ্পতিবাৰ 1 জুলাই 2009', 'thursday 1 july 2009'),
+        param('as', 'বৃহস্পতিবাৰ 1 জুলাই 2009', 'thursday 1 july 2009'),


Incorrect previous test: I know the language

gavishpoddar · 2021-07-07T09:56:55Z

tests/test_languages.py

@@ -270,7 +270,7 @@ def setUp(self):

        # bs-Latn
        param('bs-Latn', "23 septembar 1879, petak", "23 september 1879 friday"),
-        param('bs-Latn', "subota 1 avg 2009 02:27 popodne", "saturday 1 august 2009 02:27 pm"),
+        param('bs-Latn', "subota 1 aug 2009 02:27 popodne", "saturday 1 august 2009 02:27 pm"),


Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T09:57:35Z

tests/test_languages.py

-        param('ce', "6 январь 1987 пӏераскан де", "6 january 1987 friday"),
-        param('ce', "оршотан де 3 июль 1890", "monday 3 july 1890"),
+        param('ce', "6 январь 1987 пӏераска", "6 january 1987 friday"),
+        param('ce', "оршот де 3 июль 1890", "monday 3 july 1890"),


Chechen Language: Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T10:01:13Z

tests/test_languages.py

-        param('kl', "2 martsi 2001 ataasinngorneq", "2 march 2001 monday"),
-        param('kl', "pin 1 oktoberi 1901", "wednesday 1 october 1901"),
+        param('kl', "2 marsi 2001 ataasinngorneq", "2 march 2001 monday"),
+        param('kl', "pin 1 oktobari 1901", "wednesday 1 october 1901"),


Kalaallisut; Greenlandic: Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T10:02:24Z

tests/test_languages.py


        # kln
        param('kln', "3 ng'atyaato koang'wan 10:09 kooskoliny", "3 february thursday 10:09 pm"),
        param('kln', "kipsuunde nebo aeng' 14 2009 kos", "december 14 2009 wednesday"),

        # kok
-        param('kok', "1 नोव्हेंबर 2000 आदित्यवार 01:19 मनं", "1 november 2000 sunday 01:19 pm"),
+        param('kok', "1 नोव्हेंबर 2000 आयतार 01:19", "1 november 2000 sunday 01:19"),


Konkani (Indian language): Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T10:02:54Z

tests/test_languages.py

-        param('qu', "5 pauqar waray 1878 miércoles", "5 march 1878 wednesday"),
-        param('qu', "6 int 2009 domingo", "6 june 2009 sunday"),
+        param('qu', "5 marzo 1878 miércoles", "5 march 1878 wednesday"),
+        param('qu', "6 jun 2009 domingo", "6 june 2009 sunday"),


Quechua : Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T10:03:15Z

tests/test_languages.py

-        param('so', "sab 5 bisha saddexaad 1765 11:08 gn", "saturday 5 march 1765 11:08 pm"),
-        param('so', "16 lit 2008 axd", "16 december 2008 sunday"),
+        param('so', "sabti 5 bisha saddexaad 1765 11:08 gd", "saturday 5 march 1765 11:08 pm"),
+        param('so', "16 desembar 2008 axd", "16 december 2008 sunday"),


Somali: Verified via google translation.

gavishpoddar · 2021-07-07T10:05:54Z

tests/test_languages.py

@@ -741,7 +741,7 @@ def setUp(self):
        param('sv', "onsdag 16 mars 08:15 eftermiddag", "wednesday 16 march 08:15 pm"),

        # sw
-        param('sw', "5 mei 1994 jumapili 10:17 asubuhi", "5 may 1994 sunday 10:17 am"),
+        param('sw', "5 mei 1994 jumapili 10:17", "5 may 1994 sunday 10:17"),


Swahili: asubuhi means in the morning . Verified via google translation.

gavishpoddar

Fixing tests

gavishpoddar · 2021-07-07T10:52:32Z

tests/test_languages.py

@@ -1159,7 +1159,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('dav', "15 juma", "15 week"),
        # de
        param('de', "nächstes jahr", "in 1 year"),
-        param('de', "letzte woche 04:25 nachm", "1 week ago 04:25 pm"),
+        param('de', "vor einer Woche 04:25 nachm", "1 week ago 04:25 pm"),


German: Verified via google translate

gavishpoddar · 2021-07-07T10:53:35Z

tests/test_languages.py

@@ -1139,7 +1139,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('cgg', "5 omwaka", "5 year"),
        # chr
        param('chr', "ᎯᎠ ᎢᏯᏔᏬᏍᏔᏅ", "0 minute ago"),
-        param('chr', "ᎾᎿ 8 ᎧᎸᎢ ᏥᎨᏒ", "8 month ago"),
+        param('chr', "8 ꭷꮈ ꮵꭸꮢ", "8 month ago"),


Cherokee: Can't be verified but new CLDR 39 data updates this.

gavishpoddar · 2021-07-07T10:55:08Z

tests/test_languages.py

@@ -1197,7 +1197,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('et', "1 a pärast", "in 1 year"),
        param('et', "4 tunni eest", "4 hour ago"),
        # eu
-        param('eu', "aurreko hilabetea", "1 month ago"),
+        param('eu', "aurreko hilabetean", "1 month ago"),


Basque: Verified via google translate.

gavishpoddar · 2021-07-07T10:56:41Z

tests/test_languages.py

@@ -1266,7 +1266,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('id', "dalam 43 menit", "in 43 minute"),
        param('id', "dlm 23 dtk", "in 23 second"),
        # ig
-        param('ig', "nnyaafụ", "1 day ago"),
+        param('ig', "ụnyaahụ", "1 day ago"),


Igbo: Partially correct. Verified via google translate. Better than previous.

gavishpoddar · 2021-07-07T10:58:16Z

tests/test_languages.py

@@ -1240,7 +1240,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('gsw', "moorn", "in 1 day"),
        param('gsw', "geschter", "1 day ago"),
        # gu
-        param('gu', "2 વર્ષ પહેલા", "2 year ago"),
+        param('gu', "2 વર્ષ પહેલાં", "2 year ago"),


Gujarati: Same output via google translate for પહેલા or પહેલાં

gavishpoddar · 2021-07-07T10:59:28Z

tests/test_languages.py

@@ -1420,7 +1420,7 @@ def test_translation(self, shortname, datetime_string, expected_translation):
        param('ms', "bulan depan", "in 1 month"),
        # mt
        param('mt', "ix-xahar li għadda", "1 month ago"),
-        param('mt', "2 sena ilu", "2 year ago"),
+        param('mt', "2 snin ilu", "2 year ago"),


Maltese: Same output via google translate for sena or snin

gavishpoddar · 2021-07-07T11:00:52Z

tests/test_languages.py

-        param('nn', "for 5 minutter siden", "5 minute ago"),
-        param('nn', "om 3 uker", "in 3 week"),
+        param('nn', "for 5 min sidan", "5 minute ago"),
+        param('nn', "om 3 veke", "in 3 week"),


Norwegian Nynorsk: Can't be verified but new CLDR 39 data updates this. Google translate has no support for Nynorsk

gavishpoddar · 2021-07-07T11:02:00Z

tests/test_search.py

@@ -608,8 +608,7 @@ def test_splitting_of_not_parsed(self, shortname, string, expected, settings=Non

        # Hindi
        param('hi',
-              'जुलाई 1937 में, मार्को-पोलो ब्रिज हादसे का बहाना लेकर जापान ने चीन पर हमला कर दिया और चीनी साम्राज्य '
-              'की राजधानी बीजिंग पर कब्जा कर लिया,'),
+              'जुलाई 1937 में, मार्को-पोलो ब्रिज हादसे का बहाना की राजधानी बीजिंग पर कब्जा कर लिया. '),


Hindi: Test failed because of incorrect language detection.

gavishpoddar · 2021-07-07T11:04:19Z

At this point, 7 tests are failing in tests/test_freshness_date_parser.py.

I am unable to fix them please help.

@noviluni

lopuhin · 2021-07-08T11:54:13Z

@gavishpoddar the builds for this PR were not enabled (it's a newish github feature), sorry about that - just enabled them.

Fixing translation

…dateparser into pr/gavishpoddar/963

Fixing translation

codecov · 2021-10-09T21:40:21Z

Codecov Report

Merging #941 (4580337) into master (507dc6d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #941   +/-   ##
=======================================
  Coverage   98.29%   98.29%           
=======================================
  Files         234      234           
  Lines        2694     2700    +6     
=======================================
+ Hits         2648     2654    +6     
  Misses         46       46

Impacted Files	Coverage Δ
dateparser/languages/locale.py	`98.71% <100.00%> (+0.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 507dc6d...4580337. Read the comment docs.

Updating CLDR Data

ed9fa75

gavishpoddar changed the title ~~WIP : Updating CLDR Data &~~ [WIP] : Updating CLDR Data & Jul 3, 2021

gavishpoddar changed the title ~~[WIP] : Updating CLDR Data &~~ [WIP] : Updating CLDR data Jul 3, 2021

gavishpoddar added 4 commits July 4, 2021 20:26

Fixing CLDR release version

6b2ed03

Update CLDR data with referance (scrapinghub#826)

6dafbf9

WIP: Fixing tests

4927a95

Fixing tests : translation

299e727

gavishpoddar commented Jul 7, 2021

View reviewed changes

gavishpoddar added 5 commits July 7, 2021 10:25

Fixing tests

6eb7989

"de" translation issue

0aeb685

Fixing test: freshness

d52edbe

fixing tests : languages

ae1d1c2

Removing temporary test file

8340494

gavishpoddar commented Jul 7, 2021

View reviewed changes

gavishpoddar changed the title ~~[WIP] : Updating CLDR data~~ [WIP] Updating CLDR data Jul 20, 2021

gavishpoddar mentioned this pull request Aug 9, 2021

Optional Language Detect #932

Merged

21 tasks

gavishpoddar added 10 commits August 13, 2021 18:46

Fixes scrapinghub#771

261d3d5

updating tests

47ed2f9

adding test for scrapinghub#500

391ae3d

adding tests

0a52a0e

Merge branch 'scrapinghub:master' into fixing-translation

922ee90

Merge branch 'scrapinghub:master' into fixing-translation

455dd48

Merge branch 'master' into fixing-translation

9c22080

Merge pull request #6 from gavishpoddar/fixing-translation

078d8bb

Fixing translation

updates

a9f8c75

Merge branch 'fixing-translation' of https://github.com/gavishpoddar/…

c3ff334

…dateparser into pr/gavishpoddar/963

gavishpoddar added 3 commits October 9, 2021 21:31

fixes

3813d38

updating tests

ff66b2c

Merge pull request #7 from gavishpoddar/fixing-translation

4580337

Fixing translation

whalebot-helmsman mentioned this pull request Nov 2, 2021

Incorrect hungary date parsing #1015

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Updating CLDR data #941

[WIP] Updating CLDR data #941

gavishpoddar commented Jul 3, 2021

gavishpoddar commented Jul 3, 2021

noviluni commented Jul 4, 2021

gavishpoddar left a comment •

edited

gavishpoddar Jul 6, 2021

gavishpoddar Jul 6, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar left a comment

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar Jul 7, 2021

gavishpoddar commented Jul 7, 2021

lopuhin commented Jul 8, 2021

codecov bot commented Oct 9, 2021

[WIP] Updating CLDR data #941

Are you sure you want to change the base?

[WIP] Updating CLDR data #941

Conversation

gavishpoddar commented Jul 3, 2021

gavishpoddar commented Jul 3, 2021

noviluni commented Jul 4, 2021

gavishpoddar left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gavishpoddar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gavishpoddar commented Jul 7, 2021

lopuhin commented Jul 8, 2021

codecov bot commented Oct 9, 2021

Codecov Report

gavishpoddar left a comment •

edited