WIP: Reimplementing `search_dates` #945

gavishpoddar · 2021-07-16T22:30:54Z

Reimplementing and simplifying `search_dates`

A reimplemented and simplified search_dates which more directly uses dateparser.parse, improves accuracy and fixes many bugs

New Feature:

search_first_date - searches and returns the first date from the given text.

NOTE: This PR is inspired by the previous implementation of search_dates and #931.

TODO

gavishpoddar · 2021-07-16T23:03:29Z

This PR will also solve many issues with the `search_dates`.

date has year inside brackets #924
search_dates() wrong result for 12:xx am #894
Errors in processing dates in Russian text #918
search_dates cannot find complete date #521
False positives when searching dates #582
search_dates is... #681
J, K, W, X, Y letters confuse the parser #774
Mistake search_dates for ru language #706 (Correct datetime object but wrong string) (related issue 2)
Parse whole words only #856 (Correct datetime object but wrong string) (related issue 2)

Somewhat address.

Month and Month Year (e.g. April and May 2019) will return None #507 (Now returns date object but not range)
Date ambiguities during search_dates #689 (Now returnes datetime.datetime(2019, 2, 10, 0, 0) but the issue is not clear)
Search dates failing to work due to lack of commas? #695
Dates not getting recognised or wrongly recognised if there are some numbers in the text #787 (Now returnes datetime object but the issue is not clear)

Some other issue fixed:

2021-08-04T14:21:37+05:30

would parse 05:30 only after this PR

2021-08-04T14:21:37 and 05:30

NOTE: THIS LIST IS NO LONGER ACCURATE ANYMORE

gavishpoddar · 2021-07-17T11:58:24Z

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

noviluni · 2021-07-19T07:14:00Z

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

Hi @gavishpoddar, using translated chunks makes sense. 👍

codecov · 2021-07-19T07:16:08Z

Codecov Report

Base: 98.23% // Head: 98.10% // Decreases project coverage by -0.12% ⚠️

Coverage data is based on head (99e66c6) compared to base (0f5d9c5).
Patch coverage: 96.12% of modified lines in pull request are covered.

❗ Current head 99e66c6 differs from pull request most recent head 2935aae. Consider uploading reports for the commit 2935aae to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #945      +/-   ##
==========================================
- Coverage   98.23%   98.10%   -0.13%     
==========================================
  Files         232      235       +3     
  Lines        2604     2692      +88     
==========================================
+ Hits         2558     2641      +83     
- Misses         46       51       +5

Impacted Files	Coverage Δ
dateparser/search/search.py	`94.73% <94.33%> (-4.64%)`	⬇️
dateparser/search/__init__.py	`100.00% <100.00%> (ø)`
dateparser/search/languages.py	`100.00% <100.00%> (ø)`
dateparser/date_parser.py	`93.75% <0.00%> (-0.19%)`	⬇️
dateparser/date.py	`99.24% <0.00%> (-0.04%)`	⬇️
dateparser/parser.py	`99.01% <0.00%> (-0.01%)`	⬇️
dateparser/utils/__init__.py	`95.27% <0.00%> (ø)`
dateparser/languages/locale.py	`98.69% <0.00%> (ø)`
dateparser/conf.py	`100.00% <0.00%> (ø)`
dateparser/__init__.py	`100.00% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

gavishpoddar · 2021-07-21T18:15:06Z

Hi @noviluni, I have made the changes for tests to work. Can you please approve the workflow approval?

lopuhin · 2021-07-22T06:36:28Z

@gavishpoddar workflows approved 👍

gavishpoddar · 2021-08-03T17:55:57Z

Hi @noviluni, @lopuhin, @kishan3
The new search_dates implementation is nearing its completion.

Code reviews would be great to receive.

gavishpoddar · 2021-08-03T17:59:00Z

I need help with one test

StaticTzInfo 'UTC' is expected.
UTC is received from new search_dates

dateparser/languages/locale.py

gavishpoddar · 2021-08-06T10:40:32Z

Hi after some changes the codes are very much compatible with the old search_dates. Some tests are modified as the working of the search_dates has been changed.

Currently, this PR is left with some docs changes (just replacing the current with new docs and search_first_date).

gavishpoddar · 2021-08-24T14:07:45Z

Replaced previous search_dates with the new implimentaion

lopuhin

Thanks @gavishpoddar , I left some comments regarding tests. Please tell if you need advice with how to implement xfail.

dateparser/search/__init__.py

tests/test_search.py

…rser into search_dates

lopuhin

Thanks @gavishpoddar a few docs suggestions

dateparser/search/__init__.py

Co-authored-by: Konstantin Lopuhin <kostia.lopuhin@gmail.com>

gavishpoddar · 2021-09-07T16:58:25Z

Hey, this PR is updated with the #932

dateparser/search/__init__.py

…rser into search_dates

Eckii24 · 2022-08-06T14:20:12Z

Hi @gavishpoddar,

I found your PR to improve the behavior of search_dates.
There has been no further progress for almost a year. What is the current status of this PR?

Is there anything I can do to help get this PR merged?

Gallaecio · 2022-12-01T10:39:32Z

dateparser/search/search.py

    def __init__(self):
-        self.loader = LocaleDataLoader()
-        self.available_language_map = self.loader.get_locale_map()
-        self.search = _ExactLanguageSearch(self.loader)


The removal of DateSearchWithDetection.search is backward-incompatible.

I can create a shortcut the make DateSearchWithDetection.search and add a deprecation warning or simply rename.

Please suggest a preferred action.

I think keeping the old objects around with their old names, logging a warning when used (i.e. by exposing them through a property whose getter logs a warning), would be ideal.

Implimenting new search_dates

02220da

gavishpoddar mentioned this pull request Jul 16, 2021

Fixing #894 and correcting tests #901

Closed

gavishpoddar changed the title ~~[WIP] Reimplementing search_dates & fixing search translation~~ [WIP] Reimplementing search_dates & fixing search translation Jul 17, 2021

gavishpoddar added 2 commits July 21, 2021 07:22

Fixing DATE_ORDER, implimenting deep_search, tests

f933d3a

Unproving _joint_parse with data_carry accurate_return_text, deep_search

77727b5

gavishpoddar added 2 commits July 22, 2021 20:41

implementing _final_text_clean()

e7f38e8

Simplifying text_clean and modifying tests

962066c

gavishpoddar mentioned this pull request Jul 26, 2021

search_dates : Issues paring dates with period #952

Open

gavishpoddar added 5 commits July 28, 2021 12:15

Implementing relative date

624ac8e

Fixing tests

42ca6f6

secondary_split_implimentation

51749a2

positional args to keyword argument

f5e4635

Micro fixes

121b15f

gavishpoddar added 3 commits August 3, 2021 18:04

Removing codes now part of scrapinghub#953

2cd93f0

adding check_settings

006d2a5

implimenting double_punctuation_split

10404c9

noviluni reviewed Aug 5, 2021

View reviewed changes

dateparser/languages/locale.py Outdated Show resolved Hide resolved

gavishpoddar added 2 commits August 6, 2021 09:00

Updating docs and removing test (TMP)

22596e0

cleaning code, adding tests, improving coverage

b799dfb

gavishpoddar requested a review from noviluni August 6, 2021 10:40

gavishpoddar changed the title ~~[WIP] Reimplementing search_dates & fixing search translation~~ Reimplementing search_dates & fixing search translation Aug 6, 2021

adding test

5dabc62

lopuhin reviewed Aug 27, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

tests/test_search.py Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

gavishpoddar added 5 commits August 27, 2021 09:52

fixing doc string

ab1778d

fixing doc string

14adf89

Merge branch 'search_dates' of https://github.com/gavishpoddar/datepa…

d57223a

…rser into search_dates

updating xfail

88afa30

updating tests

9209f3d

lopuhin reviewed Sep 1, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

dateparser/search/__init__.py Outdated Show resolved Hide resolved

gavishpoddar and others added 4 commits September 1, 2021 15:32

Apply suggestions from code review

85254e0

Co-authored-by: Konstantin Lopuhin <kostia.lopuhin@gmail.com>

Merge branch 'master' into search_dates

e4604e6

Updates

4f119dd

Fixing upstraem merges

e6da4be

lopuhin reviewed Sep 8, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

gavishpoddar added 5 commits September 9, 2021 06:29

DateSearch -> DateSearchWithDetection

f6116bf

Merge branch 'scrapinghub:master' into search_dates

0525cdc

updating test with xfail

96b91c0

Merge branch 'search_dates' of https://github.com/gavishpoddar/datepa…

b9d12f3

…rser into search_dates

minor fixes

99e66c6

serhii73 closed this Dec 1, 2022

serhii73 reopened this Dec 1, 2022

serhii73 requested review from wRAR, serhii73 and Gallaecio December 1, 2022 08:47

serhii73 mentioned this pull request Dec 1, 2022

[WIP] add find_date function and tests: finds the first date #931

Open

5 tasks

Gallaecio reviewed Dec 1, 2022

View reviewed changes

gavishpoddar changed the title ~~Reimplementing search_dates~~ WIP: Reimplementing search_dates Dec 7, 2022

Merge branch 'master' into search_dates

2935aae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Reimplementing `search_dates` #945

WIP: Reimplementing `search_dates` #945

gavishpoddar commented Jul 16, 2021 •

edited

gavishpoddar commented Jul 16, 2021 •

edited

gavishpoddar commented Jul 17, 2021

noviluni commented Jul 19, 2021 •

edited

codecov bot commented Jul 19, 2021 •

edited

gavishpoddar commented Jul 21, 2021

lopuhin commented Jul 22, 2021

gavishpoddar commented Aug 3, 2021

gavishpoddar commented Aug 3, 2021

gavishpoddar commented Aug 6, 2021 •

edited

gavishpoddar commented Aug 24, 2021

lopuhin left a comment

lopuhin left a comment

gavishpoddar commented Sep 7, 2021

Eckii24 commented Aug 6, 2022

Gallaecio Dec 1, 2022

gavishpoddar Dec 1, 2022 •

edited

Gallaecio Dec 1, 2022

WIP: Reimplementing search_dates #945

Are you sure you want to change the base?

WIP: Reimplementing search_dates #945

Conversation

gavishpoddar commented Jul 16, 2021 • edited

Reimplementing and simplifying search_dates

TODO

gavishpoddar commented Jul 16, 2021 • edited

This PR will also solve many issues with the search_dates.

Somewhat address.

Some other issue fixed:

gavishpoddar commented Jul 17, 2021

noviluni commented Jul 19, 2021 • edited

codecov bot commented Jul 19, 2021 • edited

Codecov Report

gavishpoddar commented Jul 21, 2021

lopuhin commented Jul 22, 2021

gavishpoddar commented Aug 3, 2021

gavishpoddar commented Aug 3, 2021

gavishpoddar commented Aug 6, 2021 • edited

gavishpoddar commented Aug 24, 2021

lopuhin left a comment

Choose a reason for hiding this comment

lopuhin left a comment

Choose a reason for hiding this comment

gavishpoddar commented Sep 7, 2021

Eckii24 commented Aug 6, 2022

Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

gavishpoddar Dec 1, 2022 • edited

Choose a reason for hiding this comment

Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

WIP: Reimplementing `search_dates` #945

WIP: Reimplementing `search_dates` #945

gavishpoddar commented Jul 16, 2021 •

edited

Reimplementing and simplifying `search_dates`

gavishpoddar commented Jul 16, 2021 •

edited

This PR will also solve many issues with the `search_dates`.

noviluni commented Jul 19, 2021 •

edited

codecov bot commented Jul 19, 2021 •

edited

gavishpoddar commented Aug 6, 2021 •

edited

gavishpoddar Dec 1, 2022 •

edited