Checker: change image check #3312
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the master branch, the checker starts to stream the response and cut the connection. However it creates a lot of read error, which are false negative. I don't know how to fix the issue. This commit change the checker to download the whole image. The error reporting is also changed to report only one line, instead of the whole stacktrace. Also, if a timeout occurs, the checker waits for one second before retry. Do note that I have not test the checker running in background. This feature seems forgotten and lack of interrest despite the initial move few years ago.
dalf
force-pushed
the
checker_image_proxy
branch
from
100a3a6
to
f49d1a9
Compare
return42
requested changes
Mar 9, 2024
There was a problem hiding this comment.
Thanks for this PR .. when I test:
make search.checker.artic
the checker terminates prematurely with this exception:
Engine artic Checking
Traceback (most recent call last):
File "local/py3/bin/searxng-checker", line 33, in <module>
sys.exit(load_entry_point('searxng', 'console_scripts', 'searxng-checker')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/search/checker/__main__.py", line 115, in main
run(args.engine_name_list, args.verbose)
File "searx/search/checker/__main__.py", line 73, in run
checker.run()
File "searx/search/checker/impl.py", line 439, in run
self.run_test(test_name)
File "searx/search/checker/impl.py", line 425, in run_test
rct_list = [self.get_result_container_tests(test_name, search_query) for search_query in search_query_list]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/search/checker/impl.py", line 419, in get_result_container_tests
result_container_check.check_basic()
File "searx/search/checker/impl.py", line 273, in check_basic
self._check_results(results)
File "searx/search/checker/impl.py", line 248, in _check_results
self._check_result(result)
File "searx/search/checker/impl.py", line 241, in _check_result
elif not _is_url_image(result.get('img_src')):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/search/checker/impl.py", line 124, in _is_url_image
return _download_and_check_if_image(image_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/search/checker/impl.py", line 76, in _download_and_check_if_image
r = network.get(
^^^^^^^^^^^^
File "searx/network/__init__.py", line 165, in get
return request('get', url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/network/__init__.py", line 96, in request
return future.result(timeout)
^^^^^^^^^^^^^^^^^^^^^^
File "~/.asdf/installs/python/3.12.0/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "~/.asdf/installs/python/3.12.0/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/searx/network/network.py", line 290, in request
return await self.call_client(False, method, url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/network/network.py", line 273, in call_client
return Network.patch_response(response, do_raise_for_httperror)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx/network/network.py", line 246, in patch_response
raise_for_httperror(response)
File "searx/network/raise_for_httperror.py", line 75, in raise_for_httperror
raise SearxEngineAccessDeniedException(message='HTTP error ' + str(resp.status_code))
searx.exceptions.SearxEngineAccessDeniedException: HTTP error 403, suspended_time=86400
make: *** [Makefile:50: search.checker.artic] Error 1
|
The issue exists before this PR: there is an exception in the async code which is not catched by the sync code. |
|
@return42 may I ask you to run |
|
NOTE: artic was the first engine where a
on master the test works, at the end there is a issue report (with the false negatives): |
What can I do / how can I help further .. did you read my last comment from above? |
May be you assume that I know the code since I wrote it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
In the master branch, the checker starts to stream the response and cut the connection. However it creates a lot of read error, which are false negative. I don't know how to fix the issue.
This commit change the checker to download the whole image. The error reporting is also changed to report only one line, instead of the whole stacktrace.
Do note that I have not test the checker running in background. This feature seems forgotten and lack of interrest despite the initial move few years ago.
Why is this change important?
How to test this PR locally?
make search.checker.brave.imagesmake search.checker.google_images(_not.).Author's checklist
Related issues
Close #3311