New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
issue #6323: add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter #6324
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6324 +/- ##
==========================================
+ Coverage 83.92% 88.88% +4.95%
==========================================
Files 161 162 +1
Lines 11972 11981 +9
Branches 1865 1930 +65
==========================================
+ Hits 10048 10649 +601
+ Misses 1588 980 -608
- Partials 336 352 +16
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should go into scrapy.utils.log
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. That's where I tried to put it first but I get a circular import
error.
..\scrapy\__init__.py:15: in <module>
from scrapy.spiders import Spider
..\scrapy\spiders\__init__.py:16: in <module>
from scrapy.utils.log import SpiderLoggerAdapter
..\scrapy\utils\log.py:13: in <module>
from scrapy.settings import Settings
..\scrapy\settings\__init__.py:23: in <module>
from scrapy.settings import default_settings
..\scrapy\settings\default_settings.py:322: in <module>
USER_AGENT = f'Scrapy/{import_module("scrapy").__version__} (+https://scrapy.org)'
E AttributeError: partially initialized module 'scrapy' has no attribute '__version__' (most likely due to a circular import)
Maybe i'm wrong and it's solvable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't like having a separate module for this, I see these options:
- Move it to scrapy.spiders (probably no technical downsides as it's not useful anywhere else?).
- Remove the SpiderLoggerAdapter import from the scrapy.spiders top-level, moving it under
if TYPE_CHECKING
and insidedef logger()
. - (likely a worse idea) Change some other imports in the loop.
I would go with 2 in general but probably with 1 in this case.
@Gallaecio second opinion?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done the second option
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be honest importing SpiderLoggerAdapter on every single log call looks a bit inefficient, imho.
But it's obviously up to you guys
Is it possible to add a test that checks that the Spider logging methods work as expected? |
Do you mean a test that's making sure general logging messages(not "extra") of Spider work as expected? |
Added more tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
fixes #6323
SpiderLoggerAdapter prioritizes self.extra over logged extra so it works as it worked before.
spider
in extra will always beself