Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrawlerProcess: initialize the reactor only once #5436

Merged
merged 3 commits into from
Mar 25, 2022

Conversation

Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 2, 2022

Fixes #5435

Pending:

  • Test coverage
  • Check if CrawlerRunner is working as expected (should it initialize the reactor in any scenario?)

@codecov
Copy link

codecov bot commented Mar 2, 2022

Codecov Report

Merging #5436 (edbf397) into 2.6 (23537a0) will decrease coverage by 0.07%.
The diff coverage is 100.00%.

❗ Current head edbf397 differs from pull request most recent head 3bf6bae. Consider uploading reports for the commit 3bf6bae to get more accurate results

@@            Coverage Diff             @@
##              2.6    #5436      +/-   ##
==========================================
- Coverage   88.77%   88.70%   -0.08%     
==========================================
  Files         163      163              
  Lines       10666    10667       +1     
  Branches     1818     1788      -30     
==========================================
- Hits         9469     9462       -7     
- Misses        922      929       +7     
- Partials      275      276       +1     
Impacted Files Coverage Δ
scrapy/crawler.py 88.82% <100.00%> (+0.18%) ⬆️
scrapy/core/downloader/handlers/__init__.py 83.63% <0.00%> (-9.10%) ⬇️
scrapy/downloadermiddlewares/cookies.py 95.78% <0.00%> (-2.11%) ⬇️
scrapy/shell.py 67.96% <0.00%> (-0.79%) ⬇️
scrapy/http/response/__init__.py 97.43% <0.00%> (-0.04%) ⬇️
scrapy/http/request/__init__.py 97.77% <0.00%> (-0.03%) ⬇️
scrapy/utils/python.py 87.64% <0.00%> (ø)
scrapy/commands/check.py 71.01% <0.00%> (ø)

@Gallaecio Gallaecio marked this pull request as ready for review March 2, 2022 15:50
scrapy/crawler.py Outdated Show resolved Hide resolved
@Gallaecio Gallaecio changed the base branch from master to 2.6 March 2, 2022 16:29
@Gallaecio Gallaecio force-pushed the fix-crawlerprocess-regression branch from 08b0fdf to 3bf6bae Compare March 2, 2022 16:30
@Gallaecio Gallaecio modified the milestone: 2.6.2 Mar 4, 2022
@Gallaecio
Copy link
Member Author

CrawlerRunner is fine, and unlike CrawlerProcess, existing tests would have failed had it initialized the reactor.

@elacuesta elacuesta changed the title CrawlerProcess: initiate the reactor only once CrawlerProcess: initialize the reactor only once Mar 10, 2022
@odimko
Copy link

odimko commented Apr 13, 2022

@wRAR when is it going to be released?

@Gallaecio
Copy link
Member Author

2 weeks ago according to plan :)

We have had some busy weeks. My current hopes are to release before the end of the month.

@odimko
Copy link

odimko commented Apr 13, 2022

Awesome @Gallaecio 💯
thanks for letting me know. Looking forward to 👍🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2.6.0 breaks calling multiple Spider in CrawlerProcess()
3 participants