[434] Allow to filter jobs in ZyteJobsComparisonMonitor by close_reason

[shafiq-muhammad]

Adds limit for nested dict stats computation

Added a new setting SPIDERMON_JOBS_COMPARISON_CLOSE_REASONS to allow ZyteJobsComparisonMonitor to filter by the ScrapyCloud jobs with their close_reason stat.

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 79.48%. Comparing base (6ea9f67) to head (9df1949).

Files	Patch %	Lines
spidermon/contrib/scrapy/monitors/monitors.py	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #440      +/-   ##
==========================================
- Coverage   79.51%   79.48%   -0.04%     
==========================================
  Files          76       76              
  Lines        3237     3242       +5     
  Branches      537      539       +2     
==========================================
+ Hits         2574     2577       +3     
- Misses        593      594       +1     
- Partials       70       71       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[curita]

Changes look good to me!

We should add some unittest to cover the new addition, and it would be nice to run some test job in ScrapyCloud to verify these changes (Installing Spidermon in the requierements.txt from git+https://github.com/shafiq-muhammad/spidermon@shafiq-434 should get you these changes). After that's addressed we should be good to go 👍

[aaneto]

Just a quick FYI @shafiq-muhammad @curita, I believe the pipeline is failing due to a regression from pytest 8.1.1 to 8.2.0.

My validation was:

• Creating an empty .env for the project and running pip install tox and tox -e base
• Noticing the errors, and setting pytest==8.1.1 on tox.ini
• Running tox -e base again, with success this time.

Hope this helps (and is correct regardless of environment).

[VMRuiz]

Yes, it looks like Pytest 8.2.0 release has caused a lot of troubles according to their issues page.
@shafiq-muhammad , Could you please pin pytest to 8.1.1 until they release a new version?