Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix some thread safety issues and improve concurrency tests #537

Merged
merged 10 commits into from Feb 23, 2022
Merged

Fix some thread safety issues and improve concurrency tests #537

merged 10 commits into from Feb 23, 2022

Conversation

JWCook
Copy link
Member

@JWCook JWCook commented Feb 21, 2022

Related to #515.

Tests

  • Add stress test shortcut for local development (nox -e stress)
  • Use fewer unique responses and more total requests to increase concurrent cache hits
  • Add multiprocess tests (in addition to multithreaded tests)

Fixes

  • Handle CorruptGridFile error in GridFS backend
  • Fix race condition with SQLiteDict.clear(), when dropping and recreating SQLite tables in multiple threads
  • Fix some additional file-based race conditions in filesystem and GridFS backends
    • Note: These still do not implement a full file locking system, and aren't ideal for highly parallel applications
  • Fix potential race condition in remove_expired_responses() for SQLite, Filesystem, and GridFS backends

@JWCook JWCook added tests Unit and integration tests performance labels Feb 21, 2022
@JWCook JWCook added this to the v0.9 milestone Feb 21, 2022
@JWCook JWCook added the bug label Feb 21, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #537 (15e1154) into master (6e595f3) will decrease coverage by 0.12%.
The diff coverage is 95.23%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #537      +/-   ##
==========================================
- Coverage   98.54%   98.41%   -0.13%     
==========================================
  Files          18       18              
  Lines        1303     1325      +22     
  Branches      207      216       +9     
==========================================
+ Hits         1284     1304      +20     
- Misses         10       12       +2     
  Partials        9        9              
Impacted Files Coverage Δ
requests_cache/backends/gridfs.py 96.29% <91.66%> (-3.71%) ⬇️
requests_cache/backends/filesystem.py 96.05% <100.00%> (+0.40%) ⬆️
requests_cache/backends/sqlite.py 98.72% <100.00%> (+0.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e595f3...15e1154. Read the comment docs.

@JWCook JWCook changed the title Improve concurrency tests Fix some thread safety issues and improve concurrency tests Feb 23, 2022
@JWCook JWCook marked this pull request as ready for review February 23, 2022 01:46
@JWCook JWCook merged commit cc5305f into requests-cache:master Feb 23, 2022
@JWCook JWCook deleted the concurrency-tests branch February 23, 2022 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug performance tests Unit and integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants