Releases: celery/pytest-celery
v1.0.0
v1.0.0 Official Release
User Manual: https://pytest-celery.readthedocs.io/en/latest/
PyPI: https://pypi.org/project/pytest-celery/
Source: https://github.com/celery/pytest-celery/
Install with pip install -U "pytest-celery[all]".
Key Highlights
Simple
The plugin provides a single entry point to the test case and makes sure everything is configured according to the selected architecture and requirements.
By default, all of the supported architecture components are added to a matrix of all possible combinations.
Pytest will generate a test case for each combination, and will run it in an isolated environment.
This allows separation of concerns, and makes it simple to access different architectures in a single test case, for example:
def test_hello_world(celery_setup: CeleryTestSetup):
assert celery_setup.ready()
This code will generate test cases for all possible combinations of the supported brokers and backends, using the latest
version of Celery. Under the context of the test, each combination will be available as a celery_setup fixture,
with access to all of the required components, and will run in an isolated environment.
Flexible
The plugin is highly configurable, and can be used to test a wide range of Celery architectures.
It can be configured to use a specific version of Celery, or to use a specific version of a broker or backend.
It can also be configured to use a custom broker or backend, or to use a custom Celery application.
For basic usage, the plugin provides default components that can be configured and extended.
For more advanced use cases, the plugin uses the pytest fixtures mechanism to allow injecting custom components into the environment and build a custom Celery architecture for your project.
For example, see the rabbitmq-management example, which demonstrates how to replace the default broker matrix with a single RabbitMQ Management broker.
Fast
The plugin is designed to run tests in parallel using isolated environments. It supports the pytest-xdist plugin to run tests in parallel and scales well with available resources to improve the overall test suite performance.
Annotated
The codebase is fully annotated with type hints and is tested with mypy to ensure type safety across the board, allowing for a better development experience.
Supports
-
Workers
- Latest Celery version.
- Custom worker.
-
Brokers
- RabbitMQ.
- Redis.
- Custom broker.
-
Backends
- Redis.
- Memcached.
- Custom backend.
-
Clusters
- Worker clusters.
- Broker clusters.
- Backend clusters.
Features
Architecture Injection
By default, a set of predefined components is used to build the Celery architecture for each test.
Each built-in component can be either configured or completely replaced with a custom implementation.
Architecture Injection can be done at different layers, and can be used to replace only specific elements of the architecture pipeline, or to replace the entire pipeline altogether.
Docker Based
The plugin uses docker containers to build the Celery architecture for each test.
This means that the plugin is not limited to specific versions and can be used to test potentially any Celery setup.
It uses the pytest-docker-tools plugin to manage the docker containers which is useful for accessing the docker containers in the test case during the test run and assert on their state with high granularity.
Batteries Included
The plugin provides a set of built-in components that can be used to test ideas quickly.
You can start with the default settings and gradually modify the configurations to fine-tune the test environment. By focusing on the test case, you can quickly iterate and test ideas, without wasting time on the overhead of setting up different environments manually.
Code Generation
One of the challenges in testing production Celery applications is the need to inject testing infrastructure into the Celery worker container at runtime. The plugin provides a Code Generation mechanism that can be used to inject code into the Celery worker container at runtime according to the test case. This opens the door to a wide range of testing scenarios, and allows higher level of control over the tested Celery application.
Isolated Environments
Each test case is executed in an isolated environment. This means that each test case has its own Celery architecture, and is not affected by other test cases. Tests may run in parallel and take care of tearing down themselves when they are done, regardless of the test result.
Tests as First-Class Citizens
The plugin is designed to enhance testing capabilities by treating tests as first-class citizens. It uses advanced mechanisms to encapsulate the complexity of setting up a Celery environment, thus allowing the developer to focus on the test case itself and leave the hard lifting to the plugin.
Extensible
The plugin is designed to be extensible to fit a wide range of use cases and provides a set of built-in components that can be extended to fit more advanced use cases.
It's based on the S.O.L.I.D principles and provides APIs for developing high-quality test suites. It combines the sophistication of the pytest fixtures mechanism with OOP principles to create separation of concerns between each layer of the infrastructure and its elements, which allow a higher level of granularity and control when extending the plugin.
What's Changed
Full Changelog: v1.0.0rc4...v1.0.0
Contributors
Assets 2
v1.0.0rc4
What's Changed
- Improved the celery bug report doc by @Nusnus in #272
- Hotfix: Cleanups by @Nusnus in #273
- Bump cleanpy from 0.4.0 to 0.5.0 by @dependabot in #274
- Hotfix: Doc typo by @Nusnus in #275
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #276
- Build(deps-dev): Bump types-redis from 4.6.0.20240311 to 4.6.0.20240409 by @dependabot in #277
- Hotfix: Added gevent extra to built in worker by @Nusnus in #278
- Docs Enhancements by @Nusnus in #279
- Doc: Removed "Documentation under construction!" by @Nusnus in #280
- Hotfix: Lint error fix by @Nusnus in #281
- Updated sponsorship configurations by @Nusnus in #282
- Docs: Added main README documentation by @Nusnus in #283
- Prepare for (pre) release: v1.0.0rc4 by @Nusnus in #284
- Fixed PyPI README.rst errors that failed the automatic deployment to PyPI by @Nusnus in #285
Full Changelog: v1.0.0rc3...v1.0.0rc4
Contributors
Assets 2
v1.0.0rc3
Contributors
Assets 2
v1.0.0rc2
What's Changed
- Fixed CD by @Nusnus in #257
- Hotfix: New default worker tasks by @Nusnus in #259
- Hotfix: Added celery bug report snippet to CI by @Nusnus in #260
- New Doc: docs/devguide/sphinx.rst by @Nusnus in #261
- New Doc: docs/devguide/release.rst by @Nusnus in #262
- Hotfix: Enhanced Release Guide by @Nusnus in #263
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #265
- Bump pytest-cov from 4.1.0 to 5.0.0 by @dependabot in #264
- Prepare for (pre) release: v1.0.0rc2 by @Nusnus in #266
Full Changelog: v1.0.0rc1...v1.0.0rc2
Contributors
Assets 2
v1.0.0rc1
After so long, the first release candidate for v1.0.0 !
Highlights
- Getting Started documentation is ready!
- User Guide documentation is ready!
- New Examples
- worker_pool: Using
geventandsolopools. - vhost: Using a single Redis instance for both broker & backend, each on different vhost.
- hybrid_setup: Executing a canvas on Celery 5 and Celery 4 workers, collaborating to process their respective segments of the workflow. This involves utilizing distinct pools and queues, along with two brokers set up with failover configurations.
- worker_pool: Using
In addition, the documentation provides a new way to report a Celery bug, which will be integrated into the standard bug report template at a later stage: Standalone Celery Bug Report.
Lastly, from now on, the release dates for the plugin will be described on the Milestones page.
What's Changed
- Added .github/CODEOWNERS by @Nusnus in #240
- New Example: examples/worker_pool by @Nusnus in #242
- Bumping Dependencies by @Nusnus in #243
- New Doc: docs/userguide/celery-bug-report.rst by @Nusnus in #244
- Bump coverage from 7.4.3 to 7.4.4 by @dependabot in #245
- New Example: examples/hybrid_setup by @Nusnus in #246
- Bugfix: node.config() didn't allow overriding by @Nusnus in #247
- Hotfix: Allow overriding node celery config with kwargs by @Nusnus in #249
- New Example: examples/vhost by @Nusnus in #248
- Bump black from 24.2.0 to 24.3.0 by @dependabot in #250
- New Doc: Useful Resources by @Nusnus in #251
- New Doc: Local Dev & Tox guides by @Nusnus in #253
- Hotfix: Fixed lint error in main by @Nusnus in #254
- Stabilized CI by @Nusnus in #255
- Prepare for (pre) release: v1.0.0rc1 by @Nusnus in #256
Full Changelog: v1.0.0b4...v1.0.0rc1
Contributors
Assets 2
v1.0.0b4
What's Changed
- Bump mypy from 1.8.0 to 1.9.0 by @dependabot in #234
- Bump types-redis from 4.6.0.20240218 to 4.6.0.20240311 by @dependabot in #236
- Bump pytest from 8.0.2 to 8.1.1 by @dependabot in #237
- Bump redis from 5.0.2 to 5.0.3 by @dependabot in #235
- Hotfix: Removed after_setup_logger signal handler from the default worker app.py by @Nusnus in #238
- Prepare for (pre) release: v1.0.0b4 by @Nusnus in #239
Full Changelog: v1.0.0b3...v1.0.0b4
Contributors
Assets 2
v1.0.0b3
The celery package is now installed by default again, reverting the decision from v1.0.0b2.
To install the plugin in your environment, please follow the new Advanced Installation Guide.
What's Changed
- Bump black from 24.1.1 to 24.2.0 by @dependabot in #202
- Feature: Support for dynamic worker args by @Nusnus in #203
- Hotfix: Fixed django example by @Nusnus in #204
- Bump pytest from 8.0.0 to 8.0.1 by @dependabot in #207
- Bump types-redis from 4.6.0.20240106 to 4.6.0.20240218 by @dependabot in #208
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #209
- Bump coverage from 7.4.1 to 7.4.2 by @dependabot in #210
- Bugfix: Added missing imports from plugin by @Nusnus in #212
- Bugfix: Missing pytest-celery installation in built-in Celery worker by @Nusnus in #213
- New Example: myutils by @Nusnus in #211
- Bump cryptography from 42.0.2 to 42.0.4 by @dependabot in #214
- More documentation by @Nusnus in #219
- poetry update --with test,dev,ci,docs by @Nusnus in #220
- Change: Removed "celery" extra by @Nusnus in #221
- Revert "Bugfix: Missing pytest-celery installation in built-in Celery worker" by @Nusnus in #222
- Refactored Dependecies Management by @Nusnus in #225
- Cleanup for #225 (6575778) by @Nusnus in #227
- Advanced Installation Guide and Test Setup Matrix Guide by @Nusnus in #228
- More documentation by @Nusnus in #229
- Hotfix: Increased CI triggering changes coverage by @Nusnus in #232
- Bump pytest-subtests from 0.11.0 to 0.12.1 by @dependabot in #231
- Added "SubTests" annotation by @Nusnus in #233
- Prepare for (pre) release: v1.0.0b3 by @Nusnus in #223
Full Changelog: v1.0.0b2...v1.0.0b3
Contributors
Assets 2
v1.0.0b2
Celery is no longer installed by default. Install manually in your test environment, or use the “celery” extra.
What's Changed
- Bump sphinx-celery from 2.1.1 to 2.1.3 by @dependabot in #168
- Added examples/range by @Nusnus in #169
- Added examples/rabbitmq_management by @Nusnus in #170
- Cleanups and hotfixes by @Nusnus in #171
- Do not install Celery by default, use extra [celery] instead by @Nusnus in #172
- Fixed pip deprecation warning about egg fragment with a non-PEP 508 name by @Nusnus in #173
- Reduced installed packages in tox and CI by @Nusnus in #174
- Bump black from 23.12.1 to 24.1.0 by @dependabot in #175
- Packages upgrade by @Nusnus in #176
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #178
- Added initial devguide by @Nusnus in #179
- Bump pytest-sugar from 0.9.7 to 1.0.0 by @dependabot in #181
- Fix typos discovered by codespell by @cclauss in #183
- Added codespell to pre-commit by @Nusnus in #184
- Enhancing Pytest Celery with BSD-Licensed Software by @imSanko in #185
- Upgrade to codecov/codecov-action@v4 by @cclauss in #186
- Initial Getting Started tutorial by @Nusnus in #187
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #188
- Moved examples from getting-started to userguide by @Nusnus in #189
- Added “Next-Steps” to the Getting Started tutorial by @Nusnus in #190
- Hotfix: Moved "Useful Resources" from "Getting Started" to "User Guide" by @Nusnus in #191
- Documentation Enhancement by @Nusnus in #192
- Added documentation to all vendors fixtures by @Nusnus in #193
- Revert "Upgrade to codecov/codecov-action@v4" by @Nusnus in #195
- Bump setuptools from 69.0.3 to 69.1.0 by @dependabot in #194
- Bump codecov/codecov-action from 3 to 4 by @dependabot in #180
- Bugfix: Disabled clusters raised exception on access from setup instead of returning None by @Nusnus in #196
- Bugfix: Disabled clusters raised exception on len(celery_setup) by @Nusnus in #197
- Bugfix: Typo celery_version -> default_worker_celery_version in range example by @Nusnus in #198
- Bugfix: Renamed env() -> initial_env() by @Nusnus in #199
- Removed pytest-lazy-fixture dependency by @Nusnus in #200
- Bump pytest from 7.4.4 to 8.0.0 by @dependabot in #177
- Prepare for (pre) release: v1.0.0b2 by @Nusnus in #201
New Contributors
Full Changelog: v1.0.0b1...v1.0.0b2
Contributors
Assets 2
v1.0.0b1
v1.0.0a12 [YANKED]
What's Changed
- Fixed Installation Errors by @Nusnus in #144
- Updated Community standards by @Nusnus in #145
- Added initial GitHub issues templates by @Nusnus in #146
- Funding by @Nusnus in #147
- Bugfix: WORKER_DOCKERFILE_ROOTDIR now uses resource_filename() by @Nusnus in #150
- poetry add setuptools by @Nusnus in #151
- Added PYTHONPATH=/app to default worker env vars by @Nusnus in #153
- Fixed bug where default env were overriding custom env vars by @Nusnus in #154
- Added Django Example by @Nusnus in #152
- Prepare for (pre) release: v1.0.0a12 by @Nusnus in #155
Full Changelog: v1.0.0a11...v1.0.0a12