Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bugfix: when set config result_expires = 0, chord.get will hang. #6373

Merged
merged 3 commits into from Sep 30, 2020

Conversation

laixintao
Copy link
Contributor

@laixintao laixintao commented Sep 29, 2020

EXPIRE key 0 will delete a key in redis, then chord will never get the
result.

fix: #5237

Description

From the doc:

A value of None or 0 means results will never expire (depending on backend specifications).

https://docs.celeryproject.org/en/stable/userguide/configuration.html#std-setting-result_expires

But here only checked with None, 0 was missed.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: celery#5237
@lgtm-com
Copy link

lgtm-com bot commented Sep 29, 2020

This pull request fixes 2 alerts when merging 8d0600e into ea37db1 - view on LGTM.com

fixed alerts:

  • 1 for Non-exception in 'except' clause
  • 1 for Module is imported with 'import' and 'import from'

auvipy
auvipy previously requested changes Sep 30, 2020
Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also add a unit test for this change?

@laixintao
Copy link
Contributor Author

sure

@laixintao
Copy link
Contributor Author

I just asset that expire shouldn't call when config with zero.

@laixintao
Copy link
Contributor Author

So I wrote a testcase, without this PR, it failed:

➜ t2 celery git:(bugfix/chord-expire) ✗ pytest t/unit/backends/test_redis.py
================================ test session starts ================================
platform darwin -- Python 3.8.2, pytest-6.1.0, py-1.9.0, pluggy-0.13.1
rootdir: /Users/laixintao/Program/celery, configfile: setup.cfg
plugins: celery-0.0.0a1, case-1.5.3, timeout-1.4.2
collected 65 items

t/unit/backends/test_redis.py ..........................................F.... [ 72%]
..................                                                            [100%]

===================================== FAILURES ======================================
__________ test_RedisBackend.test_on_chord_part_return_expire_set_to_zero ___________

self = <t.unit.backends.test_redis.test_RedisBackend object at 0x10bc9b7f0>
restore = <MagicMock name='restore' id='4493273168'>

    @patch('celery.result.GroupResult.restore')
    def test_on_chord_part_return_expire_set_to_zero(self, restore):
        old_expires = self.b.expires
        self.b.expires = 0
        tasks = [self.create_task(i) for i in range(10)]

        for i in range(10):
            self.b.on_chord_part_return(tasks[i].request, states.SUCCESS, i)
            assert self.b.client.zadd.call_count
            self.b.client.zadd.reset_mock()
        assert self.b.client.zrangebyscore.call_count
        jkey = self.b.get_key_for_group('group_id', '.j')
        tkey = self.b.get_key_for_group('group_id', '.t')
        self.b.client.delete.assert_has_calls([call(jkey), call(tkey)])
>       self.b.client.expire.assert_not_called()

t/unit/backends/test_redis.py:729:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Mock name='Redis.expire' id='4493402320'>

    def assert_not_called(self):
        """assert that the mock was never called.
        """
        if self.call_count != 0:
            msg = ("Expected '%s' to not have been called. Called %s times.%s"
                   % (self._mock_name or 'mock',
                      self.call_count,
                      self._calls_repr()))
>           raise AssertionError(msg)
E           AssertionError: Expected 'expire' to not have been called. Called 20 times.
E           Calls: [call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0),
E            call(b'celery-taskset-meta-group_id.j', 0),
E            call(b'celery-taskset-meta-group_id.t', 0)].

/usr/local/var/pyenv/versions/3.8.2/lib/python3.8/unittest/mock.py:874: AssertionError
============================== short test summary info ==============================
FAILED t/unit/backends/test_redis.py::test_RedisBackend::test_on_chord_part_return_expire_set_to_zero
=========================== 1 failed, 64 passed in 1.26s ============================

With this PR, it passed! ✨ I think it's enough, what do you think? @auvipy

Copy link
Member

@thedrow thedrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thank you!

@thedrow thedrow merged commit ce4f759 into celery:master Sep 30, 2020
sihrc added a commit to IndicoDataSolutions/celery that referenced this pull request Dec 23, 2020
* Remove defaults for unsupported Python runtimes.

* Remove obsolete test.

* Doc pytest plugin (celery#6289)

* update to new pytest name

* doc pytest plugin

* trim heading to the length of the new pytest name

* add warning against use of sort key on dynamodb table, closes celery#6332

* Remove celery.five and bump vine dep (celery#6338)

* improv: Replace `five.values` with `dict.values`

* improv: Use `time.monotonic()` in kombu tests

Also in the docs where it is used to demonstrate `memcache` timeouts.

* rm: Delete `celery.five`

`vine.five` is no longer present in `vine >= 5`.

* triv: Remove refs to `celery.five` in docs, &c

* build: Bump `vine` dependency to 5.0+

* Wheels are no longer universal.

* Remove failing before_install step.

* Update changelog.

* Bump version: 5.0.0rc2 → 5.0.0rc3

* Fix release date.

* Remove unused import.

* Correctly skip these tests when the relevant dependency is missing.

* Expose retry_policy for Redis result backend

Rather than adding a new top-level config option, I have used a new key
in the already existing setting `result_backend_transport_options`.

Closes celery#6166

* Update changelog for 4.3.1.

* fix typo (celery#6346)

* Travis CI: Test Python 3.9 release candidate 1 (celery#6328)

* Travis CI: Test Python 3.9 release candidate 1

* fixup! Travis CI: matrix --> jobs

* fixup! Fix indentation error

* fixup! tox.ini: 3.9 --> 3.9-dev

* Fix test failure in Python 3.9RC1.

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Fix the broken celery upgrade settings command.

* Fix celery migrate settings options.

* Remove Riak result backend settings.

* Rephrase to mention that 3.5 is also EOL.

* Add a note about the removal of the Riak result backend.

* Fix examples of starting a worker in comments (celery#6331)

* Remove deprecated function from app.log.Logging.

* Migration guide.

* Document breaking changes for the CLI in the Whats New document.

* Add a "port code to Python 3" migration step.

* Update supported Python versions in the introduction document.

* Update bash completion.

* Add note about new shell completion.

* Update daemonization docs.

* Remove amqp backend. (celery#6360)

Fixes celery#6356.

* Warn when deprecated settings are used (celery#6353)

* Warn when deprecated settings are used.

* Mention deprecation in docs.

* Refer to the right place in the documentation.

* Complete What's New.

* Add wall of contributors.

* Update codename.

* Fix alt text.

* isort.

* PyPy 3.7 is currently in alpha.

No need for that sentence.

* Mention the new pytest-celery plugin.

* Mention retry policy for the redis result backend.

* Fix phrasing.

* Mention ordered group results are now the default.

* pyupgrade.

* Complete release notes.

* Bump version: 5.0.0rc3 → 5.0.0

* Happify linters.

* Specify utf-8 as the encoding for log files.

Fixes celery#5144.

* Fixed some typos in readme

* Fix custom headers propagation for protocol 1 hybrid messages

* Retry after race during schema creation in database backend (celery#6298)

* Retry after race during schema creation in database backend

Fixes celery#6296

This race condition does not commonly present, since the schema creation
only needs to happen once per database. It's more likely to appear in
e.g. a test suite that uses a new database each time.

For context of the sleep times I chose, the schema creation takes ~50 ms
on my laptop.

I did a simulated test run of 50 concurrent calls to MetaData.create_all
repeated 200 times and the number of retries was:

- 0 retries: 8717x
- 1 retry:   1279x
- 2 retries  4x

* Add test for prepare_models retry error condition

* Add name to contributors

* Update daemonizing.rst

Fix daemonizing documentation for issue celery#6363 to put `multi` before `-A`

* Revert "Update daemonizing.rst" (celery#6376)

This reverts commit 96ec6db.

* bugfix: when set config result_expires = 0, chord.get will hang. (celery#6373)

* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: celery#5237

* test: add testcase for expire when set config with zero.

* Display a custom error message whenever an attempt to use -A or --app as a sub-command option was made.

Fixes celery#6363

* Remove test dependencies for Python 2.7.

* Restore the celery worker --without-{gossip,mingle,heartbeat} flags (celery#6365)

In the previously used argparse arguments framework, these three options were
used as flags.

Since 5.0.0, they are options which need to take an argument (whose only
sensible value would be "true"). The error message coming up is also (very)
hard to understand, when running the celery worker command with an odd number
of flags:

  Error: Unable to parse extra configuration from command line.
  Reason: not enough values to unpack (expected 2, got 1)

When the celery worker is run with an even number of flags, the last one is
considered as an argument of the previous one, which is a subtle bug.

* Provide clearer error messages when app fails to load.

* fix pytest plugin registration documentation (celery#6387)

* fix pytest plugin registration documentation

* Update docs/userguide/testing.rst

Co-authored-by: Thomas Grainger <tagrain@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Contains a workaround for the capitalized configuration issue (celery#6385)

* Contains a workaround for the capitalized configuration issue

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove old explanation regarding `absolute_import` (celery#6390)

Resolves celery#6389.

* Update canvas.rst (celery#6392)

* Update canvas.rst

Tiny fixes.

* Update docs/userguide/canvas.rst

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove duplicate words from docs (celery#6398)

Remove the duplicate usage of “required” in documentation (specifically, `introduction.rst`).

* Allow lowercase log levels. (celery#6396)

Fixes celery#6395.

* Detach now correctly passes options with more than one word. (celery#6394)

When specifying options such as `-E` the detached worker should receive the `--task-events` option.
Instead it got the `--task_events` option which doesn't exist and therefore silently failed.

This fixes celery#6362.

* The celery multi command now works as expected. (celery#6388)

* Contains the missed change requested by @thedrow

* Added a some celery configuration examples.

* fixed loglevel info->INFO in docs

* return list instead set in CommaSeparatedList

_broadcast method of kombu Mailbox. does not support set
https://github.com/celery/kombu/blob/7b2578b19ba4b1989b722f6f6e7efee2a1a4d86a/kombu/pidbox.py#L319

* Rewrite detaching logic (celery#6401)

* Rewrite detaching logic.

* Ignore empty arguments.

* Ensure the SystemD services are up to date.

* fix: Pass back real result for single task chains

When chains are delayed, they are first frozen as part of preparation
which causes the sub-tasks to also be frozen. Afterward, the final (0th
since we reverse the tasks/result order when freezing) result object
from the freezing process would be passed back to the caller. This
caused problems in signaling completion of groups contained in chains
because the group relies on a promise which is fulfilled by a barrier
linked to each of its applied subtasks. By constructing two
`GroupResult` objects (one during freezing, one when the chain sub-tasks
are applied), this resulted in there being two promises; only one of
which would actually be fulfilled by the group subtasks.

This change ensures that in the special case where a chain has a single
task, we pass back the result object constructed when the task was
actually applied. When that single child is a group which does not get
unrolled (ie. contains more than one child itself), this ensures that we
pass back a `GroupResult` object which will actually be fulfilled. The
caller can then await the result confidently!

* fix: Retain `group_id` when tasks get re-frozen

When a group task which is part of a chain was to be delayed by
`trace_task()`, it would be reconstructed from the serialized request.
Normally, this sets the `group_id` of encapsulated tasks to the ID of
the group being instantiated. However, in the specific situation of a
group that is the last task in a chain which contributes to the
completion of a chord, it is essential that the group ID of the top-most
group is used instead. This top-most group ID is used by the redis
backend to track the completions of "final elements" of a chord in the
`on_chord_part_return()` implementation. By overwriting the group ID
which was already set in the `options` dictionaries of the child tasks
being deserialized, the chord accounting done by the redis backend would
be made inaccurate and chords would never complete.

This change alters how options are overridden for signatures to ensure
that if a `group_id` has already been set, it cannot be overridden.
Since group ID should be generally opaque to users, this should not be
disruptive.

* fix: Count chord "final elements" correctly

This change amends the implementation of `chord.__length_hint__()` to
ensure that all child task types are correctly counted. Specifically:

 * all sub-tasks of a group are counted recursively
 * the final task of a chain is counted recursively
 * the body of a chord is counted recursively
 * all other simple signatures count as a single "final element"

There is also a deserialisation step if a `dict` is seen while counting
the final elements in a chord, however this should become less important
with the merge of celery#6342 which ensures that tasks are recursively
deserialized by `.from_dict()`.

* test: Add more integration tests for groups

These tests are intended to show that group unrolling should be
respected in various ways by all backends. They should make it more
clear what behaviour we should be expecting from nested canvas
components and ensure that all the implementations (mostly relevant to
chords and `on_chord_part_return()` code) behave sensibly.

* test: Fix old markings for chord tests

* fix: Make KV-store backends respect chord size

This avoids an issue where the `on_chord_part_return()` implementation
would check the the length of the result of a chain ending in a nested
group. This would manifest in behaviour where a worker would be blocked
waiting for for the result object it holds to complete since it would
attempt to `.join()` the result object. In situations with plenty of
workers, this wouldn't really cause any noticable issue apart from some
latency or unpredictable failures - but in concurrency constrained
situations like the integrations tests, it causes deadlocks.

We know from previous commits in this series that chord completion is
more complex than just waiting for a direct child, so we correct the
`size` value in `BaseKeyValueStoreBackend.on_chord_part_return()` to
respect the `chord_size` value from the request, falling back to the
length of the `deps` if that value is missing for some reason (this is
necessary to keep a number of the tests happy but it's not clear to me
if that will ever be the case in real life situations).

* fix: Retain chord header result structure in Redis

This change fixes the chord result flattening issue which manifested
when using the Redis backend due to its deliberate throwing away of
information about the header result structure. Rather than assuming that
all results which contribute to the finalisation of a chord should be
siblings, this change checks if any are complex (ie. `GroupResult`s) and
falls back to behaviour similar to that implemented in the
`KeyValueStoreBackend` which restores the original `GroupResult` object
and `join()`s it.

We retain the original behaviour which is billed as an optimisation in
f09b041. We could behave better in the complex header result case by not
bothering to stash the results of contributing tasks under the `.j` zset
since we won't be using them, but without checking for the presence of
the complex group result on every `on_chord_part_return()` call, we
can't be sure that we won't need those stashed results later on. This
would be an opportunity for optimisation in future if we were to use an
`EVAL` to only do the `zadd()` if the group result key doesn't exist.
However, avoiding the result encoding work in `on_chord_part_return()`
would be more complicated. For now, it's not worth the brainpower.

This change also slightly refactors the redis backend unit tests to make
it easier to build fixtures and hit both the complex and simple result
structure cases.

* Update obsolete --loglevel argument values in docs

* Set logfile, not loglevel.

* Mention removed deprecated modules in the release notes.

Fixes celery#6406.

* Copy __annotations__ when creating tasks

This will allow getting type hints.

Fixes celery#6186.

* test: Improve chord body group index freezing test

Add more elements to the body so we can verify that the `group_index`
counts up from 0 as expected. This change adds the `pytest-subtests`
package as a test dependency so we can define partially independent
subtests within test functions.

* test: Use all() for subtask checks in canvas tests

When we expect all of the tasks in some iterable to meet a conditional,
we should make that clear by using `all(condition for ...)`.

* test: Add more tests for `from_dict()` variants

Notably, this exposed the bug tracked in celery#6341 where groups are not
deeply deserialized by `group.from_dict()`.

* fix: Ensure group tasks are deeply deserialised

Fixes celery#6341

* Fix `celery shell` command

* predefined_queues_urls -> predefined_queues

* Update changelog.

* Bump version: 5.0.0 → 5.0.1

* [Fix celery#6361] Fixing documentation for RabbitMQ task_queue_ha_policy

* Fix _autodiscover_tasks_from_fixups function

* fixup! Fix _autodiscover_tasks_from_fixups function

* Correct configuration item: CELERY_RESULT_EXPIRES

Related issue: celery#4050
celery#4050 (comment)

* Flush worker prints, notably the banner

In some cases (kubernetes, root) the banner is only printed at the
end of the process execution, instead of at the beginning.

* [Fix celery#6361] Remove RabbitMQ ha_policy from queue

* ci: Fix TOXENV for pypy3 unit tests

Fixes celery#6409

* ci: Move Python 3.9 test base from dev to release

* docs: fix celery beat settings

* move to travis-ci.com

* fix: Ensure default fairness maps to `SCHED_FAIR` (celery#6447)

Fixes celery#6386

* Preserve callbacks when replacing a task with a chain (celery#6189)

* Preserve callbacks when replacing a task with a chain.

* Preserve callbacks when replacing a task with a chain.

* Added tests.

* Update celery/app/task.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Mark test as flaky.

* Fix race condition in CI.

* fix: Run linked tasks in original slot for replace

This change alters the handling of linked tasks for chains which are
used as the argument to a `.replace()` call for a task which itself has
a chain of signatures to call once it completes. We ensure that the
linked callback is not only retained but also called at the appropiate
point in the newly reconstructed chain comprised of tasks from both the
replacement chain and the tail of the encapsulating chain of the task
being replaced.

We amend some tests to validate this behaviour better and ensure that
call/errbacks behave as expected if the encapsulating chain has either
set. One test is marked with an `xfail` since errbacks of encapsulating
chains are not currently called as expected due to some ambiguity in
when an errback of a replaced task should be dropped or not (celery#6441).

Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix minor documentation omission (celery#6453)

Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>

* Fix max_retries override on self.retry (celery#6436)

* Fix max_retries override

* Fix max_retries override

* Fix max_retries override

* Update exceptions.py

typo

* Update autoretry.py

typo

* Update task.py

Prevent exception unpacking for tasks without autoretry_for

* Update test_tasks.py

Unit test

* Update test_tasks.py

Added a new test

* Update autoretry.py

Fox for explicit raise in tasks

* Update test_tasks.py

* Update autoretry.py

* Update task.py

* Update exceptions.py

* Update task.py

* Happify linter.

* Raise proper error when replacing with an empty chain. (celery#6452)

Fixes celery#6451.

* Update changelog.

* Bump version: 5.0.1 → 5.0.2

* Update daemonizing.rst

Improved systemd documentation for auto-start of the service, and mention the possibility to depend on RabbitMQ service. Also add Restart=always for Celery Beat example

* Update celerybeat.service

* Fix old celery beat variables

Change made 5 days ago in 7c3da03 is faulty, the correct celery beat variables do start with `CELERYBEAT` and not `CELERY_BEAT`

* Fix formatting.

* Fix formatting.

* fix: Make `--workdir` eager for early handling (celery#6457)

This change makes the `--workdir` options an eager one which `click`
will process early for us, before any of the others. At the same time,
we add a callback which ensures that the `chdir()` is run during
handling of the argument so that all subsequent actions (e.g. app
loading) occur in the specified working directory.

Fixes celery#6445

* Fix example.

Fixes celery#6459.

* When using the MongoDB backend, don't cleanup if result_expires is 0 or None. (celery#6462)

Fixes celery#6450.

* Add missing space (celery#6468)

* Fix passing queues into purge command (celery#6469)

In current wersion calling `celery --app my.celery_app purge -Q queue_name` is failing with following trace:

```
    names = (queues or set(app.amqp.queues.keys())) - exclude_queues
TypeError: unsupported operand type(s) for -: 'list' and 'list'
```

Becouse code is expecting set and `queues` is actually a list.

Here is a fix.

* Change donations sidebar to direct users to OpenCollective.

* Added pytest to extras.

Missed in 9a6c292.

* Restore app.start() and app.worker_main() (celery#6481)

* Restore `app.start()` and `app.worker_main()`.

* Update celery/app/base.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix spelling error.

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* fix: `node_format()` logfile before detaching

Fixes celery#6426

* Multithreaded backend (celery#6416)

* Cache backend to thread local storage instead of global variable

* Cache oid to thread local storage instead of global variable

* Improve code returning thread_local data

* Move thread local storage to Celery class, introduced thread_oid and added unittests

* Remove python2 compatibility code

* Restore ability to extend the CLI with new sub-commands.

* Adjust documentation to demonstrate how to introduce sub-command plugins in 5.x.

Fixes celery#6439.

* autopep8 & isort.

* Linters now run using Python 3.9.

* Fix apply_async() in Calling Tasks userguide

* Fix dead links in contributing guide (celery#6506)

* Fix inconsistency in documentation for `link_error` (celery#6505)

* Make documentation of link_error consistent

Fixes celery#4099

* Fix undefined variable in example

* Add to contributors list

* Update testing.rst (celery#6507)

Use double back ticks for some code examples, so that quotes don't get converted into smart-quotes.

celery#6497

* Don't upgrade click to 8.x since click-repl doesn't support it yet.

Fixes celery#6511.
Upstream issue: click-contrib/click-repl#72

* Update documentation on changes to custom CLI options in 5.0.

Fixes celery#6380.

* update step to install homebrew

* redis: Support Sentinel with SSL

Use the SentinelManagedSSLConnection when SSL is enabled for the
transport. The redis-py project doesn't have a connection class for
SSL+Sentinel yet. So, create a class in redis.py to add that
functionality.

* Revert "redis: Support Sentinel with SSL" (celery#6518)

This reverts commit 18a0963.

* Reintroduce support for custom preload options (celery#6516)

* Restore preload options.

Fixes celery#6307.

* Document breaking changes for preload options in 5.0.

Fixes celery#6379.

* Changelog for 5.0.3.

* Bump version: 5.0.2 → 5.0.3

* Added integration tests for calling a task (celery#6523)

* DummyClient of cache+memory:// backend now shares state between threads (celery#6524)

* isort.

* Update changelog.

* Bump version: 5.0.3 → 5.0.4

* Change deprecated from collections import Mapping/MutableMapping to from collections.abc ... (celery#6532)

* fix celery#6047

* Fix type error in S3 backend (celery#6537)

* Convert key from bytes to str

* Add unit test for S3 delete of key with type bytes

* events.py: Remove duplicate decorator in wrong place (celery#6543)

`@handle_preload_options` was specified twice as a decorator of `events`, once at the top (wrong) and once at the bottom (right).
This fixes the `celery events` commands and also `celery --help`

* Update changelog.

* Bump version: 5.0.4 → 5.0.5

* ADD: indico additions - trails

* FIX: remove dev.txt dependencies

* ADD: handle revoke failures

* ADD: trailer_request support and better drain resolution

* ADD: merge options was overriding link_error values

* PATCH: DLX and reject behaviour

* FIX: amqp dependencies

Co-authored-by: Omer Katz <omer.drow@gmail.com>
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Martin Paulus <mpaulus@lequest.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: qiaocc <jasonqiao36@gmail.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Weiliang Li <to.be.impressive@gmail.com>
Co-authored-by: Akash Agrawal <akashrocksha@gmail.com>
Co-authored-by: Michal Kuffa <michal.kuffa@sentry.io>
Co-authored-by: Frazer McLean <frazer@frazermclean.co.uk>
Co-authored-by: Maarten Fonville <mfonville@users.noreply.github.com>
Co-authored-by: laixintao <laixintaoo@gmail.com>
Co-authored-by: Nicolas Dandrimont <nicolas@dandrimont.eu>
Co-authored-by: Bas ten Berge <bas+github@tenberge-ict.nl>
Co-authored-by: Zvi Baratz <z.baratz@gmail.com>
Co-authored-by: Justinas Petuchovas <justinas.petuchovas@gmail.com>
Co-authored-by: bastb <bas@tenberge-ict.nl>
Co-authored-by: Artem Bernatskyi <artem.bernatskyy@gmail.com>
Co-authored-by: ZubAnt <ya.zubarevanton@yandex.ru>
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>
Co-authored-by: David Pärsson <david@parsson.se>
Co-authored-by: Anthony Lukach <anthonylukach@gmail.com>
Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com>
Co-authored-by: Stepan Henek <stepan@henek.name>
Co-authored-by: KexZh <hanbaobao2005@gmail.com>
Co-authored-by: Thomas Riccardi <thomas@deepomatic.com>
Co-authored-by: Egor Sergeevich Poderiagin <egor@crazyrussian.pro>
Co-authored-by: Asif Saif Uddin (Auvi) <auvipy@gmail.com>
Co-authored-by: Lewis M. Kabui <13940255+lewisemm@users.noreply.github.com>
Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com>
Co-authored-by: Mathieu Rollet <matletix@gmail.com>
Co-authored-by: Mike DePalatis <depalatis@gmail.com>
Co-authored-by: partizan <serg.partizan@gmail.com>
Co-authored-by: Nick Pope <nick.pope@flightdataservices.com>
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
Co-authored-by: Matus Valo <matusvalo@gmail.com>
Co-authored-by: henribru <6639509+henribru@users.noreply.github.com>
Co-authored-by: Stuart Axon <stuaxo2@yahoo.com>
Co-authored-by: Sonya Chhabra <sonyakc.2007@gmail.com>
Co-authored-by: AbdealiJK <abdealikothari@gmail.com>
Co-authored-by: František Zatloukal <Zatloukal.Frantisek@gmail.com>
Co-authored-by: elonzh <elonzh@qq.com>
Co-authored-by: Sven Koitka <razorx89@users.noreply.github.com>
Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>
sihrc added a commit to IndicoDataSolutions/celery that referenced this pull request Feb 17, 2021
* Remove defaults for unsupported Python runtimes.

* Remove obsolete test.

* Doc pytest plugin (celery#6289)

* update to new pytest name

* doc pytest plugin

* trim heading to the length of the new pytest name

* add warning against use of sort key on dynamodb table, closes celery#6332

* Remove celery.five and bump vine dep (celery#6338)

* improv: Replace `five.values` with `dict.values`

* improv: Use `time.monotonic()` in kombu tests

Also in the docs where it is used to demonstrate `memcache` timeouts.

* rm: Delete `celery.five`

`vine.five` is no longer present in `vine >= 5`.

* triv: Remove refs to `celery.five` in docs, &c

* build: Bump `vine` dependency to 5.0+

* Wheels are no longer universal.

* Remove failing before_install step.

* Update changelog.

* Bump version: 5.0.0rc2 → 5.0.0rc3

* Fix release date.

* Remove unused import.

* Correctly skip these tests when the relevant dependency is missing.

* Expose retry_policy for Redis result backend

Rather than adding a new top-level config option, I have used a new key
in the already existing setting `result_backend_transport_options`.

Closes celery#6166

* Update changelog for 4.3.1.

* fix typo (celery#6346)

* Travis CI: Test Python 3.9 release candidate 1 (celery#6328)

* Travis CI: Test Python 3.9 release candidate 1

* fixup! Travis CI: matrix --> jobs

* fixup! Fix indentation error

* fixup! tox.ini: 3.9 --> 3.9-dev

* Fix test failure in Python 3.9RC1.

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Fix the broken celery upgrade settings command.

* Fix celery migrate settings options.

* Remove Riak result backend settings.

* Rephrase to mention that 3.5 is also EOL.

* Add a note about the removal of the Riak result backend.

* Fix examples of starting a worker in comments (celery#6331)

* Remove deprecated function from app.log.Logging.

* Migration guide.

* Document breaking changes for the CLI in the Whats New document.

* Add a "port code to Python 3" migration step.

* Update supported Python versions in the introduction document.

* Update bash completion.

* Add note about new shell completion.

* Update daemonization docs.

* Remove amqp backend. (celery#6360)

Fixes celery#6356.

* Warn when deprecated settings are used (celery#6353)

* Warn when deprecated settings are used.

* Mention deprecation in docs.

* Refer to the right place in the documentation.

* Complete What's New.

* Add wall of contributors.

* Update codename.

* Fix alt text.

* isort.

* PyPy 3.7 is currently in alpha.

No need for that sentence.

* Mention the new pytest-celery plugin.

* Mention retry policy for the redis result backend.

* Fix phrasing.

* Mention ordered group results are now the default.

* pyupgrade.

* Complete release notes.

* Bump version: 5.0.0rc3 → 5.0.0

* Happify linters.

* Specify utf-8 as the encoding for log files.

Fixes celery#5144.

* Fixed some typos in readme

* Fix custom headers propagation for protocol 1 hybrid messages

* Retry after race during schema creation in database backend (celery#6298)

* Retry after race during schema creation in database backend

Fixes celery#6296

This race condition does not commonly present, since the schema creation
only needs to happen once per database. It's more likely to appear in
e.g. a test suite that uses a new database each time.

For context of the sleep times I chose, the schema creation takes ~50 ms
on my laptop.

I did a simulated test run of 50 concurrent calls to MetaData.create_all
repeated 200 times and the number of retries was:

- 0 retries: 8717x
- 1 retry:   1279x
- 2 retries  4x

* Add test for prepare_models retry error condition

* Add name to contributors

* Update daemonizing.rst

Fix daemonizing documentation for issue celery#6363 to put `multi` before `-A`

* Revert "Update daemonizing.rst" (celery#6376)

This reverts commit 96ec6db.

* bugfix: when set config result_expires = 0, chord.get will hang. (celery#6373)

* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: celery#5237

* test: add testcase for expire when set config with zero.

* Display a custom error message whenever an attempt to use -A or --app as a sub-command option was made.

Fixes celery#6363

* Remove test dependencies for Python 2.7.

* Restore the celery worker --without-{gossip,mingle,heartbeat} flags (celery#6365)

In the previously used argparse arguments framework, these three options were
used as flags.

Since 5.0.0, they are options which need to take an argument (whose only
sensible value would be "true"). The error message coming up is also (very)
hard to understand, when running the celery worker command with an odd number
of flags:

  Error: Unable to parse extra configuration from command line.
  Reason: not enough values to unpack (expected 2, got 1)

When the celery worker is run with an even number of flags, the last one is
considered as an argument of the previous one, which is a subtle bug.

* Provide clearer error messages when app fails to load.

* fix pytest plugin registration documentation (celery#6387)

* fix pytest plugin registration documentation

* Update docs/userguide/testing.rst

Co-authored-by: Thomas Grainger <tagrain@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Contains a workaround for the capitalized configuration issue (celery#6385)

* Contains a workaround for the capitalized configuration issue

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove old explanation regarding `absolute_import` (celery#6390)

Resolves celery#6389.

* Update canvas.rst (celery#6392)

* Update canvas.rst

Tiny fixes.

* Update docs/userguide/canvas.rst

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove duplicate words from docs (celery#6398)

Remove the duplicate usage of “required” in documentation (specifically, `introduction.rst`).

* Allow lowercase log levels. (celery#6396)

Fixes celery#6395.

* Detach now correctly passes options with more than one word. (celery#6394)

When specifying options such as `-E` the detached worker should receive the `--task-events` option.
Instead it got the `--task_events` option which doesn't exist and therefore silently failed.

This fixes celery#6362.

* The celery multi command now works as expected. (celery#6388)

* Contains the missed change requested by @thedrow

* Added a some celery configuration examples.

* fixed loglevel info->INFO in docs

* return list instead set in CommaSeparatedList

_broadcast method of kombu Mailbox. does not support set
https://github.com/celery/kombu/blob/7b2578b19ba4b1989b722f6f6e7efee2a1a4d86a/kombu/pidbox.py#L319

* Rewrite detaching logic (celery#6401)

* Rewrite detaching logic.

* Ignore empty arguments.

* Ensure the SystemD services are up to date.

* fix: Pass back real result for single task chains

When chains are delayed, they are first frozen as part of preparation
which causes the sub-tasks to also be frozen. Afterward, the final (0th
since we reverse the tasks/result order when freezing) result object
from the freezing process would be passed back to the caller. This
caused problems in signaling completion of groups contained in chains
because the group relies on a promise which is fulfilled by a barrier
linked to each of its applied subtasks. By constructing two
`GroupResult` objects (one during freezing, one when the chain sub-tasks
are applied), this resulted in there being two promises; only one of
which would actually be fulfilled by the group subtasks.

This change ensures that in the special case where a chain has a single
task, we pass back the result object constructed when the task was
actually applied. When that single child is a group which does not get
unrolled (ie. contains more than one child itself), this ensures that we
pass back a `GroupResult` object which will actually be fulfilled. The
caller can then await the result confidently!

* fix: Retain `group_id` when tasks get re-frozen

When a group task which is part of a chain was to be delayed by
`trace_task()`, it would be reconstructed from the serialized request.
Normally, this sets the `group_id` of encapsulated tasks to the ID of
the group being instantiated. However, in the specific situation of a
group that is the last task in a chain which contributes to the
completion of a chord, it is essential that the group ID of the top-most
group is used instead. This top-most group ID is used by the redis
backend to track the completions of "final elements" of a chord in the
`on_chord_part_return()` implementation. By overwriting the group ID
which was already set in the `options` dictionaries of the child tasks
being deserialized, the chord accounting done by the redis backend would
be made inaccurate and chords would never complete.

This change alters how options are overridden for signatures to ensure
that if a `group_id` has already been set, it cannot be overridden.
Since group ID should be generally opaque to users, this should not be
disruptive.

* fix: Count chord "final elements" correctly

This change amends the implementation of `chord.__length_hint__()` to
ensure that all child task types are correctly counted. Specifically:

 * all sub-tasks of a group are counted recursively
 * the final task of a chain is counted recursively
 * the body of a chord is counted recursively
 * all other simple signatures count as a single "final element"

There is also a deserialisation step if a `dict` is seen while counting
the final elements in a chord, however this should become less important
with the merge of celery#6342 which ensures that tasks are recursively
deserialized by `.from_dict()`.

* test: Add more integration tests for groups

These tests are intended to show that group unrolling should be
respected in various ways by all backends. They should make it more
clear what behaviour we should be expecting from nested canvas
components and ensure that all the implementations (mostly relevant to
chords and `on_chord_part_return()` code) behave sensibly.

* test: Fix old markings for chord tests

* fix: Make KV-store backends respect chord size

This avoids an issue where the `on_chord_part_return()` implementation
would check the the length of the result of a chain ending in a nested
group. This would manifest in behaviour where a worker would be blocked
waiting for for the result object it holds to complete since it would
attempt to `.join()` the result object. In situations with plenty of
workers, this wouldn't really cause any noticable issue apart from some
latency or unpredictable failures - but in concurrency constrained
situations like the integrations tests, it causes deadlocks.

We know from previous commits in this series that chord completion is
more complex than just waiting for a direct child, so we correct the
`size` value in `BaseKeyValueStoreBackend.on_chord_part_return()` to
respect the `chord_size` value from the request, falling back to the
length of the `deps` if that value is missing for some reason (this is
necessary to keep a number of the tests happy but it's not clear to me
if that will ever be the case in real life situations).

* fix: Retain chord header result structure in Redis

This change fixes the chord result flattening issue which manifested
when using the Redis backend due to its deliberate throwing away of
information about the header result structure. Rather than assuming that
all results which contribute to the finalisation of a chord should be
siblings, this change checks if any are complex (ie. `GroupResult`s) and
falls back to behaviour similar to that implemented in the
`KeyValueStoreBackend` which restores the original `GroupResult` object
and `join()`s it.

We retain the original behaviour which is billed as an optimisation in
f09b041. We could behave better in the complex header result case by not
bothering to stash the results of contributing tasks under the `.j` zset
since we won't be using them, but without checking for the presence of
the complex group result on every `on_chord_part_return()` call, we
can't be sure that we won't need those stashed results later on. This
would be an opportunity for optimisation in future if we were to use an
`EVAL` to only do the `zadd()` if the group result key doesn't exist.
However, avoiding the result encoding work in `on_chord_part_return()`
would be more complicated. For now, it's not worth the brainpower.

This change also slightly refactors the redis backend unit tests to make
it easier to build fixtures and hit both the complex and simple result
structure cases.

* Update obsolete --loglevel argument values in docs

* Set logfile, not loglevel.

* Mention removed deprecated modules in the release notes.

Fixes celery#6406.

* Copy __annotations__ when creating tasks

This will allow getting type hints.

Fixes celery#6186.

* test: Improve chord body group index freezing test

Add more elements to the body so we can verify that the `group_index`
counts up from 0 as expected. This change adds the `pytest-subtests`
package as a test dependency so we can define partially independent
subtests within test functions.

* test: Use all() for subtask checks in canvas tests

When we expect all of the tasks in some iterable to meet a conditional,
we should make that clear by using `all(condition for ...)`.

* test: Add more tests for `from_dict()` variants

Notably, this exposed the bug tracked in celery#6341 where groups are not
deeply deserialized by `group.from_dict()`.

* fix: Ensure group tasks are deeply deserialised

Fixes celery#6341

* Fix `celery shell` command

* predefined_queues_urls -> predefined_queues

* Update changelog.

* Bump version: 5.0.0 → 5.0.1

* [Fix celery#6361] Fixing documentation for RabbitMQ task_queue_ha_policy

* Fix _autodiscover_tasks_from_fixups function

* fixup! Fix _autodiscover_tasks_from_fixups function

* Correct configuration item: CELERY_RESULT_EXPIRES

Related issue: celery#4050
celery#4050 (comment)

* Flush worker prints, notably the banner

In some cases (kubernetes, root) the banner is only printed at the
end of the process execution, instead of at the beginning.

* [Fix celery#6361] Remove RabbitMQ ha_policy from queue

* ci: Fix TOXENV for pypy3 unit tests

Fixes celery#6409

* ci: Move Python 3.9 test base from dev to release

* docs: fix celery beat settings

* move to travis-ci.com

* fix: Ensure default fairness maps to `SCHED_FAIR` (celery#6447)

Fixes celery#6386

* Preserve callbacks when replacing a task with a chain (celery#6189)

* Preserve callbacks when replacing a task with a chain.

* Preserve callbacks when replacing a task with a chain.

* Added tests.

* Update celery/app/task.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Mark test as flaky.

* Fix race condition in CI.

* fix: Run linked tasks in original slot for replace

This change alters the handling of linked tasks for chains which are
used as the argument to a `.replace()` call for a task which itself has
a chain of signatures to call once it completes. We ensure that the
linked callback is not only retained but also called at the appropiate
point in the newly reconstructed chain comprised of tasks from both the
replacement chain and the tail of the encapsulating chain of the task
being replaced.

We amend some tests to validate this behaviour better and ensure that
call/errbacks behave as expected if the encapsulating chain has either
set. One test is marked with an `xfail` since errbacks of encapsulating
chains are not currently called as expected due to some ambiguity in
when an errback of a replaced task should be dropped or not (celery#6441).

Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix minor documentation omission (celery#6453)

Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>

* Fix max_retries override on self.retry (celery#6436)

* Fix max_retries override

* Fix max_retries override

* Fix max_retries override

* Update exceptions.py

typo

* Update autoretry.py

typo

* Update task.py

Prevent exception unpacking for tasks without autoretry_for

* Update test_tasks.py

Unit test

* Update test_tasks.py

Added a new test

* Update autoretry.py

Fox for explicit raise in tasks

* Update test_tasks.py

* Update autoretry.py

* Update task.py

* Update exceptions.py

* Update task.py

* Happify linter.

* Raise proper error when replacing with an empty chain. (celery#6452)

Fixes celery#6451.

* Update changelog.

* Bump version: 5.0.1 → 5.0.2

* Update daemonizing.rst

Improved systemd documentation for auto-start of the service, and mention the possibility to depend on RabbitMQ service. Also add Restart=always for Celery Beat example

* Update celerybeat.service

* Fix old celery beat variables

Change made 5 days ago in 7c3da03 is faulty, the correct celery beat variables do start with `CELERYBEAT` and not `CELERY_BEAT`

* Fix formatting.

* Fix formatting.

* fix: Make `--workdir` eager for early handling (celery#6457)

This change makes the `--workdir` options an eager one which `click`
will process early for us, before any of the others. At the same time,
we add a callback which ensures that the `chdir()` is run during
handling of the argument so that all subsequent actions (e.g. app
loading) occur in the specified working directory.

Fixes celery#6445

* Fix example.

Fixes celery#6459.

* When using the MongoDB backend, don't cleanup if result_expires is 0 or None. (celery#6462)

Fixes celery#6450.

* Add missing space (celery#6468)

* Fix passing queues into purge command (celery#6469)

In current wersion calling `celery --app my.celery_app purge -Q queue_name` is failing with following trace:

```
    names = (queues or set(app.amqp.queues.keys())) - exclude_queues
TypeError: unsupported operand type(s) for -: 'list' and 'list'
```

Becouse code is expecting set and `queues` is actually a list.

Here is a fix.

* Change donations sidebar to direct users to OpenCollective.

* Added pytest to extras.

Missed in 9a6c292.

* Restore app.start() and app.worker_main() (celery#6481)

* Restore `app.start()` and `app.worker_main()`.

* Update celery/app/base.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix spelling error.

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* fix: `node_format()` logfile before detaching

Fixes celery#6426

* Multithreaded backend (celery#6416)

* Cache backend to thread local storage instead of global variable

* Cache oid to thread local storage instead of global variable

* Improve code returning thread_local data

* Move thread local storage to Celery class, introduced thread_oid and added unittests

* Remove python2 compatibility code

* Restore ability to extend the CLI with new sub-commands.

* Adjust documentation to demonstrate how to introduce sub-command plugins in 5.x.

Fixes celery#6439.

* autopep8 & isort.

* Linters now run using Python 3.9.

* Fix apply_async() in Calling Tasks userguide

* Fix dead links in contributing guide (celery#6506)

* Fix inconsistency in documentation for `link_error` (celery#6505)

* Make documentation of link_error consistent

Fixes celery#4099

* Fix undefined variable in example

* Add to contributors list

* Update testing.rst (celery#6507)

Use double back ticks for some code examples, so that quotes don't get converted into smart-quotes.

celery#6497

* Don't upgrade click to 8.x since click-repl doesn't support it yet.

Fixes celery#6511.
Upstream issue: click-contrib/click-repl#72

* Update documentation on changes to custom CLI options in 5.0.

Fixes celery#6380.

* update step to install homebrew

* redis: Support Sentinel with SSL

Use the SentinelManagedSSLConnection when SSL is enabled for the
transport. The redis-py project doesn't have a connection class for
SSL+Sentinel yet. So, create a class in redis.py to add that
functionality.

* Revert "redis: Support Sentinel with SSL" (celery#6518)

This reverts commit 18a0963.

* Reintroduce support for custom preload options (celery#6516)

* Restore preload options.

Fixes celery#6307.

* Document breaking changes for preload options in 5.0.

Fixes celery#6379.

* Changelog for 5.0.3.

* Bump version: 5.0.2 → 5.0.3

* Added integration tests for calling a task (celery#6523)

* DummyClient of cache+memory:// backend now shares state between threads (celery#6524)

* isort.

* Update changelog.

* Bump version: 5.0.3 → 5.0.4

* Change deprecated from collections import Mapping/MutableMapping to from collections.abc ... (celery#6532)

* fix celery#6047

* Fix type error in S3 backend (celery#6537)

* Convert key from bytes to str

* Add unit test for S3 delete of key with type bytes

* events.py: Remove duplicate decorator in wrong place (celery#6543)

`@handle_preload_options` was specified twice as a decorator of `events`, once at the top (wrong) and once at the bottom (right).
This fixes the `celery events` commands and also `celery --help`

* Update changelog.

* Bump version: 5.0.4 → 5.0.5

* ADD: indico additions - trails

* FIX: remove dev.txt dependencies

* ADD: handle revoke failures

* ADD: trailer_request support and better drain resolution

* ADD: merge options was overriding link_error values

* PATCH: DLX and reject behaviour

* FIX: amqp dependencies

Co-authored-by: Omer Katz <omer.drow@gmail.com>
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Martin Paulus <mpaulus@lequest.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: qiaocc <jasonqiao36@gmail.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Weiliang Li <to.be.impressive@gmail.com>
Co-authored-by: Akash Agrawal <akashrocksha@gmail.com>
Co-authored-by: Michal Kuffa <michal.kuffa@sentry.io>
Co-authored-by: Frazer McLean <frazer@frazermclean.co.uk>
Co-authored-by: Maarten Fonville <mfonville@users.noreply.github.com>
Co-authored-by: laixintao <laixintaoo@gmail.com>
Co-authored-by: Nicolas Dandrimont <nicolas@dandrimont.eu>
Co-authored-by: Bas ten Berge <bas+github@tenberge-ict.nl>
Co-authored-by: Zvi Baratz <z.baratz@gmail.com>
Co-authored-by: Justinas Petuchovas <justinas.petuchovas@gmail.com>
Co-authored-by: bastb <bas@tenberge-ict.nl>
Co-authored-by: Artem Bernatskyi <artem.bernatskyy@gmail.com>
Co-authored-by: ZubAnt <ya.zubarevanton@yandex.ru>
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>
Co-authored-by: David Pärsson <david@parsson.se>
Co-authored-by: Anthony Lukach <anthonylukach@gmail.com>
Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com>
Co-authored-by: Stepan Henek <stepan@henek.name>
Co-authored-by: KexZh <hanbaobao2005@gmail.com>
Co-authored-by: Thomas Riccardi <thomas@deepomatic.com>
Co-authored-by: Egor Sergeevich Poderiagin <egor@crazyrussian.pro>
Co-authored-by: Asif Saif Uddin (Auvi) <auvipy@gmail.com>
Co-authored-by: Lewis M. Kabui <13940255+lewisemm@users.noreply.github.com>
Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com>
Co-authored-by: Mathieu Rollet <matletix@gmail.com>
Co-authored-by: Mike DePalatis <depalatis@gmail.com>
Co-authored-by: partizan <serg.partizan@gmail.com>
Co-authored-by: Nick Pope <nick.pope@flightdataservices.com>
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
Co-authored-by: Matus Valo <matusvalo@gmail.com>
Co-authored-by: henribru <6639509+henribru@users.noreply.github.com>
Co-authored-by: Stuart Axon <stuaxo2@yahoo.com>
Co-authored-by: Sonya Chhabra <sonyakc.2007@gmail.com>
Co-authored-by: AbdealiJK <abdealikothari@gmail.com>
Co-authored-by: František Zatloukal <Zatloukal.Frantisek@gmail.com>
Co-authored-by: elonzh <elonzh@qq.com>
Co-authored-by: Sven Koitka <razorx89@users.noreply.github.com>
Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>
sihrc added a commit to IndicoDataSolutions/celery that referenced this pull request Feb 23, 2021
* Remove defaults for unsupported Python runtimes.

* Remove obsolete test.

* Doc pytest plugin (celery#6289)

* update to new pytest name

* doc pytest plugin

* trim heading to the length of the new pytest name

* add warning against use of sort key on dynamodb table, closes celery#6332

* Remove celery.five and bump vine dep (celery#6338)

* improv: Replace `five.values` with `dict.values`

* improv: Use `time.monotonic()` in kombu tests

Also in the docs where it is used to demonstrate `memcache` timeouts.

* rm: Delete `celery.five`

`vine.five` is no longer present in `vine >= 5`.

* triv: Remove refs to `celery.five` in docs, &c

* build: Bump `vine` dependency to 5.0+

* Wheels are no longer universal.

* Remove failing before_install step.

* Update changelog.

* Bump version: 5.0.0rc2 → 5.0.0rc3

* Fix release date.

* Remove unused import.

* Correctly skip these tests when the relevant dependency is missing.

* Expose retry_policy for Redis result backend

Rather than adding a new top-level config option, I have used a new key
in the already existing setting `result_backend_transport_options`.

Closes celery#6166

* Update changelog for 4.3.1.

* fix typo (celery#6346)

* Travis CI: Test Python 3.9 release candidate 1 (celery#6328)

* Travis CI: Test Python 3.9 release candidate 1

* fixup! Travis CI: matrix --> jobs

* fixup! Fix indentation error

* fixup! tox.ini: 3.9 --> 3.9-dev

* Fix test failure in Python 3.9RC1.

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Fix the broken celery upgrade settings command.

* Fix celery migrate settings options.

* Remove Riak result backend settings.

* Rephrase to mention that 3.5 is also EOL.

* Add a note about the removal of the Riak result backend.

* Fix examples of starting a worker in comments (celery#6331)

* Remove deprecated function from app.log.Logging.

* Migration guide.

* Document breaking changes for the CLI in the Whats New document.

* Add a "port code to Python 3" migration step.

* Update supported Python versions in the introduction document.

* Update bash completion.

* Add note about new shell completion.

* Update daemonization docs.

* Remove amqp backend. (celery#6360)

Fixes celery#6356.

* Warn when deprecated settings are used (celery#6353)

* Warn when deprecated settings are used.

* Mention deprecation in docs.

* Refer to the right place in the documentation.

* Complete What's New.

* Add wall of contributors.

* Update codename.

* Fix alt text.

* isort.

* PyPy 3.7 is currently in alpha.

No need for that sentence.

* Mention the new pytest-celery plugin.

* Mention retry policy for the redis result backend.

* Fix phrasing.

* Mention ordered group results are now the default.

* pyupgrade.

* Complete release notes.

* Bump version: 5.0.0rc3 → 5.0.0

* Happify linters.

* Specify utf-8 as the encoding for log files.

Fixes celery#5144.

* Fixed some typos in readme

* Fix custom headers propagation for protocol 1 hybrid messages

* Retry after race during schema creation in database backend (celery#6298)

* Retry after race during schema creation in database backend

Fixes celery#6296

This race condition does not commonly present, since the schema creation
only needs to happen once per database. It's more likely to appear in
e.g. a test suite that uses a new database each time.

For context of the sleep times I chose, the schema creation takes ~50 ms
on my laptop.

I did a simulated test run of 50 concurrent calls to MetaData.create_all
repeated 200 times and the number of retries was:

- 0 retries: 8717x
- 1 retry:   1279x
- 2 retries  4x

* Add test for prepare_models retry error condition

* Add name to contributors

* Update daemonizing.rst

Fix daemonizing documentation for issue celery#6363 to put `multi` before `-A`

* Revert "Update daemonizing.rst" (celery#6376)

This reverts commit 96ec6db.

* bugfix: when set config result_expires = 0, chord.get will hang. (celery#6373)

* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: celery#5237

* test: add testcase for expire when set config with zero.

* Display a custom error message whenever an attempt to use -A or --app as a sub-command option was made.

Fixes celery#6363

* Remove test dependencies for Python 2.7.

* Restore the celery worker --without-{gossip,mingle,heartbeat} flags (celery#6365)

In the previously used argparse arguments framework, these three options were
used as flags.

Since 5.0.0, they are options which need to take an argument (whose only
sensible value would be "true"). The error message coming up is also (very)
hard to understand, when running the celery worker command with an odd number
of flags:

  Error: Unable to parse extra configuration from command line.
  Reason: not enough values to unpack (expected 2, got 1)

When the celery worker is run with an even number of flags, the last one is
considered as an argument of the previous one, which is a subtle bug.

* Provide clearer error messages when app fails to load.

* fix pytest plugin registration documentation (celery#6387)

* fix pytest plugin registration documentation

* Update docs/userguide/testing.rst

Co-authored-by: Thomas Grainger <tagrain@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Contains a workaround for the capitalized configuration issue (celery#6385)

* Contains a workaround for the capitalized configuration issue

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Update celery/apps/worker.py

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove old explanation regarding `absolute_import` (celery#6390)

Resolves celery#6389.

* Update canvas.rst (celery#6392)

* Update canvas.rst

Tiny fixes.

* Update docs/userguide/canvas.rst

Co-authored-by: Omer Katz <omer.drow@gmail.com>

Co-authored-by: Omer Katz <omer.drow@gmail.com>

* Remove duplicate words from docs (celery#6398)

Remove the duplicate usage of “required” in documentation (specifically, `introduction.rst`).

* Allow lowercase log levels. (celery#6396)

Fixes celery#6395.

* Detach now correctly passes options with more than one word. (celery#6394)

When specifying options such as `-E` the detached worker should receive the `--task-events` option.
Instead it got the `--task_events` option which doesn't exist and therefore silently failed.

This fixes celery#6362.

* The celery multi command now works as expected. (celery#6388)

* Contains the missed change requested by @thedrow

* Added a some celery configuration examples.

* fixed loglevel info->INFO in docs

* return list instead set in CommaSeparatedList

_broadcast method of kombu Mailbox. does not support set
https://github.com/celery/kombu/blob/7b2578b19ba4b1989b722f6f6e7efee2a1a4d86a/kombu/pidbox.py#L319

* Rewrite detaching logic (celery#6401)

* Rewrite detaching logic.

* Ignore empty arguments.

* Ensure the SystemD services are up to date.

* fix: Pass back real result for single task chains

When chains are delayed, they are first frozen as part of preparation
which causes the sub-tasks to also be frozen. Afterward, the final (0th
since we reverse the tasks/result order when freezing) result object
from the freezing process would be passed back to the caller. This
caused problems in signaling completion of groups contained in chains
because the group relies on a promise which is fulfilled by a barrier
linked to each of its applied subtasks. By constructing two
`GroupResult` objects (one during freezing, one when the chain sub-tasks
are applied), this resulted in there being two promises; only one of
which would actually be fulfilled by the group subtasks.

This change ensures that in the special case where a chain has a single
task, we pass back the result object constructed when the task was
actually applied. When that single child is a group which does not get
unrolled (ie. contains more than one child itself), this ensures that we
pass back a `GroupResult` object which will actually be fulfilled. The
caller can then await the result confidently!

* fix: Retain `group_id` when tasks get re-frozen

When a group task which is part of a chain was to be delayed by
`trace_task()`, it would be reconstructed from the serialized request.
Normally, this sets the `group_id` of encapsulated tasks to the ID of
the group being instantiated. However, in the specific situation of a
group that is the last task in a chain which contributes to the
completion of a chord, it is essential that the group ID of the top-most
group is used instead. This top-most group ID is used by the redis
backend to track the completions of "final elements" of a chord in the
`on_chord_part_return()` implementation. By overwriting the group ID
which was already set in the `options` dictionaries of the child tasks
being deserialized, the chord accounting done by the redis backend would
be made inaccurate and chords would never complete.

This change alters how options are overridden for signatures to ensure
that if a `group_id` has already been set, it cannot be overridden.
Since group ID should be generally opaque to users, this should not be
disruptive.

* fix: Count chord "final elements" correctly

This change amends the implementation of `chord.__length_hint__()` to
ensure that all child task types are correctly counted. Specifically:

 * all sub-tasks of a group are counted recursively
 * the final task of a chain is counted recursively
 * the body of a chord is counted recursively
 * all other simple signatures count as a single "final element"

There is also a deserialisation step if a `dict` is seen while counting
the final elements in a chord, however this should become less important
with the merge of celery#6342 which ensures that tasks are recursively
deserialized by `.from_dict()`.

* test: Add more integration tests for groups

These tests are intended to show that group unrolling should be
respected in various ways by all backends. They should make it more
clear what behaviour we should be expecting from nested canvas
components and ensure that all the implementations (mostly relevant to
chords and `on_chord_part_return()` code) behave sensibly.

* test: Fix old markings for chord tests

* fix: Make KV-store backends respect chord size

This avoids an issue where the `on_chord_part_return()` implementation
would check the the length of the result of a chain ending in a nested
group. This would manifest in behaviour where a worker would be blocked
waiting for for the result object it holds to complete since it would
attempt to `.join()` the result object. In situations with plenty of
workers, this wouldn't really cause any noticable issue apart from some
latency or unpredictable failures - but in concurrency constrained
situations like the integrations tests, it causes deadlocks.

We know from previous commits in this series that chord completion is
more complex than just waiting for a direct child, so we correct the
`size` value in `BaseKeyValueStoreBackend.on_chord_part_return()` to
respect the `chord_size` value from the request, falling back to the
length of the `deps` if that value is missing for some reason (this is
necessary to keep a number of the tests happy but it's not clear to me
if that will ever be the case in real life situations).

* fix: Retain chord header result structure in Redis

This change fixes the chord result flattening issue which manifested
when using the Redis backend due to its deliberate throwing away of
information about the header result structure. Rather than assuming that
all results which contribute to the finalisation of a chord should be
siblings, this change checks if any are complex (ie. `GroupResult`s) and
falls back to behaviour similar to that implemented in the
`KeyValueStoreBackend` which restores the original `GroupResult` object
and `join()`s it.

We retain the original behaviour which is billed as an optimisation in
f09b041. We could behave better in the complex header result case by not
bothering to stash the results of contributing tasks under the `.j` zset
since we won't be using them, but without checking for the presence of
the complex group result on every `on_chord_part_return()` call, we
can't be sure that we won't need those stashed results later on. This
would be an opportunity for optimisation in future if we were to use an
`EVAL` to only do the `zadd()` if the group result key doesn't exist.
However, avoiding the result encoding work in `on_chord_part_return()`
would be more complicated. For now, it's not worth the brainpower.

This change also slightly refactors the redis backend unit tests to make
it easier to build fixtures and hit both the complex and simple result
structure cases.

* Update obsolete --loglevel argument values in docs

* Set logfile, not loglevel.

* Mention removed deprecated modules in the release notes.

Fixes celery#6406.

* Copy __annotations__ when creating tasks

This will allow getting type hints.

Fixes celery#6186.

* test: Improve chord body group index freezing test

Add more elements to the body so we can verify that the `group_index`
counts up from 0 as expected. This change adds the `pytest-subtests`
package as a test dependency so we can define partially independent
subtests within test functions.

* test: Use all() for subtask checks in canvas tests

When we expect all of the tasks in some iterable to meet a conditional,
we should make that clear by using `all(condition for ...)`.

* test: Add more tests for `from_dict()` variants

Notably, this exposed the bug tracked in celery#6341 where groups are not
deeply deserialized by `group.from_dict()`.

* fix: Ensure group tasks are deeply deserialised

Fixes celery#6341

* Fix `celery shell` command

* predefined_queues_urls -> predefined_queues

* Update changelog.

* Bump version: 5.0.0 → 5.0.1

* [Fix celery#6361] Fixing documentation for RabbitMQ task_queue_ha_policy

* Fix _autodiscover_tasks_from_fixups function

* fixup! Fix _autodiscover_tasks_from_fixups function

* Correct configuration item: CELERY_RESULT_EXPIRES

Related issue: celery#4050
celery#4050 (comment)

* Flush worker prints, notably the banner

In some cases (kubernetes, root) the banner is only printed at the
end of the process execution, instead of at the beginning.

* [Fix celery#6361] Remove RabbitMQ ha_policy from queue

* ci: Fix TOXENV for pypy3 unit tests

Fixes celery#6409

* ci: Move Python 3.9 test base from dev to release

* docs: fix celery beat settings

* move to travis-ci.com

* fix: Ensure default fairness maps to `SCHED_FAIR` (celery#6447)

Fixes celery#6386

* Preserve callbacks when replacing a task with a chain (celery#6189)

* Preserve callbacks when replacing a task with a chain.

* Preserve callbacks when replacing a task with a chain.

* Added tests.

* Update celery/app/task.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Mark test as flaky.

* Fix race condition in CI.

* fix: Run linked tasks in original slot for replace

This change alters the handling of linked tasks for chains which are
used as the argument to a `.replace()` call for a task which itself has
a chain of signatures to call once it completes. We ensure that the
linked callback is not only retained but also called at the appropiate
point in the newly reconstructed chain comprised of tasks from both the
replacement chain and the tail of the encapsulating chain of the task
being replaced.

We amend some tests to validate this behaviour better and ensure that
call/errbacks behave as expected if the encapsulating chain has either
set. One test is marked with an `xfail` since errbacks of encapsulating
chains are not currently called as expected due to some ambiguity in
when an errback of a replaced task should be dropped or not (celery#6441).

Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix minor documentation omission (celery#6453)

Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>

* Fix max_retries override on self.retry (celery#6436)

* Fix max_retries override

* Fix max_retries override

* Fix max_retries override

* Update exceptions.py

typo

* Update autoretry.py

typo

* Update task.py

Prevent exception unpacking for tasks without autoretry_for

* Update test_tasks.py

Unit test

* Update test_tasks.py

Added a new test

* Update autoretry.py

Fox for explicit raise in tasks

* Update test_tasks.py

* Update autoretry.py

* Update task.py

* Update exceptions.py

* Update task.py

* Happify linter.

* Raise proper error when replacing with an empty chain. (celery#6452)

Fixes celery#6451.

* Update changelog.

* Bump version: 5.0.1 → 5.0.2

* Update daemonizing.rst

Improved systemd documentation for auto-start of the service, and mention the possibility to depend on RabbitMQ service. Also add Restart=always for Celery Beat example

* Update celerybeat.service

* Fix old celery beat variables

Change made 5 days ago in 7c3da03 is faulty, the correct celery beat variables do start with `CELERYBEAT` and not `CELERY_BEAT`

* Fix formatting.

* Fix formatting.

* fix: Make `--workdir` eager for early handling (celery#6457)

This change makes the `--workdir` options an eager one which `click`
will process early for us, before any of the others. At the same time,
we add a callback which ensures that the `chdir()` is run during
handling of the argument so that all subsequent actions (e.g. app
loading) occur in the specified working directory.

Fixes celery#6445

* Fix example.

Fixes celery#6459.

* When using the MongoDB backend, don't cleanup if result_expires is 0 or None. (celery#6462)

Fixes celery#6450.

* Add missing space (celery#6468)

* Fix passing queues into purge command (celery#6469)

In current wersion calling `celery --app my.celery_app purge -Q queue_name` is failing with following trace:

```
    names = (queues or set(app.amqp.queues.keys())) - exclude_queues
TypeError: unsupported operand type(s) for -: 'list' and 'list'
```

Becouse code is expecting set and `queues` is actually a list.

Here is a fix.

* Change donations sidebar to direct users to OpenCollective.

* Added pytest to extras.

Missed in 9a6c292.

* Restore app.start() and app.worker_main() (celery#6481)

* Restore `app.start()` and `app.worker_main()`.

* Update celery/app/base.py

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* Fix spelling error.

Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>

* fix: `node_format()` logfile before detaching

Fixes celery#6426

* Multithreaded backend (celery#6416)

* Cache backend to thread local storage instead of global variable

* Cache oid to thread local storage instead of global variable

* Improve code returning thread_local data

* Move thread local storage to Celery class, introduced thread_oid and added unittests

* Remove python2 compatibility code

* Restore ability to extend the CLI with new sub-commands.

* Adjust documentation to demonstrate how to introduce sub-command plugins in 5.x.

Fixes celery#6439.

* autopep8 & isort.

* Linters now run using Python 3.9.

* Fix apply_async() in Calling Tasks userguide

* Fix dead links in contributing guide (celery#6506)

* Fix inconsistency in documentation for `link_error` (celery#6505)

* Make documentation of link_error consistent

Fixes celery#4099

* Fix undefined variable in example

* Add to contributors list

* Update testing.rst (celery#6507)

Use double back ticks for some code examples, so that quotes don't get converted into smart-quotes.

celery#6497

* Don't upgrade click to 8.x since click-repl doesn't support it yet.

Fixes celery#6511.
Upstream issue: click-contrib/click-repl#72

* Update documentation on changes to custom CLI options in 5.0.

Fixes celery#6380.

* update step to install homebrew

* redis: Support Sentinel with SSL

Use the SentinelManagedSSLConnection when SSL is enabled for the
transport. The redis-py project doesn't have a connection class for
SSL+Sentinel yet. So, create a class in redis.py to add that
functionality.

* Revert "redis: Support Sentinel with SSL" (celery#6518)

This reverts commit 18a0963.

* Reintroduce support for custom preload options (celery#6516)

* Restore preload options.

Fixes celery#6307.

* Document breaking changes for preload options in 5.0.

Fixes celery#6379.

* Changelog for 5.0.3.

* Bump version: 5.0.2 → 5.0.3

* Added integration tests for calling a task (celery#6523)

* DummyClient of cache+memory:// backend now shares state between threads (celery#6524)

* isort.

* Update changelog.

* Bump version: 5.0.3 → 5.0.4

* Change deprecated from collections import Mapping/MutableMapping to from collections.abc ... (celery#6532)

* fix celery#6047

* Fix type error in S3 backend (celery#6537)

* Convert key from bytes to str

* Add unit test for S3 delete of key with type bytes

* events.py: Remove duplicate decorator in wrong place (celery#6543)

`@handle_preload_options` was specified twice as a decorator of `events`, once at the top (wrong) and once at the bottom (right).
This fixes the `celery events` commands and also `celery --help`

* Update changelog.

* Bump version: 5.0.4 → 5.0.5

* ADD: indico additions - trails

* FIX: remove dev.txt dependencies

* ADD: handle revoke failures

* ADD: trailer_request support and better drain resolution

* ADD: merge options was overriding link_error values

* PATCH: DLX and reject behaviour

* FIX: amqp dependencies

Co-authored-by: Omer Katz <omer.drow@gmail.com>
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Martin Paulus <mpaulus@lequest.com>
Co-authored-by: maybe-sybr <58414429+maybe-sybr@users.noreply.github.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: qiaocc <jasonqiao36@gmail.com>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Weiliang Li <to.be.impressive@gmail.com>
Co-authored-by: Akash Agrawal <akashrocksha@gmail.com>
Co-authored-by: Michal Kuffa <michal.kuffa@sentry.io>
Co-authored-by: Frazer McLean <frazer@frazermclean.co.uk>
Co-authored-by: Maarten Fonville <mfonville@users.noreply.github.com>
Co-authored-by: laixintao <laixintaoo@gmail.com>
Co-authored-by: Nicolas Dandrimont <nicolas@dandrimont.eu>
Co-authored-by: Bas ten Berge <bas+github@tenberge-ict.nl>
Co-authored-by: Zvi Baratz <z.baratz@gmail.com>
Co-authored-by: Justinas Petuchovas <justinas.petuchovas@gmail.com>
Co-authored-by: bastb <bas@tenberge-ict.nl>
Co-authored-by: Artem Bernatskyi <artem.bernatskyy@gmail.com>
Co-authored-by: ZubAnt <ya.zubarevanton@yandex.ru>
Co-authored-by: Lewis Kabui <lewisemm@users.noreply.github.com>
Co-authored-by: David Pärsson <david@parsson.se>
Co-authored-by: Anthony Lukach <anthonylukach@gmail.com>
Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com>
Co-authored-by: Stepan Henek <stepan@henek.name>
Co-authored-by: KexZh <hanbaobao2005@gmail.com>
Co-authored-by: Thomas Riccardi <thomas@deepomatic.com>
Co-authored-by: Egor Sergeevich Poderiagin <egor@crazyrussian.pro>
Co-authored-by: Asif Saif Uddin (Auvi) <auvipy@gmail.com>
Co-authored-by: Lewis M. Kabui <13940255+lewisemm@users.noreply.github.com>
Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com>
Co-authored-by: Mathieu Rollet <matletix@gmail.com>
Co-authored-by: Mike DePalatis <depalatis@gmail.com>
Co-authored-by: partizan <serg.partizan@gmail.com>
Co-authored-by: Nick Pope <nick.pope@flightdataservices.com>
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
Co-authored-by: Matus Valo <matusvalo@gmail.com>
Co-authored-by: henribru <6639509+henribru@users.noreply.github.com>
Co-authored-by: Stuart Axon <stuaxo2@yahoo.com>
Co-authored-by: Sonya Chhabra <sonyakc.2007@gmail.com>
Co-authored-by: AbdealiJK <abdealikothari@gmail.com>
Co-authored-by: František Zatloukal <Zatloukal.Frantisek@gmail.com>
Co-authored-by: elonzh <elonzh@qq.com>
Co-authored-by: Sven Koitka <razorx89@users.noreply.github.com>
Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>
thedrow pushed a commit that referenced this pull request Jun 1, 2021
* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: #5237

* test: add testcase for expire when set config with zero.
thedrow pushed a commit that referenced this pull request Jun 1, 2021
* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: #5237

* test: add testcase for expire when set config with zero.
jeyrce pushed a commit to jeyrce/celery that referenced this pull request Aug 25, 2021
…ery#6373)

* bugfix: when set config result_expires = 0, chord.get will hang.

`EXPIRE key 0` will delete a key in redis, then chord will never get the
result.

fix: celery#5237

* test: add testcase for expire when set config with zero.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chord callback method can not execute when result_expires =0 or None
3 participants