Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Messenger worker going to "idle" state without exit code #54698

Open
mitalcoi opened this issue Apr 22, 2024 · 1 comment
Open

Messenger worker going to "idle" state without exit code #54698

mitalcoi opened this issue Apr 22, 2024 · 1 comment

Comments

@mitalcoi
Copy link

mitalcoi commented Apr 22, 2024

Symfony version(s) affected

7.0

Description

We are using symfony messenger directly through kubernates command, without supervisor, example of configuration:

command:
  enabled: true
  cli: '["/bin/sh"]'
  args: '["-c", "php bin/console messenger:consume default --time-limit=3600 --memory-limit=256M"]'

We expecting that after reaching 3600 seconds since starting messenger will finish work with exit code "0" and pod will gracefully reload out worker - and in most cases it happens as it should, but sometimes worker freezes out without any exit code
The latest what we see in STDOUT is:
{"message":"Worker stopped due to time limit of 3600s exceeded","context":{},"level":200,"level_name":"INFO","channel":"messenger","datetime":"2024-04-10T08:39:37.697238+00:00","extra":{}}
after that time of latest logs will stay as it forever. Meanwhile kubernates still treating pod with this idle state as normally working.

How to reproduce

Issue normally happened rare, that's why it is hard to provide exact steps to reproduce issue. To reproduce need to run pod with provided configuration and wait when worker will be go to this "idle" state. It can take few hours or even days, issue have sporadical nature

Possible Solution

We have two possible strategy to resolve that issue:

  1. Use messenger with supervisord
  2. Handle WorkerStoppedEvent and through exception on time limit to force pod recreating

Both solutions not ideal

Additional Context

Is it really important to manage workers with supervisor, or it is just recommendation? What motivation behind this recommendation from official symfony docs?

@javaDeveloperKid
Copy link
Contributor

Is it really important to manage workers with supervisor, or it is just recommendation? What motivation behind this recommendation from official symfony docs?

It's just an example. You can manage your worker however you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants