Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultMessageListenerContainer does not log an error/warning when consumer tasks have been rejected #27451

Closed
grubeninspekteur opened this issue Sep 22, 2021 · 1 comment
Assignees
Labels
in: messaging Issues in messaging modules (jms, messaging) status: backported An issue that has been backported to maintenance branches type: enhancement A general enhancement
Milestone

Comments

@grubeninspekteur
Copy link

grubeninspekteur commented Sep 22, 2021

Affects: 5.1.2, 5.3.9

Background: We experienced an issue due to DefaultMessageListenerContainer no longer receiving messages. This was because of a misunderstanding on our part how ThreadPoolExecutorFactoryBean works. We used following settings for the executor:

corePoolSize=5
maximumPoolSize=10
queueCapacity=1

which was injected into a DefaultMessageListenerContainer with these parameters:

concurrentConsumers=1
maxConcurrentConsumers=10
maxMessagesPerTask=1

The goal was to have a dynamic scaling of threads aligned with the scaling of consumers; however we shouldn't have set neither a queue capacity nor a maximum pool size and rather used something like Executors.newCachedThreadPool(); after all, the container will take care of not scheduling more than maxConcurrentConsumers * 2 threads in the worst case (since each task reschedules itself if all are busy).

Our problem was reproducible given this test, the log is available here (I killed the process after no more messages were being received).

The issue: There are some rejected task debug level log entries due to the executor misconfiguration, but no warn or error entries. I would expect at least the error "All scheduled consumers have been paused, probably due to tasks having been rejected. Check your thread pool configuration! Manual recovery necessary through a start() call." at DefaultMessageListenerContainer.java#L1180-L1183 to appear. I don't understand why it didn't, since everything happens inside locks held on the lifecycleMonitor.

On a side note, why is the rejection of tasks logged only at debug level (not even info)? After all, dropping below concurrentConsumers raises a warning (this works in the test if I set concurrentConsumers to 2). Is there any use case of configuring an executor that occasionally rejects tasks?

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Sep 22, 2021
@jhoeller jhoeller self-assigned this Sep 22, 2021
@jhoeller jhoeller added in: messaging Issues in messaging modules (jms, messaging) type: enhancement A general enhancement and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Sep 22, 2021
@jhoeller jhoeller added this to the 5.3.11 milestone Sep 22, 2021
@jhoeller
Copy link
Contributor

Not sure why the error did not get logged, but in any case, each individual rejection should get logged at warn level at least. While executor implementations may temporarily reject tasks for any reason, it really isn't the norm, in particular not with the common executor implementations out there. I'll therefore backport the log level change to 5.2.18 as well.

@jhoeller jhoeller added the for: backport-to-5.2.x Marks an issue as a candidate for backport to 5.2.x label Sep 23, 2021
@spring-projects-issues spring-projects-issues added status: backported An issue that has been backported to maintenance branches and removed for: backport-to-5.2.x Marks an issue as a candidate for backport to 5.2.x labels Sep 23, 2021
@jhoeller jhoeller changed the title DefaultMessageListenerContainer does not log an error when all consumer tasks have been rejected DefaultMessageListenerContainer does not log an error/warning when consumer tasks have been rejected Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: messaging Issues in messaging modules (jms, messaging) status: backported An issue that has been backported to maintenance branches type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

3 participants