Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Used Queue for STOMP listener handover; Improved connection loss recovery #1474

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Used Queue for STOMP listener handover; Improved connection loss recovery #1474

wants to merge 1 commit into from
+607 −180

Conversation

andy-maier
Copy link
Member

@andy-maier andy-maier commented May 10, 2024 •
edited

For details, see the commit message.

WIP TODOs:

  • Change notifications() to return (without exception) when receiver.close() is called.
  • Test with zhmc-log-forwarder

@andy-maier andy-maier self-assigned this May 10, 2024
@andy-maier andy-maier added this to the 1.15.0 milestone May 10, 2024
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch 13 times, most recently from 096f873 to f43985a Compare May 17, 2024 12:46
@andy-maier andy-maier changed the title [WIP] Added stomp retry/timeout config; stomp >7; Documented stomp exceptions Stomp retry/timeout config; stomp >7; Improved recovery May 17, 2024
@andy-maier andy-maier modified the milestones: 1.15.0, 1.16.0 Jun 7, 2024
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch 3 times, most recently from b8c7d0b to b1def25 Compare June 8, 2024 05:04
@andy-maier andy-maier changed the title Stomp retry/timeout config; stomp >7; Improved recovery Stomp retry/timeout config; Improved recovery Jun 8, 2024
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch 7 times, most recently from 15dbce4 to 5ba3f69 Compare June 8, 2024 06:51
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch 3 times, most recently from a0e1172 to 64da72f Compare June 9, 2024 06:37
@andy-maier andy-maier changed the title Stomp retry/timeout config; Improved recovery Used Queue for STOMP listener handover; Improved connection loss recovery Jun 9, 2024
@andy-maier andy-maier linked an issue Jun 9, 2024 that may be closed by this pull request
Use Queue in handover of notification listener #1502
Open
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch 8 times, most recently from 93fa392 to 3ca4f1c Compare June 10, 2024 11:46
@andy-maier
Used Queue for STOMP listener handover; Improved connection loss reco…
0e48802 
…very

Details:

* Replaced the event-based handover of a single item from the notification
  listener thread to the caller's thread with a Python Queue, because that
  is more reliable.

* Added new methods 'connect()' and 'is_connected()' to the
  'NotificationReceiver' class.

* Added exceptions 'NotificationConnectionError' and
  'NotificationSubscriptionError'.

* The 'NotificationReceiver.notifications()' method now continues running
  when there are no notifications, and only ever returns when exceptions are
  raised. The method can be brought to raise 'NotificationConnectionError'
  through 'NotificationReceiver.close()'

* Added proper detection of STOMP connection loss if STOMP heartbeating is
  enabled. The connection loss is surfaced by raising
  'NotificationConnectionError' in 'NotificationReceiver.notifications()'.
  This allows users to retry 'NotificationReceiver.notifications()' upon
  connection loss.

* Added the missing event methods to the _NotificationListener class that
  are defined in stomp.ConnectionListener, in case they are ever invoked.
  The _NotificationListener class continues not to inherit from
  stomp.ConnectionListener in order to be continue to be able to lazy-import
  stomp.

* In the connected() event method in the _NotificationListener class, added
  a log message for the heartbeat parameters returned by the HMC.

* Changed the detection for connection loss to be in the notifications()
  method, and no longer in the disconnected() event method of the listener,
  since it turned out that that event method is called not on connection
  loss, but after the connection is re-established. This allowed for the
  first time to properly recover from connection loss.

* Added a new public constant 'STOMP_MIN_CONNECTION_CHECK_TIME' that defines
  the minimum time between checks for STOMP connection loss. The check time
  increases from this minimum with as the heartbeat receive time gets larger.

* Added more log messages around connection / disconnect.

* Added an internal debug flag DEBUG_HEARTBEATS to produce log messages for
  each heartbeat sent or received.

* Documented the stomp exceptions that can be raised from public zhmcclient
  methods.

Signed-off-by: Andreas Maier <maiera@de.ibm.com>
@andy-maier andy-maier force-pushed the andy/improve-stomp-exceptions branch from 3ca4f1c to 0e48802 Compare June 10, 2024 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@andy-maier andy-maier

Labels
under work
Projects
None yet
Milestone
1.16.0
Development

Successfully merging this pull request may close these issues.

Use Queue in handover of notification listener
1 participant
@andy-maier