[HA] Can't update or insert instance

[deric]

Describe the bug

icingadb ends up in a dead-lock state. None instance can't update or write. After icingadb service restart seems to be ok.

active icingadb instance:

Jan 24 07:13:01 master02: runtime-updates: Upserted 84 ServiceState items
Jan 24 07:13:01 master02: overdue-sync: Synced 50 service overdue indicators
Jan 24 07:14:13 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:15 master02: history-sync: Synced 111 state history items
Jan 24 07:14:15 master02: history-sync: Synced 19 notification history items
Jan 24 07:14:15 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:17 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:19 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:21 master02: runtime-updates: Upserted 6 ServiceState items
Jan 24 07:14:22 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:24 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:32 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:33 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:38 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:14:40 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:12 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:12 master02: Handing over
Jan 24 07:15:13 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:15 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:17 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:19 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:21 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:24 master02: high-availability: Can't update or insert instance. Retrying
Jan 24 07:15:26 master02: high-availability: Another instance is active
Jan 24 07:15:26 master02: high-availability: Another instance is active

second instance:

Jan 24 07:15:26 master01: Taking over
Jan 24 07:15:26 master01: Starting config sync
Jan 24 07:15:26 master01: Starting overdue sync
Jan 24 07:15:26 master01: Starting initial state sync
Jan 24 07:15:27 master01: config-sync: Updating 599 items of type host state
Jan 24 07:15:32 master01: config-sync: Updating 172572 items of type service state
Jan 24 07:15:35 master01: Starting config runtime updates sync
Jan 24 07:15:35 master01: config-sync: Finished config sync in 9.218197441s
Jan 24 07:15:36 master01: history-sync: Synced 829 state history items
Jan 24 07:15:36 master01: history-sync: Synced 303 notification history items
Jan 24 07:15:46 master01: overdue-sync: Synced 31 service overdue indicators
Jan 24 07:17:02 master01: high-availability: Can't update or insert instance. Retrying
Jan 24 07:17:04 master01: high-availability: Can't update or insert instance. Retrying
Jan 24 07:17:06 master01: high-availability: Can't update or insert instance. Retrying
Jan 24 07:17:08 master01: high-availability: Can't update or insert instance. Retrying

To Reproduce

Not sure. All components (redis, postgresql backend) seems to be active.

Expected behavior

icingadb should be able to write to a redis instance (when redis is available).

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Icinga DB version: v1.1.1
• Icinga 2 version: r2.14.2-1
• Operating System and version: Debian 12

[yhabteab]

Hii @deric, thanks for reporting! Can you reproduce the problem easily? If so, can you please provide us the original logs? In your logs above, some relevant information has been cut off, but I'm not sure if this is a copy-paste error or if it's systemd's fault.

Jan 24 07:17:02 master01: high-availability: Can't update or insert instance. Retrying

Right after Retrying in the same line there should be further information, such as the actual database error etc.