FireFly gets stuck processing events from an unknown listener

[awrichar]

For each blockchain event batch, FireFly attempts to re-query the blockchain connector to get the listener name from the listener ID. If it happens that the listener has actually been deleted by the time FireFly performs this request, then FireFly refuses to process the batch. However, this causes the connector to redeliver the batch indefinitely, causing FireFly to get stuck in a loop where it cannot process the events.

[2024-02-01T22:42:03.789Z] ERROR Rejecting batch due error: FF10111: Error from ethereum connector: {"error":"FF21046: Event listener '018d65ec-322c-2286-ee27-a92f8ad51122' not found"} pid=1 proto=ethereum role=event-loop

[awrichar]

Possible solutions:

• have blockchain connectors proactively add the listener name, to avoid this query
• gracefully handle this specific error in FireFly and drop the batch
• make blockchain connector realize the listener is gone and stop delivering the event