limit amount of messages received by a consumer

[aaleksandrov]

Hi!
We're using this library with https://github.com/cometd/cometd-nodejs-client for fetching server-to-server from Salesforce. And when Salesforce publishes many messages (5k+) our consumer can't process it, runs out of memory etc.
Is it possible to limit amount of messages received by consumer? I cannot find anything regarding limits in cometd

[sbordet]

CometD is notifying message listeners synchronously: it reads from the network, it calls the listeners and does not read from the network again until the listeners return.

This naturally applies backpressure so that the client does not explode.

If the server sends down a single enormous message, or a single array of messages of an enormous length, then it's the server assuming too much of what the client can do, and I would see that as a server-side application mistake (it should split the enormous message, or split the array into smaller arrays).

I'm not that familiar with SalesForce message stream, but you should look into whether SalesForce allows you to configure how messages are sent from the server. For example, you may subscribe to less channels, or ask to be notified less frequently, etc.

[sbordet]

FTR, in the case of many small messages, it may be possible to write an extension where the client declares its demand, similar to reactive streams.

This will require collaboration of the server, and delivery via /meta/connect only.

The client will send demand: 10 the server will have an extension similar to the ack extension where the messages are kept around and only 10 are delivered via /meta/connect. The client receives the 10, and sends a new /meta/connect with demand: 15, etc.

This moves the queuing problem to the server, where it's already handled by queue listeners, although it needs to be carefully coded - if the client is really slow, what can the server do? disconnecting the client will just make the client connect again, making it even slower. But again this is logic that only applications can implement, not CometD.

Needs some experimentation to see if there are unforeseen issues, but looks promising.

[surajitgit]

Hi.. I'm using CometD-Java Client 4.0.9 version. I'm stuck with something..
We have microservice which subscribes and continuously listens to Salesforce cometD channel, once a message is available in the channel, listener microservice immediately consumes it but the issue is, if we scale up listener MS instance more than one, both the instances would consume copy of the same message hence duplicate records are being produced wrongly.
Is there anything we could perform which would restrict other instance to consume the same message i.e. already consumed by other instance?

[sbordet]

You have 2 subscribers to the same channel, so for the server these are 2 different subscribers and it will send the same message to both.

It's only on the client that you know that the 2 subscribers are in fact the same subscriber.

There are plenty solutions for this architectural problem, from shared in-memory queues, to shared RDBMS, to shared storages such as Redis and such, etc. but it's in general outside the scope of CometD -- there are too many variations due to the application behavior (may or not tolerate duplicates), the architecture, etc. that CometD cannot cover them all.