PIP 37: [pulsar-client] support large message size #4400

rdhabalia · 2019-05-29T01:40:42Z

Motivation

Addressing PIP-37 to support large size of messages in pulsar.

Modification

This PR supports large message size for the non-shared subscription and it has only client side changes.
I will create a separate PR to support large size message for shared-subscription.

jiazhai · 2019-05-30T01:15:23Z

@rdhabalia There are some PRs on going for PIP-36, will it partly solve this issue?

rdhabalia · 2019-05-30T01:40:22Z

@jiazhai

There are some PRs on going for PIP-36, will it partly solve this issue?

No, right now, pulsar can only support up to 5MB message size and PIP-36 doesn't support large message but it allows broker to configure max message size from 0 to 5MB. we need it for customers who want to publish large message into large-scale specialized pulsar instance for which txn might not be feasible option. it's similar like chunking in kafka mentioned here.

jiazhai · 2019-06-19T03:07:05Z

Hi @rdhabalia, sorry for the delay. After PIP36 (#4247 ), pulsar could support message size large than 5MB.

rdhabalia · 2019-10-23T21:31:57Z

@jiazhai sorry for late response as I had missed your comment.

After PIP36 (#4247 ), pulsar could support message size large than 5MB.

That's incorrect. #4247 is not a useful feature and that could be used for a specific usecase but we can not use for large scale and multi tenant system where same bookie is serving topics from different tenants.
For example, If client wants to publish message with 100MB and if broker allows to store one message with 100MB into bookie then bookie will max out in 1 or 2 messages and it can directly impact to other tenants for publish and draining backlog. Also, it will be hard to force throttling for one message with 100MB payload size. I have also described this scenario into the PIP.
Right now, we have multiple users who are chunking the payload at producer side and merging at consumer side and they need this feature to avoid such complexity.
So, we need this feature available for multiple users.

rdhabalia · 2019-10-24T18:15:43Z

I think rebasing this PR takes lot of time as txn and other changes have changed a lot after this PR. @merlimat can you please review it as multiple users need this feature and #4247 and txn is not useful for multiple reasons.

rdhabalia · 2020-01-29T20:41:52Z

we have many users waiting for this feature. Can you please review it. @merlimat @sijie @jiazhai @nkurihar @jai1

sijie · 2020-01-31T06:00:16Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java

+     * recommended configuration at pulsar producer and consumer. Recommendation to use this feature:
+     * 
+     * <pre>
+     * 1. This feature is right now only supported by non-shared subscription and persistent-topic.


This makes me feel that we are not on the right track for implementing this feature. If a user is required to do these many things in order to enable this feature, it seems that we are not doing something correctly. Also, this seems to make the producer and consumer logic very complicated.

Can this chunking feature be implemented in a high level? E.g. over a producer wrapper or a consumer wrapper? E.g. you can chunk a large message into smaller messages and add the chunk id in the message properties.

It makes harder to keep this responsibility in wrapper because it also requires to stitch messages by handling all failure scenarios. So, it's more logical to keep this responsibility with in client which makes it more simpler to handle them. Other system also does similar thing at client side.
Also, many customers do such chunking and stitching at application level for their pipeline to transport large messages without impacting cluster and other tenants. But they also have to take care message-sizing, chunking, dedup and failure scenario which make harder to solve this problem. Therefore, it makes user's life easier if this feature is part of the client lib.
Right now, all the users need this feature for the streaming usecase which needs exclusive/failure subscription to consume data and push it to grid or perform Online analytical processing.. so, we have introduced this feature for one type of subscription for now and will see in future if we need it for other type as well.

If a user is required to do these many things in order to enable this feature,

User can use this feature with default values as well but here, we have added documentation to utilize this feature with better performance by tuning configuration. So, there shouldn't be complexity to support this feature and it should be straight forward to use it. In fact, this feature is more useful when pulsar is deployed on large scale system and user can deal with large size of messages without impacting other tenants.

rdhabalia · 2020-03-09T23:31:04Z

we have many customers need this feature so, can someone please review the PR and can we merge the PR soon.

sijie · 2020-03-09T23:48:30Z

@jiazhai @codelipenghui can you help review @rdhabalia 's change?

jiazhai · 2020-04-08T01:13:34Z

use for large scale and multi tenant system where same bookie is serving topics from different tenants
@rdhabalia Agree with you on the use case. +1 overall lgtm.

codelipenghui · 2020-05-25T14:53:18Z

@rdhabalia Would you please resolve the conflicts? So that we can onboard this PR in 2.6.0 release.

codelipenghui · 2020-05-29T04:41:28Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerBuilderImpl.java

@@ -96,6 +96,9 @@ private ProducerBuilderImpl(PulsarClientImpl client, ProducerConfigurationData c

    @Override
    public CompletableFuture<Producer<T>> createAsync() {
+        // config validation
+        checkArgument(!(conf.isBatchingEnabled() && conf.isChunkingEnabled()),


@rdhabalia The change looks good to me. About to enable the chunk, is it possible to add a config for the max chunk size and enable batching at the same time. If the message size is greater than the max chunk size and the chunk is enabled, we send messages with chunks, If the message is size is smaller than the max chunk size, we also can add them to a batch and if chunk is enabled, we make sure the match size can not exceed the max chunk size. So that it can be better applied when the user cannot confirm the message size for example from 100 bytes to 10MB. What do you think?

@codelipenghui we can do but it will be little tricky to cover all usecases as we use batch-container if batching is enabled and combination of non-batch and batching will be little tricky. I have already put good amount of efforts to rebase and doing perf testing with changes. so, let's merge this PR and I will create a issue to add this feature and address it in separate PR as it will also require some effort to cover all unknowns.
So, can you please review this PR again as I am done with rebasing.

rdhabalia · 2020-06-02T01:06:44Z

I just rebased all the files and I can again see conflicts.. :-) please review so, I will rebase it again and merge it soon.

fix producer fix ref counts add timeouts add validation fix recycling fix stats review fix test fix test fix send message and expiry-consumer-config fix schema test fix chunk properties

* PIP 37: [pulsar-client] support large message size fix producer fix ref counts add timeouts add validation fix recycling fix stats review fix test fix test fix send message and expiry-consumer-config fix schema test fix chunk properties * fix test

### Motivation The PR #4400 introduced new params on the pulsar-client consume command but it broked some old ones. ### Modifications This PR fixes this problem to restore the good param behavior.

* PIP 37: [pulsar-client] support large message size fix producer fix ref counts add timeouts add validation fix recycling fix stats review fix test fix test fix send message and expiry-consumer-config fix schema test fix chunk properties * fix test

### Motivation The PR apache#4400 introduced new params on the pulsar-client consume command but it broked some old ones. ### Modifications This PR fixes this problem to restore the good param behavior.

### Motivation The PR #4400 introduced new params on the pulsar-client consume command but it broked some old ones. ### Modifications This PR fixes this problem to restore the good param behavior. (cherry picked from commit a4a12d1)

rdhabalia added the area/client label May 29, 2019

rdhabalia added this to the 2.4.0 milestone May 29, 2019

rdhabalia requested review from merlimat, sijie, jiazhai, jai1, codelipenghui and massakam May 29, 2019 01:40

rdhabalia self-assigned this May 29, 2019

merlimat modified the milestones: 2.4.0, 2.5.0 May 29, 2019

rdhabalia force-pushed the msg_chunk branch from 29fb4b9 to 1b0873d Compare May 29, 2019 17:37

sijie modified the milestones: 2.5.0, 2.6.0 Nov 25, 2019

sijie reviewed Jan 31, 2020

View reviewed changes

sijie mentioned this pull request May 20, 2020

[discussion] Pulsar release 2.6.0 #5819

Closed

rdhabalia force-pushed the msg_chunk branch 4 times, most recently from c52ab25 to 595467a Compare May 28, 2020 07:03

rdhabalia changed the title ~~PIP 37: [pulsar-client] support large message size~~ [WIP-Rebase] PIP 37: [pulsar-client] support large message size May 28, 2020

codelipenghui reviewed May 29, 2020

View reviewed changes

rdhabalia force-pushed the msg_chunk branch from 3e2d491 to 2daf5ce Compare May 31, 2020 23:25

rdhabalia changed the title ~~[WIP-Rebase] PIP 37: [pulsar-client] support large message size~~ PIP 37: [pulsar-client] support large message size May 31, 2020

rdhabalia force-pushed the msg_chunk branch from 9e971e4 to 4ebc700 Compare June 2, 2020 01:09

codelipenghui approved these changes Jun 2, 2020

View reviewed changes

rdhabalia force-pushed the msg_chunk branch from 4ebc700 to 486f316 Compare June 2, 2020 02:55

fix test

d8e4d49

rdhabalia merged commit 870a637 into apache:master Jun 2, 2020

rdhabalia mentioned this pull request Jun 2, 2020

PIP 37-Follow up: Chunk message with batching enabled #7138

Open

sijie mentioned this pull request Jun 2, 2020

ISSUE-7138: PIP 37-Follow up: Chunk message with batching enabled streamnative/pulsar-archived#1045

Open

Mefl added a commit to Mefl/pulsar that referenced this pull request Aug 11, 2020

Fix the params broken by PR apache#4400

f5732e8

Mefl mentioned this pull request Aug 11, 2020

[Pulsar-client-cli] fix some params on consumer broken by #4400 (regex, initialSouscriptionPosition) #7795

Merged

1 task

flowchartsman mentioned this pull request Jan 28, 2021

Support Large Message Size (PIP-37 / "chunking") apache/pulsar-client-go#456

Closed

sijie mentioned this pull request Jan 28, 2021

ISSUE-456: Support Large Message Size (PIP-37 / "chunking") streamnative/pulsar-client-go#147

Open

michaeljmarshall mentioned this pull request Mar 26, 2021

[Question] chuckedMessageRate vs chunkedMessageRate - Is chucked a typo? #10046

Closed

sijie mentioned this pull request Mar 26, 2021

ISSUE-10046: [Question] chuckedMessageRate vs chunkedMessageRate - Is chucked a typo? streamnative/pulsar-archived#2332

Closed

BewareMyPower mentioned this pull request Jan 5, 2022

[C++] PIP 37: Support large message size #13627

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PIP 37: [pulsar-client] support large message size #4400

PIP 37: [pulsar-client] support large message size #4400

rdhabalia commented May 29, 2019

Uh oh!

jiazhai commented May 30, 2019

Uh oh!

rdhabalia commented May 30, 2019

Uh oh!

jiazhai commented Jun 19, 2019

Uh oh!

rdhabalia commented Oct 23, 2019

Uh oh!

rdhabalia commented Oct 24, 2019

Uh oh!

rdhabalia commented Jan 29, 2020

Uh oh!

sijie Jan 31, 2020

Uh oh!

rdhabalia Feb 1, 2020

Uh oh!

rdhabalia commented Mar 9, 2020

Uh oh!

sijie commented Mar 9, 2020

Uh oh!

jiazhai commented Apr 8, 2020

Uh oh!

codelipenghui commented May 25, 2020

Uh oh!

codelipenghui May 29, 2020

Uh oh!

rdhabalia Jun 1, 2020

Uh oh!

rdhabalia commented Jun 2, 2020

Uh oh!

PIP 37: [pulsar-client] support large message size #4400

PIP 37: [pulsar-client] support large message size #4400

Conversation

rdhabalia commented May 29, 2019

Motivation

Modification

Uh oh!

jiazhai commented May 30, 2019

Uh oh!

rdhabalia commented May 30, 2019

Uh oh!

jiazhai commented Jun 19, 2019

Uh oh!

rdhabalia commented Oct 23, 2019

Uh oh!

rdhabalia commented Oct 24, 2019

Uh oh!

rdhabalia commented Jan 29, 2020

Uh oh!

sijie Jan 31, 2020

Choose a reason for hiding this comment

Uh oh!

rdhabalia Feb 1, 2020

Choose a reason for hiding this comment

Uh oh!

rdhabalia commented Mar 9, 2020

Uh oh!

sijie commented Mar 9, 2020

Uh oh!

jiazhai commented Apr 8, 2020

Uh oh!

codelipenghui commented May 25, 2020

Uh oh!

codelipenghui May 29, 2020

Choose a reason for hiding this comment

Uh oh!

rdhabalia Jun 1, 2020

Choose a reason for hiding this comment

Uh oh!

rdhabalia commented Jun 2, 2020

Uh oh!