Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[package management service] fix wrappedBuffer always using the same block of memory #11782

Conversation

freeznet
Copy link
Contributor

@freeznet freeznet commented Aug 26, 2021

Fixes #11781

Motivation

When uploading file content to bk, the BookKeeperPackagesStorage always use the same block of memory to read the file content and build up a bunch of zero-copy ByteBuf.

This causes the ByteBufs having the same block of memory and break the file content.

Modifications

Alloc new memory block in each iteration.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

For contributor

For this PR, do we need to update docs?

  • If yes, please update docs or create a follow-up issue if you need help.

  • If no, please explain why.

For committer

For this PR, do we need to update docs?

  • If yes,

    • if you update docs in this PR, label this PR with the doc label.

    • if you plan to update docs later, label this PR with the doc-required label.

    • if you need help on updating docs, create a follow-up issue with the doc-required label.

  • If no, label this PR with the no-need-doc label and explain why.

internal bug fix

@freeznet freeznet requested a review from zymap August 26, 2021 01:03
@freeznet freeznet self-assigned this Aug 26, 2021
@freeznet freeznet changed the title [package management service] fix wrappedBuffer always having the same block of memory [package management service] fix wrappedBuffer always using the same block of memory Aug 26, 2021
@zymap zymap added release/2.8.2 type/bug The PR fixed a bug or issue reported a bug labels Aug 26, 2021
@zymap zymap added this to the 2.9.0 milestone Aug 26, 2021
@codelipenghui codelipenghui merged commit 12aef52 into apache:master Aug 26, 2021
hangc0276 pushed a commit that referenced this pull request Aug 26, 2021
@hangc0276 hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 26, 2021
@lhotari
Copy link
Member

lhotari commented Aug 27, 2021

Good catch @freeznet ! I also added "release/2.7.4" since this looks like an important fix.

@lhotari
Copy link
Member

lhotari commented Aug 27, 2021

or does it only apply to 2.8.x + ? I see that the change was made to DLOutputStream, but I guess that's not used asynchronously by Pulsar Functions upload, so that wouldn't be impacted?

@freeznet
Copy link
Contributor Author

@lhotari thanks for asking. from my understanding this fix only applies to package management service, and not with Pulsar Functions upload, so it might not affect release 2.7.x

bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-picked/branch-2.8 Archived: 2.8 is end of life doc-not-needed Your PR changes do not impact docs release/2.8.1 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[package management service] inequality file issue when having large file content
5 participants