Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support magicless frame #151

Closed
rdicroce opened this issue Nov 23, 2020 · 3 comments
Closed

Support magicless frame #151

rdicroce opened this issue Nov 23, 2020 · 3 comments

Comments

@rdicroce
Copy link
Contributor

I'm using this in a messaging scenario where I already have a flag to tell me which compression algorithm is being used. So Zstd's 4-byte magic identifier is just wasting bytes and I'd like to omit it. The Zstd API already supports doing this (see the API docs about ZSTD_DCtx_setFormat and ZSTD_f_zstd1_magicless) so the JNI wrapper just needs to expose that functionality.

@luben
Copy link
Owner

luben commented Nov 23, 2020

Hi, may be in the future - these params are still experimental.

@rdicroce
Copy link
Contributor Author

Are you saying you won't implement any API labeled as experimental? Because the Zstd API docs say that some experimental APIs are "planned to remain in the static_only section indefinitely" and I suspect magicless is one of them. It has been available for over 3 years, and the stance of the Zstd maintainers seems to be that they don't want to make magicless frames too easy to obtain. Labeling it as an "experimental" API is presumably part of that. But I doubt they'll ever remove it altogether, because there are certain situations (like mine) where it's useful.

@luben
Copy link
Owner

luben commented Nov 23, 2020

Agree that it may be useful. Currently I don't compile it with STATIC_ONLY and this may need some substantial change to the build process. I will explore it

rdicroce added a commit to rdicroce/zstd-jni that referenced this issue Oct 12, 2022
@luben luben closed this as completed in 4a46d81 Oct 15, 2022
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Oct 27, 2022
### What changes were proposed in this pull request?
This pr aims to upgrade zstd-jni to 1.5.2-5

### Why are the changes needed?
This version start to support magic less data frames:

- luben/zstd-jni#151
- luben/zstd-jni#235

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes #38412 from LuciferYang/zstd-1.5.2-5.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
SandishKumarHN pushed a commit to SandishKumarHN/spark that referenced this issue Dec 12, 2022
### What changes were proposed in this pull request?
This pr aims to upgrade zstd-jni to 1.5.2-5

### Why are the changes needed?
This version start to support magic less data frames:

- luben/zstd-jni#151
- luben/zstd-jni#235

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes apache#38412 from LuciferYang/zstd-1.5.2-5.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants