Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LZO #41

Open
martindurant opened this issue Feb 25, 2021 · 9 comments · May be fixed by #52
Open

Add LZO #41

martindurant opened this issue Feb 25, 2021 · 9 comments · May be fixed by #52

Comments

@martindurant
Copy link

A rarer compression algorithm for parquet is LZO. This library seems to do it: https://badboy.github.io/minilzo-rs/minilzo/index.html

@messense
Copy link
Contributor

messense commented Feb 28, 2021

Note that there is a pure Rust version: https://github.com/gmg137/minilzo-rs

Never mind, the project's description A pure rust implementation bound to the C version of minilzo. is confusing.

@martindurant
Copy link
Author

martindurant commented Feb 28, 2021 via email

@milesgranger
Copy link
Owner

I was thinking that as well. Do you know how it works if it's translated from original C implementation? I've used c2rust before with various degrees of success. But not really knowledgeable if that avenue still causes licensing conflicts?

@milesgranger
Copy link
Owner

lzokay is a C++ implementation, released under MIT; suggesting, at least to me, it would be acceptable. 🤔

@martindurant
Copy link
Author

martindurant commented Feb 28, 2021 via email

@milesgranger
Copy link
Owner

I've successfully been able to port part of the C implementation to pure Rust, but with GPLv2 it seems very restrictive, covering even 'work derived'. But I see python-lzo to which fastparquet has an optional dependency is GPLv2, but fastparquet remains Apache 2. If this is acceptable, then I guess there isn't an issue for the rust implementation to have GPLv2 and cramjam to have MIT then, as it's basically the same relationship that fastparquet currently has, no? Maybe I'll just email the gentleman and see about at least a LGPL for the rust implementation. Worst case, cramjam is dual licensed under both depending on if lzo is included. 🙄

@martindurant
Copy link
Author

There is a big difference between having an optional dependency on a library published and built by someone else, and directly embedding that library into your own code. That would make cramjam GPL, and I'm not sure about fastparquet (because cramjam would be a required dependency).

I cannot find a non-GPL LZO implementation, the original reference C impl was GPL. Maybe close this issue and ignore the problem.

@milesgranger
Copy link
Owner

Okay. Again, I'm a bit of a license novice here, just trying to understand; I was having ball with the rust implementation and would love to have it as a separate crate cramjam could make use of. And from what I understand, if I can at least publish a separate crate as LGPL, cramjam could link to it and maintain its current license; to do this, as I see it, would need permission from the author as I don't think GPLv2 can go directly LGPL. Feel free to close this issue if you'd like, but I'd still like to see if it can be done; by poking the author of lzo about it.

@martindurant
Copy link
Author

If you link dynamically, as opposed to including in your own build, that's definitely OK.

would need permission from the author as I don't think GPLv2 can go directly LGPL

This I don't know.

I am just-above-novice...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants