Moving from zerocopy to bytemuck

[Kerollmops]

This library currently uses the zerocopy crate to provide some useful codecs for common types and types which implements the AsBytes/FromBytes traits (i.e. u8, [u8]). I find it complex/impossible to contribute to this crate as there is no repository for this crate only as it has been developed for the new fuchsia OS; sources of this crate are mixed with the source of the OS (in a sub-repository at least).

The bytemuck library seems to be much easier to contribute to; it is hosted on Github, seems to be much more popular than the former (114k downloads by month compared to 15k). It brings a better API, at least for heed, as it can return information on which kind of problem happens when a cast fails, it would simplify some codecs.

I looked into the crate and found that the change, despite breaking, will be easy. I also found out that we can cast slices of different types by using the try_cast_slice function.

I am just not sure how to expose integers as big-endian, ensuring that the global order is preserved. We will maybe need to create codecs for every integer type like what zerocopy has done using byteorder maybe.

[joshlf]

Hi @Kerollmops , just wanted to let you know that zerocopy is on GitHub now in case there are still zerocopy features that would be useful to you (like byteorder-aware types). If not, of course disregard this message 🙂

[Kerollmops]

Thank you very much @joshlf,

Is there a page that compares both crates feature-wise? The only issue that I found is this one on bytemuck. I'll evaluate both crates.

Note that I will release a new version of heed with a lot of things rewritten and this choice between heed and bytemuck will be the last choice.

[joshlf]

Is there a page that compares both crates feature-wise? The only issue that I found is this one on bytemuck. I'll evaluate both crates.

At this point, we've both adopted features from each other, so it really depends on your specific use cases. If you can describe specific use cases, I can try to speak to what the two crates provide for those use cases (with the obvious bias that I know zerocopy a lot better than I know bytemuck), but I don't maintain a comparison page. For the current state of zerocopy, I'd check out our latest alpha release. For the future state, I'd check out our Q4 2022 roadmap (that roadmap is best effort, so definitely don't make any decisions that rely on that roadmap coming to fruition on schedule).

I'd also love to hear any feedback you have or feature requests! Even just "I want to do <vague, high-level description> and I'm not sure how" is very helpful.

[Kerollmops]

Hey @joshlf,

I'm sorry for not getting back to you sooner. I just released a new version of heed v0.20.0-alpha.0 to let people test the version of the BytesEncode/BytesDecode traits backed by bytemuck. However, as we will also test it at Meilisearch inside milli, we will be able to ensure that it is correct and works great. If we find issues, we can roll back and use zerocopy or ask you for some advice.