prepare fastfield format for null index

[PSeitz]

With the upcoming null handling/sparse data, the format will change for fast fields.
Serialize a null index footer that reflects a Full column, to avoid compatibility issues for the next tantivy version

[fulmicoton]

I don't understand what this PR is about. Can you edit the description and the commit message? @PSeitz

[PSeitz]

I don't understand what this PR is about. Can you edit the description and the commit message? @PSeitz

I modified the commit message

[fulmicoton]

I don't think this is putting us on the right track for backward compatibility.

Here are the issue I have:

• The footer is constant length.
• It implements stuff for one future that is not there.
• The footer is 11 bytes long and replicated for every column. once we have sparse columns we may end up with a lot of columns. 11 bytes is a bit wasteful (not so terrible but well)

I think we could...
End the fastfield file by a codec version and a magic number.

A codec object would have a fn load(&self, data: OwnedBytes) -> Result<_, _> instead of the load method in the fastfield_codecs crate.

[PSeitz]

It implements stuff for one future that is not there.

In just describes the current state, which will be compatible with the future.

The footer is 11 bytes long and replicated for every column. once we have sparse columns we may end up with a lot of columns. 11 bytes is a bit wasteful (not so terrible but well)

It's 18bytes, that's ~2kb for 100columns, I don't think this will ever be relevant. No problem to change it though.

I think we could...
End the fastfield file by a codec version and a magic number.

I think that's a good idea in general and should be added.

The downside to do that instead, is to introduce a new version and with it all the annoying version handling when one will be fine.

A codec object would have a fn load(&self, data: OwnedBytes) -> Result<_, _> instead of the load method in the fastfield_codecs crate.

We already have that

fn open_from_bytes(mut data: OwnedBytes, header: NormalizedHeader) -> io::Result<Self::Reader>

[codecov-commenter]

Codecov Report

Merging #1691 (7ce6abe) into main (600548f) will decrease coverage by 0.00%.
The diff coverage is 92.07%.

@@            Coverage Diff             @@
##             main    #1691      +/-   ##
==========================================
- Coverage   94.05%   94.05%   -0.01%     
==========================================
  Files         256      258       +2     
  Lines       49253    49400     +147     
==========================================
+ Hits        46324    46461     +137     
- Misses       2929     2939      +10     

Impacted Files	Coverage Δ
fastfield_codecs/src/format_version.rs	59.25% <59.25%> (ø)
fastfield_codecs/src/null_index_footer.rs	97.77% <97.77%> (ø)
common/src/serialize.rs	86.90% <100.00%> (+0.48%)	⬆️
fastfield_codecs/src/compact_space/mod.rs	96.78% <100.00%> (+0.02%)	⬆️
fastfield_codecs/src/lib.rs	98.89% <100.00%> (+<0.01%)	⬆️
fastfield_codecs/src/serialize.rs	87.44% <100.00%> (+0.80%)	⬆️
ownedbytes/src/lib.rs	98.65% <100.00%> (+0.02%)	⬆️
src/fastfield/mod.rs	99.73% <100.00%> (ø)
src/schema/schema.rs	98.79% <0.00%> (+0.15%)	⬆️
src/fastfield/multivalued/mod.rs	99.22% <0.00%> (+0.77%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[fulmicoton]

Can you switch to encoding end-start instead of end