Make it possible to index a field without fieldnorms

[fulmicoton]

Right now indexed fields come with a fieldnorm. This fieldnorm is used for BM25 scoring.

Bm25 is not always relevant. This ticket is about adding an indexing option for which fieldnorm data is not retained.

[lpouget]

Hi,

I started working on this here: #1001

I didn't see the previous work on 7cb018c and I took a different approach.

In previous work, the data is serialized even if field has no fieldnorm.

In mine, I didn't serialize the fieldnorm if there is none for the field (https://github.com/lpouget/tantivy/blob/784c9a4afade9f72456f8368e9687e85b0d5fcaa/src/indexer/segment_writer.rs#L215).

It's implies you didn't always have a fieldnorm reader (or have an empty default one).

Is it too radical ?

[fulmicoton]

Your approach is better! Let's forget my previous work and let me review your code tomorrow.

[lpouget]

Thx but I have a little problem with my approach :D

Idk how to handle this method with my approach:

fn compute_total_num_tokens(readers: &[SegmentReader], field: Field) -> crate::Result<u64> {
    let mut total_tokens = 0u64;
    let mut count: [usize; 256] = [0; 256];
    for reader in readers {
        if reader.has_deletes() {
            // if there are deletes, then we use an approximation
            // using the fieldnorm
            let fieldnorms_reader = reader.get_fieldnorms_reader(field)?;
            for doc in reader.doc_ids_alive() {
                let fieldnorm_id = fieldnorms_reader.fieldnorm_id(doc);
                count[fieldnorm_id as usize] += 1;
            }
        } else {
            total_tokens += reader.inverted_index(field)?.total_num_tokens();
        }
    }
    Ok(total_tokens
        + count
            .iter()
            .cloned()
            .enumerate()
            .map(|(fieldnorm_ord, count)| {
                count as u64 * u64::from(FieldNormReader::id_to_fieldnorm(fieldnorm_ord as u8))
            })
            .sum::<u64>())
}

because I will have an empty fieldnorms_reader so I will not be able to call fieldnorms_reader.fieldnorm_id(doc) to increment counter accuratly :/

and btw just wait I remove the draft flag before reviewing because I want to use your approach to handle the option.

[fulmicoton]

First of all explanation about what this ufnction is about.

It computes an approximate overall number of tokens for a field in a given segment when there is a merge.
A tantivy index serializes this information within the index.

It makes it possible to compute the average fieldnorm for bm25.

This value is approximate after a merge because it relies on fieldnormid which itself encodes the fieldnorm in a single byte, using some kind of logarithmic scale.

Now that what it is...

Is it legitimate for a user to disable fieldnorm and still try to use BM25?
Well, it is a little bit weird, but I could see that happen. BM25 should still give importance weights between fields for instance. Maybe only one field does not have fieldnorm too.

One thing we could do is, when there are no fieldnorm availabe, to extrapolate the total_num_tokens using the previous total_num_tokens and their number of deletes.

In other word,

Here num_doc and max_doc refer to methods you have in SegmentReader mean respectively.

• the number of non-deleted docs in the segment
• the number of docs in the segment, including deleted docs as if they were not deleted.

[fulmicoton]

Closed see #1001