zstd: Improve zstd best efficiency

[klauspost]

@greatroar

Before/after compared to master:

enwik8	zskp	4	100000000	29798035	5596	17.04
enwik8	zskp	4	100000000	29298617	5678	16.79

silesia.tar	zskp	4	211947520	59904436	10622	19.03
silesia.tar	zskp	4	211947520	59311059	10818	18.68

TS40.txt	zskp	4	400000000	123504912	22005	17.34
TS40.txt	zskp	4	400000000	121367006	23661	16.12

apache.log	zskp	4	2622574440	113537595	21239	117.76
apache.log	zskp	4	2622574440	110106407	24301	102.92

github-ranks-backup.bin	zskp	4	1862623243	377801114	75884	23.41
github-ranks-backup.bin	zskp	4	1862623243	373322075	76540	23.21

Doubles table size.

[greatroar]

I'd reconstructed most of e283cac and I can confirm that it produces about 1.6% smaller files with acceptable speed loss (though almost double the memory use, I guess).

[greatroar]

Does this help? I'd tried to check for overlap between the best match and the candidate before the first load3232, but found that it slows compression down. From memory, what I did was

overlaps := m.rep > 0 && offset >= m.offset && offset < m.offset+m.length
if s-offset >= e.maxMatchOff || overlaps || load3232(src, offset) != first {
    return
}

[klauspost]

The cheap version of your check is to check if the offset is the same. I had it at some point but very few hits. Also tried keeping all previously tested offsets - but horrible.

This piece of code does something else. It is a bit easier to explain with an example.

Say we have a match of 50 bytes already.
When we check if a new match is better, we start by checking if bytes [42...46] match. If they don't, we will at best get a match that is 45 bytes, which will always be worse than the 50 byte match, without further tests.

In most cases we can reject on this. Some do the entire matchlen in reverse first and forward afterwards. Too tedious for me.

We could check 8 bytes - but I don't think that will make much difference.

[klauspost]

I tried

		improve := func(m *match, offset int32, s int32, first uint32, rep int32) {
			if offset == m.offset {
				if m.rep > 0 || rep < 0 {
					// Existing is repeat or new isn't...
					return
				}
				// If base offset matches we can use it as is.
				if m.s == s {
					// We can just use the repeat value
					m.rep = rep
					// Recalc...
					m.estBits(bitsPerByte)
					return
				}
			}

Only "apache.log" improved (it has very long matches). The rest were slower.

I would rather improve the ones that are slow than the ones that are already fast.

[greatroar]

LGTM.