s2 snappy compression panic

[jnyi]

hi @klauspost,

Just want to report a weird panic issue from a consumer project of this fancy repo (stack traces included). thanos-io/thanos#6942, we've seen s2/writer panic with latest go runtime v1.21.+ and tried a few version of this go mod:

• v1.17.4
• v1.17.1
• v1.16.7

Would appreciate if you have any valuable insights.

[klauspost]

Interesting. We (in MinIO) have sporadically seen similar crashes, but without a clear trace back to this library.

We only experienced it it Go 1.20 and later, so we have been forced to stay at Go 1.19 when we see it.

This looks like an interesting lead. I will investigate and see if I can do semi-reliable reproducers.

[mhoffm-aiven]

FWIW: we have been seeing it in our thanos built with go 1.21.3 too.

[klauspost]

@mhoffm-aiven Yeah - versions from 1.20 and forward. It seems like there is something that can cause runtime issues with these.

We have only had a few crashes, so we'd been unable to correlate it to any code. But it seems like the overlap with your issue narrows it down considerably.

[klauspost]

We may need a second common factor - could be something like an incoming signal, since it doesn't generally reproduce.

[mhoffm-aiven]

So my random observation was that it was happening far more often ( in my thanos storage gateway ) if we were using in-memory cache, so if the thanos process was using tons of memory. I moved to an external redis-cache and that helped with crashes in the storage gateway ( we still have some in the router component ). It might be somewhat related to memory pressure maybe.

[klauspost]

@mhoffm-aiven Yes. It does seem related to GC events.

My hunch is that it is a combination of goroutine preemption and the stack not being in expected state correct through/after assembly calls.

[klauspost]

I will open a golang issue and see if people with some more internal knowledge can assist.

[klauspost]

Submitted golang/go#64781 which also includes some of the issues we've seen at MinIO.

[klauspost]

@jnyi Could you try https://github.com/klauspost/compress/releases/tag/v1.17.7 or the latest?

It could be that #930 fixed this issue.