Runtime Panic (index out of range) when running query

[vanugrah]

Hello!

I enabled the thanos promql engine in our staging environment and I see the following runtime panic occurring:

level=error ts=2023-04-07T04:05:54.210299148Z caller=engine.go:361 msg="runtime panic in engine" expr="group by (container, dc, namespace, node, owner_kind, owner_name, pod) ((group without (host_ip, pod_ip) (kube_pod_info{job=\"kube-state-metrics\"} > 0)) * on (dc, namespace, pod) group_left (owner_kind, owner_name) (kube_pod_owner{job=\"kube-state-metrics\",pod!=\"\"} > 0) * on (dc, namespace, pod) group_right (node, owner_kind, owner_name) kube_pod_container_info{container!=\"\",job=\"kube-state-metrics\"})" err="runtime error: index out of range [4359] with length 4359" stacktrace="goroutine 9818884 [running]:\ngithub.com/thanos-community/promql-engine/engine.recoverEngine({0x2bd95c0, 0xc00052cdc0}, {0x2bf7820, 0xc0626c8060}, 0xc0626c4d00)\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:359 +0xc6\npanic({0x2507ae0, 0xc04c77d818})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).join(0xc0626de3c0?, 0xc011730000?, 0x1108?, 0x0?, 0xc042dc86c0, {0xc0626c45c0, 0x3, 0x4})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:242 +0x71a\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).initOutputs(0xc0626de3c0, {0x2bf4140, 0xc0626c4d40})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:109 +0x2a5\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x2e\nsync.(*Once).doSlow(0x0?, 0x0?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/binary.(*vectorOperator).Series(0xc0626de3c0?, {0x2bf4140?, 0xc0626c4d40?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/binary/vector.go:78 +0x7e\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables(0xc00002a8f0, {0x2bf4140?, 0xc0626c4d40?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:200 +0x51\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).initializeTables(0xc00002a8f0, {0x2bf4140, 0xc0626c4d40})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:168 +0x45\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series.func1()\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x2e\nsync.(*Once).doSlow(0x0?, 0x7f38f2ecd0c8?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/thanos-community/promql-engine/execution/aggregate.(*aggregate).Series(0xc00002a8f0?, {0x2bf4140?, 0xc0626c4d40?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/aggregate/hashaggregate.go:91 +0x7e\ngithub.com/thanos-community/promql-engine/execution/exchange.(*concurrencyOperator).Series(0x2bf41e8?, {0x2bf4140?, 0xc0626c4d40?})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/execution/exchange/concurrent.go:41 +0x2c\ngithub.com/thanos-community/promql-engine/engine.(*compatibilityQuery).Exec(0xc0626d3d10, {0x2bf41e8, 0xc0626c6390})\n\t/go/pkg/mod/github.com/thanos-community/promql-engine@v0.0.0-20221115192426-d68c8139c070/engine/engine.go:204 +0x1c2\ngithub.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).query(0xc00022a870, 0xc061a05e00)\n\t/app/pkg/api/query/v1.go:441 +0x92b\ngithub.com/thanos-io/thanos/pkg/api.GetInstr.func1.1({0x2be9960, 0xc036e4aa10}, 0x4?)\n\t/app/pkg/api/api.go:211 +0x50\nnet/http.HandlerFunc.ServeHTTP(0xc0626a82f0?, {0x2be9960?, 0xc036e4aa10?}, 0x2bce3bc?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1({0x2be9960?, 0xc036e4aa10}, 0xc061a05e00)\n\t/app/pkg/logging/http.go:69 +0x3b8\nnet/http.HandlerFunc.ServeHTTP(0x2bf41e8?, {0x2be9960?, 0xc036e4aa10?}, 0x2bcec38?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1({0x2be9960, 0xc036e4aa10}, 0xc061a05d00)\n\t/app/pkg/server/http/middleware/request_id.go:40 +0x542\nnet/http.HandlerFunc.ServeHTTP(0x2184f60?, {0x2be9960?, 0xc036e4aa10?}, 0x4?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x2bedd40, 0xc0608ddf00}, 0x490001?)\n\t/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x26f\nnet/http.HandlerFunc.ServeHTTP(0x1?, {0x2bedd40?, 0xc0608ddf00?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7f38f374df70?, 0xc04f6f2e10}, 0xc061a05d00)\n\t/app/pkg/extprom/http/instrument_server.go:75 +0x10b\nnet/http.HandlerFunc.ServeHTTP(0x7f38f374df70?, {0x7f38f374df70?, 0xc04f6f2e10?}, 0xc0626c6060?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7f38f374df70?, 0xc04f6f2dc0?}, 0xc061a05d00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:288 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x7f38f374df70?, {0x7f38f374df70?, 0xc04f6f2dc0?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7f38f374df70?, 0xc04f6f2d70?}, 0xc061a05d00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xb8\nnet/http.HandlerFunc.ServeHTTP(0x22c9b80?, {0x7f38f374df70?, 0xc04f6f2d70?}, 0x6?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/extprom/http.instrumentHandlerInFlight.func1({0x7f38f374df70, 0xc04f6f2d70}, 0xc061a05d00)\n\t/app/pkg/extprom/http/instrument_server.go:162 +0x169\nnet/http.HandlerFunc.ServeHTTP(0x2bf1310?, {0x7f38f374df70?, 0xc04f6f2d70?}, 0xc02cbe1698?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2bf1310?, 0xc0015321c0?}, 0xc061a05d00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:238 +0xc5\nnet/http.HandlerFunc.ServeHTTP(0x2bf41e8?, {0x2bf1310?, 0xc0015321c0?}, 0xc06269de90?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2bf1310, 0xc0015321c0}, 0xc061a05c00)\n\t/app/pkg/tracing/http.go:62 +0x9a2\ngithub.com/prometheus/common/route.(*Router).handle.func1({0x2bf1310, 0xc0015321c0}, 0xc061a05b00, {0x0, 0x0, 0x478d4e?})\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:83 +0x18d\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc001118c00, {0x2bf1310, 0xc0015321c0}, 0xc061a05b00)\n\t/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x81c\ngithub.com/prometheus/common/route.(*Router).ServeHTTP(0xc0015321c0?, {0x2bf1310?, 0xc0015321c0?}, 0x268c113?)\n\t/go/pkg/mod/github.com/prometheus/common@v0.37.1/route/route.go:126 +0x26\nnet/http.(*ServeMux).ServeHTTP(0x0?, {0x2bf1310, 0xc0015321c0}, 0xc061a05b00)\n\t/usr/local/go/src/net/http/server.go:2487 +0x149\nnet/http.serverHandler.ServeHTTP({0x2be7b80?}, {0x2bf1310, 0xc0015321c0}, 0xc061a05b00)\n\t/usr/local/go/src/net/http/server.go:2947 +0x30c\nnet/http.(*conn).serve(0xc00296ab40, {0x2bf41e8, 0xc000948300})\n\t/usr/local/go/src/net/http/server.go:1991 +0x607\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:3102 +0x4db\n"

The query that caused this panic is as follows:

group by (container, dc, namespace, node, owner_kind, owner_name, pod) (
    (group without (host_ip, pod_ip) (kube_pod_info{job=\"kube-state-metrics\"} > 0)) 
    * on (dc, namespace, pod) group_left (owner_kind, owner_name) 
    (kube_pod_owner{job=\"kube-state-metrics\",pod!=\"\"} > 0) 
    * on (dc, namespace, pod) group_right (node, owner_kind, owner_name) 
    kube_pod_container_info{container!=\"\",job=\"kube-state-metrics\"}
)

[saswatamcode]

The promql-engine version from the panic seems to be an older Thanos version v0.30.2. Can you try upgrading queriers to v0.31.0 and see if the panic still occurs? There have been quite a few fixes between versions.