[rhobs-logs] Disable tracing to temporarily stop panic errors

[periklis]

We are currently observing panic errors on Loki components for all sampled request by Jaeger. The root cause for this issue is when using the UserID and OrgID in exemplars. See:

• panic: exemplar labels have 85 runes, exceeding the limit of 64 grafana/loki#6667
• fix prometehus client exemplar panic weaveworks/common#246

The following PR disables trace sampling for all Loki components.

2022-07-19 08:02:19.917503 I | http: panic serving 10.129.8.49:35634: exemplar labels have 87 runes, exceeding the limit of 64
goroutine 1058109 [running]:
net/http.(*conn).serve.func1()
/usr/lib/golang/src/net/http/server.go:1802 +0xb9
panic({0x1e333c0, 0xc0014feb50})
/usr/lib/golang/src/runtime/panic.go:1047 +0x266
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
/go/src/github.com/grafana/loki/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:150 +0x139
panic({0x1e333c0, 0xc0014feb50})
/usr/lib/golang/src/runtime/panic.go:1038 +0x215
github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar(0xc00072b950, 0x40ad9d, 0x0, 0xc0012fd0b0)
/go/src/github.com/grafana/loki/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:434 +0xb1
github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar(0xc00072b950, 0x3f5df6374bbe10ea, 0x21ea5f3)
/go/src/github.com/grafana/loki/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:314 +0xcd
github.com/weaveworks/common/instrument.ObserveWithExemplar({0x26729e8, 0xc0012fc960}, {0x7f8f8919e3c0, 0xc00072b950}, 0x0)
/go/src/github.com/grafana/loki/vendor/github.com/weaveworks/common/instrument/instrument.go:78 +0x202
github.com/weaveworks/common/middleware.Instrument.Wrap.func1({0x264f3e0, 0xc0012215e0}, 0xc000b04900)
/go/src/github.com/grafana/loki/vendor/github.com/weaveworks/common/middleware/instrument.go:76 +0x66e
net/http.HandlerFunc.ServeHTTP(0xc000b04900, {0x264f3e0, 0xc0012215e0}, 0x0)
/usr/lib/golang/src/net/http/server.go:2047 +0x2f
github.com/weaveworks/common/middleware.Log.Wrap.func1({0x2653610, 0xc0012214a0}, 0xc000b04900)
/go/src/github.com/grafana/loki/vendor/github.com/weaveworks/common/middleware/logging.go:55 +0x287
net/http.HandlerFunc.ServeHTTP(0x40d2c7, {0x2653610, 0xc0012214a0}, 0xc0007ebd01)
/usr/lib/golang/src/net/http/server.go:2047 +0x2f
github.com/weaveworks/common/middleware.Log.Wrap.func1({0x2653610, 0xc0012214a0}, 0xc000b04900)
/go/src/github.com/grafana/loki/vendor/github.com/weaveworks/common/middleware/logging.go:55 +0x287
net/http.HandlerFunc.ServeHTTP(0x40d2c7, {0x2653610, 0xc0012214a0}, 0xc0007ebd01)
/usr/lib/golang/src/net/http/server.go:2047 +0x2f
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x26515a0, 0xc000a3e000}, 0xc000b04700)
/go/src/github.com/grafana/loki/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:154 +0x62d
net/http.HandlerFunc.ServeHTTP(0xc000b04600, {0x26515a0, 0xc000a3e000}, 0x7b0c10)
/usr/lib/golang/src/net/http/server.go:2047 +0x2f
github.com/grafana/loki/pkg/loki.(*Loki).initServer.func2.1({0x26515a0, 0xc000a3e000}, 0xc000b04600)
/go/src/github.com/grafana/loki/pkg/loki/modules.go:131 +0x4de
net/http.HandlerFunc.ServeHTTP(0x0, {0x26515a0, 0xc000a3e000}, 0x7b0e60)
/usr/lib/golang/src/net/http/server.go:2047 +0x2f
net/http.serverHandler.ServeHTTP({0x2648d70}, {0x26515a0, 0xc000a3e000}, 0xc000b04600)
/usr/lib/golang/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc0004c6320, {0x26729e8, 0xc0005f8f00})
/usr/lib/golang/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
/usr/lib/golang/src/net/http/server.go:3034 +0x4e8

[douglascamata]

Is this the minimum set of things we can disable to make this work? Wouldn't it be enough to just disable tracing through its boolean flag and leave all the configuration there?

[periklis]

Is this the minimum set of things we can disable to make this work? Wouldn't it be enough to just disable tracing through its boolean flag and leave all the configuration there?

Sure we can disable the flag only, but I though why keeping the sidecars idling if they don't sample anything.

[douglascamata]

🚢