Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Service down by out of memory, and restart fail with io.wait and head_wal read wal(size= 0) #13884

Open
yqjdcyy opened this issue Apr 3, 2024 · 1 comment
Open

Service down by out of memory, and restart fail with io.wait and head_wal read wal(size= 0) #13884

yqjdcyy opened this issue Apr 3, 2024 · 1 comment

Comments

@yqjdcyy
Copy link

yqjdcyy commented Apr 3, 2024

What did you do?

The node of prometheus work for scrape jobs on docker.
It down and restart fail for serval times, until clear wal.

What did you expect to see?

How to fix it, or correct the configuration.

What did you see instead? Under which circumstances?

Seem to write wal fail (the size of lastest 0000x file is zero) when out of memory, and then restart fail by reading the wal.

Note: sometime it logs "Error on ingesting out-of-order samples", but these sample had been droped.

System information

Linux 3.10.0-327.el7.x86_64 x86_64

Prometheus version

build user:       root@920118f645b7
  build date:       20230623-15:09:49
  go version:       go1.20.5
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

global:
  scrape_interval: 30s
  external_labels:
    namespace: custom-scrape-itest
# scrape_configs: 11 job

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

## docker ps -a | grep promethe
ede10ec2c2fa        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   7 hours ago         Exited (255) About an hour ago   9090/tcp            iumbtest_custom_scrape_prometheus.1.tdvussclcxhyjco4r10li5z41
bbbd91aca512        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   7 hours ago         Exited (2) 7 hours ago                               iumbtest_custom_scrape_prometheus.1.n7u8hn4d28icdbqagyv9xafwi
ac43b1045eec        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   7 hours ago         Exited (137) 7 hours ago                             iumbtest_custom_scrape_prometheus.1.tm6hsyd8pa6odjp6rhuq9ml78
147f6ba1344d        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   7 hours ago         Exited (2) 7 hours ago                               iumbtest_custom_scrape_prometheus.1.muj1zmn5zsj8ir1x8uxu914lo
1d221d5faaa6        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   7 hours ago         Exited (137) 7 hours ago                             iumbtest_custom_scrape_prometheus.1.re24g5xqyt6yn6td8ke2m7loq
f4b76a523f8c        prom/prometheus:v2.45.0                   "/bin/prometheus --c…"   29 hours ago        Exited (255) 24 hours ago        9090/tcp            iumbtest_custom_scrape_prometheus.1.ptfsr7ruec5gnbqazkenvkubo


## Logs
### f4b76a523f8c/ Crash
> memory reach the ceiling/ 14G, ssh don't work too

{"log":"ts=2024-04-01T22:03:37.477Z caller=head.go:684 level=info component=tsdb msg=\"Replaying WAL, this may take a while\"\n","stream":"stderr","time":"2024-04-01T22:03:37.47726269Z"}
{"log":"ts=2024-04-01T22:03:37.478Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=275\n","stream":"stderr","time":"2024-04-01T22:03:37.478945899Z"}
{"log":"ts=2024-04-01T22:04:08.275Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=1 maxSegment=275\n","stream":"stderr","time":"2024-04-01T22:04:08.276074243Z"}
{"log":"ts=2024-04-01T22:04:31.752Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=2 maxSegment=275\n","stream":"stderr","time":"2024-04-01T22:04:31.753092689Z"}

### 1d221d5faaa6/ ReStart
> without exception

{"log":"ts=2024-04-02T19:27:06.021Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=19\n","stream":"stderr","time":"2024-04-02T19:27:06.021449414Z"}
{"log":"ts=2024-04-02T19:27:38.448Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=1 maxSegment=19\n","stream":"stderr","time":"2024-04-02T19:27:38.448771201Z"}
{"log":"ts=2024-04-02T19:28:00.386Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=2 maxSegment=19\n","stream":"stderr","time":"2024-04-02T19:28:00.387157734Z"}

### 147f6ba1344d/ ReStart
> out of memory + IO wait+ wal.read

{"log":"ts=2024-04-02T19:28:40.842Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=20\n","stream":"stderr","time":"2024-04-02T19:28:40.842372417Z"}
{"log":"ts=2024-04-02T19:29:12.989Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=1 maxSegment=20\n","stream":"stderr","time":"2024-04-02T19:29:12.990114644Z"}
{"log":"ts=2024-04-02T19:29:36.303Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=2 maxSegment=20\n","stream":"stderr","time":"2024-04-02T19:29:36.304020691Z"}

{"log":"fatal error: runtime: out of memory\n","stream":"stderr","time":"2024-04-02T19:30:11.645978778Z"}
{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.65456112Z"}
{"log":"runtime stack:\n","stream":"stderr","time":"2024-04-02T19:30:11.654577388Z"}
{"log":"runtime.throw({0x34e9140?, 0x0?})\n","stream":"stderr","time":"2024-04-02T19:30:11.654581145Z"}
{"log":"\u0009/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000113df8 sp=0xc000113dc8 pc=0x438e9d\n","stream":"stderr","time":"2024-04-02T19:30:11.654584345Z"}
{"log":"runtime.sysMapOS(0xc3ad400000, 0x400000?)\n","stream":"stderr","time":"2024-04-02T19:30:11.655575581Z"}
{"log":"\u0009/usr/local/go/src/runtime/mem_linux.go:187 +0x11b fp=0xc000113e40 sp=0xc000113df8 pc=0x4183bb\n","stream":"stderr","time":"2024-04-02T19:30:11.655579165Z"}
{"log":"runtime.sysMap(0xffff800000000000?, 0x7fffffffffff?, 0x7f56ef96f000?)\n","stream":"stderr","time":"2024-04-02T19:30:11.655582396Z"}
{"log":"\u0009/usr/local/go/src/runtime/mem.go:142 +0x35 fp=0xc000113e70 sp=0xc000113e40 pc=0x417d95\n","stream":"stderr","time":"2024-04-02T19:30:11.655585764Z"}
{"log":"runtime.(*mheap).grow(0x57345a0, 0xa?)\n","stream":"stderr","time":"2024-04-02T19:30:11.655592244Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:1522 +0x252 fp=0xc000113ee8 sp=0xc000113e70 pc=0x429eb2\n","stream":"stderr","time":"2024-04-02T19:30:11.655595345Z"}
{"log":"runtime.(*mheap).allocSpan(0x57345a0, 0xa, 0x0, 0x3f?)\n","stream":"stderr","time":"2024-04-02T19:30:11.655598474Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:1243 +0x1b7 fp=0xc000113f80 sp=0xc000113ee8 pc=0x4295f7\n","stream":"stderr","time":"2024-04-02T19:30:11.655601559Z"}
{"log":"runtime.(*mheap).alloc.func1()\n","stream":"stderr","time":"2024-04-02T19:30:11.656911735Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:961 +0x65 fp=0xc000113fc8 sp=0xc000113f80 pc=0x4290a5\n","stream":"stderr","time":"2024-04-02T19:30:11.657066227Z"}
{"log":"runtime.systemstack()\n","stream":"stderr","time":"2024-04-02T19:30:11.657503925Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc000113fd0 sp=0xc000113fc8 pc=0x46dd29\n","stream":"stderr","time":"2024-04-02T19:30:11.657547881Z"}

{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.657873681Z"}
{"log":"goroutine 247 [running]:\n","stream":"stderr","time":"2024-04-02T19:30:11.657893718Z"}
{"log":"runtime.systemstack_switch()\n","stream":"stderr","time":"2024-04-02T19:30:11.658147991Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc000a66a00 sp=0xc000a669f8 pc=0x46dcc0\n","stream":"stderr","time":"2024-04-02T19:30:11.658167628Z"}
{"log":"runtime.(*mheap).alloc(0x14000?, 0xa?, 0xc0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.658765462Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:955 +0x65 fp=0xc000a66a48 sp=0xc000a66a00 pc=0x428fe5\n","stream":"stderr","time":"2024-04-02T19:30:11.658779638Z"}
{"log":"runtime.(*mcache).allocLarge(0x403c85?, 0x13fd0, 0x0)\n","stream":"stderr","time":"2024-04-02T19:30:11.659026971Z"}
{"log":"\u0009/usr/local/go/src/runtime/mcache.go:234 +0x85 fp=0xc000a66a90 sp=0xc000a66a48 pc=0x416e25\n","stream":"stderr","time":"2024-04-02T19:30:11.659038141Z"}
{"log":"runtime.mallocgc(0x13fd0, 0x30f2440, 0x1)\n","stream":"stderr","time":"2024-04-02T19:30:11.66256472Z"}
{"log":"\u0009/usr/local/go/src/runtime/malloc.go:1053 +0x4fe fp=0xc000a66af8 sp=0xc000a66a90 pc=0x40db3e\n","stream":"stderr","time":"2024-04-02T19:30:11.662575085Z"}
{"log":"runtime.newarray(0x7f552ae70078?, 0x7f56ef98ff18?)\n","stream":"stderr","time":"2024-04-02T19:30:11.662579035Z"}

.....

{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.691647026Z"}
{"log":"goroutine 511 [IO wait]:\n","stream":"stderr","time":"2024-04-02T19:30:11.691652559Z"}
{"log":"runtime.gopark(0x99?, 0xb?, 0x0?, 0x0?, 0x34?)\n","stream":"stderr","time":"2024-04-02T19:30:11.691655926Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc319ca98a0 sp=0xc319ca9880 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.691659094Z"}
{"log":"runtime.netpollblock(0x484925?, 0x40482f?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.691662379Z"}
{"log":"\u0009/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc319ca98d8 sp=0xc319ca98a0 pc=0x434497\n","stream":"stderr","time":"2024-04-02T19:30:11.691666606Z"}
{"log":"internal/poll.runtime_pollWait(0x7f56c85ca6c8, 0x72)\n","stream":"stderr","time":"2024-04-02T19:30:11.691670164Z"}
{"log":"\u0009/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc319ca98f8 sp=0xc319ca98d8 pc=0x469fc9\n","stream":"stderr","time":"2024-04-02T19:30:11.691673184Z"}
{"log":"internal/poll.(*pollDesc).wait(0xc00057ac00?, 0xc10d796000?, 0x0)\n","stream":"stderr","time":"2024-04-02T19:30:11.691677371Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc319ca9920 sp=0xc319ca98f8 pc=0x4e87b2\n","stream":"stderr","time":"2024-04-02T19:30:11.691681236Z"}
{"log":"internal/poll.(*pollDesc).waitRead(...)\n","stream":"stderr","time":"2024-04-02T19:30:11.691694968Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_poll_runtime.go:89\n","stream":"stderr","time":"2024-04-02T19:30:11.691702049Z"}
{"log":"internal/poll.(*FD).Read(0xc00057ac00, {0xc10d796000, 0x1000, 0x1000})\n","stream":"stderr","time":"2024-04-02T19:30:11.691717392Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc319ca99b8 sp=0xc319ca9920 pc=0x4e9b99\n","stream":"stderr","time":"2024-04-02T19:30:11.691727766Z"}
{"log":"net.(*netFD).Read(0xc00057ac00, {0xc10d796000?, 0x4e8ca6?, 0x7f56c85ca7b0?})\n","stream":"stderr","time":"2024-04-02T19:30:11.691732356Z"}
{"log":"\u0009/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc319ca9a00 sp=0xc319ca99b8 pc=0x52d469\n","stream":"stderr","time":"2024-04-02T19:30:11.691742631Z"}
{"log":"net.(*conn).Read(0xc105809958, {0xc10d796000?, 0x0?, 0x0?})\n","stream":"stderr","time":"2024-04-02T19:30:11.691755644Z"}
{"log":"\u0009/usr/local/go/src/net/net.go:183 +0x45 fp=0xc319ca9a48 sp=0xc319ca9a00 pc=0x53f8c5\n","stream":"stderr","time":"2024-04-02T19:30:11.691771572Z"}
{"log":"net.(*TCPConn).Read(0xc17b355bab1b0b2d?, {0xc10d796000?, 0x571ace0?, 0xc319ca9a68?})\n","stream":"stderr","time":"2024-04-02T19:30:11.691807297Z"}
{"log":"\u0009\u003cautogenerated\u003e:1 +0x29 fp=0xc319ca9a78 sp=0xc319ca9a48 pc=0x552529\n","stream":"stderr","time":"2024-04-02T19:30:11.691818237Z"}
{"log":"golang.org/x/net/netutil.(*limitListenerConn).Read(0xc00057ac00?, {0xc10d796000?, 0xc0009a6720?, 0xc10d780c38?})\n","stream":"stderr","time":"2024-04-02T19:30:11.691824964Z"}
{"log":"\u0009\u003cautogenerated\u003e:1 +0x2f fp=0xc319ca9aa8 sp=0xc319ca9a78 pc=0x2892f6f\n","stream":"stderr","time":"2024-04-02T19:30:11.691848034Z"}
{"log":"github.com/mwitkow/go-conntrack.(*serverConnTracker).Read(0xc10d780c30?, {0xc10d796000?, 0x5526c9?, 0xc105809958?})\n","stream":"stderr","time":"2024-04-02T19:30:11.691867029Z"}
{"log":"\u0009\u003cautogenerated\u003e:1 +0x2f fp=0xc319ca9ad8 sp=0xc319ca9aa8 pc=0x90370f\n","stream":"stderr","time":"2024-04-02T19:30:11.691880654Z"}
{"log":"net/http.(*connReader).Read(0xc10d780c30, {0xc10d796000, 0x1000, 0x1000})\n","stream":"stderr","time":"2024-04-02T19:30:11.691895774Z"}
{"log":"\u0009/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc319ca9b28 sp=0xc319ca9ad8 pc=0x700a31\n","stream":"stderr","time":"2024-04-02T19:30:11.691904048Z"}
{"log":"bufio.(*Reader).fill(0xc0005f4a80)\n","stream":"stderr","time":"2024-04-02T19:30:11.691912724Z"}
{"log":"\u0009/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc319ca9b60 sp=0xc319ca9b28 pc=0x682e7f\n","stream":"stderr","time":"2024-04-02T19:30:11.691934962Z"}
{"log":"bufio.(*Reader).Peek(0xc0005f4a80, 0x4)\n","stream":"stderr","time":"2024-04-02T19:30:11.691942224Z"}
{"log":"\u0009/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc319ca9b80 sp=0xc319ca9b60 pc=0x682fdd\n","stream":"stderr","time":"2024-04-02T19:30:11.691955928Z"}
{"log":"net/http.(*conn).serve(0xc1a4ced290, {0x3d35750, 0xc000c26120})\n","stream":"stderr","time":"2024-04-02T19:30:11.691993249Z"}
{"log":"\u0009/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc319ca9fb8 sp=0xc319ca9b80 pc=0x7069fc\n","stream":"stderr","time":"2024-04-02T19:30:11.692002974Z"}
{"log":"net/http.(*Server).Serve.func3()\n","stream":"stderr","time":"2024-04-02T19:30:11.692010752Z"}
{"log":"\u0009/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc319ca9fe0 sp=0xc319ca9fb8 pc=0x70bcce\n","stream":"stderr","time":"2024-04-02T19:30:11.692017087Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.692022859Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc319ca9fe8 sp=0xc319ca9fe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.692027754Z"}
{"log":"created by net/http.(*Server).Serve\n","stream":"stderr","time":"2024-04-02T19:30:11.692033252Z"}
{"log":"\u0009/usr/local/go/src/net/http/server.go:3089 +0x5ed\n","stream":"stderr","time":"2024-04-02T19:30:11.692037832Z"}
{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.692041342Z"}
{"log":"goroutine 629 [chan receive]:\n","stream":"stderr","time":"2024-04-02T19:30:11.692060519Z"}
{"log":"runtime.gopark(0xc0009d8900?, 0xc000b31d88?, 0x29?, 0x3d?, 0x18ea00decf8?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692067389Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000b31ce0 sp=0xc000b31cc0 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.692092544Z"}
{"log":"runtime.chanrecv(0xc000b1e660, 0xc000b31f18, 0x1)\n","stream":"stderr","time":"2024-04-02T19:30:11.692100819Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000b31d70 sp=0xc000b31ce0 pc=0x406f9d\n","stream":"stderr","time":"2024-04-02T19:30:11.692139852Z"}
{"log":"runtime.chanrecv2(0xc0003af400?, 0xc3ad3fa4d0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692146134Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000b31d98 sp=0xc000b31d70 pc=0x406ad8\n","stream":"stderr","time":"2024-04-02T19:30:11.692150007Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*walSubsetProcessor).processWALSamples(0xc15d7c62b8, 0xc0003af400, 0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692181024Z"}
{"log":"\u0009/app/tsdb/head_wal.go:569 +0x16d fp=0xc000b31f78 sp=0xc000b31d98 pc=0x275426d\n","stream":"stderr","time":"2024-04-02T19:30:11.692189634Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func2(0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692193077Z"}
{"log":"\u0009/app/tsdb/head_wal.go:106 +0x4d fp=0xc000b31fc8 sp=0xc000b31f78 pc=0x27534cd\n","stream":"stderr","time":"2024-04-02T19:30:11.692197339Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func5()\n","stream":"stderr","time":"2024-04-02T19:30:11.692200904Z"}
{"log":"\u0009/app/tsdb/head_wal.go:111 +0x2a fp=0xc000b31fe0 sp=0xc000b31fc8 pc=0x275344a\n","stream":"stderr","time":"2024-04-02T19:30:11.692205312Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.69223254Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000b31fe8 sp=0xc000b31fe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.692238149Z"}
{"log":"created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL\n","stream":"stderr","time":"2024-04-02T19:30:11.692241687Z"}
{"log":"\u0009/app/tsdb/head_wal.go:105 +0x31f\n","stream":"stderr","time":"2024-04-02T19:30:11.692245682Z"}

{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.692249132Z"}
{"log":"goroutine 630 [chan receive]:\n","stream":"stderr","time":"2024-04-02T19:30:11.692252542Z"}
{"log":"runtime.gopark(0xc0009d8cc0?, 0xc000b2fd88?, 0x29?, 0x3d?, 0x18e9ff7e458?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692256704Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000b2fce0 sp=0xc000b2fcc0 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.692265404Z"}
{"log":"runtime.chanrecv(0xc000b1e780, 0xc000b2ff18, 0x1)\n","stream":"stderr","time":"2024-04-02T19:30:11.692286219Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000b2fd70 sp=0xc000b2fce0 pc=0x406f9d\n","stream":"stderr","time":"2024-04-02T19:30:11.69229172Z"}
{"log":"runtime.chanrecv2(0xc0003af400?, 0xc3ad3fa580?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692296057Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000b2fd98 sp=0xc000b2fd70 pc=0x406ad8\n","stream":"stderr","time":"2024-04-02T19:30:11.692309424Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*walSubsetProcessor).processWALSamples(0xc15d7c62d0, 0xc0003af400, 0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692320752Z"}
{"log":"\u0009/app/tsdb/head_wal.go:569 +0x16d fp=0xc000b2ff78 sp=0xc000b2fd98 pc=0x275426d\n","stream":"stderr","time":"2024-04-02T19:30:11.692341302Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func2(0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692345597Z"}
{"log":"\u0009/app/tsdb/head_wal.go:106 +0x4d fp=0xc000b2ffc8 sp=0xc000b2ff78 pc=0x27534cd\n","stream":"stderr","time":"2024-04-02T19:30:11.69234978Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func5()\n","stream":"stderr","time":"2024-04-02T19:30:11.692353387Z"}
{"log":"\u0009/app/tsdb/head_wal.go:111 +0x2a fp=0xc000b2ffe0 sp=0xc000b2ffc8 pc=0x275344a\n","stream":"stderr","time":"2024-04-02T19:30:11.692371592Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.692378132Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000b2ffe8 sp=0xc000b2ffe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.692381649Z"}
{"log":"created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL\n","stream":"stderr","time":"2024-04-02T19:30:11.692385759Z"}
{"log":"\u0009/app/tsdb/head_wal.go:105 +0x31f\n","stream":"stderr","time":"2024-04-02T19:30:11.692389249Z"}


{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.69239229Z"}
{"log":"goroutine 631 [chan receive]:\n","stream":"stderr","time":"2024-04-02T19:30:11.692396043Z"}
{"log":"runtime.gopark(0xc000c195c0?, 0xc014e87d88?, 0x29?, 0x3d?, 0x18e9ff874e0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692411033Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc014e87ce0 sp=0xc014e87cc0 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.692423882Z"}
{"log":"runtime.chanrecv(0xc000b1e8a0, 0xc014e87f18, 0x1)\n","stream":"stderr","time":"2024-04-02T19:30:11.69245722Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc014e87d70 sp=0xc014e87ce0 pc=0x406f9d\n","stream":"stderr","time":"2024-04-02T19:30:11.692464439Z"}
{"log":"runtime.chanrecv2(0xc0003af400?, 0xc3ad3fa630?)\n","stream":"stderr","time":"2024-04-02T19:30:11.69247585Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc014e87d98 sp=0xc014e87d70 pc=0x406ad8\n","stream":"stderr","time":"2024-04-02T19:30:11.692482477Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*walSubsetProcessor).processWALSamples(0xc15d7c62e8, 0xc0003af400, 0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692486572Z"}
{"log":"\u0009/app/tsdb/head_wal.go:569 +0x16d fp=0xc014e87f78 sp=0xc014e87d98 pc=0x275426d\n","stream":"stderr","time":"2024-04-02T19:30:11.69249085Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func2(0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692510132Z"}
{"log":"\u0009/app/tsdb/head_wal.go:106 +0x4d fp=0xc014e87fc8 sp=0xc014e87f78 pc=0x27534cd\n","stream":"stderr","time":"2024-04-02T19:30:11.69251593Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func5()\n","stream":"stderr","time":"2024-04-02T19:30:11.69251939Z"}
{"log":"\u0009/app/tsdb/head_wal.go:111 +0x2a fp=0xc014e87fe0 sp=0xc014e87fc8 pc=0x275344a\n","stream":"stderr","time":"2024-04-02T19:30:11.692523822Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.692527242Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc014e87fe8 sp=0xc014e87fe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.69253852Z"}
{"log":"created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL\n","stream":"stderr","time":"2024-04-02T19:30:11.692542552Z"}
{"log":"\u0009/app/tsdb/head_wal.go:105 +0x31f\n","stream":"stderr","time":"2024-04-02T19:30:11.692546579Z"}


{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.692550213Z"}
{"log":"goroutine 632 [chan receive]:\n","stream":"stderr","time":"2024-04-02T19:30:11.692554052Z"}
{"log":"runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692566977Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0efc9f5d0 sp=0xc0efc9f5b0 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.692578603Z"}
{"log":"runtime.chanrecv(0xc000b1e9c0, 0xc0efc9f788, 0x1)\n","stream":"stderr","time":"2024-04-02T19:30:11.692591287Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0efc9f660 sp=0xc0efc9f5d0 pc=0x406f9d\n","stream":"stderr","time":"2024-04-02T19:30:11.692598465Z"}
{"log":"runtime.chanrecv2(0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692602075Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0efc9f688 sp=0xc0efc9f660 pc=0x406ad8\n","stream":"stderr","time":"2024-04-02T19:30:11.692608877Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func3(0x0?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692616932Z"}
{"log":"\u0009/app/tsdb/head_wal.go:119 +0xbc fp=0xc0efc9f7c8 sp=0xc0efc9f688 pc=0x27530dc\n","stream":"stderr","time":"2024-04-02T19:30:11.692626103Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func6()\n","stream":"stderr","time":"2024-04-02T19:30:11.692646363Z"}
{"log":"\u0009/app/tsdb/head_wal.go:136 +0x2a fp=0xc0efc9f7e0 sp=0xc0efc9f7c8 pc=0x2752fea\n","stream":"stderr","time":"2024-04-02T19:30:11.692652465Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.69265587Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0efc9f7e8 sp=0xc0efc9f7e0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.692666933Z"}
{"log":"created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL\n","stream":"stderr","time":"2024-04-02T19:30:11.692670865Z"}
{"log":"\u0009/app/tsdb/head_wal.go:116 +0x6ae\n","stream":"stderr","time":"2024-04-02T19:30:11.692673853Z"}

{"log":"\n","stream":"stderr","time":"2024-04-02T19:30:11.692676903Z"}
{"log":"goroutine 633 [chan send]:\n","stream":"stderr","time":"2024-04-02T19:30:11.692875409Z"}
{"log":"runtime.gopark(0x47b6a5?, 0x0?, 0x60?, 0xbc?, 0xc15d464e40?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692883935Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc15d464dc0 sp=0xc15d464da0 pc=0x43bbf6\n","stream":"stderr","time":"2024-04-02T19:30:11.692887655Z"}
{"log":"runtime.chansend(0xc000b1e060, 0xc15d464fb8, 0x1, 0x1?)\n","stream":"stderr","time":"2024-04-02T19:30:11.69289238Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:259 +0x42e fp=0xc15d464e48 sp=0xc15d464dc0 pc=0x40618e\n","stream":"stderr","time":"2024-04-02T19:30:11.692906112Z"}
{"log":"runtime.chansend1(0xc27b200000?, 0x5?)\n","stream":"stderr","time":"2024-04-02T19:30:11.692912679Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:145 +0x1d fp=0xc15d464e78 sp=0xc15d464e48 pc=0x405d3d\n","stream":"stderr","time":"2024-04-02T19:30:11.692916603Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL.func4()\n","stream":"stderr","time":"2024-04-02T19:30:11.692920215Z"}
{"log":"\u0009/app/tsdb/head_wal.go:167 +0x27f fp=0xc15d464fe0 sp=0xc15d464e78 pc=0x275233f\n","stream":"stderr","time":"2024-04-02T19:30:11.69292521Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:30:11.692929505Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc15d464fe8 sp=0xc15d464fe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:30:11.692947879Z"}
{"log":"created by github.com/prometheus/prometheus/tsdb.(*Head).loadWAL\n","stream":"stderr","time":"2024-04-02T19:30:11.692960089Z"}
{"log":"\u0009/app/tsdb/head_wal.go:138 +0x81f\n","stream":"stderr","time":"2024-04-02T19:30:11.692963412Z"}


### ac43b1045eec/ ReStart
> without exception

{"log":"ts=2024-04-02T19:30:19.584Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=21\n","stream":"stderr","time":"2024-04-02T19:30:19.585075672Z"}
{"log":"ts=2024-04-02T19:30:52.136Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=1 maxSegment=21\n","stream":"stderr","time":"2024-04-02T19:30:52.137481527Z"}
{"log":"ts=2024-04-02T19:31:15.598Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=2 maxSegment=21\n","stream":"stderr","time":"2024-04-02T19:31:15.598929524Z"}


### bbbd91aca512/ Crash without restart
> out of memory+ wal+ io.wait


{"log":"ts=2024-04-02T19:32:16.464Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=22\n","stream":"stderr","time":"2024-04-02T19:32:16.464315089Z"}
{"log":"ts=2024-04-02T19:32:49.074Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=1 maxSegment=22\n","stream":"stderr","time":"2024-04-02T19:32:49.074511013Z"}
{"log":"ts=2024-04-02T19:33:12.339Z caller=head.go:755 level=info component=tsdb msg=\"WAL segment loaded\" segment=2 maxSegment=22\n","stream":"stderr","time":"2024-04-02T19:33:12.340180494Z"}

{"log":"fatal error: runtime: out of memory\n","stream":"stderr","time":"2024-04-02T19:33:47.645489588Z"}
{"log":"\n","stream":"stderr","time":"2024-04-02T19:33:47.648009813Z"}
{"log":"runtime stack:\n","stream":"stderr","time":"2024-04-02T19:33:47.648024033Z"}
{"log":"runtime.throw({0x34e9140?, 0x0?})\n","stream":"stderr","time":"2024-04-02T19:33:47.663509632Z"}
{"log":"\u0009/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc00008bdf8 sp=0xc00008bdc8 pc=0x438e9d\n","stream":"stderr","time":"2024-04-02T19:33:47.666428858Z"}
{"log":"runtime.sysMapOS(0xc3a9c00000, 0x400000?)\n","stream":"stderr","time":"2024-04-02T19:33:47.682643335Z"}
{"log":"\u0009/usr/local/go/src/runtime/mem_linux.go:187 +0x11b fp=0xc00008be40 sp=0xc00008bdf8 pc=0x4183bb\n","stream":"stderr","time":"2024-04-02T19:33:47.685212727Z"}
{"log":"runtime.sysMap(0xffff800000000000?, 0x7fffffffffff?, 0x7f855206a000?)\n","stream":"stderr","time":"2024-04-02T19:33:47.70137973Z"}
{"log":"\u0009/usr/local/go/src/runtime/mem.go:142 +0x35 fp=0xc00008be70 sp=0xc00008be40 pc=0x417d95\n","stream":"stderr","time":"2024-04-02T19:33:47.703810284Z"}
{"log":"runtime.(*mheap).grow(0x57345a0, 0xa?)\n","stream":"stderr","time":"2024-04-02T19:33:47.705884933Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:1522 +0x252 fp=0xc00008bee8 sp=0xc00008be70 pc=0x429eb2\n","stream":"stderr","time":"2024-04-02T19:33:47.706724185Z"}
{"log":"runtime.(*mheap).allocSpan(0x57345a0, 0xa, 0x0, 0xbf?)\n","stream":"stderr","time":"2024-04-02T19:33:47.723233059Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:1243 +0x1b7 fp=0xc00008bf80 sp=0xc00008bee8 pc=0x4295f7\n","stream":"stderr","time":"2024-04-02T19:33:47.725511761Z"}
{"log":"runtime.(*mheap).alloc.func1()\n","stream":"stderr","time":"2024-04-02T19:33:47.725777339Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:961 +0x65 fp=0xc00008bfc8 sp=0xc00008bf80 pc=0x4290a5\n","stream":"stderr","time":"2024-04-02T19:33:47.725781427Z"}
{"log":"runtime.systemstack()\n","stream":"stderr","time":"2024-04-02T19:33:47.737395442Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc00008bfd0 sp=0xc00008bfc8 pc=0x46dd29\n","stream":"stderr","time":"2024-04-02T19:33:47.742621344Z"}
{"log":"\n","stream":"stderr","time":"2024-04-02T19:33:47.742809237Z"}
{"log":"goroutine 292 [running]:\n","stream":"stderr","time":"2024-04-02T19:33:47.742812912Z"}
{"log":"runtime.systemstack_switch()\n","stream":"stderr","time":"2024-04-02T19:33:47.742820542Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc0005e6a00 sp=0xc0005e69f8 pc=0x46dcc0\n","stream":"stderr","time":"2024-04-02T19:33:47.742824107Z"}
{"log":"runtime.(*mheap).alloc(0x14000?, 0xa?, 0xc0?)\n","stream":"stderr","time":"2024-04-02T19:33:47.742827334Z"}
{"log":"\u0009/usr/local/go/src/runtime/mheap.go:955 +0x65 fp=0xc0005e6a48 sp=0xc0005e6a00 pc=0x428fe5\n","stream":"stderr","time":"2024-04-02T19:33:47.742849236Z"}
{"log":"runtime.(*mcache).allocLarge(0x403c85?, 0x13fd0, 0x0)\n","stream":"stderr","time":"2024-04-02T19:33:47.742852276Z"}
{"log":"\u0009/usr/local/go/src/runtime/mcache.go:234 +0x85 fp=0xc0005e6a90 sp=0xc0005e6a48 pc=0x416e25\n","stream":"stderr","time":"2024-04-02T19:33:47.742855113Z"}
{"log":"runtime.mallocgc(0x13fd0, 0x30f2440, 0x1)\n","stream":"stderr","time":"2024-04-02T19:33:47.742858194Z"}
{"log":"\u0009/usr/local/go/src/runtime/malloc.go:1053 +0x4fe fp=0xc0005e6af8 sp=0xc0005e6a90 pc=0x40db3e\n","stream":"stderr","time":"2024-04-02T19:33:47.742860982Z"}
{"log":"runtime.newarray(0x7f838d5ca148?, 0x7f855208a108?)\n","stream":"stderr","time":"2024-04-02T19:33:47.742863901Z"}
{"log":"\u0009/usr/local/go/src/runtime/malloc.go:1276 +0x52 fp=0xc0005e6b20 sp=0xc0005e6af8 pc=0x40e0b2\n","stream":"stderr","time":"2024-04-02T19:33:47.742866834Z"}
{"log":"runtime.makeBucketArray(0xc3a9b60d38?, 0x0?, 0xc0005e6bd0?)\n","stream":"stderr","time":"2024-04-02T19:33:47.742869778Z"}
{"log":"\u0009/usr/local/go/src/runtime/map.go:363 +0x18e fp=0xc0005e6b60 sp=0xc0005e6b20 pc=0x40ef0e\n","stream":"stderr","time":"2024-04-02T19:33:47.742872594Z"}
{"log":"runtime.hashGrow(0x1?, 0xc000a58e40)\n","stream":"stderr","time":"2024-04-02T19:33:47.742890957Z"}
{"log":"\u0009/usr/local/go/src/runtime/map.go:1051 +0x79 fp=0xc0005e6ba0 sp=0xc0005e6b60 pc=0x4109b9\n","stream":"stderr","time":"2024-04-02T19:33:47.742893708Z"}
{"log":"runtime.mapassign_fast64(0x3020a20, 0xc000a58e40, 0x2a6948bec54bc471)\n","stream":"stderr","time":"2024-04-02T19:33:47.742896544Z"}
{"log":"\u0009/usr/local/go/src/runtime/map_fast64.go:157 +0xdc fp=0xc0005e6be0 sp=0xc0005e6ba0 pc=0x41249c\n","stream":"stderr","time":"2024-04-02T19:33:47.742899327Z"}
{"log":"github.com/prometheus/prometheus/tsdb.seriesHashmap.set(...)\n","stream":"stderr","time":"2024-04-02T19:33:47.742902113Z"}
{"log":"\u0009/app/tsdb/head.go:1656\n","stream":"stderr","time":"2024-04-02T19:33:47.742904968Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet(0xc0006ee3c0, 0x2a6948bec54bc471, {{0xc373e13d10?, 0xc000581748?}}, 0xc0005e6d00)\n","stream":"stderr","time":"2024-04-02T19:33:47.742907759Z"}
{"log":"\u0009/app/tsdb/head.go:1851 +0x525 fp=0xc0005e6cc0 sp=0xc0005e6be0 pc=0x2741445\n","stream":"stderr","time":"2024-04-02T19:33:47.742910699Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).getOrCreateWithID(0xc000648400, 0xd0d4fc, 0xe4?, {{0xc373e13d10?, 0xe3?}})\n","stream":"stderr","time":"2024-04-02T19:33:47.742929165Z"}
{"log":"\u0009/app/tsdb/head.go:1616 +0x86 fp=0xc0005e6d38 sp=0xc0005e6cc0 pc=0x2740006\n","stream":"stderr","time":"2024-04-02T19:33:47.742931828Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).loadWAL(0xc000648400, 0xc176180000, 0x3d08140?, 0xc00001c1e0, 0xc00001c210)\n","stream":"stderr","time":"2024-04-02T19:33:47.742934414Z"}
{"log":"\u0009/app/tsdb/head_wal.go:240 +0x226a fp=0xc0005e72c0 sp=0xc0005e6d38 pc=0x27517ca\n","stream":"stderr","time":"2024-04-02T19:33:47.742937108Z"}
{"log":"github.com/prometheus/prometheus/tsdb.(*Head).Init(0xc000648400, 0x3d27140?)\n","stream":"stderr","time":"2024-04-02T19:33:47.742955189Z"}
{"log":"\u0009/app/tsdb/head.go:748 +0x1a9f fp=0xc0005e7a80 sp=0xc0005e72c0 pc=0x27398bf\n","stream":"stderr","time":"2024-04-02T19:33:47.742957942Z"}
{"log":"github.com/prometheus/prometheus/tsdb.open({0x7ffdf32fcf5c, 0xb}, {0x3d0af20, 0xc0008efb80}, {0x3d27140?, 0xc00013cf00}, 0xc0007120a0, {0xc000207ea0, 0xa, 0xa}, ...)\n","stream":"stderr","time":"2024-04-02T19:33:47.743787991Z"}
{"log":"\u0009/app/tsdb/db.go:888 +0xe27 fp=0xc0005e7ce8 sp=0xc0005e7a80 pc=0x2724c87\n","stream":"stderr","time":"2024-04-02T19:33:47.743794171Z"}
{"log":"github.com/prometheus/prometheus/tsdb.Open({0x7ffdf32fcf5c, 0xb}, {0x3d0af20, 0xc0008efb80}, {0x3d27140, 0xc00013cf00}, 0xc000133ac0?, 0x4?)\n","stream":"stderr","time":"2024-04-02T19:33:47.744673599Z"}
{"log":"\u0009/app/tsdb/db.go:681 +0x9b fp=0xc0005e7d58 sp=0xc0005e7ce8 pc=0x2723bbb\n","stream":"stderr","time":"2024-04-02T19:33:47.744684564Z"}
{"log":"main.openDBWithMetrics({0x7ffdf32fcf5c, 0xb}, {0x3d0afc0?, 0xc000ac0d80?}, {0x3d27140?, 0xc00013cf00?}, 0x0?, 0x0?)\n","stream":"stderr","time":"2024-04-02T19:33:47.746033845Z"}
{"log":"\u0009/app/cmd/prometheus/main.go:1166 +0x10d fp=0xc0005e7e38 sp=0xc0005e7d58 pc=0x28c846d\n","stream":"stderr","time":"2024-04-02T19:33:47.746042425Z"}
{"log":"main.main.func24()\n","stream":"stderr","time":"2024-04-02T19:33:47.748177199Z"}
{"log":"\u0009/app/cmd/prometheus/main.go:1031 +0x24b fp=0xc0005e7f88 sp=0xc0005e7e38 pc=0x28c5c0b\n","stream":"stderr","time":"2024-04-02T19:33:47.748187133Z"}
{"log":"github.com/oklog/run.(*Group).Run.func1({0xc000200840?, 0xc0000e9e90?})\n","stream":"stderr","time":"2024-04-02T19:33:47.752939773Z"}
{"log":"\u0009/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0005e7fc0 sp=0xc0005e7f88 pc=0x9040ef\n","stream":"stderr","time":"2024-04-02T19:33:47.752951354Z"}
{"log":"github.com/oklog/run.(*Group).Run.func2()\n","stream":"stderr","time":"2024-04-02T19:33:47.752966549Z"}
{"log":"\u0009/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0005e7fe0 sp=0xc0005e7fc0 pc=0x90408e\n","stream":"stderr","time":"2024-04-02T19:33:47.752969997Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2024-04-02T19:33:47.752973157Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005e7fe8 sp=0xc0005e7fe0 pc=0x46fda1\n","stream":"stderr","time":"2024-04-02T19:33:47.752976249Z"}
{"log":"created by github.com/oklog/run.(*Group).Run\n","stream":"stderr","time":"2024-04-02T19:33:47.752979379Z"}
{"log":"\u0009/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69\n","stream":"stderr","time":"2024-04-02T19:33:47.752982323Z"}
@bwplotka
Copy link
Member

Is it perhaps related to #13939, so if the Prometheus OOMs you have to delete the WAL (and reduce your scrape load) OR change memory limits if you want to recover from the OOM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yqjdcyy @bwplotka