-
Notifications
You must be signed in to change notification settings - Fork 432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"Killed" in postgres logs and wal-g hangs when push wals to Minio #810
Comments
Thanks for so detailed report! It seems like long hunted S3 problem. It was somewhat fixed in 0.2.17, but instead of hanging it now just stalls for long but finite time. It is still very annoying. I hope your report will help to shed nore light on the problem. Right now I'm not able to debug anything except tuna sandwich, but I hope to get back to tgus issue soon. Thanks! Cheers 🍻 |
I believe it's the same problem as in #656 |
Any news on this? Is this enough for you to replicate the issue? |
Please install v0.2.19 Your report is very good, but it contains traces of a manifestation that is fixed in 0.2.17. Thanks! |
Hey thanks! Just created our own spilo container with latest wal-g with the following Dockerfile. We have the same setup as before, just with new version of wal-g. We didn't observe any problems during wal-push. However backup-push fails almost all the time, here is the logs with WALG_LOG_LEVEL=DEVEL
Sorry it is long and probably unuseful but still ... We'll try to add some load to it to see if that changes anything. |
Probably, it was killed by OOM killer? If so it would be great to know goroutine stacks to understand where mem consumption happens. |
It is running on kubernetes with requests and limits enforced. It would go OOM, k8s would have totally killed the pod if it would you beyond its limits.
I’ll still try to bump up that request and limit just to be sure and try to get a stack trace (even though nothing is logged beyond what I’ve included).
Thanks a lot for your help 😄
Le 13 déc. 2020 à 18:37 +0100, Andrey Borodin <notifications@github.com>, a écrit :
Probably, it was killed by OOM killer? If so it would be great to know goroutine stacks to understand where mem consumption happens.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#810 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC62SRYQSKGDQ4IA6XDKDSDSUT3WZANCNFSM4UF3WQOQ>.
|
Thanks @vfrans ! It would be very good to know why it was killed and is it connected with S3 stalls. We want to fix these S3 problems before releasing 1.0. |
Hello everybody,
We use Minio as our S3 backend to archive WALs using wal-g.
I does work pretty correctly, however, when under load wal-g hangs on some WALs archiving.
No errors are reported on the Minio side weirdly.
We use postgres-operator v1.5.0 with spilo-12:1.6-p5
Also using wal-g version v0.2.15
Here are the postgres logs:
Much other "killed" mentions in the logs.
Here is the minio deployment configuration:
Do you have any clue?
Thanks a lot in advance,
Best regards,
The text was updated successfully, but these errors were encountered: