Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a more aggressive GC watchdog to help in bursty memory usage situations #8798

Open
3 tasks done
Jorropo opened this issue Mar 17, 2022 · 2 comments · May be fixed by #9451
Open
3 tasks done

Add a more aggressive GC watchdog to help in bursty memory usage situations #8798

Jorropo opened this issue Mar 17, 2022 · 2 comments · May be fixed by #9451
Labels
kind/enhancement A net-new feature or improvement to an existing feature P2 Medium: Good to have, but can wait until someone steps up

Comments

@Jorropo
Copy link
Contributor

Jorropo commented Mar 17, 2022

Checklist

  • My issue is specific & actionable.
  • I am not suggesting a protocol enhancement.
  • I have searched on the issue tracker for my issue.

Description

In #8797 it happen often that bitswap use 20+Gigs of ram and will continue even after all requests has been canceled / finished.

However due to how golang GC's work, that lead to an accoutumance to high memory usage, and the GC wont get agressive to reclaim space.

(note the way the GC works, is that the GC tries to run when it is a factor X above the last GC size, so if your last size was 20Gig, golang might as well wait for 40gig of memory usage before gcing)

I belive lotus use https://github.com/raulk/go-watchdog to help in memory bursty conditions, maybe we should try it ? (I'm creating the issue to log any experiments)

@Jorropo Jorropo added the kind/enhancement A net-new feature or improvement to an existing feature label Mar 17, 2022
@BigLep BigLep added the need/author-input Needs input from the original author label Mar 18, 2022
@BigLep
Copy link
Contributor

BigLep commented Mar 18, 2022

@Jorropo : cool, sounds good. We look forward to hearing what you learn from experimenting.

@BigLep BigLep added the P2 Medium: Good to have, but can wait until someone steps up label Jun 3, 2022
@BigLep BigLep removed the need/author-input Needs input from the original author label Jun 10, 2022
@Jorropo
Copy link
Contributor Author

Jorropo commented Nov 30, 2022

Update: go implemented this somewhat recently however it's disabled by default.
So this could be as is as calling debug.SetMemoryLimit with 80% of system ram (or some other metric) when Kubo starts.

Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Dec 5, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue May 3, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue May 4, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
Jorropo added a commit to Jorropo/go-ipfs that referenced this issue Jun 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB.
Then you open the profile and it is using half of that, this is due to the default GOGC=200%.
That means, go will only run the GC once it's twice as being as the previous alive set.

This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.

Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out,
it's not healthy to run at the edge of the limit because the GC will continously run killing performance.
So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.

Closes: ipfs#8798
@BigLep BigLep mentioned this issue Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature P2 Medium: Good to have, but can wait until someone steps up
Projects
Status: 🥞 Todo
Development

Successfully merging a pull request may close this issue.

2 participants