Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manually compact GC before fork #2093

Merged
merged 1 commit into from Dec 18, 2019
Merged

Manually compact GC before fork #2093

merged 1 commit into from Dec 18, 2019

Conversation

schneems
Copy link
Contributor

@schneems schneems commented Dec 18, 2019

Rub 2.7.0 introduced GC.compact which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other before_fork hooks are called.

Description

Please describe your pull request. Thank you for contributing! You're the best.

Your checklist for this pull request

  • I have reviewed the guidelines for contributing to this repository.
  • I have added an entry to History.md if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added [changelog skip] the pull request title.
  • I have added appropriate tests if this PR fixes a bug or adds a feature.
  • My pull request is 100 lines added/removed or less so that it can be easily reviewed.
  • If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
  • If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
  • I have updated the documentation accordingly.
  • All new and existing tests passed, including Rubocop.

@MSP-Greg
Copy link
Member

JFYI, using ruby master in actions:

https://github.com/ruby/psych/blob/master/.github/workflows/ubuntu-rvm.yml

@liaden
Copy link

liaden commented Dec 18, 2019

Would it be worthwhile to add a GC.start to increase the resulting compactness? Edit: Nevermind, it already invokes GC https://github.com/ruby/ruby/blob/master/gc.c#L8438

At work, we often will use eye to monitor child processes for memory bloat and have eye kill them. It would be beneficial to have this only done on the first fork instead of every time we fork.

History.md Outdated Show resolved Hide resolved
@schneems schneems force-pushed the schneems/compact-gc branch 2 times, most recently from 9c46fea to 020e4d8 Compare December 18, 2019 17:23
@schneems
Copy link
Contributor Author

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8 with just

Run actions/setup-ruby@v1
  with:
    ruby-version: 2.7.0-preview3
##[error]Version 2.7.0-preview3 not found
##[error]Node run failed with exit code 1

Any idea how to debug or what needs to be changed @MSP-Greg ?

@MSP-Greg
Copy link
Member

@schneems

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8

At present, standard GitHub actions just has Ruby 2.4.x, 2.5.x, & 2.6.x installed. Anything else needs code using a container, installing and using rvm, or a special action step.

@dentarg & PR #2087 adds Ruby 2.3 in a container. The Ruby/Psych code I linked to adds ruby-head using rvm. I can add 2.3 & ruby-head to Windows, but that won't run this code as fork is not available...

Rub 2.7.0 introduced `GC.compact` which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

- https://www.ruby-lang.org/en/news/2019/12/17/ruby-2-7-0-rc1-released/
- https://www.youtube.com/watch?v=1F3gXYhQsAY

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other `before_fork` hooks are called.
@MSP-Greg
Copy link
Member

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

@schneems
Copy link
Contributor Author

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

Seems good!

@schneems schneems merged commit b646fc5 into master Dec 18, 2019
@dentarg dentarg deleted the schneems/compact-gc branch February 10, 2021 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants