Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explore: Adjust the number of workers based on generation #22

Open
casperisfine opened this issue Jan 31, 2023 · 0 comments
Open

Explore: Adjust the number of workers based on generation #22

casperisfine opened this issue Jan 31, 2023 · 0 comments

Comments

@casperisfine
Copy link
Contributor

This remain to be seen whether we'll actually need this, but would be worth exploring.

The assumption is that the first few generation will invalidate shared pages rather quickly, as such the peak memory usage at first might get quite high.

A possible solution to this would be to start with a lower number of workers to compensate for the lower CoW effectiveness, and then increase the number of workers of processes with each new generation.

This would mean pitchfork needs a bit of a warm up before achieving peak throughput.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant