New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Socket removed after restart #1988
Comments
@mcg, Trying to tie all this together. I'm looking at #1842 (comment), and confused, as that's Puma 4.0.1, and it seems to be working. Hence, what am I missing? If you have any logs, they may be helpful... |
@MSP-Greg #1842 (comment) isn't working for me, that's my non-working config. Logs look "normal" tho the socket gets removed: puma.service log
puma_access.log
puma_error.log
|
@gingerime @mcg A decision was made to remove UNIXSocket files, and the change took effect in 4.2.0. I believe this may affect systemd when (and only when) a stop/start is done. I think, but I'm not sure, that it should not affect puma 'phased-restart' or puma 'restart'. JFYI: Puma 'phased-restart' leaves all the listener sockets open. Puma 'restart' replaces itself via Kernel.exec, and doesn't remove the UNIXSocket files. If the UNIXSocket files exists, but is not connected to a UNIXServer, Puma will delete it and start as it should. |
This has been broken since, 4.0.0. not 4.2.0. This effects systemd when used as documented with socket activation and hot restarts. If this change is intentional, it needs to be documented and marked as a breaking change. |
@MSP-Greg It's tricky for me to reproduce this, but I believe this also affects phased-restart EDIT: to clarify, my systemd script runs EDIT2: my repro was focusing on the restart (not phased-restart) issue so far, which was easier to reproduce... I can try to also reproduce the phased-restart issue, but this might be trickier. |
Referencing @gingerlime repo that seems to reproduce this here so it's not lost. https://github.com/gingerlime/puma-test-socket |
May be an unrelated code change but while trying to create a reproducible VM for this I notice that the
For 3.21
|
Thanks for the info. All this is helpful. But, I'm only windows locally, and using systemd is something that hasn't been implemented in CI yet. Maybe another Actions workflow can test it, not sure... From above:
From @gingerlime's https://github.com/gingerlime/puma-test-socket
Those are different. As I tried to mention above, the code paths for start/stop/start, start/restart, and start/phased-restart are all different. Also, Puma's existing CI is mostly 'signal', but people seem to be using pumactl socket control with systemd. I mention this because I've been working in my Puma fork on an updated test framework, and there are some differences between control behavior, dependent on whether signal or socket used. So, at least from my perspective, it's very important to know what commands/protocl were used with logs showing problems. IOW, start/stop/start, start/restart, or start/phased-restart, and whether signal or socket. Small point, I've seen 3.12 mentioned, but 3.12.0 and 3.12.1 were released about 9 months apart. I haven't really reviewed the differences, but the exact version may be helpful to know.... I can't speak for @nateberkopec, but I suspect he feels the same way I do. I strongly want to get Puma working and stable in all configurations. For instance, I've got a lot of test changes (improvements?) , but I don't want to push them until Puma is bug free and stable... Finally, what you've shown above I'll see if I can repo without systemd... |
The documented systemd config is using signals. It seems almost like a waste of time to try and reproduce without systemd, at least for my issue as it's using systemd's socket activation. |
Thanks. I've looked at the systemd files a bit more, and found some systemd man pages (haven't read yet). A recent change affected closing UNIXSocket files, PR #1992 reverts the recent change. There is one more place where this is done, it was added between 3.12.1 and 4.0.0. I haven't reverted that, but if this doesn't change anything, we can try that next. I'm assuming that systemd will consider UNIXSockets 'handled' if the file exists, and that the issue now is that because the files have been removed, it's trying to 'hold' them? |
Thanks for checking. I know this is a PITA. I mentioned I'm working on revising some of the test framework, and I've added a lot of tests just because. Puma is stable, this issue is particular to using Puma with systemd. Excepting the log issue that I haven't looked at. The other place where files are unlinked only happens when Otherwise, the remaining Sanity check. You're issue is with start(1)/stop/start(2) and start(2) is missing the socket? |
@MSP-Greg Correct. Starting with a system that Puma is not running on. If I |
Thanks. Not going crazy.
The above seems to indicate that Puma is being killed by maybe I don't know if you can try a JFYI, I'm looking at |
I also meet this issue without using systemd |
|
Thanks for checking. After looking at both Puma & systemd docs last night, I may have an idea where the issue is, but that's based on some assumptions that may be incorrect. I need to check some things with Puma first, as I'm going to see if I can mimic the LISTEN_FDS used by systemd, as that's where the info re activated sockets is contained. Overall, there is one issue that I am confused about regarding start/stop/start operations not working. The implication is that systemd is providing a different environment on the second start than it is on the first. Why is that? JFYI,
BTW, sorry for all the info. but months from now, if there are issues with systemd, maybe I or someone else may find them helpful... |
Found a bad workaround. |
Perhaps this helps find where the socket is being deleted. With socket deletion blocked by
|
Thanks. Hate to ask, are you sure you tested off of #1992? The first diff shown in the first commit removes |
With #1992
|
Thank you. Remember how I said in #1988 (comment) that there was a 2nd instance where the files were unlinked, but I thought it didn't affect this use case? I was wrong. EDIT: You've been very helpful and patient. Thank you. Personally, I'd like to try to mimic some of the environment settings used by systemd in the test suite, I may have some questions. |
I think I know what changes are needed for this. Puma has always worked fine if the UNIXSocket files were not removed, so I think we should remove all code that does so. Maybe even test for that. But, I think there are some duplicate test method names, so maybe we should use file names that also include the test class... You're working on some refactoring, I'm doing the same with the tests and also Maybe get this, #1986, and whatever else patched, do a point release, let the dust settle, and start back on the refactoring work? Also, maybe set JRuby back to 'allow-failure' until it can be made stable? After all the work on tests, always having to check logs, etc, I'd really like to see a few green checks... |
Using systemd and hot reloading as described(https://github.com/puma/puma/blob/master/docs/systemd.md#socket-activation) the puma socket file is removed when
systemctl restart puma.service
is issues. This worked until Puma version > 3.12.ruby 2.6.3
puma 4.2.0
Puma config:
puma.service:
puma.socket:
To Reproduce
systemctl restart puma.service
The text was updated successfully, but these errors were encountered: