New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection closed prematurely #1529
Comments
You could do something like this: <?php
include('Net/SSH2.php');
$ssh = new Net_SSH2('www.website.com');
$ssh->login('username', 'password');
$ssh->write("ping 127.0.0.1\n");
echo $ssh->exec('sleep 3600s; ls -latr'); |
@terrafrost Thanks, but really is there is or gonna be a fix for this? I currently solved my problem by increasing ClientAliveInterval. All good now. |
idk why libssh2 doesn't time out whereas phpseclib does. What'd be more useful is if PuTTY doesn't time out as I can get the SSH packet logs from that. That said, in lieu of having looked into it, it's quite possible that other SSH clients that don't time out are employing multiple threads. ie. one thread waits for responses whereas another thread sends periodic SSH2_MSG_IGNORE packets or something. In that scenario, duplicating that behavior would be quite difficult as PHP isn't multi threaded. Anyway, I'll try to look into it at some point. |
Hi, I got this when using it on
and there is no ssh log, any solution? |
I added a https://github.com/terrafrost/phpseclib/tree/keepalive-2.0 LMK if it solves the problem for y'all. Thanks! |
Hi.. I downloaded |
Did it send NET_SSH2_MSG_IGNORE packets every x seconds as you'd expect? You can verify this by doing The NET_SSH2_MSG_IGNORE approach is what PuTTY does when you enter in a non-zero value for "Seconds between keepalives (0 to turn off)" in the Connections "tab". OpenSSH seems to employ another approach - it sends a keepalive@openssh.com SSH2_MSG_CHANNEL_REQUEST which will most likely result in the other end sending a SSH2_MSG_CHANNEL_FAILURE response. I tried to reproduce the issue locally by running a long running command. I disabled ServerAliveInterval and ServerAliveCountMax (or rather, I set them to 0) and then tried to run a long running command and I didn't get any timeouts with either PuTTY or the OpenSSH client: https://superuser.com/q/1588942/172193 In the case of PuTTY the server kept on sending the keepalive@openssh.com SSH2_MSG_CHANNEL_REQUEST even after ServerAliveCountMax had been passed. That's probably what's happening with the OpenSSH client as well. Like in neither cases is the client actually doing anything to keep the connection alive - it's the server that's doing all the work. But in your case the server I guess isn't doing any work after ServerAliveCountMax has been hit? If so then that is not something that I've been able to reproduce... |
@terrafrost I'm getting a similar issue with version 3 of the library when trying to execute a long running script (with constant output so it's not an idle connection). I tried implementing the keepalive-2.0 changes but I don't directly see any keep alive packages being send. If I read the code correctly the keepalive "NET_SSH2_MSG_IGNORE" is only send when I believe your last paragraph is correct since I checked the putty settings and see that the keep alive there is 0 (no keep alive is send) for the session that I use to connect to this server and timeouts are happening. So something must be missing to keep the connection alive. I'm willing to do some research but my knowledge of the SSH protocol and working with streams is pretty limited. |
@terrafrost I was just able to execute a 50 minutes script by sending the NET_SSH2_MSG_IGNORE package at the keepalive intervals. As soon as I have some time I will implement it on a fork so you can have a look and share me your opinion on it. I've made a fork with my changes here: |
@Echron - I'll check it out - thanks!! |
This also seems to have been working in version 2.0.13. I am currently updating a project using laravelcollective/remote and updating has this issue but the old version of the application does not. |
@terrafrost Off the record: are you using any automated (end to end) tests? Besides this ticket that is covering a rare bug/functionality, I've been going through your code and there are so many cases and statements. I see in you are planning on refactoring a lot, there should be some ways to test most of this. |
@Echron - there are unit tests but a lot of the edge cases I've encountered over the years there's no easy way to test. Consider the window size handling stuff. I have absolutely no idea how to control the window size that OpenSSH uses. I suppose I could modify the OpenSSH source code and then have 15 different compiled versions but I am not going to do that. A lot of issues I've never actually been able to test first hand either. Someone posts log files and I develop a hypothesis from those log files and then I make a change in my fork of phpseclib and I ask the person if that change fixes the issue for them. If it does I merge it in, in many cases, never having been able to reproduce it directly myself. I suppose I could write a pure-PHP SSH server to test some of these conditions, but again, I am absolutely not going to do that. That would be a gargantuan task that I'm not even sure would be that beneficial as all I'd be doing is emulating my understanding of what problem server is doing (which might not be what problem servers are actually doing) |
@Echron - like even this one. I can't even reproduce this issue. https://superuser.com/q/1588942/172193 documents my attempt to do so which was a bit of a dead end. If I can't reproduce on my personal laptop then it stands to reason that reproducing it on Travis CI or any other CI platform is going to be beyond my ability! |
@terrafrost This does not happen when I run ssh directly with popen. I had to switch to this in the meantime since I have some ssh commands that run and do not return any data back for a long time. |
@Xachman - that doesn't help me reproduce the issue. Anyway, as I said, I'll look at this hopefully within the next few days. My most immediate priority is getting phpseclib unit tested on PHP 8, which has now been out for like a week. |
@terrafrost Sorry I thought the issue itself was pretty easy to reproduce with the sleep command you posted without the ping. Are you saying you have never experienced this issue? I'm sure I could throw together a environment where this happens using docker if that's the case. |
I'm able to semi reproduce the issue with this Vagrantfile (it's easier to setup SSH with Vagrant than it is with Docker):
BUT https://github.com/terrafrost/phpseclib/tree/keepalive-2.0 works with that Vagrantfile:
If I comment out And yet, despite working for me, it doesn't work for @Echron . So that's what I'm not able to reproduce. idk how to set OpenSSH up such that my proposed fix doesn't work... |
I backported Echron's keep-alive code to the 1.0 branch here: https://github.com/terrafrost/phpseclib/tree/alt-keep-alive-1.0 In comparing that to my own implementation (https://github.com/terrafrost/phpseclib/tree/keepalive-1.0)... Consider the scenario where the timeout is 0 (ie. nothing should timeout) and keep alive is non-zero. Echron's implementation checks to see if more than x seconds have passed since the last keep-alive packet was sent and, if so, sends the keep-alive packet. My implementation, otoh, sends the keep-alive packet after waiting for up to x seconds for the next packet to become available (via the select system call). x, in this case, is the keep-alive amount. It looks like his implementation is largely adapted from my own. Using the test code and Vagrantfile I posted in my last post, here's the relevant parts of the SSH logs from Echron's implementation:
Here's the relevant portion from my implementation:
So one thing to note is that my implementation sends a NET_SSH2_MSG_IGNORE message every 2s (which is consistent with the setKeepAlive parameter). Echron's implementation sends it out every 3s (ClientAliveInterval in the Vagrantfile). ie. it waits until it's gotten a NET_SSH2_MSG_CHANNEL_REQUEST packet to send it's NET_SSH2_MSG_IGNORE. This is because his code can't actually "pause" it's So what about when the phpseclib keep-alive interval is greater than the server's ClientAliveInterval? eg. instead of In this case both implementations get a "Connection closed prematurely" error. I'll mull it over some more but I'm currently leaning towards going with my implementation, altho I'd still like to get it working for the use case where both are currently failing. |
Another scenario to consider: what if ClientAliveCountMax is 0? My implementation works (with For the scenario when the keep-alive interval is greater than the server's ClientAliveInterval... I think making it so phpseclib return a NET_SSH2_MSG_IGNORE packet when the server sends a keep-alive@openssh.com NET_SSH2_MSG_CHANNEL_REQUEST would be good. That's not going to work if ClientAliveCountMax is also 0 but there's only so much that I figure can realistically be done! |
I have four new branches with these improvements: https://github.com/terrafrost/phpseclib/tree/keepalive-1.0 The unit tests are running right now. If they pass I'll merge this into phpseclib/phpseclib this evening and will release a new version of 2.0 and do the maiden release of 3.0. In the mean time, feel free to test these changes out! |
https://github.com/Xachman/test-ssh Using this test environment with the standard php:7 environment and ubuntu trusty as the environment you ssh into the current phpseclib 2.0 connection drops with $ssh->exec('sleep 300; ls -al'); before outputting the ls. Then trying with the keepalive-2.0 branch it just seemed to drop faster. running ssh root@ubuntu "sleep 300; ls -al" works fine. |
I'm not seeing
You could empirically test that by doing |
I tried with |
I use Docker a fair amount (see https://github.com/phpseclib/docker-php) but have never used docker-compose. I did the following:
Here's what I got in response:
I've Google'd that without a ton of success. None of the following links seem very helpful: https://forums.docker.com/t/docker-compose-yml-is-invalid-because-unsupported-config/51463 From the links I gather indentation matters but I feel like if all I did was In lieu of that... can you do Thanks! |
Well, actually,
|
Looking at your code I see one other thing you should do... you'll need to adjust the timeout as well. You can disable the timeout by doing If you're still having issues update https://github.com/Xachman/test-ssh/blob/main/php-ssh/test_ssh.php accordingly and commit your changes. Since you're going this Docker route I feel like I shouldn't have to copy / past anything other than the URL to reproduce the issue - |
@terrafrost I added the setTimeout and that did work was able to sleep for 300 seconds and output the ls. changes are https://github.com/Xachman/test-ssh/blob/main/php-ssh/test_ssh.php Adding the code if anyone needed the example. And I deleted that.
I added a docker-compose version string to the docker-compose.yml in case that was the issue. Could be that you are using an older version of docker-compose but I'm not 100%. If you want to try it out you just make an id_rsa in a folder called build in php-ssh and put the id_rsa.pub in a folder called build in ubuntu. php-ssh could be able to ssh into ubuntu using the docker network service name ubuntu as the host. |
My last comment was a mistake if you got the email on it. Ran the wrong test. |
I was able to confirm the keep alive changes do keep the connection alive to the original AWS EC2 instance I had this issue with. I had to run the test again because my original test used the connection to the local ubuntu container. My environment does not replicate the issue. I was confused because of the set timeout part. These changes do fix the issue for my use case though. Since I came from laravelcollective/remote I will make an issue there once these changes get merged and I would assume they aren't going to be immediately implemented there. |
Thanks for the feedback!! |
The branches have been merged. I'll do a new release shortly! |
I installed 3.0.0 tested it on the servers that gave issues before and everything seems to work really well. Good work! |
Encountered exception: Connection closed prematurely in line: 3195 |
2021-11-02 19:55:39 - getting data from xxrouterhost |
@insiderinside - can you post the SSH logs? You can get them by doing If you could also post your code too, obviously redacting any passwords / public keys or any other sensitive info you might have, that'd be helpful. |
Using:
PHP 7.3.21
phpseclib 2.0.29
I'm trying to run a very long running command (like 1 hour or more) with this function, as this function runs more than ClientAliveInterval * ClientAliveCountMax ~ 10 mins (default) I get error Connection closed prematurely.
I switched to this library from php ssh2 extension, back there I had no problem with this.
https://www.php.net/manual/en/function.ssh2-exec.php
Is there any command or setting to send server that we are still alive as client?
Thanks in advance.
The text was updated successfully, but these errors were encountered: