Fix Channel.close() race condition in pipe.py #2274

pettitpeon · 2023-07-23T19:32:21Z

self._closed = True
was moved to the top. Built-in assignment is python is atomic and there is no need to protect it with a lock.

If there is a Bad file descriptor (errno == 9). We can safely disregard it if ._closed is set

`self._closed = True` was moved to the top. Built-in assignment is python is atomic and there is no need to protect it with a lock. If there is a `Bad file descriptor` (errno == 9). We can safely disregard it if `._closed` is set

pettitpeon · 2023-07-25T08:56:51Z

Fixes this BUG
#2271

bskinn

Couple of questions about the implementation. Thanks for the PR!

bskinn · 2023-08-02T03:33:32Z

paramiko/pipe.py

+        try:
+            os.write(self._wfd, b"*")
+        except OSError as e:
+            if e.errno == 9 and self._closed:
+                # The pipe was closed, no need to do anything
+                return
+            raise e


Do we not also need something analogous to protect the os.read(self._rfd, 1) call above in clear()?

(We might not; again, just raising the question.)

From the pipe's perspective, yes. And it would be wise to protect it. However, the way the pipe is used, this does not happen. The same thread

Opens,

clears and

closes

the pipe. So there is no race condition between .clear() and `.close().
A second thread sets it.

Same request, please add a comment here explaining why a similar race catch is not (currently) required here.

bskinn · 2023-08-02T03:33:33Z

paramiko/pipe.py

        os.close(self._rfd)
        os.close(self._wfd)


@pettitpeon, is there any chance of either of these two .close() calls failing, making self._closed == True a possibly inaccurate representation of the pipe state?

(I would think that a failure of either of these two calls would signal an error state for the pipe such that self._closed == True is a better state than == False... but I thought I'd raise the question.)

is there any chance of either of these two .close() calls failing?

Yes, it could happen. If either .close() fails, it would raise an OSError. In that case some FD might stay open, but the Pipe is not usable anyways. There is no use-case of using it after .close(). Worst case scenario, we leak resources until the application exits. The most probable scenario is that the close() exception is not handled and your program terminates anyways.
A small improvement would be to have single trys on each .close(). In that case if, the first fails and leaks, we might still close the second one correctly.
In any case, I see .close() errors as non-handable, we close() file descriptors as a "best-effort" to release the resource, but cannot really do anything if it fails.

making self._closed == True a possibly inaccurate representation of the pipe state?

Not really, after close() has been called, the object is disposed and no one should it anyways. The "broken state" is meaningless.

Last, the pipe is most probably a child of Channel and when the channel gets destroyed it tries to re-close, which fails if the pipe was previously closed, but it is disregarded by a catch-all try. This is OK, and aligns with the idea that it is a non-handable error.

paramiko/paramiko/channel.py

Line 136 in f4d7fec

def __del__(self):

bskinn · 2023-08-02T03:39:16Z

paramiko/pipe.py

+        try:
+            os.write(self._wfd, b"*")
+        except OSError as e:
+            if e.errno == 9 and self._closed:


Is it possible for this ifcondition to ever be True?

We've already trapped for self._closed == True with the first if in the method, so it seems like we'd only ever hit this point with self._closed == False.

Yes, the flowing (race condition) can happen

The write thread sees ._closed == False and continues

The close thread closes the descriptors before the write thread writes on line 67

The write fails to write() because the FD has been closed.

Aha, this is the crux of the race condition, got it.

For the benefit of future readers of the code, please add a comment here describing the race condition as you just described it.

Otherwise we risk someone thinking the same thing I did and removing this.

bskinn · 2023-08-02T03:45:36Z

I would think it would be challenging to do, but: is there any good way to write a test for this?

I would figure that any such test would require setting up a test harness with a server to connect to, and then run the open/close cycle you describe in the issue for some period of time long enough to trigger the race condition 90%+ of the time... but that would make for a flaky test, and that'd be not great.

Interested in any thoughts you have on it, though.

bskinn · 2023-08-02T03:46:36Z

Oh, also -- I added Needs changelog/docs because this would need an entry in /sites/www/changelog.rst.

pettitpeon · 2023-08-02T08:49:35Z

I would think it would be challenging to do, but: is there any good way to write a test for this?

I will look into this. It might be possible by tunneling/forwarding an ssh connection and sending ssh echos in a loop.

pettitpeon · 2023-08-02T12:26:23Z

I would think it would be challenging to do, but: is there any good way to write a test for this?

I would figure that any such test would require setting up a test harness with a server to connect to, and then run the open/close cycle you describe in the issue for some period of time long enough to trigger the race condition 90%+ of the time... but that would make for a flaky test, and that'd be not great.

Interested in any thoughts you have on it, though.

Unfortunately, I could not think of a way of easily reproducing the bug. I tried tunneling SSH connections through a local forward (demo/forward.py), but SSH is too smart and rejects connections from the server side if it is being spammed (DoS attack mitigation). So I cannot open and close connections fast enough to run into the race condition.

pettitpeon · 2023-08-02T12:55:09Z

/sites/www/changelog.rst

I fixed the review issues. I do not know what to to about the changelog.

paramiko/pipe.py

pettitpeon · 2023-08-17T13:52:02Z

@bskinn @jun66j5 What else shall be done to merge? I fixed all the issues

paramiko/pipe.py

bskinn · 2023-08-17T14:18:53Z

@bskinn @jun66j5 What else shall be done to merge? I fixed all the issues

Minor tweak to the PR, and then I think it's ready to flag for review by bitprophet.

It still needs a CHANGELOG entry - you said you're not sure what to do about that... take a look at /www/sites/changelog.rst to see the syntax & template of entries added for other bugfixes, and adapt to provide a description of this bugfix.

Co-authored-by: Brian Skinn <brian.skinn@gmail.com>

pettitpeon · 2023-08-30T08:33:11Z

@bskinn @jun66j5 What else shall be done to merge? I fixed all the issues

Minor tweak to the PR, and then I think it's ready to flag for review by bitprophet.

It still needs a CHANGELOG entry - you said you're not sure what to do about that... take a look at /www/sites/changelog.rst to see the syntax & template of entries added for other bugfixes, and adapt to provide a description of this bugfix.

Gonna give it a try today
--> DONE. thanks for the orientation. Anything else left to do?

pettitpeon · 2023-09-19T09:38:23Z

Dear @bskinn @jun66j5 @bitprophet, What else is needed for the merge?

thanks!

pettitpeon · 2023-10-10T09:40:09Z

Hi all again! please review the PR when possible

jun66j5 · 2023-10-11T02:50:16Z

paramiko/pipe.py

+        # "best effort" approach
+        try:
+            os.close(self._rfd)
+        except Exception:


Why OSError is not used rather than Exception?

In this case we want to be as broad as possible in the try/except. If any exception is raised during the closing of the rd file descriptor, we still want to guarantee the closing of the wr file descriptor. Catching only one exception type could be too narrow and leave the pipe in an invalid state (one descriptor open and the other one closed). But I won't be able to test my changes on Windows

In this case we want to be as broad as possible in the try/except.

I'm unsure your comment. What I'm saying is:

try: os.close(self._rfd) except Exception: pass try: os.close(self._wfd) except Exception: pass

should be

try: os.close(self._rfd) except OSError: pass try: os.close(self._wfd) except OSError: pass

Yes, the problem with catching OSError is that if os.close() raises a different exception for whatever unknow reason, the pipe could go into an invalid state and potentially leak opened file descriptors.

try: os.close(self._rfd) # imagine this throws RuntimeError except OSError: pass try: os.close(self._wfd) # in that case, the _wrd would not be closed except OSError: pass

I'll leave these excepts as a decision for bitprophet.

jun66j5 · 2023-10-11T02:57:30Z

This PR is to fix issues in PosixPipe. I think WindowsPipe has the same issue. Any plans to fix it?

pettitpeon · 2023-10-11T07:08:17Z

This PR is to fix issues in PosixPipe. I think WindowsPipe has the same issue. Any plans to fix it?

TBH, I did not look into the windows pipe since I use paramiko only in linux. If desired I can look into it and replicate the changes in the WindowsPipe. Still I'd like to merge this fix asap to remove my local patches

bskinn · 2023-10-17T04:06:31Z

I figure this is up to bitprophet whether he wants to merge with just the PosixPipe fix and leave the WindowsPipe fix for a later PR, or roll them all up in one.

bskinn

Formatting nit.

sites/www/changelog.rst

bskinn · 2023-10-17T04:11:37Z

It looks like the two failing CircleCI checks were due to an error on CircleCI's side. Likely all is ok with the test suite here.

Co-authored-by: Brian Skinn <brian.skinn@gmail.com>

pettitpeon · 2023-10-17T08:33:03Z

It looks like the two failing CircleCI checks were due to an error on CircleCI's side. Likely all is ok with the test suite here.

Thanks I applied your formatting suggestion. I'll await the final review. Cheers

bskinn · 2023-10-17T13:23:47Z

Great, thanks! And, thanks for your patience here. bitprophet is in the middle of a dry spell in terms of his open source bandwidth, which has slowed everything down across his projects.

Fix Channel.close() race condition in pipe.py

44cf229

`self._closed = True` was moved to the top. Built-in assignment is python is atomic and there is no need to protect it with a lock. If there is a `Bad file descriptor` (errno == 9). We can safely disregard it if `._closed` is set

pettitpeon mentioned this pull request Jul 23, 2023

[BUG] - forward.py demo: pipe write() after close() [Race Condition] #2271

Open

fix typo

f4d7fec

bskinn reviewed Aug 2, 2023

View reviewed changes

bskinn added Bug Needs changelog/docs labels Aug 2, 2023

pettitpeon added 3 commits August 2, 2023 14:41

Fix review issues

e4d9af9

Fix typo

9b5776a

Fix flake8

70741ac

jun66j5 reviewed Aug 2, 2023

View reviewed changes

paramiko/pipe.py Outdated Show resolved Hide resolved

Use errno.EBADF

6a6f6fb

bskinn suggested changes Aug 17, 2023

View reviewed changes

paramiko/pipe.py Outdated Show resolved Hide resolved

paramiko/pipe.py Outdated Show resolved Hide resolved

pettitpeon and others added 2 commits August 30, 2023 10:32

Update paramiko/pipe.py

bf887b2

Co-authored-by: Brian Skinn <brian.skinn@gmail.com>

Update paramiko/pipe.py

a092146

Co-authored-by: Brian Skinn <brian.skinn@gmail.com>

pettitpeon added 2 commits August 30, 2023 12:11

Update changelog.rst

3cf92eb

Update changelog.rst

d9fd423

jun66j5 reviewed Oct 11, 2023

View reviewed changes

bskinn added Ready for review and removed Needs changelog/docs labels Oct 17, 2023

bskinn reviewed Oct 17, 2023

View reviewed changes

sites/www/changelog.rst Outdated Show resolved Hide resolved

Update sites/www/changelog.rst

917e1ec

Co-authored-by: Brian Skinn <brian.skinn@gmail.com>

Fix Channel.close() race condition in pipe.py #2274

Are you sure you want to change the base?

Fix Channel.close() race condition in pipe.py #2274

Conversation

pettitpeon commented Jul 23, 2023

pettitpeon commented Jul 25, 2023

bskinn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pettitpeon Aug 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskinn commented Aug 2, 2023

bskinn commented Aug 2, 2023

pettitpeon commented Aug 2, 2023

pettitpeon commented Aug 2, 2023

pettitpeon commented Aug 2, 2023

pettitpeon commented Aug 17, 2023

bskinn commented Aug 17, 2023 • edited

pettitpeon commented Aug 30, 2023 • edited

pettitpeon commented Sep 19, 2023

pettitpeon commented Oct 10, 2023

Choose a reason for hiding this comment

pettitpeon Oct 11, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskinn Oct 17, 2023 • edited

Choose a reason for hiding this comment

jun66j5 commented Oct 11, 2023

pettitpeon commented Oct 11, 2023

bskinn commented Oct 17, 2023

bskinn left a comment

Choose a reason for hiding this comment

bskinn commented Oct 17, 2023

pettitpeon commented Oct 17, 2023

bskinn commented Oct 17, 2023

pettitpeon Aug 2, 2023 •

edited

bskinn commented Aug 17, 2023 •

edited

pettitpeon commented Aug 30, 2023 •

edited

pettitpeon Oct 11, 2023 •

edited

bskinn Oct 17, 2023 •

edited