Examine handling of u option (Unbuffered stdio)

[codewarrior0]

In pyi_pylib_set_runtime_opts, we handle the u option by calling setbuf and _setmode on the stdio streams. This is probably a complete no-op on Windows. These set the buffering options for the stdio of the C Run-time library (CRT) that is linked to the bootloader, but the Python DLL is linked to a different version of the CRT. It will create its own stdio streams wrapping the underlying Windows HANDLEs, and the HANDLEs themselves have no such buffering.

We need to get the CRT functions and stdio objects that the Python DLL is linked to in order to change the buffering mode, or else find a way to change the buffering mode through Python API calls or executing Python code.

[codewarrior0]

I haven't looked at this much because I think it's a very low priority. Windows users generally create GUI apps and don't care much about the terminal output, which is probably why nobody has complained about it.

Possible python code to execute after the pylib is initialized looks something like this:

import ctypes
NULL = ctypes.c_void_p(0)
crt = ctypes.cdll.msvcrt
if UNBUFFERED:
  crt.setbuf(0, NULL)
  crt.setbuf(1, NULL)
  crt.setbuf(2, NULL)

The ctypes.cdll.msvcrt correctly loads the CRT linked to the pylib rather than the one linked to the bootloader.

[codewarrior0]

Actually it's a bit trickier than that. On Python 2, python relies on the buffering of the underlying FILE * provided by the CRT, but on Python 3, it bypasses the FILE * and operates on OS file descriptors directly via the _io.BufferedWriter class. It will take some time to come up with a solution for both 2 and 3.

[htgoebel]

This does not seem to have any priority to anybody, thus I'm removing it from the milestones plan.

[stevievb]

This is an issue for me. I'm on windows. I have two python programs built with pyinstaller. I'm trying to run one of them as a subprocess of the other one, and monitor the real time output of the subprocess via a PIPE. Because the stdout is always buffered regardless of the -u option, I can't communicate between the two processes in real time.

@codewarrior0 - Is the code snippet you pasted above valid for python 2? If so, where exactly could I stick that to makes things work?

[paniabhisek]

I've created a python 3-django package using pyinstaller.

As mentioned here, I've added options = [('u', None, 'OPTION')] and options in exe variable in the spec file.

After that I'm creating a docker image which will use the above package to start. But I've no idea whether the server has started or not unless I hit any request when it shows Running on ... log.

Is the unbuffered option works here. If not what can I do to make it work ?

[htgoebel]

One needs to learn how Python 2.7, 3.4, 3.5, 3.6 and 3.7 handle this option in Python's C-code. Then one needs to provide a pull-request.

[Legorooj]

Pinging.

[capsocrates]

I am encountering this issue as well.

[jharrang]

It would be helpful to remove the 'u' runtime option from the documentation if it is not working or clarify the subset of conditions under which it works. The current documentation is very misleading given that this appears to be functionally not yet implemented.

[htgoebel]

@jharrang Pull request welcome :-)

[jharrang]

@jharrang Pull request welcome :-)

@htgoebel - Perhaps you unintentionally missed the point? I'm not suggesting that this needs to be implemented. While that would be great, it's obviously not been a priority.

What should be a high priority and is an easy fix would be updating the documentation to better reflect the current state of the package. I am not sufficiently well-versed in the history of this issue or the current state of the codebase to make that revision, but I'm happy to open a new issue so that it can be assigned to someone who is if you prefer. Thanks!

[htgoebel]

@jharrang Even for the documentation a pull-request is welcome. Or for the "Known issues section of the Changelog".

[CDFriend]

Same problem here. Would love to see a fix for this.