New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mitmdump stdout never reaches parent process #6757
Comments
It's also interesting that the Python docs regarding
Aren't we using "the text layer"? |
It behaves the same with the this ChatGTP Python parent process (just to rule out funny Node.js things): import subprocess
def run_command(command):
try:
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
env=dict(PYTHONUNBUFFERED="1"),
)
while True:
stdout_line = process.stdout.readline().decode().strip()
if stdout_line:
print("stdout:", stdout_line)
# Check if both pipes are closed and process has terminated
if not stdout_line and process.poll() is not None:
break
except Exception as e:
print("Error:", e)
command = "/home/alex/Downloads/mitmproxy-10.2.4-linux-x86_64/mitmdump"
# command = "/home/alex/Downloads/mitmproxy-9.0.1-linux/mitmdump"
# command = "/home/alex/Downloads/mitmproxy-8.1.1-linux/mitmdump"
# command = "/home/alex/Downloads/mitmproxy-7.0.4-linux/mitmdump"
# command = "/home/alex/Downloads/mitmproxy-6.0.2-linux/mitmdump"
# command = "/home/alex/forks/mitmproxy/venv/bin/mitmdump"
run_command(command) |
I don't have a solution but I thought I'd just add my 2 cents since I recently encountered a similar issue when writing a NodeJS script. My first attempt was also setting Obviously it's a dirty hack (and also not portable) but if someone needs a quick workaround on a system that has access to script binary it does the job |
Thank you for your input! I think the reason it works is because from what I understand
That would be my main problem with it. I assume you could achieve similar things using https://github.com/microsoft/node-pty, but I would rather not go that route and add yet another layer that can break. |
Are you sure you set PYTHONUNBUFFERED correctly? Without PYTHONUNBUFFERED stdout is indeed buffered:
But with PYTHONUNBUFFERED stdout seems to be flushed ok:
I have tested on Kali linux with python3 & mitmproxy from packages:
And with latest version of mitmproxy installed with pip in a virtualenv, it behaves the same.
|
I've provided both a Node.js and Python script, I guess I'm setting it correctly in there? Since |
I am not an expert on nodejs… Just offered some observations regarding stdout buffering behaviour of mitmproxy without pty but with PYTHONUNBUFFERED=1. As far as I can tell it behaves as expected. Maybe you should spawn /usr/bin/env instead of mitmproxy and assess its output to see if env gets set as expected. |
I will try your python test script later |
What OS do you use? |
Ubuntu 23.10. So I'd assume Kali should behave the same. |
Your python script invoking mitmproxy works, it captures stdout of mitmdump on my kali. |
I can't reproduce these results at all using ssh -T when I'm sshing into a machine using Mint 21.1 so something must be different on Kali then?
And then the same thing using PYTHONUNBUFFERED
I am using bash though not zsh |
strange. well, maybe try bash. and try getting yourself Kali VM. |
Honestly no idea what's going on but even on fresh Kali VM I'm getting the same results
And then running the same command with PYTHONUNBUFFERED. I only increased sleep length to be safe and make sure I'm not reading file too fast.
|
Curious ;) I still get the same results as yesterday.
My kali is much older. Can you point me to the exact Kali you've downloaded? I get myself fresh one too. |
I got the VirtualBox image on this page https://www.kali.org/get-kali/#kali-virtual-machines. Version is |
I don't have VirtualBox, installed a fresh on ProxMox. Still works for me:
Can you dump all including env vars in one go?
|
This is using system's mitmproxy but I also tested it with latest mitmdump (10.2.4) |
Well, there are still differences between our systems still, but nothing major:
We are likely to use different SSH clients, let's take those out of the equation. Make test.sh script on kali and run it via ssh localhost launched from the graphical console directly, like this: If you still get different results, I suggest maybe you get yourself the exact same image as I did https://kali.download/base-images/kali-weekly/kali-linux-2024-W15-qemu-amd64.7z (SHA1 5dd0c36fd089ecd217432992a3bd13dacdc88f07). |
Ok now I'm finally at a point where I can reproduce your results :). So |
Thank you two for looking into this! It became a little hard to follow, can you confirm that you are both working with the latest mitmproxy downloaded from https://mitmproxy.org/downloads/#10.2.4/ ? Because I don't know how exactly Kali packages mitmproxy (https://gitlab.com/kalilinux/packages/mitmproxy) and if they are using PyInstaller at all. Juding from issues such as https://bugs.kali.org/view.php?id=8695 I don't think they are, so it's closer to running the venv (which respects |
Sorry I didn't even realize there might be a difference so all this testing was kind of pointless 😬 What I did was tested on @Prinzhorn Your assumption seems to be correct. I rechecked both Kali builds again and in both of them |
Nice, thanks for confirming! Now I guess all we need is a minimal repro to get this fixed in PyInstaller (I didn't find anything related in their issue tracker). I gave up setting up PyInstaller locally, but if it doesn't conflict with your system (on Ubuntu none of the suggested ways to install made it run successfully) then this should be enough to produce a minimal binary: hello.py (via ChatGPT) import time
def main():
while True:
print("Hello, world! Press Ctrl+C to exit.")
time.sleep(1)
if __name__ == "__main__":
main() pyinstaller --onefile hello.py
./dist/hello Now if this |
I took a shot at testing this and I'm getting mixed results. Here's the pyinstaller version I used for reference
Now for the weird part: if I run the binary (with redirected output) and then kill it with SIGTERM or SIGKILL it doesn't flush anything at all. The only way I'm able to get it to flush things is when terminating it with SIGINT. This makes me think I'm doing something wrong but I'm not sure what? I've been able to kind of reproduce the mitmdump behavior. Excuse the long shell one-liner but as soon as I put this into a script file and run it, I can't get it to flush any output no matter how I kill it.
Again adding PYTHONUNBUFFERED doesn't do anything
I know this is a bit all over the place but hopefully someone more knowledgeable can provide some insight into what's happening. |
Looks like we can configure this in the PyInstaller spec #6821 In the end it's a tiny change, thanks for helping me figure this out! |
Problem Description
I'm spawning mitmdump from Node.js as a child process. Starting with mitmdump 8 stdout never reaches the parent. Only when I kill the mitmdump process will I finally get its output.
Steps to reproduce the behavior:
index.mjs
node index.mjs
killall mitmdump
Version 6 + 7: I get stdout
Version 8 + 9 + 10: I don't get stdout, only after I kill the mitmdump process
I assume it must be PyInstaller, because spawning directly from
venv/bin
works too (but only withPYTHONUNBUFFERED=1
). However, if I add the following addon I get all stdout:So I assume PyInstaller does not implement/respect
PYTHONUNBUFFERED
and mitmproxy does not explicitly flush.Looking at the PyInstaller issue tracker I assume I must be the first human to spawn a PyInstaller process? Or maybe it's how we use PyInstaller and we can actually configure this on our end?
The text was updated successfully, but these errors were encountered: