won't log localhost traffic on windows

[cmedcoff]

I'm attempting to execute what I would think to be a very common use case - intercepting and logging all HTTP traffic on localhost servers for debugging/demonstration purposes. For example I want to run an OAuth confidential client (HTTP server), an authorization server and a resource server on localhost host and capture all frontend and backend HTTP traffic to demonstrate and/or troubleshoot a OAuth code grant use case/flow. For example the code for all of these are available from

https://github.com/oauthinaction/oauth-in-action-code/

Specifcally

https://github.com/oauthinaction/oauth-in-action-code/tree/master/exercises/ch-3-ex-1

But of course it shouldn't matter if this is node, python, .NET or whatever.

When running "mitmproxy" non of the traffic shows up - only traffic/noise I'm not interesting in leaving my "localhost".

Steps to reproduce the behavior:

1. create a python virtual environment and install using "python -m venv pip install mitmproxy"
2. install mitmproxy certs as document - I won't reproduce the instruction here
3. open a windows command prompt and/or terminal, activate the virtual environment and run "mitmproxy"
4. run all apps (e.g. "node some_node_js_script.js"
5. configure your browser to use the proxy which defaults to "localhost:8080"
6. navigate the brower to the OAuth confidential client - in my case localhost:9000
7. observe that no traffic is captured by the mitmproxy output

I've tried various modes including transparent ("mitmproxy --mode transparent") which only results in many messages messsing up the screen with "Warning: Previously unseen connection from proxy ..." not helpful, maybe in a log file but not the screen.

System Information

Mitmproxy: 10.2.4
Python: 3.12.1
OpenSSL: OpenSSL 3.2.1 30 Jan 2024
Platform: Windows-11-10.0.22631-SP0

[mhils]

observe that no traffic is captured by the mitmproxy output

This sounds like your browser client is not sending localhost traffic to the proxy - so that more of a browser configuration issue than anything we could fix in mitmproxy itself.

Have you tried local redirect mode? UX and docs aren't quite there yet, but it should work well for your use case.

[cmedcoff]

User error on my part and apologies for taking your time with my mistake.

With that corrected I can capture all of the OAuth so called "front channel traffic" in the browser. Local redirect mode e.g "mitmxproxy --mode local" doesn't work for me.

I'd still like to capture so called "back channel" traffic e.g the server to server REST calls that occur outside the browser for this OAuth flow. It looks like I should use transparent mode for this but that still results in all of these print statements spilling onto the screen reporting key errors which makes the cli screen unuable. It appears that it's so busy printing that it wont accept user command to quit, e.g. 'q', but I have to kill it off with control-C (windows).

Do I need to follow all of the steps on:

https://docs.mitmproxy.org/stable/howto-transparent/

to get transparent mode to work?

[mhils]

Transparent mode on Windows is not really supported / very janky. The intended replacement for that is local redirect mode. Can you clarify why that does not work for you / what issues you are seeing with it?

[cmedcoff]

Context - I'm running 3 node servers on "localhost" at port 9000, 9001 and 9002. These each represent an OAuth client, autorization server and protected resource in case that helps to bring clarity. I want to point my browser to the server running at 9000 and execute and OAuth code grant flow use case and see all HTTP traffic to/from these servers - via the browser or otherwise - e.g server to server calls. E.g. I want to see/capture traffic from the brower to the server at 9000, 9001 (these are done via HTTP redirects using the browser) and I want to see/capture back channel traffic, e.g. and HTTP call from 9000 to 9001 via a REST call outside the browser - e.g. server to server.

Test Scenario 1:

if I run "mitmproxy" and I set a filter for "localhost" I see what I want to see with regards to browser traffic - GET/POST, REDIRECTS and only for thoses servers on ports 9000 and 9001 (now that I've correct my browser proxy settings). I DO NOT see (backchannel) traffic for localhost:9002, but it is occuring - I can see that from the logs of the server running at 9002.

Test Scenario 2:

if I run "mitmproxy --mode local" I immediately see other HTTP traffic which is not from my use case I just described. In fact as soon as I begin the OAuth use case again, e.g. point my browser to locahost:9000, Firefox reports "The proxy server is refusing connections".

Again per our disucussion I'm attempting to pick up the back channel, server to server call, using "--mode local" in addition to what I get from "Test Scenario 1", but then I cannot execute the use case and I get all sorts of other traffic that is just noise. I do seem traffic to my local IP, but its labed with the IP rather than 'localhost'.

Appreciate any guidance.

[mhils]

if I run "mitmproxy --mode local" I immediately see other HTTP traffic which is not from my use case I just described. In fact as soon as I begin the OAuth use case again, e.g. point my browser to locahost:9000, Firefox reports "The proxy server is refusing connections".

When running mitmproxy --mode local, you must not configure any proxy settings anywhere else. mitmproxy will use a kernel driver to grab traffic (without collaboration from the respective application). "The proxy server is refusing connections" indicates that you need to undo your proxy server configuration in Firefox for this.

Once working, you can use --mode local:foo.exe,bar.exe to only grab traffic for specific executables.

[cmedcoff]

Ugh. I should have thought of/realized that ... it's hooking into the stack at a lower level.

So tried again without configuring the browser for proxy and traffic in general is captured, but not my traffic as described previously. Also "mitmproxy --mode local:node (or "node.exe") captures nothing. If I just do "mitmproxy --mode local" and the perform a get to the 9000 app, initially I see the outgoing message in the UI but no response content. If I retry, then I do see a response from the 9000 app, but the proxy does not pick up the request when triggered via a browser.

I also noticed up on exiting after performing these tests that I see many:

"TCP connection handler coroutine raised an exception"

I don't know if that to be expected.

A couple of side notes. I've tried this from and normal windows command prompt, a windows terminal and from git/bash (windows). Behavior seems to be the same. I do see the "windows-redirector" getting launched.

[mhils]

"TCP connection handler coroutine raised an exception"

Do you have more details / a traceback? This should not happen.

[cmedcoff]

I didn't get a trackback, just saw many lines of that string upon exit in the terminal. I could dig through source, but can you point/provide basic insructions for finding and/or turning on/configuring logging?

[mhils]

The best approach would probably to run mitmdump (mitmproxy without TUI or web UI), capture for a short time, and check what kind of logging you get on stdout/stderr. Does that yield anything?

[cmedcoff]

so I run the command:
mitmdump --mode local

And I see ...
[20:36:49.962] Local redirector started.

then a bunch of traffic I'm not interested in ...

[20:36:51.003][192.168.0.114:50134] client connect
< a bunch of traffic snipped>

Then I trigger my use case, I see no traffic of interest against my local server apps running on 9000, etc. Then I end with control-C and I observe this output:

TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
Traceback (most recent call last):
File "C:\Python312\Lib\asyncio\base_events.py", line 837, in call_soon_threadsafe
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError:
TCP connection handler coroutine raised an exception:
CancelledError: