Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

different output from io.FlowReader and io.FlowWrite #6801

Open
rgre26 opened this issue Apr 15, 2024 · 1 comment
Open

different output from io.FlowReader and io.FlowWrite #6801

rgre26 opened this issue Apr 15, 2024 · 1 comment
Labels
kind/triage Unclassified issues

Comments

@rgre26
Copy link

rgre26 commented Apr 15, 2024

Goal:
Reduce size of mitmdump file, by removing large response entries.

Code:

from mitmproxy import io

with open(file,'rb') as file_in:
    fr = io.FlowReader(file_in)
    with open(f'{file}.out', 'ab') as file_out:
        fw = io.FlowWriter(file_out)
            for n,f in enumerate(fr.stream()):
                fw.add(f)

Got a new file, but with different sha256, so i used diff:


1,4c1
< 9112:4:type;
< 4:http;
< 7:version;
< 2:18#9:websocket;
---
> 9052:9:websocket;
35c32,33
< 8:HTTP/2.0,}17:timestamp_created;
---
> 8:HTTP/2.0,}6:backup;
> 0:~17:timestamp_created;
44,46c42,54
< 6313:4:via2;
< 0:~11:cipher_list;
< 0:]11:cipher_name;
---
> 6277:3:via;
> 0:~19:timestamp_tcp_setup;
> 17:1692359990.453262^7:address;
> 47:37:firefox.settings.services.mozilla.com;
> 3:443#]19:timestamp_tls_setup;
> 18:1692359990.4875603^13:timestamp_end;
> 0:~15:timestamp_start;
> 18:1692359990.4108424^3:sni;
> 37:firefox.settings.services.mozilla.com;
> 11:tls_version;
> 7:TLSv1.3;
> 11:cipher_list;
> 0:]6:cipher;
49c57,58
< 16:2:h2,8:http/1.1,]16:certificate_list;
---
> 16:2:h2,8:http/1.1,]4:alpn;
> 2:h2,16:certificate_list;
  1. Is it normal, that generated output diffs original?
  2. If yes, is it possible to keep old file format when rewriting file?
@rgre26 rgre26 added the kind/triage Unclassified issues label Apr 15, 2024
@rgre26 rgre26 changed the title io.FlowReader and then io.FlowWrite generate different output different output from io.FlowReader and then io.FlowWrite Apr 15, 2024
@rgre26 rgre26 changed the title different output from io.FlowReader and then io.FlowWrite different output from io.FlowReader and io.FlowWrite Apr 15, 2024
@mhils
Copy link
Member

mhils commented Apr 15, 2024

I can't really tell from the diff: is the flow the same with just reordered keys, or are we loosing information? The latter would be a bug we should fix. The former is less urgent, but something I'd be happy to review a PR for nonetheless, at least if it is possible to fix without adding complexity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/triage Unclassified issues
Projects
None yet
Development

No branches or pull requests

2 participants