Fixing AMF parsing

[6600024d]

Problem Description

AMF ( https://en.wikipedia.org/wiki/Action_Message_Format ) parsing support vanished at some point in the early mitmproxy releases, and didn't come back. I'm planning to re-plug pyamf or some AMF lib in mitmproxy to decode AMF.

• I found when it arrived in 2012 : Optional AMF decoding support #62
• In 2017 it wasn't working https://discourse.mitmproxy.org/t/amf-support-not-working/437
• We're in 2024 and it's not there :D

CONTRIBUTING says :

If you intend to work on a larger contribution to the project, please come talk to us first.

There we go :)

Proposal

I figure out how to plug some AMF lib (the python ones are pretty old, but might work and one of them has a nice C implementation, there's surely some lib for that in ruffle.rs but I'm no cross-language packaging expert, and the spec itself doesn't seem that overly complicated, so as a last resort I'll try to write one myself).

I've got some X-AMF traffic there, not sure I need an extensive dataset as long as I can plug an existing lib.

Alternatives

Not parsing AMF with mitmproxy.

Questions

Is there anything specific I should be aware of ? I'll follow the doc otherwise. Cheers !

[mhils]

Thank you for raising an issue before starting your work! 🍰

Generally happy to add AMF parsing, my only concern here is that I'd like to avoid adding a dependency on an additional library. In particular, I'm very strongly -1 on anything that involves file format parsing in C. Both from a memory safety perspective as well as a "needs a compiler to install" perspective. Is there maybe already an existing Kaitai Struct definition for AMF somewhere? Not sure how complex AMF is, but if this can be done with Kaitai this would be by far my preferred option. :)

[6600024d]

Hey, status update: I managed to get some amf0 parsing and it works well against the https://github.com/sile/amf/tree/master/src/testdata dataset. Here's the Kaitai Struct YAML file:
amf0.v1.ksy.txt, (I couldn't find amf ksy files online, maybe I'm bad at searching) and here's what parsing amf0-ref-test.bin gives:

It's pretty verbose, maybe there are KS tricks to reduce the amount of layers.

I didn't have a chance yet to plug that in mitmproxy to observe my own intercepted data. I'm specifically wondering if it's raw amf0/amf3 objects or if they're packed in the custom amf-packet format. Because both could be used, and there's no way to tell amf-packet data from amf0/3 data without attempting to parse it. I guess I'll have to offer both, even if having amf-packet and amf as types feels superfluous, I don't foresee a viable alternative.

Are there examples of Kaitai Struct -parsed formats in mitmproxy I could borrow code from ?

[mhils]

Thanks for the update! Our Kaitai defs are in https://github.com/mitmproxy/mitmproxy/tree/main/mitmproxy/contrib/kaitaistruct , https://github.com/search?q=repo%3Amitmproxy%2Fmitmproxy+kaitaistruct&type=code should show usages. :)

…

On Wed, Mar 27, 2024, 20:46 6600024d ***@***.***> wrote: Hey, status update: I managed to get some amf0 parsing and it works well against the https://github.com/sile/amf/tree/master/src/testdata dataset. Here's the Kaitai Struct YAML file: amf0.v1.ksy.txt <https://github.com/mitmproxy/mitmproxy/files/14780372/amf0.v1.ksy.txt>, (I couldn't find amf ksy files online, maybe I'm bad at searching) and here's what parsing amf0-ref-test.bin gives: image.png (view on web) <https://github.com/mitmproxy/mitmproxy/assets/164740436/3cf546c6-27dd-49c1-be0d-1bea4705de3f> It's pretty verbose, maybe there are KS tricks to reduce the amount of layers. I didn't have a chance yet to plug that in mitmproxy to observe my own intercepted data. I'm specifically wondering if it's raw amf0/amf3 objects or if they're packed in the custom amf-packet format. Because both could be used, and there's no way to tell amf-packet data from amf0/3 data without attempting to parse it. I guess I'll have to offer both, even if having amf-packet and amf as types feels superfluous, I don't foresee a viable alternative. Are there examples of Kaitai Struct -parsed formats in mitmproxy I could borrow code from ? — Reply to this email directly, view it on GitHub <#6753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHY2PUKECCYN7FCSQTXV6DY2MHZNAVCNFSM6AAAAABFFTJC2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRTHAZTONBSGA> . You are receiving this because you commented.Message ID: ***@***.***>

[6600024d]

Hey, status update:

• amf0 : more or less OK. could be enhanced: boolean, undefined, null, references, times
• amf-packet : OK (!) (or is it ? :D)
• amf3 : just started, I'm struggling with the U29 integer format they invented, pretty much UTF-8-for-uint32. There will be the same possible enhancements as for amf0 on data types, and I'll also have to sort out some bitfield issues. (Wouldn't be fun if these bitfield flags were not applied to U29 integers, yay.)

I got some pointers from the peeps developing ruffle.rs, the flash runtime. My own dataset is amf-packet with headers packing amf0 objects which sometimes are amf3 objects. I've been pointed at some random flash games that do generate amf-packet data as well. Unless I see raw amf0 objects on the wire then the parsing root will be amf-packet.

[6600024d]

Status update

I managed to implement the U29 format, but am stuck at amf3_object_type, and can't reasonably invest more energy & time into this initiative « at this time ».

Blockers

• I couldn't get the meaning of the specification for amf3_object_type or the few implementations I could find. I tried reading sile/amf, ruffle-rs/rust-flash-lso and hydralabs/pyamf and still could not make sense of the amf3 object type, notably when compared to real-world data.
• The resources I can invest in this are too scarce. Real life. This, or I'm too stupid to read a spec. I guess I'll never know 🤷 :D.

Remaining work

• Ideally, you just have to sort the amf3_object_type structure, possibly array and dictionary will require similar handling.
• Then, plugging the spec in plugging it in mitmproxy might not be that hard. I don't know hehe.

Contribution so far

• amf-packet : implemented
• amf0 : implemented, and tested to some extent against the sile/amf dataset and some custom flash app.
• Limitation: "null", "undefined" and other JavaScript-ish objects are not well handled by Kaitai Struct so we just skip them
• Limitation: so far, there's no support for references, I'm not sure if it's possible (or worth it) to perform bookkeeping like that in Kaitai Struct.
• I managed to sort out that u29 format, which isn't the vlq_base128 found in the Kaitai Struct standard set.

Enclosed .KSY file

Anyways, here's my amf.ksy file so far, I'm releasing it as public domain for whomever wants to finish tackling this down. I'd be very honored if anyone managed to reuse my work to some extent.

amf.ksy.txt

Thanks for mitmproxy, it really is a great tool I keep recommending anytime I get a chance.