Determining the computed MIME type of a resource

[akshaysharmajs]

Adding the main mime sniffing rules to #2 based on section 7 of standards.

[akshaysharmajs]

@Gallaecio For the byte pattern in section 7.1 step1, Patterns end with a tag-terminating byte that includes 0x20 (SP), 0x3E (">")

how should I make the entires of these patterns in _patterns.py? Making two entries of all patterns one ending with 0x20 and other with 0x3E or can make some changes in the _is_match_mime_pattern or there is some precise way ??

[Gallaecio]

@Gallaecio For the byte pattern in section 7.1 step1, Patterns end with a tag-terminating byte that includes 0x20 (SP), 0x3E (">")

how should I make the entires of these patterns in _patterns.py? Making two entries of all patterns one ending with 0x20 and other with 0x3E or can make some changes in the _is_match_mime_pattern or there is some precise way ??

I think the duplicate rows approach may be the best here, being the simplest, and something that users can easily do themselves as well when wishing to do something similar for extra_types.

You could use a loop in _patterns.py to avoid the duplication in code, but since the end goals is to autogenerate _pattern.py, I think that approach would be a bit of a waste of time. For extra_types, users are free to use this approach, though, e.g.:

extra_types = tuple(
    (prefix + suffix, mask, WHITESPACE_BYTES, "text/xml")
    for prefix, mask,  in (
        (b"\x3C\x21\x44\x4F\x43\x54\x59\x50\x45\x20\x48\x54\x4D\x4C", b"\xFF\xFF\xDF\xDF\xDF\xDF\xDF\xDF\xDF\xFF\xDF\xDF\xDF\xDF\xFF"),
        (b"\x3C\x48\x54\x4D\x4C", b"\xFF\xDF\xDF\xDF\xDF\xFF"),
        # …
    )
    for suffix in (b"\x20", b"\x3E")
)

[akshaysharmajs]

I believe I am wrong but this is what I understand till step 5.1 in section 7.3.

def _sniff_mislabled_feed(input_bytes: bytes, supplied_type: Optional[Tuple[bytes]]) -> str:
    input_size = len(input_bytes)
    index = 0

    if input_size >= 3 and input_bytes[:3] == b"\xef\xbb\xbf":
        index += 3

    while index < input_size:
        while True:
            if input_bytes[index:index+1] == None:
                return supplied_type

            if input_bytes[index:index+1] == b"<":
                index += 1
                break

            if input_bytes[index:index+1] not in WHITESPACE_BYTES:
                return supplied_type

            index += 1
              

[Gallaecio]

Sounds correct to me.

The first loop (while True) aims to return the supplied type unless the file starts with < (ignoring spacing characters).

if input_bytes[index:index+1] == None will not work, though. I’ve just tested it, and it should be if input_bytes[index:index+1] == b'', or simplified as if not input_bytes[index:index+1], since:

>>> data = b"a"
>>> data[:1]
b'a'
>>> data[1:2]
b''

You could also simplify input_size >= 3 and input_bytes[:3] == b"\xef\xbb\xbf" as input_bytes[:3] == b"\xef\xbb\xbf".

[akshaysharmajs]

I am shifting _is_match_mime_pattern to _utils.py, I think it's creating a reference cycle for imports

[codecov]

Codecov Report

Merging #4 (882bc7d) into main (f656d9b) will increase coverage by 0.60%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main        #4      +/-   ##
===========================================
+ Coverage   99.39%   100.00%   +0.60%     
===========================================
  Files           3         4       +1     
  Lines         164       353     +189     
===========================================
+ Hits          163       353     +190     
+ Misses          1         0       -1     

Impacted Files	Coverage Δ
xtractmime/__init__.py	100.00% <100.00%> (+4.00%)	⬆️
xtractmime/_patterns.py	100.00% <100.00%> (ø)
xtractmime/_utils.py	100.00% <100.00%> (ø)
xtractmime/mimegroups.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f656d9b...882bc7d. Read the comment docs.

[Gallaecio]

Awesome.

I think we can merge as soon as you fix the renaming issues from the static checks and we merge #2, if you prefer to handle documentation in a separate pull request.

You might also want to do something like this to prevent 1 failure to stop an unrelated CI job: scrapy/scrapy#5200

[akshaysharmajs]

I think we can merge as soon as you fix the renaming issues from the static checks and we merge #2, if you prefer to handle documentation in a separate pull request.

Yes, handling documentation in a separate PR will be better and easier for me.

Also, please suggest a fix for the only error in typing checks in the latest commit. I think the type hint for leading bytes in extra_types requires a change.

[elacuesta]

LGTM. The only reason I'm not merging right away is because I wonder if it would be clearer to use bytes.fromhex more extensively instead of escaping hex sequences directly. It's not a blocker at all, let's just talk about it in our next meeting.

[elacuesta]

It was decided that converting the escape sequences can be done later, in a separate PR.