Support OAEP decryption with modulus above 4096

[GabMus]

Riffing off of #210 I wanted to ask if it's possible to add some feature flag or some other opt-in solution to support keys with modulus size bigger than 4096.

I understand the concerns for algorithmic denial of service, but I think having an escape hatch for when you actually want to use this key size for various reasons would be useful, and in fact it's necessary for a particular need I have 😅

[tarcieri]

Which type of keys are you asking about? RsaPublicKey? PKCS#8 or SPKI encoded keys?

RsaPublicKey has new_with_max_size and new_unchecked constructors which can be used to instantiate larger public keys.

PKCS#8 and SPKI are capped to 4096-bits, following recommendations from the X.509 ecosystem. Since these keys are potentially attacker-controlled they're a potential DoS vector if too large. See #166.

[GabMus]

Sorry, I'm in fact referring to PKCS#8 keys in this particular case. Again, as I said I'm well aware of the implication, that's why I was asking specifically for an opt-in solution, something you'd have to explicitly enable. Can this be done?

[tarcieri]

You can parse the PKCS#8 key yourself using the pkcs1 crate and call the appropriate unchecked constructors.

Adding a dedicated feature for this would be rather tricky and seems generally questionable.

Can I ask what use case you have for extremely large PKCS#8 keys?

[GabMus]

In this case it's for keys provided by third parties and we cannot change it.

Regardless the problem does not occur when initializing the RsaPrivateKey but when doing the decryption. Let me show you some example code:

let key = RsaPrivateKey::from_pkcs8_encrypted_pem(my_pem, my_password)?;
let decrypted = key.decrypt(my_oaep_padding_instance, my_encrypted_payload)?; // this errors out with ModulusTooLarge

In particular the error seems to come from RsaPrivateKey::decrypt, calling Oaep::decrypt, calling key::check_public.

This last one then calls key::check_public_with_max_size, and that's where we find the size check code:

    if public_key.n().bits() > max_size {
        return Err(Error::ModulusTooLarge);
    }

Additionally, the max_size value comes from key::RsaPublicKey::MAX_SIZE and it's hardcoded to 4096.

The idea would be to have some sort of override for this functionality, so that we can use whatever we want as a max size, or even an exact expected size.

[tarcieri]

Okay, I agree that's an edge case in OAEP decryption which should be allowed since it's a private key and not a potentially untrusted public key

[GabMus]

Any updates on this? Are you able to give me an ETA, or point me in a direction to propose a PR?