Proposal for creating pkeys from raw parameters #555

anakinj · 2022-10-19T15:48:42Z

Hi.

I'm Joakim and have never done anything else than used and read the code for this gem.

I'm involved in the ruby-jwt gem and as @bellebaum pointed out in #551 there is the need to create RSA (and other keys) from raw parameters. Due to the changes in mutability of keys in OpenSSL 3.0 it is no longer possible that easily.

This PR is a suggestion on how the interface for the OpenSSL gem could look like and a very rough implementation that probably does not work for all the key types (yet).

It's a while ago I did C so apologies for the potential messiness.

Do you think this direction and way solving the problem is feasible? Im open for all feedback.

anakinj · 2022-10-25T20:23:26Z

Got the RSA generation pretty much working, with user-friendly parameter names for the most common parameters.

Started to look into how to generate EC keys from params, turned out to be a little more complicate than to just throw bignums at it so still a WIP :)

Does this approach make sense:

Hash with the different parameters to build the key
Choose the hash iterator function based on the desired key type. Raise if there is no support.
Translate the hash keys into the OpenSSL constants.
Push the key and value into the parameter builder with correct types.
Let the builder generate the key

Also would it make sense to incrementally build this and maybe first only support RSA keys and then expanding support for other key types later?

anakinj · 2022-10-28T21:47:28Z

Continued a bit on the EC pkey generation. Seems that OpenSSL only allows passing the pub parameter as an octet string to the builder. It's a little cumbersome to use in this way instead of passing the point as-is or the x y coordinates.

Not sure if we want to go into making it convenient by:

Converting x and y into a octet string
Some custom parameter for this gem that takes the bignum that represents the point and extract x and y from that depending on the curve type.

Also unsure what other parameters do we want to support on creating, the EC object is pretty simple in Ruby. There is a bunch of different parameters that can be set for the EC pkey, some are inherited from the curve name I guess.

Im not an expert on this so all feedback is appreciated :)

For the use-case Im interested in this implementation would be good enough.

Allow EC keys to be generated based on:

curve name
private point (octet string representing x and y)
private key (unsigned integer)

anakinj · 2022-10-29T08:41:45Z

Continuing my monologue :)

This draft now supports all the PKey types in this gem (DH, DSA, RSA, EC). The EC support is only supporting the most common parameters as mentioned in some previous comment.

Now I'm wondering about backwards compatibility. Would be nice to be able to support the same interface for OpenSSL 1.1 and OpenSSL 3.0. The implementation would probably be very different for OpenSSL 1.1.

Could someone allow the CI to run on this one? Would be interesting to see all the issues. Also still open for feedback on the code part of things.

anakinj · 2022-12-22T20:50:11Z

Feel a little bad for leaving this open like this, but a little clueless on how to continue or if to continue at all.

Any pointers on the direction to take this? Try to use the mutable pkeys in openssl 1.x to handle the parameters or just support openssl 3.0 for now?

Maybe @rhenium could have some ideas?

rhenium · 2022-12-22T22:00:21Z

Thank you for working on this! Sorry for taking so long time to review.

Yes, OpenSSL::PKey.from_parameters(type, param1=>value1, ...) matches what I hope to have in this library!

As for the implementation, I want generic methods directly defined in OpenSSL::PKey to be minimum and independent of key types as much as possible. I wonder if the parameter name → value type mappings could be done without hard-coding them; OpenSSL's docs mention EVP_PKEY_fromdata_settable() and it appears to be provided for this purpose.

Try to use the mutable pkeys in openssl 1.x to handle the parameters or just support openssl 3.0 for now?

The compatibility is nice to have, but I don't find it a requirement. Once we have OpenSSL 3.0 code in C, I expect this can be implemented in Ruby with existing methods (probably in lib/openssl/pkey.rb).

anakinj · 2022-12-28T22:18:10Z

Thanks @rhenium for pointing to EVP_PKEY_fromdata_settable, now the implementation is pretty type agnostic. Left the parameter types OSSL_PARAM_OCTET_PTR and OSSL_PARAM_UTF8_PTR unsupported, did not find any key types that would support these kind of parameter types.

Still not happy with the way the aliases are handled, as said earlier my C skills are a little rusty so passing static arrays to the callback was not that straightforward so for now passing both size and pointer was the only option I could figure out, so probably going return to them soonish.

collimarco · 2023-01-04T20:43:17Z

+1 It would be extremely useful to have this method and it would make it easier to migrate some code from OpenSSL v1.1 to OpenSSL v3. Is there any chance to see this merged?

wilsonsilva · 2023-02-22T13:15:14Z

Excellent work! Meanwhile, is there a Ruby workaround?

collimarco · 2023-02-22T14:11:30Z

@wilsonsilva Probably you can find this commit useful as a start... take a look to the set_keys! method in particular.

wilsonsilva · 2023-02-28T07:43:50Z

@wilsonsilva Probably you can find this commit useful as a start... take a look to the set_keys! method in particular.

Thank you. The morning you posted this, I had already been on that repo! Thanks for writing that code.

https://github.com/wilsonsilva/nostr/blob/main/lib/nostr/crypto.rb#L111-L129

anakinj · 2023-12-09T19:35:17Z

I refreshed (and rebased) the branch a bit, curious if the tests pass on CI.

bdewater · 2024-03-22T02:11:15Z

@rhenium could you approve the CI workflow please?

junaruga · 2024-03-23T09:25:26Z

I approved the CI workflow.
@anakinj I think you can also run the GitHub Actions CI workflow on your forked repository without waiting for approval, https://github.com/anakinj/openssl/actions. For example, here is my GitHub Actions page on my forked repository.

junaruga · 2024-03-23T09:38:48Z

I see the failures on the openssl-head and openssl 3 fips cases on CI. For the failure of the openssl-head case, the 2e826d5 fixes it. So, you can pass the case by rebasing this PR on the latest master branch.

junaruga · 2024-03-23T10:05:20Z

For example below are the failures on openssl-head fips.
https://github.com/ruby/openssl/actions/runs/8085746974/job/23008578186?pr=555#step:13:110

I would like you to consider using FIPS-approved crypto algorithms or bits in your new tests as much as possible because I want to see that this new feature is supported in the FIPS cases.

As a reference, I don't think that the following code using the OpenSSL.fips_mode to skip the tests in the FIPS cases is a good practice. I am passionate about rewriting the part in the future.

openssl/test/openssl/test_provider.rb

Line 3 in a8caa63

if defined?(OpenSSL) && defined?(OpenSSL::Provider) && !OpenSSL.fips_mode

Using the method omit_on_fips is possible if there is no alternative way to pass a test in the FIPS case. But I think that it is a last resort.

anakinj · 2024-03-26T16:18:50Z

Thanks for all of the pointers. I'll look into refreshing the branch again and looking into making the tests FIPS compatible.

test/openssl/test_pkey.rb

anakinj · 2024-03-27T06:19:56Z

Thanks for the great pointers again @junaruga, the changes seem to pass CI now. Is there any adjustments that you would wish to see still?

junaruga · 2024-03-27T16:01:56Z

test/openssl/test_pkey.rb

+                                                      dmp1: source.dmp1,
+                                                      dmq1: source.dmq1,
+                                                      iqmp: source.iqmp
+                                                    )


Maybe the position of the ) can be adjusted. As a reference, below is what Rubocop style guide suggests.
https://github.com/rubocop/ruby-style-guide?tab=readme-ov-file#method-arguments-alignment

junaruga · 2024-03-27T16:13:33Z

Thanks for the great pointers again @junaruga, the changes seem to pass CI now. Is there any adjustments that you would wish to see still?

@anakinj Thank you for working for applying the FIPS-approved crypto algorithms. I think the logic for the FIPS part is good to me. I think you can squash the 6 commits to 1 commit in this PR now, because eventually I think we want to see just one commit for this PR.

Then perhaps you may need small adjustments about the coding style. While the maximum line length in the current existing code is sometimes more than 80 bytes, I may try to keep the maximum 80 bytes for the code I implement newly if I were you. @rhenium may have his opinion about the style.
https://github.com/rubocop/ruby-style-guide?tab=readme-ov-file#maximum-line-length

junaruga · 2024-03-27T16:15:59Z

test/openssl/test_pkey.rb

+
+      key = OpenSSL::PKey.from_parameters("ED25519", pub: "\xD0\x8E\xA8\x96\xB6Fbi{$k\xAC\xB8\xA2V\xF4n\xC3\xD06}R\x8A\xE6I\xA7r\xF6D{W\x84")
+      assert_instance_of OpenSSL::PKey::PKey, key
+      assert_equal "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEA0I6olrZGYml7JGusuKJW9G7D0DZ9UormSady9kR7V4Q=\n-----END PUBLIC KEY-----\n", key.public_to_pem


I think you can use Ruby's heredoc syntax for a better readability. You can refer to other parts using the heredoc in the test/openssl/test_pkey.rb.

Adjusted to have the expected public presented in heredoc. Looks way prettier

junaruga · 2024-03-27T16:31:29Z

test/openssl/test_pkey.rb

+    def test_s_from_parameters_rsa_with_n_e_and_d_given_as_integers
+      new_key = OpenSSL::PKey.from_parameters("RSA", n: 3161751493,
+                                                     e: 65537,
+                                                     d: 2064855961)


Is it intentional that you are using mixture of the symbol keys such as n: in some tests and String keys such as "n" => in other tests?

The intention was to test that keys could also be passed as strings. Maybe it would be clearer to have a dedicated test for that instead of testing it "on the side"?

Yes, I think the dedicated test for testing the symbol keys is a good idea to show the intention clearly. Checking the existing test/openssl/test_pkey.rb, there are only the cases using String keys for a hash in the file. So, in my opinion, maybe it's better to use the String keys as a default, and a dedicated test for testing the symbol keys.

Now tests are using string keys, except one dedicated test with symbol keys

junaruga · 2024-03-27T16:50:13Z

test/openssl/test_pkey.rb

+                                                      "rsa-exponent1" => source.dmp1,
+                                                      "rsa-exponent2" => source.dmq1,
+                                                      "rsa-coefficient1" => source.iqmp
+                                                    )


The thing for the ) is same with the one I mentioned in another part.

Adjusted the alignment of everything

rhenium · 2024-03-27T16:04:18Z

ext/openssl/ossl_pkey.c

+        if (cur > end)
+            break;
+        cur += snprintf(cur, end-cur, fmt, params->aliases[i].alias);
+    }


The use of snprintf() makes me a bit nervous. Could you use Ruby String for building the error message? Ruby exposes a number of functions for interacting with string, for example:

// Creates an empty string VALUE str = rb_str_new(NULL, 0); // Append to `str` using printf-style format rb_str_catf(str, "%s", params->aliases[i].alias);

ossl_raise() supports "%"PRIsVALUE format specifier for including Ruby object in the message (as apposed to "%s" for a C string).

Thanks for the pointer (and the patience), this is my first really touching a bigger gem with C extensions. So learning here.

I ended up using an array and joining with a comma as I probably would have done in Ruby also. This to avoid the complexity of checking if a comma needs to be appended or not. Think the result is pretty tidy.

rhenium · 2024-03-27T16:05:13Z

ext/openssl/ossl_pkey.c

+    else
+        from_params_args.aliases = fcc_aliases;
+
+    rb_hash_foreach(options, &add_parameter_to_builder, (VALUE) &from_params_args);


ctx will leak if rb_hash_foreach() raises an exception. Please wrap it with rb_protect() so that they can freed before escaping from this function.

Adjusted this one. There were more leaks than just this one but now I think it might be freeing everything when exceptions happen. This approach also removes the need to free things inside the iterator function.

rhenium · 2024-03-27T16:06:32Z

ext/openssl/ossl_pkey.c

+static VALUE
+ossl_pkey_s_from_parameters(int argc, VALUE *argv, VALUE self)
+{
+    #if OSSL_OPENSSL_PREREQ(3, 0, 0)


Style nit: please remove spaces before #

rhenium · 2024-03-27T16:07:47Z

test/openssl/test_pkey.rb

+      e = assert_raise(OpenSSL::PKey::PKeyError) {
+        OpenSSL::PKey.from_parameters("ASR", {})
+      }
+      assert_match(/^EVP_PKEY_CTX_new_from_name: unsupported/, e.message)


Suggested change

e = assert_raise(OpenSSL::PKey::PKeyError) {

OpenSSL::PKey.from_parameters("ASR", {})

}

assert_match(/^EVP_PKEY_CTX_new_from_name: unsupported/, e.message)

assert_raise_with_message(OpenSSL::PKey::PKeyError, /^EVP_PKEY_CTX_new_from_name: unsupported/) {

OpenSSL::PKey.from_parameters("ASR", {})

}

We can use assert_raise_with_message for this kind of test.

rhenium · 2024-03-27T16:09:10Z

test/openssl/test_pkey.rb

+    end
+
+    def test_s_from_parameters_rsa_with_openssl_internal_names
+      source  = OpenSSL::PKey::RSA.generate(2048)


Please avoid generating a new RSA/DSA/DH key because it can take a long time. For this feature, pre-generated keys (Fixtures.pkey("rsa2048"), for example, for RSA) should work.

I change the key generations to use fixtures. For the EC test I added 3 new fixtures for each EC curve used in these tests. The existing p256.pem and ec-prime256v1.pem serve the same purpose and could probably be unified, but think that could happen in separate.

junaruga · 2024-03-27T16:56:41Z

Thanks for the great pointers again @junaruga, the changes seem to pass CI now. Is there any adjustments that you would wish to see still?

@anakinj Thank you for working for applying the FIPS-approved crypto algorithms. I think the logic for the FIPS part is good to me. I think you can squash the 6 commits to 1 commit in this PR now, because eventually I think we want to see just one commit for this PR.

Then perhaps you may need small adjustments about the coding style. While the maximum line length in the current existing code is sometimes more than 80 bytes, I may try to keep the maximum 80 bytes for the code I implement newly if I were you. @rhenium may have his opinion about the style. https://github.com/rubocop/ruby-style-guide?tab=readme-ov-file#maximum-line-length

Below is the ruby/ruby's coding style. And there is no comment about the maximum line length. So, I think the style of the maximum length 80 bytes is optional, and nice to have.

https://github.com/ruby/ruby/wiki/Developer-How-To#coding-style

rhenium · 2024-03-27T17:12:00Z

I'm sorry for the slow response.

As for the method name, can we maybe have a different one than from_parameters? We currently have a similarly named method OpenSSL::PKey.generate_parameters which generates a new pkey object containing only key parameters (though I'm starting to regret that I didn't name it generate_key_parameters instead). It sounds too similar while having a different function.

This is based on EVP_PKEY_fromdata(), so from_data is my suggestion. I'm also hoping to add the counterpart to the method in future, which should correspond to EVP_PKEY_todata().

The implementatation, I'm not sure what value the alias names for RSA/DSA/DH give us. Since this is a completely new method, I think we can simply forget about the old names and only accept what EVP_PKEY_fromdata() supports.

anakinj · 2024-03-27T18:23:10Z

Thanks for all the feedback. The suggested from_data sounds great.

About the aliases, it's mainly because of convenience and keeping the consistency on how this gem has named the parameters and how the average user is subject to the parameters. I have a few arguments for and against the aliases :)

Being able to pass back the same values that are referred to for the PKey types. Think it just would be a bit confusing that sometimes the RSA parameter is p and sometimes rsa-factor1. Im a bit biased in this example but when mapping a JWK RSA key to a PKey object the transformation from p to rsa-factor1 needs to happen somewhere, so why not in the place that actually knows the relation between the different names used for different parameters.

Think the biggest challenge with all this is the need to know a bit of the internals of openssl to be able to use this method. Maybe all this could be documented for the method. With comprehensive descriptions of what these parameters also sometimes are called. Maybe there could even be some new methods on the PKey objects that use the names used internally. OpenSSL::PKey::RSA#p would be an alias for OpenSSL::PKey::RSA#rsa-factor1.

Also the aliases will cause a bit of confusion if a potential to_data or OpenSSL::PKey#data method will return different keys than passed into from_data. So for the future it would be better not to have them.

I'm fine either way, I would say you decide if aliases are valuable or not. I will comply :)

EDIT:
Also rsa-factor1 cannot be represented as a Ruby symbol that nicely :)

anakinj force-pushed the pkey-from-parameters branch from d3071b7 to 61052b4 Compare October 25, 2022 20:16

bdewater mentioned this pull request Oct 28, 2022

JWK to RSA key translation #551

Open

anakinj mentioned this pull request Oct 30, 2022

Stricter RSA key generation from jwk parameters jwt/ruby-jwt#524

Merged

rhenium mentioned this pull request Dec 18, 2022

pkey: track whether pkey is private key or not #527

Closed

rhenium mentioned this pull request Dec 28, 2022

Support OpenSSL 3.0 #369

Closed

collimarco mentioned this pull request Dec 28, 2022

Add support for OpenSSL 3 pushpad/web-push#2

Closed

anakinj force-pushed the pkey-from-parameters branch from aa6383d to 0cddb6a Compare December 28, 2022 18:54

anakinj force-pushed the pkey-from-parameters branch from 7026f0c to e3c722f Compare January 6, 2023 10:05

anakinj marked this pull request as ready for review January 6, 2023 10:13

nscott mentioned this pull request Jan 17, 2023

OpenSSL v3 Support dvci/health_cards#108

Open

anakinj mentioned this pull request Jan 31, 2023

OpenSSL 3.0 error - rsa#set_key= railslove/epics#138

Closed

anakinj force-pushed the pkey-from-parameters branch from e3c722f to 0d510ed Compare February 23, 2023 16:55

sylph01 mentioned this pull request Jul 14, 2023

pkeys are immutable on OpenSSL 3.0\e[0m (OpenSSL::PKey::PKeyError) #619

Open

This was referenced Sep 2, 2023

OpenSSL 3 - Unable to create OpenSSL::PKey instances from pem/der/asn sequences jwt/ruby-jwt#568

Closed

openssl 3: Can't construct RSA keys from any available serializable key data #679

Closed

jonmchan mentioned this pull request Oct 22, 2023

updating ruby-version, refactoring ec_crypto_suite to support OpenSSL… EthicalIdentity/fabric-gateway-ruby#19

Merged

anakinj force-pushed the pkey-from-parameters branch 3 times, most recently from a5dc485 to 70bb9d4 Compare December 9, 2023 19:29

anakinj force-pushed the pkey-from-parameters branch from a8a6f5d to dfe3d55 Compare February 17, 2024 13:31

anakinj force-pushed the pkey-from-parameters branch from dfe3d55 to 063a6e2 Compare February 28, 2024 19:04

anakinj force-pushed the pkey-from-parameters branch from 063a6e2 to ae9a2d5 Compare March 26, 2024 16:29

bdewater reviewed Mar 26, 2024

View reviewed changes

test/openssl/test_pkey.rb Outdated Show resolved Hide resolved

junaruga reviewed Mar 27, 2024

View reviewed changes

rhenium requested changes Mar 27, 2024

View reviewed changes

anakinj force-pushed the pkey-from-parameters branch from d726b4e to 762bdb1 Compare March 28, 2024 05:26

OpenSSL::PKey.from_data for generating PKey objects from user data

2b73110

anakinj force-pushed the pkey-from-parameters branch from 762bdb1 to 2b73110 Compare March 28, 2024 05:44

anakinj requested a review from rhenium April 2, 2024 13:23

Proposal for creating pkeys from raw parameters #555

Are you sure you want to change the base?

Proposal for creating pkeys from raw parameters #555

Conversation

anakinj commented Oct 19, 2022

anakinj commented Oct 25, 2022 • edited

anakinj commented Oct 28, 2022 • edited

anakinj commented Oct 29, 2022 • edited

anakinj commented Dec 22, 2022 • edited

rhenium commented Dec 22, 2022

anakinj commented Dec 28, 2022 • edited

collimarco commented Jan 4, 2023

wilsonsilva commented Feb 22, 2023

collimarco commented Feb 22, 2023

wilsonsilva commented Feb 28, 2023

anakinj commented Dec 9, 2023

bdewater commented Mar 22, 2024

junaruga commented Mar 23, 2024

junaruga commented Mar 23, 2024 • edited

junaruga commented Mar 23, 2024 • edited

anakinj commented Mar 26, 2024

anakinj commented Mar 27, 2024

Choose a reason for hiding this comment

junaruga commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junaruga Mar 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junaruga Mar 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junaruga commented Mar 27, 2024 • edited

rhenium commented Mar 27, 2024

anakinj commented Mar 27, 2024 • edited

anakinj commented Oct 25, 2022 •

edited

anakinj commented Oct 28, 2022 •

edited

anakinj commented Oct 29, 2022 •

edited

anakinj commented Dec 22, 2022 •

edited

anakinj commented Dec 28, 2022 •

edited

junaruga commented Mar 23, 2024 •

edited

junaruga commented Mar 23, 2024 •

edited

junaruga Mar 27, 2024 •

edited

junaruga Mar 27, 2024 •

edited

junaruga commented Mar 27, 2024 •

edited

anakinj commented Mar 27, 2024 •

edited