OSSL_DECODER_CTX_set_selection doesn't apply the selection value properly #20657
Comments
t8m
added
triaged: question
|
The problem is the Red Hat FIPS support in their openssl package is only related to the upstream OpenSSL FIPS module and there are some serious differences - for example the crypto-policies support. So trying to debug it with the OpenSSL FIPS module can be problematic or even impossible if the bug is due to the additional Red Hat patches. |
|
@junaruga It may be a Red Hat specific problem. in RHEL FIPS provider is auto activated when the system is in FIPS mode, and if the OpenSSL configuration file is present and loaded. So I'd look in OpenSSL initialization. |
|
Guys, thank you for your responses! I was able to reproduced this issue with not only OpenSSL 3.0 RPM package on RHEL 9 FIPS mode but also with the OpenSSL 3.0 compiled from the OpenSSL source directly without any patch files on Fedora 37. The reproducing steps above is with the OpenSSL 3.0 compiled from the source directly. Why do you think this may be a Red Hat specific problem? |
|
If this is an universal problem then I'd anyway check via strace if openssl config and providers are loaded. |
|
Thanks! I suppose this is an universal problem. Because I can reproduce this issue with OpenSSL 3.0.8 with FIPS mode compiled from the source on the GitHub Actions Ubuntu on my forked branch: wip/fips-read-test-report from the https://github.com/ruby/openssl. The CI log is here. My concern is the The log in Ubuntu case on the GitHub Actions: I just prepared the |
|
Sorry, maybe my assumption was wrong about where the problematic code is. I said the problem is the 6th call of Because I compared the result between with OpenSSL 3.0.8 FIPS enabled and OpenSSL FIPS disabled now. non-FIPS mode (FIPS mode disabled)Below steps are for the OpenSSL 3.0.8 non-FIPS mode compiled from the source without any patch files (not RPM pacakge). I compiled the OpenSSL 3.0.8 non-FIPS mode with the debug flags from the source by the commands below. Then I confirmed that the version and the FIPS provide is not available and FIPS mode is not enabled as expected. Then below is the result of running the OpenSSL Ruby binding. You see the ext/openssl/ossl_pkey.c#L133 And I debugged with GDB again for the non-FIPS mode. Here is the line soon after the 5th called Here is the backtrace. I updated the repository: https://github.com/junaruga/report-openssl-fips-read-error adding the non-FIPS mode ltrace and strace log files. |
|
I compared the input arguments I found one difference between the 2 cases. The value of the FIPS modeNon-FIPS mode |
|
I just added the new section "Reproducing steps - 2. Install Ruby and Compile OpenSSL Ruby bindings. - Install Ruby" on the first comment above, as I thought you may want to reproduce this issue on your environment. |
|
I'm looking at all this, but can't really see directly what's happening. The BIO that's passed to Can you confirm that the contents that this BIO handles are what they are supposed to be? Also, does The reason I'm asking all this is that I've started to suspect that the |
|
Sure! Let me confirm it. And I will let you know it here. |
I think so. The So, below is the state of the Then After calling the first Then soon after calling the When calling the Reference: |
|
Guys, do you have any other info do you want to see? I am happy to provide it. Thanks. |
|
When checking the The Then at the following line in the function https://github.com/openssl/openssl/blob/openssl-3.0.8/crypto/bio/bss_mem.c#L272 It is processed as read only. |
|
So these calls work fine if run with default provider but they fail with fips+base providers? If you call OSSL_PROVIDER_available("fips") and OSSL_PROVIDER_available("base") do these calls return 1? |
Yes, right. As I commented at #20657 (comment), the call of the
Yes, right. Both the I tested my testing OpenSSLs |
|
I updated my testing branch on the forked repository: https://github.com/junaruga/openssl/commits/wip/fips-read-report by adding more debug logs to print the base/fips providers and the fips-enabled, and the result of the FIPS modeNon-FIPS mode |
|
I do not think the problem is related to the bio reset call. It is fairly strange because the decoders in default and base providers are identical implementations and if fips and base providers are properly loaded there should be really no difference if you're importing 4096 bit RSA key in unencrypted PEM format. Could you please put breakpoint to rsa_d2i_PKCS8 in the debugger to see if it is called in the non-fips and fips cases and if there are any differences in the return value? |
Sure. I checked it. The Below is the working log on GDB. FIPS modeBelow is the line where the Then after stepping forward to get the return value, the The return value is NULL. I also printed the input values of the function and the local variables too. Non-FIPS modeI did the same thing with the FIPS mode case to get the return value. Below is the line where the The return value is NULL as well as the FIPS mode case. |
|
Just to avoid having to deal with Ruby, I made a test program that essentially does what your extension does, but limits itself to the problem domain. I can confirm seeing the same problem in my runs. https://gist.github.com/levitte/7a27cebdb9537ff0a59641c9a5bed53d |
|
With an OpenSSL built with BIO *trace_bio = BIO_new_fp(stderr, BIO_NOCLOSE | BIO_FP_TEXT);
OSSL_trace_set_channel(OSSL_TRACE_CATEGORY_DECODER, trace_bio);That's a lot of output, but one line that I think tells a bit of the story is this (where When running with the FIPS module, the last such line has openssl/crypto/encode_decode/decoder_pkey.c Lines 68 to 70 in 40f4884 |
|
Wow, thank you for the test program without Ruby and your investigation! I also learned the |
|
I figured it out... This is caused by the combination of a decoder in one provider and the keymgmt in another. This causes an export/import dance ('cause you must assume that they have different internal representations of keys, even if they are the same key type), seen here: openssl/crypto/encode_decode/decoder_pkey.c Lines 151 to 166 in 40f4884 However, that's not the problem per se. However, there are indeed two bugs:
I think it shouldn't be too hard to fix the first bug. The second... not so sure. |
|
The first bug that I mention is actually fairly easy to demonstrate. All I had to do was to apply this little patch: @@ -9,7 +18,8 @@
EVP_PKEY *pkey = NULL;
int pos = 0, pos2;
- dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL, 0, NULL, NULL);
+ dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL,
+ EVP_PKEY_KEYPAIR, NULL, NULL);
if (!dctx)
goto out;
if (OSSL_DECODER_CTX_set_pem_password_cb(dctx, PEM_def_callback, pass) != 1)And then I get nice runs in all cases: $ OPENSSL_CONF=./20657.cnf OPENSSL_CONF_INCLUDE=/home/levitte/.local/opt/openssl-3.0/ssl OPENSSL_MODULES=/home/levitte/.local/opt/openssl-3.0/lib/ossl-modules ./20657 key.pem
Loaded providers:
fips
base
[DEBUG] Calling ossl_pkey_read_generic from ossl_pkey_new_from_data.
[DEBUG] Calling OSSL_DECODER_from_bio 1.
40C72990E97F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: DER
[DEBUG] Calling OSSL_DECODER_from_bio 2.
Got a pkey! 0x558171f95830
It's held by the provider fips
$ ./20657 key.pem
Loaded providers:
default
[DEBUG] Calling ossl_pkey_read_generic from ossl_pkey_new_from_data.
[DEBUG] Calling OSSL_DECODER_from_bio 1.
40D7E3D7E07F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: DER
[DEBUG] Calling OSSL_DECODER_from_bio 2.
Got a pkey! 0x564cbb9970f0
It's held by the provider default |
|
Do note that said small patch to |
|
Thank you!! I am still reading your comment to understand it. And I will apply your patch above to my my OpenSSL Ruby binding code! |
|
I tested the OpenSSL Ruby bindings with your patch. The result was there is an error in the 1st call of the |
junaruga
changed the title
|
I changed this issue ticket's title to the "OSSL_DECODER_CTX_set_selection doesn't set the selection value properly on OpenSSL 3 FIPS mode.". I hope it's better to describe this issue. |
junaruga
changed the title
t8m
changed the title
t8m
added
branch: master
|
@t8m I see that you renamed this issue ticket's title to "OSSL_DECODER_CTX_set_selection doesn't apply the selection value properly", removing the "on OpenSSL 3 FIPS mode". In my understanding, this issue only happens in the FIPS mode case. Is that right? |
|
@junaruga, the issue is more generic than that. The way the DECODER functionality works, the provider implementations for it may live in a different provider than the one handling the keys themselves, and that's the combination where we have this issue. For you, it ended up being triggered by the combination of FIPS and base providers (the latter being where the DECODER implementations are), but it might as well have been the combination of the OQS and base providers. |
|
... also, |
|
@levitte thanks for your explanation! OK. I understood it. In my case there are 2 providers "base" and "fips". And the "base" provider handles the DECODER implementation, and "fips" provider handles (holds) the key. The issue happened as one of the cases that a provider handling the decoder is different from a provider handling a key. |
@levitte I tried your way to debug with enable-trace option above for the code without workaround patch. I added the following lines to your reproducing program. Then I expected a trace log in stderr. However I haven't seen it. Do you know what's wrong in my code? |
This links the program to the default libcrypto, not your build. To link with your build, you must tell gcc where it is: Also, when you run the program, remember to set |
|
@levitte Oh thanks!! However, I still don't see the trace log on your way. I added the |
|
Oh I am sorry. I found my mistake (typo) in the command above about how to print the trace log. You see that I was setting |
|
Incorrectly closed, see #20657 (comment) |
|
How is the progress about this issue? |
|
I tested this issue with the reproducing program 20657.c on the latest master branch 06a0d40 . And it seems the issue still exists on it. |
|
Now I understood why the issue happens with the help of the #20657 (comment) . It's not new information. But let me explain. CauseI used my updated reprocucer.c with the tracing logs to check it by msyelf. You can also see the tracing logs In the non-FIPS case, the log is below. 1 is ok. 0 is not ok. In the FIPS case, the log is below. The source code is below. openssl/crypto/encode_decode/decoder_lib.c Lines 768 to 780 in 06a0d40 And the openssl/crypto/encode_decode/decoder_pkey.c Lines 71 to 73 in 06a0d40 And in the openssl/crypto/encode_decode/decoder_pkey.c Lines 151 to 170 in 06a0d40 And in the How to fix?The So, my guess is that we need to update the openssl/crypto/encode_decode/decoder_lib.c Lines 166 to 179 in 06a0d40 It might be like this. However, the problem is the openssl/crypto/encode_decode/encoder_local.h Line 155 in 06a0d40 In the openssl/crypto/encode_decode/decoder_pkey.c Lines 61 to 69 in 06a0d40 |
|
In the code, when calling the following openssl/crypto/encode_decode/decoder_pkey.c Lines 156 to 167 in 06a0d40 openssl/providers/implementations/encode_decode/decode_der2key.c Lines 310 to 326 in 06a0d40 And you see the So, the |
|
I am experimenting one idea to fix this issue on the latest master branch 7a2bb21 including the commits by the #21519 related to this issue. I am working on this branch on my forked repository for that. The latest one commit on the branch is for the idea. The idea is to change the member However, the value set by the I am debugging this with my reproducer. At the following step to set the In ossl_decoder_ctx_setup_for_pkey in crypto/encode_decode/decoder_pkey.c Then when setting the But then on the following step to get the In decoder_construct_pkey in crypto/encode_decode/decoder_pkey.c |
|
What is the status of this issue? |
I read carefully to decide which is better in the openssl-users@openssl.org mailing list or opening this issue ticket on GitHub. As my question is about the OpenSSL API used in the OpenSSL Ruby bindings. I am trying to fix a bug in the OpenSSL Ruby bindings. I thought that perhaps my question is more close to the developing OpenSSL rather than using OpenSSL. But let me know if you think my case is to ask on the mailing list. I am happy to post it on it. Sorry for that.
I am debugging the OpenSSL Ruby bindings to fix a bug. Please let me know what's wrong in the code. Perhaps the APIs are wrongly called?
You can reproduce this bug by doing
git cloneon my forked repository branch: https://github.com/junaruga/openssl/tree/wip/fips-read-report that includes some debugging commits on the master branch. However, the reproducing steps are a bit complicated, please let me know if there are commands that you want me to run to find additional info.Reproducing steps
Environment
My local environment is Fedora 37. However I was able to reproduce this issue on the Ubuntu (the
ubuntu-latest) on the GitHub Actions too. And this issue also happens with the both cases of OpenSSL built from the source code without any patch files, and OpenSSL RPM package on RHEL 9.1.In the reproducing steps below, the used OpenSSL version is OpenSSL 3.0.8 compiled from the source without any patch files. The
LD_LIBRARY_PATHis used to load the OpenSSL.1. Install OpenSSL with FIPS mode option.
I compiled the openssl with fips mode, and debug flags (
-O0 -g3 -ggdb3 -gdwarf-5flags.) as I wanted to debug. But this issue happens with the openssl compiled without the debug flags.And here is the OpenSSL config file used in the later process.
Then I used this program to check if the fips mode is available.
2. Install Ruby and Compile OpenSSL Ruby bindings.
Below is the steps to install Ruby and to compile the Ruby OpenSSL bindings with the latest stable Ruby 3.2. I am compiling with the
-O0 -g3 -ggdb3 -gdwarf-5flags. You can skip the section. Note that at that time, Ruby 3.2.1 was the latest stable one. But now the Ruby 3.2.2 is the latest stable one.Install Ruby
In the case of compiling with the source on the GitHub.
Run
autoconfto generate theconfigurescript.Or in the case of downloading the Ruby 3.2.1 source article from the Ruby official website.
Then I installed the Ruby with the commands below. The
--enable-mkmf-verboseoption makes thebundle exec rake compilecommand in the later process print the C compiler (gcc) command lines.Or in the case of using Ruby RPM package on Fedora Linux, you see an error by the
bundle exec rake compilein the later process. Here is a workaround.Then you can set the
PATHfor the installed Ruby..bashrc
Compile OpenSSL Ruby bindings.
If you want to compile with the branch on my forked repository to reproduce this issue ticket:
or if you want to compile with the original repository:
Then I installed the dependency RubyGems packages.
I compiled the OpenSSL Ruby bindings.
If you want to clean to compile again by
bundle exec rake compile, you can run the command below.3. Run the command raising the error.
I created a testing pem file.
Then I ran the OpenSSL Ruby binding to read the pem file from the OpenSSL Ruby binding. In the result of the command, you see the error message "Could not parse PKey (OpenSSL::PKey::PKeyError)" that comes from the OpenSSL Ruby binding, and it comes from the following the
OSSL_DECODER_from_bio(dctx, bio)returning 0. See below.The other parts in the output, the
[DEBUG] ...is by myprintfdebugging log. And the... Input type: ...is by theERR_print_errors_fp(stdout). Theossl_pkey_read_genericfunction is called 2 times, and theOSSL_DECODER_from_biofunction is called 3 times in eachossl_pkey_read_genericfunction called.The error comes from the
OSSL_DECODER_from_bioreturning the0.https://github.com/junaruga/openssl/blob/41bc792df2cf54660264bd6fc6368044f2877e99/ext/openssl/ossl_pkey.c#L149
ext/openssl/ossl_pkey.c#L149
Debugging
ltrace
First, I captured the ltrace log by the command below. Because I think the ltrace log is good to see how the OpenSSL APIs are called in the process. You can see the
OSSL_DECODER_from_biois called totally 6 times in theltrace. So, the 6th calledOSSL_DECODER_from_biofails and causes the error. Note that unfortunately, the log is by theltracein the first part, and by therubyin the second part unfortuntely.GDB
Debug around the
OSSL_DECODER_from_bioI debugged the gdb by the command below. The reason why I am setting the
LD_LIBRARY_PATHin the gdb prompt is because the system openssl is the dependency of thegdbcommand. The gdb fails hiding the system openssl by referring to the manually installed openssl byLD_LIBRARY_PATH.After some steps, below is soon after calling the 6th
OSSL_DECODER_from_bioreturning the0as a error. And the values of the input arguments*dctxand*bioare below.And here is the backtrace.
Debug deeply in the
OSSL_DECODER_from_bioAs a reference, I stepped in the
OSSL_DECODER_from_bio. Running the GDB from the start again, then here is a part that causes the error in theOSSL_DECODER_from_bio. As theokis0, and thedecoder_processreturns the0.Here are input arguments of the function
decoder_processand local variables at the same pointcrypto/encode_decode/decoder_lib.c:747.Here is the backtrace.
Please let me know if you want to see additional information. I am happy to help for that! Thank you for reading this, and thank you for your help.
The text was updated successfully, but these errors were encountered: