SSL Handshake issue with logstash container

[armensuri]

I see this error in logstash container and the container reboots every 30+ seconds

Error Bundler::HTTPError, retrying 9/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Updated Aborted, message: Could not fetch specs from https://rubygems.org/
Validating logstash-filter-translate-3.1.0
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-filter-translate 3.1.0
Validating logstash-output-sqs-5.0.2
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-output-sqs 5.0.2

I tried the following :
apt-get update
apt-get install ca-certificates
apt-get install -y ruby
gem update --system 2.5.2
gem install bundler
gem update bundle

bundle update
bundle install

then when running this command I get the above error :
/opt/logstash/bin/logstash-plugin install --version 3.1.0 logstash-filter-translate

The logstash container that I'm using is version 2.4.1 (don't ask, we have to be on this container for a few more months)
we tried it on the latest logstash container and encountered the same issue.

Am I doing too many things? I think it should be a simple fix and I'm missing something here.

Environment :
Logstash container 2.4.1
gem env

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "gem env"
RubyGems Environment:

• RUBYGEMS VERSION: 2.5.2
• RUBY VERSION: 2.3.3 (2016-11-21 patchlevel 222) [x86_64-linux-gnu]
• INSTALLATION DIRECTORY: /var/lib/gems/2.3.0
• USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.3.0
• RUBY EXECUTABLE: /usr/bin/ruby2.3
• EXECUTABLE DIRECTORY: /usr/local/bin
• SPEC CACHE DIRECTORY: /root/.gem/specs
• SYSTEM CONFIGURATION DIRECTORY: /etc
• RUBYGEMS PLATFORMS:
  • ruby
  • x86_64-linux
• GEM PATHS:
  • /var/lib/gems/2.3.0
  • /root/.gem/ruby/2.3.0
  • /usr/lib/x86_64-linux-gnu/rubygems-integration/2.3.0
  • /usr/share/rubygems-integration/2.3.0
  • /usr/share/rubygems-integration/all
• GEM CONFIGURATION:
  • :update_sources => true
  • :verbose => true
  • :backtrace => false
  • :bulk_threshold => 1000
• REMOTE SOURCES:
  • https://rubygems.org/
• SHELL PATH:
  • /opt/logstash/bin
  • /usr/local/sbin
  • /usr/local/bin
  • /usr/sbin
  • /usr/bin
  • /sbin
  • /bin

---

bundle env
[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "bundle env"

Environment

Bundler       2.2.1
  Platforms   ruby, x86_64-linux
Ruby          2.3.3p222 (2016-11-21 revision 0) [x86_64-linux-gnu]
  Full Path   /usr/bin/ruby2.3
  Config Dir  /etc
RubyGems      2.5.2
  Gem Home    /var/lib/gems/2.3.0
  Gem Path    /root/.gem/ruby/2.3.0:/var/lib/gems/2.3.0:/usr/lib/x86_64-linux-gnu/rubygems-integration/2.3.0:/usr/share/rubygems-integration/2.3.0:/usr/share/rubygems-integration/all
  User Home   /root
  User Path   /root/.gem/ruby/2.3.0
  Bin Dir     /usr/local/bin
Tools         
  Git         not installed
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2020-12-14
Git SHA           b98d6b2035
Released Version  true

---

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "ruby -ropenssl -e 'puts OpenSSL::OPENSSL_LIBRARY_VERSION'"
OpenSSL 1.0.2l 25 May 2017

---

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "cat /etc/os-release"
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "apt-get --version"
apt 1.4.8 (amd64)
Supported modules:
*Ver: Standard .deb
*Pkg: Debian dpkg interface (Priority 30)
Pkg: Debian APT solver interface (Priority -1000)
Pkg: Debian APT planner interface (Priority -1000)
S.L: 'deb' Debian binary tree
S.L: 'deb-src' Debian source tree
Idx: Debian Source Index
Idx: Debian Package Index
Idx: Debian Translation Index
Idx: Debian dpkg status file
Idx: Debian deb file
Idx: Debian dsc file
Idx: Debian control file
Idx: EDSP scenario file
Idx: EIPP scenario file

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "curl -Lks 'https://git.io/rg-ssl' | ruby"
Here's your Ruby and OpenSSL environment:

Ruby: 2.3.3p222 (2016-11-21 revision 0) [x86_64-linux-gnu]
RubyGems: 2.5.2
Bundler: 2.2.1
Compiled with: OpenSSL 1.0.2u 20 Dec 2019
Loaded version: OpenSSL 1.0.2l 25 May 2017
SSL_CERT_FILE: /usr/lib/ssl/cert.pem
SSL_CERT_DIR: /usr/lib/ssl/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org: success ✅
RubyGems connection to rubygems.org: success ✅
Ruby net/http connection to rubygems.org: success ✅

Hooray! This Ruby can connect to rubygems.org. You are all set to use Bundler and RubyGems. 👌

[deivid-rodriguez]

@armensuri You're running a very old rubygems version. As a matter of fact, the oldest version that can be installed from any supported ruby. Can you update rubygems and see if the issue if fixed? Change the

gem update --system 2.5.2

with

gem update --system 3.2.1

[armensuri]

@deivid-rodriguez Thanks for the quick reply. I changed 2.5.2 to 3.2.1 and the result is the same :
Updating rubygems-update
Successfully installed rubygems-update-3.2.1
Parsing documentation for rubygems-update-3.2.1
Installing ri documentation for rubygems-update-3.2.1
Installing darkfish documentation for rubygems-update-3.2.1
Done installing documentation for rubygems-update after 90 seconds
Parsing documentation for rubygems-update-3.2.1
Done installing documentation for rubygems-update after 0 seconds
Installing RubyGems 3.2.1
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:43:in <top (required)>': uninitialized constant Gem::BasicSpecification (NameError) from /var/lib/gems/2.3.0/gems/rubygems-update-3.2.1/lib/rubygems.rb:1348:in require'
from /var/lib/gems/2.3.0/gems/rubygems-update-3.2.1/lib/rubygems.rb:1348:in <top (required)>' from setup.rb:23:in require'
from setup.rb:23:in `

'
Successfully installed bundler-2.2.1
Parsing documentation for bundler-2.2.1
Installing ri documentation for bundler-2.2.1
Done installing documentation for bundler after 10 seconds
1 gem installed
Updating installed gems
Nothing to update
Error Bundler::HTTPError, retrying 1/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 2/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 3/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 4/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 5/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 6/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 7/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 8/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 9/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Updated Aborted, message: Could not fetch specs from https://rubygems.org/
Validating logstash-filter-translate-3.1.0
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-filter-translate 3.1.0
Validating logstash-output-sqs-5.0.2
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-output-sqs 5.0.2

it looks like there was an error installing 3.2.1, but the message sys installed. Then when I check gem --version I see 2.5.2.1
somehow the change did not take effect.

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "curl -Lks 'https://git.io/rg-ssl' | ruby"
Here's your Ruby and OpenSSL environment:

Ruby: 2.3.3p222 (2016-11-21 revision 0) [x86_64-linux-gnu]
RubyGems: 2.5.2.1
Bundler: 2.2.1
Compiled with: OpenSSL 1.0.2u 20 Dec 2019
Loaded version: OpenSSL 1.0.2l 25 May 2017
SSL_CERT_FILE: /usr/lib/ssl/cert.pem
SSL_CERT_DIR: /usr/lib/ssl/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org: success ✅
RubyGems connection to rubygems.org: success ✅
Ruby net/http connection to rubygems.org: success ✅

Hooray! This Ruby can connect to rubygems.org. You are all set to use Bundler and RubyGems. 👌
[root@ip-161-32-32-161 ~]# vi /local/etc/ansible/roles/logstash-forwarder/files/start.sh
[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "gem --version"
2.5.2.1

[deivid-rodriguez]

Well, not really the same. In this case, it's the upgrade process of rubygems that's failing. You're using ruby provided by your OS, so upgrading rubygems like you're doing is not really supported.

I'm not sure what's going on, honestly. I guess upgrading ruby is not under your control? The latest ruby is 2.7 and the ruby installed by your OS is 2.3.

One thing I'd do is completely remove the gem update --system 2.5.2 command, since that's the rubygems version that comes with ruby 2.3 anyways, and who knows if that upgrade is messing something up 🤷‍♂️.

Also, since you have opened this ticket close after the bundler 2.2 release date, you can try changing gem install bundler to gem install bundler -v 2.1.4. Maybe pinning to the previous version helps.

[armensuri]

removed "gem update --system " and pinned bundler to 2.14
handshake error to Rubygems still exists.

Successfully installed bundler-2.1.4
Parsing documentation for bundler-2.1.4
Installing ri documentation for bundler-2.1.4
Done installing documentation for bundler after 10 seconds
1 gem installed
Error Bundler::HTTPError, retrying 1/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 2/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 3/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 4/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 5/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 6/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 7/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 8/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 9/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Updated Aborted, message: Could not fetch specs from https://rubygems.org/
Validating logstash-filter-translate-3.1.0
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-filter-translate 3.1.0

[deivid-rodriguez]

Not sure if it could be related to server side changes, do you have data about which day these errors started happening?

Maybe it could be an issue with openssl fixed in newer rubies, you could also try gem install openssl to update that 🤷‍♂️.

[armensuri]

We started noticing this around Dec 5. I think that was when the fastly certificate was updated on rubygems.

[deivid-rodriguez]

Ok, then I think you need to upgrade to a version of rubygems that got the certifcate updates. Can you try gem update --system 2.7.11?

[armensuri]

I'll add gem pristine rake to get this error out. and see. This does have the new certificate as the PR suggests.

Updating rubygems-update
Successfully installed rubygems-update-2.7.11
Parsing documentation for rubygems-update-2.7.11
Installing ri documentation for rubygems-update-2.7.11
Installing darkfish documentation for rubygems-update-2.7.11
Done installing documentation for rubygems-update after 46 seconds
Parsing documentation for rubygems-update-2.7.11
Done installing documentation for rubygems-update after 0 seconds
Installing RubyGems 2.7.11
`/usr/share/rubygems-integration/all/gems/rake-10.5.0/bin/rake` does not exist, maybe `gem pristine rake` will fix it?
Bundler 1.16.6 installed
RubyGems 2.7.11 installed
Regenerating binstubs
Parsing documentation for rubygems-2.7.11
Installing ri documentation for rubygems-2.7.11

=== 2.7.11 / 2020-12-08

Minor enhancements:

* Add GlobalSign Root CA - R3 cert and remove outdated certs. Pull request #4100
  by Aditya Prakash.

[armensuri]

Still seeing the same error after applying 2.7.11 and 2.7.10

Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Updated Aborted, message: Could not fetch specs from https://rubygems.org/
Validating logstash-filter-translate-3.1.0
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-filter-translate 3.1.0
Validating logstash-output-sqs-5.0.2
Unable to download data from https://rubygems.org - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-output-sqs 5.0.2

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "curl -Lks 'https://git.io/rg-ssl' | ruby"
Here's your Ruby and OpenSSL environment:

Ruby:           2.3.3p222 (2016-11-21 revision 0) [x86_64-linux-gnu]
RubyGems:       2.7.10
Bundler:        2.1.4
Compiled with:  OpenSSL 1.0.2u  20 Dec 2019
Loaded version: OpenSSL 1.0.2l  25 May 2017
SSL_CERT_FILE:  /usr/lib/ssl/cert.pem
SSL_CERT_DIR:   /usr/lib/ssl/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org:       success ✅
RubyGems connection to rubygems.org:      success ✅
Ruby net/http connection to rubygems.org: success ✅

Hooray! This Ruby can connect to rubygems.org. You are all set to use Bundler and RubyGems. 👌

Is there a cert folder that i need to focus on and make sure that the certs are in that location?

[deivid-rodriguez]

The certs should be inside your rubygems installation. Look for .pem files there.

It's also weird that the troubleshooting script says that you can connect to rubygems.org using both bundler and rubygems, maybe the script should do something more realistic to catch this.

[armensuri]

I think logstash-plugin is using jruby to get the plugins.

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "cat /opt/logstash/bin/logstash-plugin"
#!/bin/sh

unset CDPATH
. "$(cd `dirname $0`/..; pwd)/bin/logstash.lib.sh"
setup

# bin/plugin is a short lived ruby script thus we can use aggressive "faster starting JRuby options"
# see https://github.com/jruby/jruby/wiki/Improving-startup-time
export JRUBY_OPTS="$JRUBY_OPTS -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-noverify -X-C -Xcompile.invokedynamic=false"

ruby_exec "${LOGSTASH_HOME}/lib/pluginmanager/main.rb" "$@"

and the version of jruby installed is 1.9

is there a way for me to install the gems using ruby and not jruby avoid what logstash is doing and pull the pugins?

this script is doing its job. Its just that logstash-plugin was using jruby insetad of ruby to get the gems.

[root@ip-161-32-32-161 ~]# docker exec logstash bash -c "curl -Lks 'https://git.io/rg-ssl' | jruby"
Here's your Ruby and OpenSSL environment:

Ruby:           1.9.3p551 (2020-10-01 revision 48406) [java]
RubyGems:       2.4.8
Compiled with:  JRuby-OpenSSL 0.9.11
SSL_CERT_FILE:  /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
SSL_CERT_DIR:   /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org:       failed  ❌  (uninitialized constant Bundler)
RubyGems connection to rubygems.org:      failed  ❌  (Received fatal alert: handshake_failure (https://rubygems.org))
Ruby net/http connection to rubygems.org: failed  ❌

Unfortunately, this Ruby can't connect to rubygems.org. 😡
Even worse, we're not sure why. 😕

Here's the full error information:
OpenSSL::SSL::SSLError: Received fatal alert: handshake_failure
  org/jruby/ext/openssl/SSLSocket.java:210:in `connect'
  /usr/share/jruby/lib/ruby/1.9/net/http.rb:800:in `connect'
  org/jruby/ext/timeout/Timeout.java:98:in `timeout'
  /usr/share/jruby/lib/ruby/1.9/net/http.rb:800:in `connect'
  /usr/share/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'
  /usr/share/jruby/lib/ruby/1.9/net/http.rb:751:in `start'
  -:89:in `(root)'

You might have more luck using Mislav's SSL doctor.rb script. You can get it here:
https://github.com/mislav/ssl-tools/blob/8b3dec4/doctor.rb
Read more about the script and how to use it in this blog post:
https://mislav.net/2013/07/ruby-openssl/

[deivid-rodriguez]

I'm confused @armensuri 😞. Now you're showing me an error that uses a different ruby implementation and a different rubygems version, both 100% unsupported.

Also, could you please format your messages a bit? They are really hard to read. Here's a guide for you: https://guides.github.com/features/mastering-markdown/ 👍.

[armensuri]

I'm trying to figure out this error for a very old logstash container that we are using. I narrowed it down by looking into logstash-plugin code that is calling jruby. I was trying to fix ruby install and that looks ok, and the tool was showing the connections were ok. Now, I realized that its the jruby that is causing the issues and tried running that same tool using jruby and expectedly, seeing the connection issues. Looking into the certs folders of the jruby, they are from 2016. Tried regenerating the certs but that did not work. the version of jruby that came installed is jruby1.9 I'm trying to figure out which one is the nearest version that I can install without installing the latest. Is there a way to install the plugins that logstash is trying to install using ruby?

for example : /opt/logstash/bin/logstash-plugin install --version 3.1.0 logstash-filter-translate
can I use ruby gems installer to install that ?

[deivid-rodriguez]

I'm sorry I'm not able to help you there. I've never used logstash.

But from what I understand you original error was fixed and now you're having a jruby issue. Unfortunately, the other problem is out of our hands, the jruby & rubygems versions being used are just too old for us to provide any support. I'm sorry, but it was a matter of time that something would break there :(

In case upgrading is completely impossible, here's some links for you so you can keep digging: jruby/jruby#3764, https://www.ruby-lang.org/en/news/2014/10/27/changing-default-settings-of-ext-openssl/.

[armensuri]

Thanks David for your time, it did really help me navigate through the issues and pin point at what is causing the problem. I will keep looking for the jruby solution.

[deivid-rodriguez]

No problem, I hope you find a solution. I'll close this ticket now since we can't really do much I believe.