get_url module does not validate certificates for protocol TLSv1

[dospromptman]

Issue Type:

Bug Report

Ansible Version:

ansible 1.6

Environment:

Ubuntu 12.04 LTS

(Linux precise64 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux)

Summary:

When using the get_url module to connect to a host that does not support SSL version ssl.PROTOCOL_SSLv23 but does support ssl.PROTOCOL_TLSv1, then the validation of the server's certificate will fail because the default ssl_version for ssl.get_server_certificates() is ssl.PROTOCOL_SSLv23.

See sample code included in this issue for a demonstration of the issue and a potential fix.

Steps To Reproduce:

Attempt to create a task that uses the get_url module to download a file from a server that only supports TLSv1:

 - name: Download composer
   get_url: url=https://getcomposer.org/installer dest=/tmp/composer_installer

Expected Results:

The file at https://getcomposer.org/installer should be downloaded to /tmp/composer_installer

Actual Results:

The task fails and reports that it "Failed to validate the SSL certificate for getcomposer.org:443":

TASK: [Download composer] ***************************************************** 
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE get_url url=https://getcomposer.org/installer dest=/tmp/composer_installer
<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jbury/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=2222', '-o', 'IdentityFile=/Users/jbury/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '127.0.0.1', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653 && echo $HOME/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653'"]
<127.0.0.1> PUT /var/folders/_d/g4cj8rq92rg64w9_gpckhwd59nwjk5/T/tmpumlbrb TO /home/vagrant/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653/get_url
<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jbury/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=2222', '-o', 'IdentityFile=/Users/jbury/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '127.0.0.1', '/bin/sh -c \'sudo -k && sudo -H -S -p "[sudo via ansible, key=klacuomazgachpnkpdqeloufacronxzp] password: " -u root /bin/sh -c \'"\'"\'echo SUDO-SUCCESS-klacuomazgachpnkpdqeloufacronxzp; /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653/get_url; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1396987081.14-176538139938653/ >/dev/null 2>&1\'"\'"\'\'']
failed: [default] => {"failed": true}
msg: Failed to validate the SSL certificate for getcomposer.org:443. Use validate_certs=no or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

Below is sample code that demonstrates the issue, with the setup code borrowed from ansible/lib/ansible/module_utils/urls.py. The code also demonstrates a fix that works for both ssl.PROTOCOL_SSLv23 and ssl.PROTOCOL_TLSv1 by using SSLSocket to validate the server certificate and not trying to do the handshake (which will fail on SSL version mismatch).

I could probably submit a pull request with a fix for the SSLValidationHandler class, if the SSLSocket technique is a preferred fix.

import ssl
import tempfile
import os
import socket

paths_checked = []

paths_checked.append('/etc/ssl/certs')
paths_checked.append('/etc/pki/ca-trust/extracted/pem')
paths_checked.append('/etc/pki/tls/certs')
paths_checked.append('/usr/share/ca-certificates/cacert.org')
paths_checked.append('/etc/ansible')

tmp_fd, tmp_path = tempfile.mkstemp()

for path in paths_checked:
    if os.path.exists(path) and os.path.isdir(path):
        dir_contents = os.listdir(path)
        for f in dir_contents:
            full_path = os.path.join(path, f)
            if os.path.isfile(full_path) and os.path.splitext(f)[1] in ('.crt','.pem'):
                try:
                    cert_file = open(full_path, 'r')
                    os.write(tmp_fd, cert_file.read())
                    cert_file.close()
                except:
                    pass

tmp_ca_cert_path = tmp_path

server = 'getcomposer.org'
port = 443

try:
    ssl.get_server_certificate((server, port), ca_certs=tmp_ca_cert_path)
    print 'validated!'
except ssl.SSLError as e:
    print e

# The above will fail with the following Traceback:
#
#Traceback (most recent call last):
#  File "test_cert.py", line 31, in <module>
#    print ssl.get_server_certificate((server, port), ca_certs=tmp_ca_cert_path)
#  File "/usr/lib/python2.7/ssl.py", line 442, in get_server_certificate
#    s.connect(addr)
#  File "/usr/lib/python2.7/ssl.py", line 331, in connect
#    self._real_connect(addr, False)
#  File "/usr/lib/python2.7/ssl.py", line 324, in _real_connect
#    raise e
#ssl.ssl.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

# However, the following succeeds in validating the certificate.  Note the change in ssl_version:

try:
    ssl.get_server_certificate((server, port), ssl_version=ssl.PROTOCOL_TLSv1, ca_certs=tmp_ca_cert_path)
    print 'validated!'
except ssl.SSLError as e:
    print e

# Additionally, you can use ssl.wrap_socket() and SSLSocket.connect() to allow negotiation of the protocol.
# The following snippet also connects and validates the certificate without specifying the protocol:

try:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    ss = ssl.wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED)
    ss.connect((server, port))
    print 'validated!'
except ssl.SSLError as e:
    print e

[dospromptman]

I updated using the template. Thanks, @ansibot. You're awesome!

[jimi-c]

I'm going to go with your second method, using wrap_socket(). It seems easier than trying to iterate over a few protocol options and we won't have to extend it in the future.

[jimi-c]

The above patch should resolve the issue for you. If you continue seeing any problems related to this, or if you had any further questions regarding this issue, please let us know. Thanks!

[feanil]

@jimi-c would it be possible to port this back to 1.5.5 as well, we are currently stuck there due to the issue reported here: #9362

[jimi-c]

@feanil no, we do not backport patches to previous stable branches. You can, however, work around your issue simply by using the int filter as follows:

    - debug: msg="Y={{Y|int}}"
    - debug: msg="Y JSON={{ Y |int| to_json }}"

If you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

• https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
• https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!