SSL Verifications Started Failing

[ttutko]

Recently (within the past month), I started seeing the following errors stop me from performing a sync:

vscsync_1 | [W 200413 18:44:30 sync:146] ********************** <VSCExtensionDefinition> ceapex.docs-build (c51a76ee-a3ed-4bf4-8836-9ff198c46260) - Version: 0.1.17 vscsync_1 | Traceback (most recent call last): vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen vscsync_1 | chunked=chunked, vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request vscsync_1 | self._validate_conn(conn) vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn vscsync_1 | conn.connect() vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect vscsync_1 | ssl_context=context, vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket vscsync_1 | return context.wrap_socket(sock, server_hostname=server_hostname) vscsync_1 | File "/usr/local/lib/python3.7/ssl.py", line 423, in wrap_socket vscsync_1 | session=session vscsync_1 | File "/usr/local/lib/python3.7/ssl.py", line 870, in _create vscsync_1 | self.do_handshake() vscsync_1 | File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake vscsync_1 | self._sslobj.do_handshake() vscsync_1 | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076) vscsync_1 | vscsync_1 | During handling of the above exception, another exception occurred: vscsync_1 | vscsync_1 | Traceback (most recent call last): vscsync_1 | File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send vscsync_1 | timeout=timeout vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen vscsync_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] vscsync_1 | File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment vscsync_1 | raise MaxRetryError(_pool, url, error or ResponseError(cause)) vscsync_1 | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ceapex.gallerycdn.vsassets.io', port=443): Max retries exceeded with url: /extensions/ceapex/docs-build/0.1.17/1584603092136/Microsoft.VisualStudio.Code.Manifest (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)'))) vscsync_1 | vscsync_1 | During handling of the above exception, another exception occurred: vscsync_1 | vscsync_1 | Traceback (most recent call last): vscsync_1 | File "/opt/vscoffline/sync.py", line 569, in <module> vscsync_1 | extensions[identity].download_assets(config.artifactdir_extensions) vscsync_1 | File "/opt/vscoffline/sync.py", line 149, in download_assets vscsync_1 | self._download_asset(destination, availableasset) vscsync_1 | File "/opt/vscoffline/sync.py", line 199, in _download_asset vscsync_1 | result = self.session.get(url, allow_redirects=True, timeout=vsc.TIMEOUT) vscsync_1 | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 543, in get vscsync_1 | return self.request('GET', url, **kwargs) vscsync_1 | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request vscsync_1 | resp = self.send(prep, **send_kwargs) vscsync_1 | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send vscsync_1 | r = adapter.send(request, **kwargs) vscsync_1 | File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 514, in send vscsync_1 | raise SSLError(e, request=request) vscsync_1 | requests.exceptions.SSLError: HTTPSConnectionPool(host='ceapex.gallerycdn.vsassets.io', port=443): Max retries exceeded with url: /extensions/ceapex/docs-build/0.1.17/1584603092136/Microsoft.VisualStudio.Code.Manifest (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)'))) vscodeoffline_vscsync_1 exited with code 1

Things I've tried:

• Rebuilding the container to ensure that the latest version of requests and python base image was pulled down.
• Mapping the host machine's "ca-certificates.crt" file into the container and setting the "REQUESTS_CA_BUNDLE" environment variable to point to that file.
• If I docker exec into the alpine ash shell of the container while it's running I can confirm that the date and time is correct within the container

Accessing the url that it's trying to hit from the host system works and the expiration dates on the cert look fine. Docker exec'ing into the container and running python then doing a requests.get(<URL>) also seems to work fine and return a 200 status code.

No idea why the sync is throwing those errors. The only thing I haven't tried yet was modifying sync.py's code to turn off ssl verification for each place a request is happening.

[ttutko]

It could be somewhat related to this: urllib3/urllib3#1682

[ttutko]

I set all requests to ignore SSL verification which lead to a different error:

Traceback (most recent call last): vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request vscsync_1 | six.raise_from(e, None) vscsync_1 | File "<string>", line 3, in raise_from vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 416, in _make_request vscsync_1 | httplib_response = conn.getresponse() vscsync_1 | File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse vscsync_1 | response.begin() vscsync_1 | File "/usr/local/lib/python3.8/http/client.py", line 303, in begin vscsync_1 | version, status, reason = self._read_status() vscsync_1 | File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status vscsync_1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") vscsync_1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto vscsync_1 | return self._sock.recv_into(b) vscsync_1 | File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into vscsync_1 | return self.read(nbytes, buffer) vscsync_1 | File "/usr/local/lib/python3.8/ssl.py", line 1099, in read vscsync_1 | return self._sslobj.read(len, buffer) vscsync_1 | socket.timeout: The read operation timed out vscsync_1 | vscsync_1 | During handling of the above exception, another exception occurred: vscsync_1 | vscsync_1 | Traceback (most recent call last): vscsync_1 | File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send vscsync_1 | resp = conn.urlopen( vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen vscsync_1 | retries = retries.increment( vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 400, in increment vscsync_1 | raise six.reraise(type(error), error, _stacktrace) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise vscsync_1 | raise value vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen vscsync_1 | httplib_response = self._make_request( vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 423, in _make_request vscsync_1 | self._raise_timeout(err=e, url=url, timeout_value=read_timeout) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 330, in _raise_timeout vscsync_1 | raise ReadTimeoutError( vscsync_1 | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='sparkfountain.gallerycdn.vsassets.io', port=443): Read timed out. (read timeout=12) vscsync_1 | vscsync_1 | During handling of the above exception, another exception occurred: vscsync_1 | vscsync_1 | Traceback (most recent call last): vscsync_1 | File "/opt/vscoffline/sync.py", line 572, in <module> vscsync_1 | extensions[identity].download_assets(config.artifactdir_extensions) vscsync_1 | File "/opt/vscoffline/sync.py", line 151, in download_assets vscsync_1 | self._download_asset(destination, availableasset) vscsync_1 | File "/opt/vscoffline/sync.py", line 201, in _download_asset vscsync_1 | result = self.session.get(url, allow_redirects=True, timeout=vsc.TIMEOUT) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get vscsync_1 | return self.request('GET', url, **kwargs) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request vscsync_1 | resp = self.send(prep, **send_kwargs) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send vscsync_1 | r = adapter.send(request, **kwargs) vscsync_1 | File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 529, in send vscsync_1 | raise ReadTimeout(e, request=request) vscsync_1 | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='sparkfountain.gallerycdn.vsassets.io', port=443): Read timed out. (read timeout=12)

This leads me to believe the first error was really related to timing out and not specifically SSL expired issue.

I'm going to try increasing the timeouts. In any case, I think it would be a good feature to not fail out completely when this happens but perhaps continue on to the next extension and write out a report at the end that says something like "The following extensions could not be sync'd: " and list out any that failed.

[ttutko]

Upon further evaluation, I believe the error messages to be misleading and what was really happening was a timeout. This timeout appeared to be caused by either the VM or the host machine shutting down the network adapter when the screen would timeout due to inactivity. While I do not know why that is happening, it does not appear to be an issue with this code and I'm closing the issue.