Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ssl_verify=false doesn't work in dvc version 2.6.4 #6563

Closed
albreand opened this issue Sep 8, 2021 · 3 comments
Closed

ssl_verify=false doesn't work in dvc version 2.6.4 #6563

albreand opened this issue Sep 8, 2021 · 3 comments
Labels
A: data-sync Related to dvc get/fetch/import/pull/push fs: s3 Related to the S3 filesystem upstream Issues which need to be resolved in an upstream dependency

Comments

@albreand
Copy link

albreand commented Sep 8, 2021

When having ssl_verify set to false and executing dvc pull i get this error:

dvc pull
ERROR: unexpected error - verify_ssl, ssl_context, fingerprint and ssl parameters are mutually exclusive 

With the Version 2.0.17 we did not have this issue.

For better replicability:

$ dvc pull -v
2021-09-01 10:39:41,577 DEBUG: Preparing to transfer data from 's3://cloud' to '.dvc\cache'
2021-09-01 10:39:41,587 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-01 10:39:41,597 DEBUG: Collecting status from '.dvc\cache'
2021-09-01 10:39:42,767 DEBUG: Preparing to collect status from 's3://cloud'
2021-09-01 10:39:42,777 DEBUG: Collecting status from 's3://cloud'
2021-09-01 10:39:43,627 ERROR: unexpected error - verify_ssl, ssl_context, fingerprint and ssl parameters are mutually exclusive
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc\main.py", line 55, in main
  File "dvc\command\base.py", line 45, in do_run
  File "dvc\command\data_sync.py", line 30, in run
  File "dvc\repo\__init__.py", line 49, in wrapper
  File "dvc\repo\pull.py", line 29, in pull
  File "dvc\repo\__init__.py", line 49, in wrapper
  File "dvc\repo\fetch.py", line 67, in fetch
  File "dvc\repo\fetch.py", line 87, in _fetch
  File "dvc\data_cloud.py", line 114, in pull
  File "dvc\objects\transfer.py", line 221, in transfer
  File "dvc\objects\status.py", line 166, in compare_status
  File "dvc\objects\status.py", line 132, in status
  File "dvc\objects\db\base.py", line 501, in hashes_exist
  File "dvc\objects\db\base.py", line 303, in _estimate_remote_size
  File "dvc\objects\db\base.py", line 257, in _hashes_with_limit
  File "dvc\objects\db\base.py", line 247, in list_hashes
  File "dvc\objects\db\base.py", line 227, in _list_paths
  File "dvc\fs\fsspec_wrapper.py", line 110, in walk_files
  File "dvc\fs\fsspec_wrapper.py", line 212, in find
  File "fsspec\asyn.py", line 88, in wrapper
  File "fsspec\asyn.py", line 69, in sync
  File "fsspec\asyn.py", line 25, in _runner
  File "s3fs\core.py", line 642, in _find
  File "s3fs\core.py", line 560, in _lsdir
  File "s3fs\core.py", line 404, in set_session
  File "aiobotocore\session.py", line 37, in __aenter__
  File "aiobotocore\session.py", line 121, in _create_client
  File "aiobotocore\client.py", line 32, in create_client
  File "aiobotocore\client.py", line 88, in _get_client_args
  File "aiobotocore\args.py", line 53, in get_client_args
  File "aiobotocore\endpoint.py", line 252, in create_endpoint
  File "aiobotocore\httpsession.py", line 87, in __init__
  File "aiohttp\connector.py", line 765, in __init__
  File "aiohttp\client_reqrep.py", line 159, in _merge_ssl_params
ValueError: verify_ssl, ssl_context, fingerprint and ssl parameters are mutually exclusive
------------------------------------------------------------
2021-09-01 10:39:46,637 DEBUG: Version info for developers:
DVC version: 2.6.4 (exe)
---------------------------------
Platform: Python 3.8.10 on Windows-10-10.0.18363-SP0
Supports:
        azure (adlfs = 2021.7.1, knack = 0.8.2, azure-identity = 1.6.1),
        gdrive (pydrive2 = 1.8.1),
        gs (gcsfs = 2021.7.0),
        hdfs (pyarrow = 5.0.0),
        webhdfs (hdfs = 2.5.8),
        http (requests = 2.26.0),
        https (requests = 2.26.0),
        s3 (s3fs = 2021.8.0, boto3 = 1.17.106),
       ssh (sshfs = 2021.7.1),
        oss (ossfs = 2021.7.5),
        webdav (webdav4 = 0.9.0),
        webdavs (webdav4 = 0.9.0)
Cache types: hardlink
Cache directory: NTFS on E:\
Caches: local
Remotes: s3
Workspace directory: NTFS on E:\
Repo: dvc, git
 
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-09-01 10:39:46,647 DEBUG: Analytics is enabled.
2021-09-01 10:39:47,827 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\ AppData\\Local\\Temp\\tmpgrc_fch0']'
2021-09-01 10:39:50,601 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\ AppData\\Local\\Temp\\tmpgrc_fch0']'
 
$ dvc config --list
remote.cranescloud.url=s3://scloud
remote.cranescloud.endpointurl=https:XXXXXXXX.vpce.amazonaws.com
remote.cranescloud.ssl_verify=false
core.remote=cloud
cache.slow_link_warning=false
@isidentical isidentical self-assigned this Sep 8, 2021
@isidentical isidentical added fs: s3 Related to the S3 filesystem upstream Issues which need to be resolved in an upstream dependency labels Sep 8, 2021
@isidentical
Copy link
Contributor

Reported to aiobotocore at aio-libs/aiobotocore#883

@efiop
Copy link
Member

efiop commented Nov 9, 2021

For the record: the fix got merged and released in aiobotocore, but s3fs is pinned for now to the older one fsspec/s3fs#528 🙁

@daavoo daavoo added the A: data-sync Related to dvc get/fetch/import/pull/push label Feb 22, 2022
@dberenbaum
Copy link
Contributor

Looks like s3fs is now on aiobotocore 2.4.0 in https://github.com/fsspec/s3fs/blob/main/requirements.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: data-sync Related to dvc get/fetch/import/pull/push fs: s3 Related to the S3 filesystem upstream Issues which need to be resolved in an upstream dependency
Projects
None yet
Development

No branches or pull requests

5 participants