Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kedro-datasets: Rewrite the s3 datasets test #584

Open
ankatiyar opened this issue Feb 28, 2024 · 0 comments
Open

kedro-datasets: Rewrite the s3 datasets test #584

ankatiyar opened this issue Feb 28, 2024 · 0 comments

Comments

@ankatiyar
Copy link
Contributor

Description

Related: #463 #360

Context

Some xfail tests since bumping s3fs version -

  • tests/dask/test_parquet_dataset.py- test_pass_credentials
  • tests/spark/test_spark_dataset.py - test_no_version
  • tests/netcdf/test_netcdf_dataset.py- test_pass_credentials

More tests skipped in #360 which load the dataset from s3 object -
This is because NetCDFDataset creates a local copy of the remote data to load them in a temp_path. This uses self._fs.get() which is causing the tests to hang indefinitely because it doesn't play well with the mocked s3 objects.

Possible Implementation

Possible Alternatives

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant