BigQuery: numpy release breaks Python 2.7 unit tests #8549
Assignees
Labels
api: bigquery
Issues related to the BigQuery API.
priority: p1
Important issue which blocks shipping the next release. Will be fixed prior to next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Comments
tswast
added
api: bigquery
|
Why are we getting a pre-release version? |
|
I'm not able to reproduce locally (on Mac). Maybe the bad package was deleted already? |
tswast
added
priority: p1
|
I can reproduce on Linux with a fresh Python 2 virtualenv. |
|
It's fastparquet's fault. |
|
@tswast on my Linux box (Ubuntu 16.04), I see failures on 3.5+ too: $ nox -e unit-3.5
nox > Running session unit-3.5
nox > Creating virtualenv using python3.5 in /home/tseaver/projects/agendaless/Google/src/gcp/bigquery/.nox/unit-3-5
nox > pip install --upgrade mock pytest pytest-cov
nox > pip install --upgrade -e ../api_core[grpc]
nox > pip install --upgrade -e ../core
nox > pip install --upgrade -e .[all]
nox > pip install --upgrade ipython
nox > py.test --quiet --cov=google.cloud.bigquery --cov=tests.unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=97 tests/unit
........................................................................ [ 6%]
........................................................................ [ 12%]
..............................................FF........................ [ 18%]
........................................................................ [ 25%]
........................................................................ [ 31%]
........................................................................ [ 37%]
........................................................................ [ 44%]
........................................................................ [ 50%]
........................................................................ [ 56%]
........................................................................ [ 62%]
..........................................................F.........F... [ 69%]
........................................................................ [ 75%]
........................................................................ [ 81%]
........................................................................ [ 88%]
..............................................F......................... [ 94%]
..F.............................................................. [100%]
=================================== FAILURES ===================================
__________________ test_dataframe_to_parquet_without_pyarrow ___________________
module_under_test = <module 'google.cloud.bigquery._pandas_helpers' from '/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/google/cloud/bigquery/_pandas_helpers.py'>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f505a4d0d68>
@pytest.mark.skipIf(pandas is None, "Requires `pandas`")
def test_dataframe_to_parquet_without_pyarrow(module_under_test, monkeypatch):
monkeypatch.setattr(module_under_test, "pyarrow", None)
with pytest.raises(ValueError) as exc:
module_under_test.dataframe_to_parquet(pandas.DataFrame(), (), None)
> assert "pyarrow is required" in str(exc)
E AssertionError: assert 'pyarrow is required' in '<ExceptionInfo ValueError tblen=2>'
E + where '<ExceptionInfo ValueError tblen=2>' = str(<ExceptionInfo ValueError tblen=2>)
tests/unit/test__pandas_helpers.py:538: AssertionError
_________________ test_dataframe_to_parquet_w_missing_columns __________________
module_under_test = <module 'google.cloud.bigquery._pandas_helpers' from '/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/google/cloud/bigquery/_pandas_helpers.py'>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f505a527390>
@pytest.mark.skipIf(pandas is None, "Requires `pandas`")
@pytest.mark.skipIf(pyarrow is None, "Requires `pyarrow`")
def test_dataframe_to_parquet_w_missing_columns(module_under_test, monkeypatch):
with pytest.raises(ValueError) as exc:
module_under_test.dataframe_to_parquet(
pandas.DataFrame(), (schema.SchemaField("not_found", "STRING"),), None
)
> assert "columns in schema must match" in str(exc)
E AssertionError: assert 'columns in schema must match' in '<ExceptionInfo ValueError tblen=3>'
E + where '<ExceptionInfo ValueError tblen=3>' = str(<ExceptionInfo ValueError tblen=3>)
tests/unit/test__pandas_helpers.py:548: AssertionError
_____________ test__make_bqstorage_client_true_raises_import_error _____________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f505a388f60>
def test__make_bqstorage_client_true_raises_import_error(monkeypatch):
monkeypatch.setattr(magics, "bigquery_storage_v1beta1", None)
credentials_mock = mock.create_autospec(
google.auth.credentials.Credentials, instance=True
)
with pytest.raises(ImportError) as exc:
magics._make_bqstorage_client(True, credentials_mock)
> assert "google-cloud-bigquery-storage" in str(exc)
E AssertionError: assert 'google-cloud-bigquery-storage' in '<ExceptionInfo ImportError tblen=2>'
E + where '<ExceptionInfo ImportError tblen=2>' = str(<ExceptionInfo ImportError tblen=2>)
tests/unit/test_magics.py:279: AssertionError
______________ test_bigquery_magic_w_maximum_bytes_billed_invalid ______________
@pytest.mark.usefixtures("ipython_interactive")
def test_bigquery_magic_w_maximum_bytes_billed_invalid():
ip = IPython.get_ipython()
ip.extension_manager.load_extension("google.cloud.bigquery")
magics.context._project = None
sql = "SELECT 17 AS num"
with pytest.raises(ValueError):
> ip.run_cell_magic("bigquery", "--maximum_bytes_billed=abc", sql)
tests/unit/test_magics.py:565:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.nox/unit-3-5/lib/python3.5/site-packages/IPython/core/interactiveshell.py:2358: in run_cell_magic
result = fn(*args, **kwargs)
google/cloud/bigquery/magics.py:400: in _cell_magic
default_query_job_config=context.default_query_job_config,
google/cloud/bigquery/client.py:167: in __init__
project=project, credentials=credentials, _http=_http
../core/google/cloud/client.py:226: in __init__
_ClientProjectMixin.__init__(self, project=project)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <google.cloud.bigquery.client.Client object at 0x7f505a37f320>
project = None
def __init__(self, project=None):
project = self._determine_default(project)
if project is None:
raise EnvironmentError(
> "Project was not passed and could not be "
"determined from the environment."
)
E OSError: Project was not passed and could not be determined from the environment.
../core/google/cloud/client.py:181: OSError
------------------------------ Captured log call -------------------------------
WARNING google.auth._default:_default.py:314 No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
WARNING google.auth._default:_default.py:314 No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
______ TestRowIterator.test_to_dataframe_w_bqstorage_raises_import_error _______
self = <tests.unit.test_table.TestRowIterator testMethod=test_to_dataframe_w_bqstorage_raises_import_error>
@unittest.skipIf(pandas is None, "Requires `pandas`")
@unittest.skipIf(
bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
)
def test_to_dataframe_w_bqstorage_raises_import_error(self):
from google.cloud.bigquery import table as mut
bqstorage_client = mock.create_autospec(
bigquery_storage_v1beta1.BigQueryStorageClient
)
path = "/foo"
api_request = mock.Mock(return_value={"rows": []})
row_iterator = mut.RowIterator(
_mock_client(), api_request, path, [], table=mut.Table("proj.dset.tbl")
)
with mock.patch.object(mut, "bigquery_storage_v1beta1", None), pytest.raises(
ValueError
) as exc:
row_iterator.to_dataframe(bqstorage_client=bqstorage_client)
> assert mut._NO_BQSTORAGE_ERROR in str(exc)
E AssertionError: assert 'The google-cloud-bigquery-storage library is not installed, please install google-cloud-bigquery-storage to use bqstorage features.' in '<ExceptionInfo ValueError tblen=6>'
E + where 'The google-cloud-bigquery-storage library is not installed, please install google-cloud-bigquery-storage to use bqstorage features.' = <module 'google.cloud.bigquery.table' from '/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/google/cloud/bigquery/table.py'>._NO_BQSTORAGE_ERROR
E + and '<ExceptionInfo ValueError tblen=6>' = str(<ExceptionInfo ValueError tblen=6>)
tests/unit/test_table.py:2231: AssertionError
____________ test_table_reference_to_bqstorage_raises_import_error _____________
@unittest.skipIf(
bigquery_storage_v1beta1 is None, "Requires `google-cloud-bigquery-storage`"
)
def test_table_reference_to_bqstorage_raises_import_error():
from google.cloud.bigquery import table as mut
classes = (mut.TableReference, mut.Table, mut.TableListItem)
for cls in classes:
with mock.patch.object(mut, "bigquery_storage_v1beta1", None), pytest.raises(
ValueError
) as exc:
cls.from_string("my-project.my_dataset.my_table").to_bqstorage()
> assert mut._NO_BQSTORAGE_ERROR in str(exc)
E AssertionError: assert 'The google-cloud-bigquery-storage library is not installed, please install google-cloud-bigquery-storage to use bqstorage features.' in '<ExceptionInfo ValueError tblen=2>'
E + where 'The google-cloud-bigquery-storage library is not installed, please install google-cloud-bigquery-storage to use bqstorage features.' = <module 'google.cloud.bigquery.table' from '/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/google/cloud/bigquery/table.py'>._NO_BQSTORAGE_ERROR
E + and '<ExceptionInfo ValueError tblen=2>' = str(<ExceptionInfo ValueError tblen=2>)
tests/unit/test_table.py:2519: AssertionError
=============================== warnings summary ===============================
.nox/unit-3-5/lib/python3.5/distutils/__init__.py:4
/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/.nox/unit-3-5/lib/python3.5/distutils/__init__.py:4: PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
.nox/unit-3-5/lib/python3.5/site-packages/_pytest/mark/structures.py:332
.nox/unit-3-5/lib/python3.5/site-packages/_pytest/mark/structures.py:332
/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/.nox/unit-3-5/lib/python3.5/site-packages/_pytest/mark/structures.py:332: PytestUnknownMarkWarning: Unknown pytest.mark.skipIf - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
PytestUnknownMarkWarning,
tests/unit/test__pandas_helpers.py::test_bq_to_arrow_data_type_w_struct_unknown_subfield
/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/google/cloud/bigquery/_pandas_helpers.py:143: UserWarning: Unable to determine type for field 'field3'.
warnings.warn("Unable to determine type for field '{}'.".format(bq_field.name))
tests/unit/test_magics.py::test_bigquery_magic_w_maximum_bytes_billed_invalid
tests/unit/test_magics.py::test_bigquery_magic_w_maximum_bytes_billed_invalid
/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/.nox/unit-3-5/lib/python3.5/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
tests/unit/test_table.py::TestRowIterator::test_to_dataframe_tqdm_error
/home/tseaver/projects/agendaless/Google/src/gcp/bigquery/tests/unit/test_table.py:1634: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
df = row_iterator.to_dataframe(progress_bar_type=progress_bar_type)
-- Docs: https://docs.pytest.org/en/latest/warnings.html
Required test coverage of 97% reached. Total coverage: 99.90%
6 failed, 1139 passed, 7 warnings in 20.35 seconds
nox > Command py.test --quiet --cov=google.cloud.bigquery --cov=tests.unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=97 tests/unit failed with exit code 1
nox > Session unit-3.5 failed. |
|
I'm guessing that this is also related to the prerelease version of numpy. |
|
I see a bunch of noise about old versions of I don't think upgrading |
This was referenced Jul 1, 2019
Closed
|
I am also start getting this error as of now. It was working okay few days ago. Any clue whats going on? |
|
@omeryounus Numpy made a broken pre-release, and fastparquet (unnecessarily?) depends on a pre-release version of numpy when installed from the source distribution. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As discovered in #8548
The text was updated successfully, but these errors were encountered: