Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spark odbc / databricks integration tests #156

Closed
1 of 3 tasks
jeremyyeo opened this issue Sep 7, 2022 · 0 comments · Fixed by #159
Closed
1 of 3 tasks

Fix spark odbc / databricks integration tests #156

jeremyyeo opened this issue Sep 7, 2022 · 0 comments · Fixed by #159
Assignees
Labels
bug Something isn't working

Comments

@jeremyyeo
Copy link
Collaborator

jeremyyeo commented Sep 7, 2022

Describe the bug

The databricks integration tests are running into errors like:

23:35:24  Encountered an error:
Runtime Error
  Runtime Error
    Database Error
      failed to connect
23:35:24  Traceback (most recent call last):
  File "/home/dbt_test_user/project/venv/lib/python3.8/site-packages/dbt/adapters/spark/connections.py", line 436, in open
    conn = pyodbc.connect(connection_str, autocommit=True)
pyodbc.Error: ('HY000', '[HY000] [unixODBC][Simba][ODBC] (11560) Unable to locate SQLGetPrivateProfileString function. (11560) (SQLDriverConnect)')

This is due to pyodbc 4.0.34 (mkleehammer/pyodbc#1079) - the fix is to pin it dbt-labs/dbt-spark#398.

Steps to reproduce

I actually pulled the image (https://github.com/dbt-labs/dbt-external-tables/blob/main/.circleci/config.yml#L52) locally and ran it. If we install dbt-spark[ODBC] via:

pip install dbt-spark[ODBC] --upgrade --pre

image

If you then install pyodbc==4.0.32 then it comes right (alternatively pip install dbt-spark[ODBC]==1.2).

Expected results

Databricks tests to be passing.

Actual results

Databricks tests running into error shown above.

Screenshots and log output

https://app.circleci.com/pipelines/github/dbt-labs/dbt-external-tables/234/workflows/073054ce-0ebb-4491-9163-aaac7f98c1c0/jobs/1070

System information

The contents of your packages.yml file:

Which database are you using dbt with?

  • redshift
  • snowflake
  • other (specify: Databricks via Spark ODBC)

The output of dbt --version:

Core:
  - installed: 1.3.0-b2
  - latest:    1.2.1    - Ahead of latest version!

Plugins:
  - spark: 1.3.0b2 - Ahead of latest version!

The operating system you're using:

$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

The output of python --version:

3.8.6

Additional context

Pretty straightforward fix to pin pyodbc (or actually pip install dbt-spark[ODBC]==1.2 would work too.

@jeremyyeo jeremyyeo added bug Something isn't working triage labels Sep 7, 2022
@jeremyyeo jeremyyeo self-assigned this Sep 7, 2022
@jeremyyeo jeremyyeo removed the triage label Sep 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant