Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the package versions in the CI-generated wheels #1121

Merged

Conversation

keitherskine
Copy link
Collaborator

The wheels generated by cibuildwheel in our CI pipeline currently have the wrong pyodbc package version id.s For example, the file "pyodbc-4.0.dev0-cp310-cp310-win_amd64.whl". "4.0.dev0" is incorrect and should be something like "4.0.35". The wheels are given the id "4.0.dev0" because setup.py is unable to find any tags in the available git repo. When building pyodbc locally on laptops, setup.py usually figures out the version from the git tags in the local repo. setup.py is unable to figure this out in the CI pipeline because Github Actions does not include the tags when it fetches the repo. It uses a command similar to:

/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +cafa37e846e09b36ebf821ac69b6f7d35f423baa:refs/remotes/origin/fix-labels-on-cicd-wheels

Note the options --no-tags and --depth=1.

To get around this, I have introduced a constant to setup.py called VERSION, currently of value "4.0.35". When setup.py is called within the CI pipelines, setup.py will detect this and use the VERSION constant. This constant will naturally have to be manually maintained but that shouldn't be too onerous.

Whilst here, I have added a basic pyproject.toml file to help pyodbc be more PEP517-compliant. We should probably make more use of pyproject.toml in the future, transferring the static elements of setup.py into it at some point.

Also, a tweak to .gitignore.

@keitherskine
Copy link
Collaborator Author

Just to emphasize, once this PR is merged, we should be able to let cibuildwheel in Github Actions build all our wheels for us going forward, for uploading to PyPi. cibuildwheel has the big advantage of being able to build wheels for a wide range of operating systems, and any future ones too.

cibuildwheel won't build Python 2.7 wheels anymore but we can get AppVeyor to generate one last set of 2.7 Windows wheels if needed (by setting APVYR_GENERATE_WHEELS to "true" in the AppVeyor portal). As I mentioned before, maintaining support for Python 2.7 is becoming a bit problematic. Not many other Python packages, utilities, and applications support 2.7 anymore. It might be time to drop 2.7 and move on to version 5 of pyodbc.

..or else PyPi will reject them:

Binary wheel 'pyodbc-4.0.35-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'

If repairing, must "exclude" all the .so files from the wheel so they don't interfere with any existing installations.
@keitherskine
Copy link
Collaborator Author

keitherskine commented Nov 12, 2022

FYI, I've uploaded a deployment based on this branch to the Test PyPi index, here:
https://test.pypi.org/project/pyodbc-keef-v1/
The installed package name is the same, i.e. "pyodbc", it's just a different catalog entry in the Python Package Index.

All the wheels were generated by the CI pipelines. Feel free to try installing them at your leisure:
pip install -i https://test.pypi.org/simple/ pyodbc-keef-v1

Note, the linux wheels do not include any .so files anymore.

Copy link
Collaborator

@gordthompson gordthompson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job, Keith! Good catch on the PiPI naming thing.

Wheels from the PyPI test site install just fine under Py3.11 on Windows and Py3.10 on Ubuntu 22.04. As expected, pyodbc won't run unless unixODBC is installed on Linux (verifying that the .so files are not bundled in the wheel).

So +1 from me!

ping @mkleehammer re: this and #1096

@keitherskine
Copy link
Collaborator Author

Thanks, Gord. I too have checked the install works from the Test PyPi on Windows 64-bit (Python 2.7, 3.6-3.11), Linux Ubuntu 20.04 (Python 3.10), and Mac (not ARM) (Python 3.8). I'll merge this PR now to get it in the master branch, but if we need to make changes subject to review by Kleehammer, that will be easy enough. It'll be great to drop Python 2.7 at some point but it'll be even better to fix the Linux wheels through version 4.0.35 in the meantime.

@keitherskine keitherskine merged commit ec1e3a7 into mkleehammer:master Nov 13, 2022
@keitherskine
Copy link
Collaborator Author

Further to this PR, here are my thoughts on how a new release should be deployed. Happy to hear any feedback on this. Maybe add this to the Wiki?

  1. The package version for the new release should already be defined in setup.py (see the VERSION constant). For example "5.2.0". If that value is not correct, change it and commit.

  2. Choose the commit that will be used for the release. This will almost always be the head of the master branch, but it can be any commit.

  3. In your local pyodbc repository, create an annotated git tag for the chosen commit, as follows:
    (If you are not already on the commit git checkout <commit hash>)
    git tag -a <package version>
    You will be prompted to enter a message for the tag. Give a short description of the release, a full description will be added to the Github release notes later.
    Push this tag to Github.

  4. Delete all the files in your top-level "dist" directory (create the "dist" directory if necessary).

  5. From the Github Actions page, find the commit in the "Ubuntu build" workflow and download the two zipped artifacts "sdist" and "wheels" into your "dist" directory. These artifacts cover all the files for Python 3.
    https://github.com/mkleehammer/pyodbc/actions/workflows/ubuntu_build.yml

  6. From the AppVeyor History page, find the commit and download the two artifacts for Python 2.7 (both 32-bit and 64-bit) from the build page into your "dist" directory.
    https://ci.appveyor.com/project/mkleehammer/pyodbc/history
    (If there are no artifacts, go to the Environment page in Settings, and set the environment variable "APVYR_GENERATE_WHEELS" to "true", then re-build the commit from the build page)

  7. In your "dist" directory, unzip the two zip files "sdist" and "wheels" to retrieve the sdist/wheel files within them. Delete the zip files. Don't unzip the AppVeyor .tar.gz files.

  8. From the command prompt, navigate to the top-level pyodbc directory, and then upload the contents of the "dist" directory to the Test PyPi index:
    twine upload -r testpypi dist/*
    Check https://test.pypi.org/project/pyodbc/ to make sure the files were uploaded successfully.

  9. If that is successful, upload the contents of the "dist" directory to the real PyPi index.
    Bear in mind, you get only one shot at this! You can't replace files in PyPi, so make sure you get this right first time:
    twine upload dist/*
    Check https://pypi.org/project/pyodbc/ to make sure the files were uploaded successfully.

  10. Create release notes on the Github portal, based on the git tag created earlier. This will be the main source of information about the release so make it reasonably comprehensive.
    https://github.com/mkleehammer/pyodbc/releases

  11. Finally, increment the VERSION constant in setup.py to the next release, and commit. For example, from "5.2.0" to "5.3.0". And that's it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants