Add wheels to PyPI #219

henningpeters · 2016-04-12T08:57:05Z

First, thanks for building and maintaining ujson. We at spaCy use it pretty much everywhere where we deal with json.

While many users don't mind installing from source, now as we have with wheels a solid binary distribution format it would be great if you could adopt it and provide wheels for OS X and Windows on PyPI.

Setting up a build environment on Windows is particularly tricky as there is no simple guide you can follow that applies to all Python versions. What makes matters worse is that pip's error on a missing msvc compiler is unnecessarily cryptic and people are easily confused, see explosion/spaCy#334

The Python community is already pretty far in adopting wheels, see http://pythonwheels.com, unfortunately ujson still belongs in the not-supporting camp. As a popular Python package I urge you guys to adopt wheels.

If you would provide wheels spaCy could be installed without build system on OS X and Windows as ujson is the only non-wheel dependency missing. I guess there will be many other projects that will benefit from it as well. If you think that's not possible we will probably have to switch to a slower alternative and/or provide a graceful fallback, that's not my preference though.

If you need help setting up an automatic build system I can lend a hand. I would be also happy to provide ours (https://ci.spacy.io) to you.

Tirthankar25 · 2017-01-30T20:19:47Z

Is whl for Spacy for python 2.7 64-bit available now?

Tirthankar25 · 2017-01-31T03:08:41Z

Cannot install spacy in Anaconda python2.7-64-bit in Windows 10. Any help appreciated.

Installing collected packages: thinc, cloudpickle, spacy
  Found existing installation: thinc 6.3.0
    Uninstalling thinc-6.3.0:
      Successfully uninstalled thinc-6.3.0
  Running setup.py install for thinc ... error
...
copying thinc\extra\search.pxd -> build\lib.win-amd64-2.7\thinc\extra
    copying thinc\extra\__init__.pxd -> build\lib.win-amd64-2.7\thinc\extra
    running build_ext
    building 'thinc.linalg' extension
    error: INCLUDE environment variable is empty

    ----------------------------------------
  Rolling back uninstall of thinc

Jahaja · 2017-02-09T21:41:11Z

The reason this hasn't been done is just because it feels like such a hassle for platforms mainly used for dev. And would we do it once we'd have to continue supporting it. But we'll see. I'll look into how messy this is.

graingert · 2017-03-13T18:59:46Z

@Jahaja I'd really appreciate wheels in the package too, as it's something that blocks us from installing Sanic easily.

There's a demo for how to build them automatically for linux: https://github.com/pypa/python-manylinux-demo

Miserlou · 2017-04-24T14:23:45Z

We would very much appreciate ujson as a manylinux wheel, so that we can use it here with API Star (and Sanic) on Lambda: Miserlou/Zappa#801

audiolion · 2017-04-29T10:05:21Z

It would also help out when you cant use gcc on the server, my sysadmins dont allow gcc so I cannot build from source.

ethanhs · 2017-08-14T21:36:54Z

I have wheels for 2.7-3.6 building see #280. Im working on Windows wheels now.

ethanhs · 2017-08-14T21:52:10Z

Windows wheels are looking to be a bit more complicated. For example VS 2008 doesn't have stdint.h (which is easy enough to work around by doing the defines on Windows manually), and ultrajson.h isn't getting included correctly.

v1r · 2017-11-21T20:00:43Z

@ethanhs any update on this?

ethanhs · 2017-11-21T20:05:30Z

@v1r I have an open PR for manylinux wheels, I haven't gotten to Windows wheels. I haven't had a response on my PR so I'm worried about #291.

bityob · 2018-04-26T15:59:18Z

There are unofficial windows wheels for ujson, here.

lfd.uci.edu/~gohlke/pythonlibs is a great site, with a huge list of windows binaries for Python.

hugovk · 2020-02-24T15:18:51Z

Wheels would be great for the next release, see #352 and #355.

ethanhs · 2020-02-24T21:54:03Z

Hi @hugovk! I'm happy to reopen my PR

asottile · 2020-03-12T18:48:47Z

here's a script which builds the manylinux wheels:

#!/usr/bin/env python3
import argparse
import os.path
import re
import subprocess

VERSION = re.compile(r':: Python :: (\d\.\d)$')

EXES = []
with open('setup.py') as f:
    for line in f:
        match = VERSION.search(line)
        if match:
            major_s, minor_s = match[1].split('.')
            major, minor = int(major_s), int(minor_s)
            if (major, minor) == (2, 7):
                EXES.extend(('cp27-cp27mu', 'cp27-cp27m'))
            elif (major, minor) < (3, 8):
                EXES.append(f'cp{major}{minor}-cp{major}{minor}m')
            else:
                EXES.append(f'cp{major}{minor}-cp{major}{minor}')


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('version')
    args = parser.parse_args()

    pkg = f'ujson=={args.version}'

    os.makedirs('dist', exist_ok=True)
    for exe in EXES:
        pip = f'/opt/python/{exe}/bin/pip'

        if subprocess.call((
                'docker', 'run', '--rm',
                # so files are not root-owned
                '--user', f'{os.getuid()}:{os.getgid()}',
                '--volume', f'{os.path.abspath("dist")}:/dist:rw',
                'quay.io/pypa/manylinux1_x86_64:latest',
                'bash', '-euxc',
                f'{pip} wheel -w /tmp/wheels --no-deps {pkg} && '
                f'auditwheel repair -w /dist /tmp/wheels/*.whl',
        )):
            return 1
    return 0

if __name__ == '__main__':
    exit(main())

feel free to steal / adjust it

asottile · 2020-03-12T18:49:32Z

output:

$ ./bin/build-manylinux 2.0.1
...
$ ls dist/
ujson-2.0.1-cp27-cp27m-manylinux1_x86_64.whl
ujson-2.0.1-cp27-cp27mu-manylinux1_x86_64.whl
ujson-2.0.1-cp35-cp35m-manylinux1_x86_64.whl
ujson-2.0.1-cp36-cp36m-manylinux1_x86_64.whl
ujson-2.0.1-cp37-cp37m-manylinux1_x86_64.whl
ujson-2.0.1-cp38-cp38-manylinux1_x86_64.whl

vltr · 2020-03-12T19:57:03Z

I really prefer to create wheel files using docker images for greater binary compatibility. You can see how I did that in a Cython project of mine. Also, the bash script that does the actual build. I kind of "borrowed" the idea from here 😅

asottile · 2020-03-12T19:58:43Z

@vltr that's exactly what the script does if you read it ;)

vltr · 2020-03-12T20:01:58Z

@asottile argh, I'm really sorry, I think I passed through the docker call 😬

hugovk · 2020-03-12T21:39:37Z

Thanks! I've added that to the repo in #379. Once that's merged, I suggest doing this:

Upload the wheels to TestPyPI for a sanity test
Upload the wheels to production PyPI

goanpeca · 2020-04-06T18:02:28Z

Windows

wgordon17 · 2020-04-13T20:01:16Z

Any luck with MacOS wheels?

ifduyue · 2020-04-29T11:21:37Z

Maybe consider cibuildwheel?
It supports building wheels in GitHub Action for Mac, Linux and Windows.

bwoodsend · 2020-05-05T14:58:45Z

Hi, I've set up a few C extension wheel building pipelines on TravisCI before. I could do this pretty easily for ujson and resolve this issue permanently. Would you be ok for me to do this?

goanpeca · 2020-05-05T15:01:07Z

It would be better to just use Github Actions like the current situation, which makes things simpler and avoids having to deal with multiple CI systems.

bwoodsend · 2020-05-05T15:08:29Z

Ok, no problem. Let me know if you change your mind. Travis is pretty easy.

goanpeca · 2020-05-05T15:09:43Z

Ok, no problem. Let me know if you change your mind.

Not my call :-)

Travis is pretty easy.

So are github actions 🤷

bwoodsend · 2020-05-05T15:27:54Z

So are github actions

Alright fair enough. I'd just want to see this resolved.

hugovk · 2020-05-05T15:33:57Z

Yes please! GitHub Actions is preferred, but Travis CI is fine too!

vltr · 2020-05-05T15:35:13Z

@hugovk and everyone else, sorry for not being able to see this (yet). Family issues + Covid-19 + work adaptations made any of my extra time to vanish into thin air 😬

hugovk · 2020-05-05T15:45:56Z

@vltr No problem whatsoever, all the best.

bwoodsend · 2020-05-05T17:49:17Z

OK. I've setup a github action and got it working. @goanpeca I take back what I said about TravisCI - github actions are much easier. I'll to a pull request in a bit.

Are we still trying to support Python 2.7? It's causing issues on Mac.

goanpeca · 2020-05-05T18:06:42Z

Awesome @bwoodsend thanks for working on it :-)

hugovk · 2020-05-05T18:58:59Z

Thank you! No need to support 2.7 any more, it's just been dropped in #404.

hugovk · 2020-05-06T14:48:45Z

Thanks to @bwoodsend we now have Windows and macOS wheels for Python 3.5-3.8 ready for testing!

It would be great if some people could test out -- but not in production -- the wheels, especially on Windows.

You can find them here:

https://test.pypi.org/project/ujson/2.0.4.dev47/#files

hugovk · 2020-05-06T14:50:52Z

✅ Smoke test on macOS:

import sys

print(sys.version)
import ujson

print(ujson.__version__)
print(ujson.dumps([{"key": "value"}, 81, True]))

python3.5 -m pip install ujson-2.0.4.dev47-cp35-cp35m-macosx_10_13_x86_64.whl
python3.6 -m pip install ujson-2.0.4.dev47-cp36-cp36m-macosx_10_13_x86_64.whl
python3.7 -m pip install ujson-2.0.4.dev47-cp37-cp37m-macosx_10_13_x86_64.whl
python3.8 -m pip install ujson-2.0.4.dev47-cp38-cp38-macosx_10_13_x86_64.whl

python3.5 smoke.py
python3.6 smoke.py
python3.7 smoke.py
python3.8 smoke.py

Processing ./ujson-2.0.4.dev47-cp35-cp35m-macosx_10_13_x86_64.whl
Installing collected packages: ujson
Successfully installed ujson-2.0.4.dev47
Processing ./ujson-2.0.4.dev47-cp36-cp36m-macosx_10_13_x86_64.whl
Installing collected packages: ujson
Successfully installed ujson-2.0.4.dev47
Processing ./ujson-2.0.4.dev47-cp37-cp37m-macosx_10_13_x86_64.whl
Installing collected packages: ujson
Successfully installed ujson-2.0.4.dev47
WARNING: You are using pip version 20.0.2; however, version 20.1 is available.
You should consider upgrading via the '/Users/hugo/.pyenv/versions/3.7.7/bin/python3.7 -m pip install --upgrade pip' command.
Requirement already satisfied: ujson==2.0.4.dev47 from file:///private/tmp/Downloads/ujson-2.0.4.dev47-cp38-cp38-macosx_10_13_x86_64.whl in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (2.0.4.dev47)
3.5.9 (default, Mar 22 2020, 15:35:53)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)]
2.0.4.dev47
[{"key":"value"},81,true]
3.6.10 (default, Mar 22 2020, 15:27:16)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)]
2.0.4.dev47
[{"key":"value"},81,true]
3.7.7 (default, Mar 22 2020, 15:42:40)
[Clang 10.0.1 (clang-1001.0.46.4)]
2.0.4.dev47
[{"key":"value"},81,true]
3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)]
2.0.4.dev47
[{"key":"value"},81,true]

hugovk · 2020-05-06T16:01:10Z

✅ Same smoke test on macOS-latest, windows-2019 and windows-2016 for Python 3.5-3.8 on GitHub Actions:

bwoodsend · 2020-05-06T16:04:29Z

I can confirm the Windows ones work. Don't ask me what's going on with my Python PATHs. @hugovk I wish my python installations were as organised as yours.

Python3.5

> λ C:\test_environments\vpl_testing\py_3_5_3\python -m pip install -i https://test.pypi.org/simple/ ujson==2.0.4.dev47
Requirement already satisfied: ujson==2.0.4.dev47 in c:\test_environments\vpl_testing\py_3_5_3\lib\site-packages
You are using pip version 9.0.1, however version 20.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

> λ C:\test_environments\vpl_testing\py_3_5_3\python.exe ujson-smoke.py
3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 16:02:32) [MSC v.1900 64 bit (AMD64)]
2.0.4.dev47
[{"key":"value"},81,true]

Python3.6

> λ C:\test_environments\vpl_testing\py_3_6_0\python -m pip install -i https://test.pypi.org/simple/ ujson==2.0.4.dev47
Looking in indexes: https://test.pypi.org/simple/
Collecting ujson==2.0.4.dev47
  Downloading https://test-files.pythonhosted.org/packages/22/02/d3eb57c4b6933794d510abb153487fb7790a7162a79bd7e9b093d3e88992/ujson-2.0.4.dev47-cp36-cp36m-win_amd64.whl (43 kB)
     |████████████████████████████████| 43 kB 213 kB/s
Installing collected packages: ujson
Successfully installed ujson-2.0.4.dev47

> λ C:\test_environments\vpl_testing\py_3_6_0\\python.exe ujson-smoke.py
3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)]
2.0.4.dev47
[{"key":"value"},81,true]

Python3.7

> λ E:\pythons\py_3_7_7\python -m pip install -i https://test.pypi.org/simple/ ujson==2.0.4.dev47
Looking in indexes: https://test.pypi.org/simple/
Collecting ujson==2.0.4.dev47
  Downloading https://test-files.pythonhosted.org/packages/0f/65/3007df5aff41c83eff7c779b5bddee48d575f3c189c410dd8cf3d03dbc20/ujson-2.0.4.dev47-cp37-cp37m-win_amd64.whl (43kB)
     |████████████████████████████████| 51kB 409kB/s
Installing collected packages: ujson
Successfully installed ujson-2.0.4.dev47

> λ E:\pythons\py_3_7_7\python ujson-smoke.py
3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)]
2.0.4.dev47
[{"key":"value"},81,true]

Python3.8 (my default).

> pip install -i https://test.pypi.org/simple/ ujson==2.0.4.dev47
Looking in indexes: https://test.pypi.org/simple/
Collecting ujson==2.0.4.dev47
  Downloading https://test-files.pythonhosted.org/packages/dd/28/cb07a3c3d1822927577978e07b6da724a07c577db7c06f00a29be24e77c3/ujson-2.0.4.dev47-cp38-cp38-win_amd64.whl (43 kB)
     |████████████████████████████████| 43 kB 274 kB/s
Installing collected packages: ujson
  Attempting uninstall: ujson
    Found existing installation: ujson 1.35
    Uninstalling ujson-1.35:
      Successfully uninstalled ujson-1.35
Successfully installed ujson-2.0.4.dev47

> λ python ujson-smoke.py
3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)]
2.0.4.dev47
[{"key":"value"},81,true]

bwoodsend · 2020-05-06T16:05:04Z

Ahh @hugovk I see you beat me to it...

hugovk · 2020-05-06T16:14:01Z

Thanks, good to see it working in other environments too :)

(I use pyenv for non-3.8 installations.)

bwoodsend · 2020-05-06T16:29:44Z

(I use pyenv for non-3.8 installations.)

Thanks for sharing that. I've found all kinds of environment managers that seem to do everything except multiple Pythons.

bwoodsend · 2020-05-06T16:33:18Z

Any idea when this can propagate to normal PyPi? I can't wait to be shot of all my statements along the lines of:

try:
    import ujson as json
except ImportError:
    import json

hugovk · 2020-05-06T16:36:08Z

Within a week or so?

I'd like to give others a few days to test out too.

bwoodsend · 2020-05-06T16:49:49Z

OK. Looking forward to it. Big thanks for letting and helping me do this. 🚀🚀

hugovk · 2020-06-03T16:52:56Z

"A week or so" became 28 days, but wheels are now released on PyPI!

https://pypi.org/project/ujson/3.0.0/#files

Thanks everyone!

hugovk mentioned this issue Feb 24, 2020

UltraJSON 2.0.0 release checklist #352

Closed

6 tasks

hugovk added the release Deploy and release label Feb 24, 2020

This was referenced Feb 27, 2020

Autodeploy to PyPI from Travis CI #357

Merged

wheel for easy windows installation #313

Closed

hugovk pinned this issue Feb 29, 2020

hugovk added the help wanted Extra attention is needed label Feb 29, 2020

hugovk mentioned this issue Feb 29, 2020

Only deploy sdist from Travis CI #359

Merged

This was referenced Mar 7, 2020

Is this project still maintained? #343

Closed

v2.0.0 fails to build on Windows #369

Closed

hugovk mentioned this issue Mar 12, 2020

Error when installing ujson in Docker image python:3.8-slim-buster #378

Closed

hugovk mentioned this issue Mar 12, 2020

Add script to build wheels with manylinux #379

Merged

bwoodsend mentioned this issue May 5, 2020

Build Windows and MacOS #405

Merged

hugovk mentioned this issue Jun 2, 2020

MacOS failure installing ujson 2.0.3 #403

Closed

hugovk closed this as completed Jun 3, 2020

hugovk unpinned this issue Sep 26, 2020

hugovk mentioned this issue Sep 26, 2020

Question: Comparison with other Python-JSON packages #428

Closed

Add wheels to PyPI #219

Add wheels to PyPI #219

Comments

henningpeters commented Apr 12, 2016

Tirthankar25 commented Jan 30, 2017

Tirthankar25 commented Jan 31, 2017 • edited by hugovk

Jahaja commented Feb 9, 2017

graingert commented Mar 13, 2017

Miserlou commented Apr 24, 2017 • edited

audiolion commented Apr 29, 2017

ethanhs commented Aug 14, 2017

ethanhs commented Aug 14, 2017

v1r commented Nov 21, 2017

ethanhs commented Nov 21, 2017

bityob commented Apr 26, 2018 • edited

hugovk commented Feb 24, 2020

ethanhs commented Feb 24, 2020

asottile commented Mar 12, 2020

asottile commented Mar 12, 2020

vltr commented Mar 12, 2020

asottile commented Mar 12, 2020

vltr commented Mar 12, 2020

hugovk commented Mar 12, 2020 • edited

goanpeca commented Apr 6, 2020

wgordon17 commented Apr 13, 2020

ifduyue commented Apr 29, 2020

bwoodsend commented May 5, 2020

goanpeca commented May 5, 2020

bwoodsend commented May 5, 2020

goanpeca commented May 5, 2020

bwoodsend commented May 5, 2020

hugovk commented May 5, 2020

vltr commented May 5, 2020

hugovk commented May 5, 2020

bwoodsend commented May 5, 2020

goanpeca commented May 5, 2020

hugovk commented May 5, 2020

hugovk commented May 6, 2020

hugovk commented May 6, 2020

hugovk commented May 6, 2020

bwoodsend commented May 6, 2020 • edited by hugovk

Python3.5

Python3.6

Python3.7

Python3.8 (my default).

bwoodsend commented May 6, 2020

hugovk commented May 6, 2020

bwoodsend commented May 6, 2020

bwoodsend commented May 6, 2020 • edited by hugovk

hugovk commented May 6, 2020

bwoodsend commented May 6, 2020

hugovk commented Jun 3, 2020 • edited

Tirthankar25 commented Jan 31, 2017 •

edited by hugovk

Miserlou commented Apr 24, 2017 •

edited

bityob commented Apr 26, 2018 •

edited

hugovk commented Mar 12, 2020 •

edited

bwoodsend commented May 6, 2020 •

edited by hugovk

bwoodsend commented May 6, 2020 •

edited by hugovk

hugovk commented Jun 3, 2020 •

edited