Pyston (faster CPython 3.8) support

[SpaceMonkeyForever]

Hi,

Pyston is a much faster version of Python that's very compatible. I have been able to run all libraries I need in Pyston without issues (using prebuilt binaries on Ubuntu 20.04). It would be hugely beneficial if I can use Pyston in Rust instead of the plain Python interpreter which is a lot slower and typically the use case for Rust is speed.

Is it possible to build PyO3 against Pyston? or just use it directly? Where to start?

[davidhewitt]

So the Pyston docs mention cryptography, which has a Rust package and apparently works.

However I also see this mention on the readme:

Pyston is API compatible but not ABI compatible with CPython. This means that C extensions will work, but they need to be recompiled.

Without a closer investigation this makes me think that it's possible some functionality may not work as intended in Pyston. If you wanted to use this with confidence in production I think some effort would be needed to verify all functionality. I could detail this if you're wanting to invest the time on it.

In the end this might require additional support in pyo3_ffi. The amount of work would depend on how much Pyston's ABI deviates from CPython.

[SpaceMonkeyForever]

HI David @davidhewitt thanks for taking the time

To be clear, I want to use Pyston FROM Rust and not the other way around. If you would tell me where to start to even attempt that, it would be great! All I need is to call a function in Python because I have the Python code for it but it's very difficult to rewrite in Rust, so I want to simply call it from Rust and I want it to be as fast as possible so I'd rather use Pyston which proved to be a lot faster than CPython for me.

[SpaceMonkeyForever]

hmm that sounds like a difficult task. Would it be easier if I use pyston-lite module? How do I tell PyO3 to use a specific Python env (where I will have pyston-lite installed)?

[birkenfeld]

I just made a try using these steps:

1. use the pyston/pyston docker container
2. apt install cargo
3. cargo new --bin pyo3test
4. create example project with Cargo.toml and src/main.rs from https://pyo3.rs/v0.17.2/#using-python-from-rust
5. build with PYO3_PYTHON=pyston PYO3_PRINT_CONFIG=1 cargo build

This printed:

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.8
  shared=true
  abi3=false
  lib_name=python3.8-pyston2.3
  lib_dir=/usr/lib
  executable=/usr/bin/pyston
  pointer_width=64
  build_flags=
  suppress_build_script_link_lines=false

which looks good.

6. build and run: PYO3_PYTHON=pyston cargo run

This gave:

     Running `target/debug/pyo3test`
Hello Unknown, I'm Python 3.8.12 (remotes/origin/release_2.3.5:4b858b5062, Sep 25 2022, 18:56:33)
[Pyston 2.3.5, GCC 9.4.0]

which also looks good.

So I can confirm that at least in principle, building against Pyston works, can startup the interpreter and run simple code. However, the devil is in the details, as @davidhewitt said, and as long as we don't officially support Pyston you'll have to carefully confirm that your code works as expected.

[SpaceMonkeyForever]

Hi @birkenfeld I really appreciate that! I think

I would much rather not use docker though. I can install Pyston myself, but I don't understand how to tell the Rust compiler to use Pyston and not the default Python installation. I'm guessing I can find out by inspecting that docker image, but I thought there should be a simple flag or config that I can edit to point to Pyston library instead of CPython library. I also want to point to a specific venv (where I installed the needed libraries), I would imagine this is something that's supported by PyO3 in general? Or maybe I'm misunderstanding how this works and I can't use venv, I think it might be using its own interpreter, in which case I have to find out how to make python libraries (e.g. numpy) available.

[adamreichold]

As written above

> export PYO3_PYTHON=/path/to/pyston

[SpaceMonkeyForever]

@adamreichold thank you. Also, for future reference, I searched the docs and found this page about configuring Python version

Thank you very much everyone, I didn't expect this level of support. I really appreciate this, it's what makes the tech community great.