Failed to execute the basic example

[GoodManWEN]

Hi everyone, love this project as it just meets my recent needs. As a 'hello world' test, however, I'm having problems running the rust-parallel example.

According to the notes in the readme I need to first run nox to create a virtual environment. Since I am not familiar with nox I created the environment myself and installed the dependency manually.

~# python3 -m venv .env
~# source .env/bin/activate
(.env) ~# pip install numpy

Then I created a new project using marutin and manually copied all the files in rust-numpy/examples/parallel/ to overwrite the original.

(.env) ~# mkdir rust_parallel
(.env) ~# cd rust_parallel
(.env) ~/rust_parallel# maturin init
(.env) ~/rust_parallel# cp -r ../rust-numpy/examples/parallel/ .

When I try to run maturin develop, I got the following error

(.env) ~/rust_parallel# maturin develop
💥 maturin failed
  Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
  Caused by: `cargo metadata` exited with an error: error: failed to load manifest for dependency `numpy`

Caused by:
  failed to read `/home/rust/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
(.env) ~/rust_parallel# ^C

The error message indicates that numpy is not installed, which is not the case, and also the pointing of Cargo.toml is not correct. I'm not quite sure which step is wrong, wondering if anyone can help, thanks.

Besides, since this is the first time experience this project, I have a few other questions.
One is that, since it is not explicitly stated in the documentation, I would like to know if this project avoids type conversion and type copy overhead between py and rust. As we know, if you pass a two-dimensional array to rust with pyo3 way, you may need to go through a type conversion from numpy.int to python.int then to rust.usize, which will consume a lot of resources when there is a lot of data, and I think it would be great if this was avoided in this project.

Another point is that due to some kind of missing clarity I'm not quite sure what the examples/parallel project is aimed at. Since its project name is 'parallel', does it mean that this project can break the GIL limit of python and get multiple cores used at the same time in one process for simultaneous computation (which is necessary in production environments). I' m interested in the par_map_collect function called in the example, which seems to serve such a role, but unfortunately I didn't find the relevant search results in the documentation.

[davidhewitt]

The example has numpy as a path dependency -

rust-numpy/examples/parallel/Cargo.toml

Line 13 in 19bfc9d

numpy = { path = "../.." }

You will need to update this to be the version of numpy you want to use.

[GoodManWEN]

The example has numpy as a path dependency -

rust-numpy/examples/parallel/Cargo.toml

Line 13 in 19bfc9d

numpy = { path = "../.." }

You will need to update this to be the version of numpy you want to use.

Thanks for the reply, but I'm not quite understand the writing way under [dependencies], does it mean that I have to point to some relying items that is not rust but python? Should I write the numpy directory under site-package here?

[adamreichold]

Thanks for the reply, but I'm not quite understand the writing way under [dependencies], does it mean that I have to point to some relying items that is not rust but python? Should I write the numpy directory under site-package here?

This is about the Rust-side dependency on the crate numpy, not the Python package. In this particular case, you need to replace the path dependency (which works when the example is part of this Git repository) by the version number to pull from crates.io, e.g.

[dependencies]
numpy = "0.16"

[GoodManWEN]

Sorry, I understand somewhat the reasons for the problems you describe, but trying has left me a bit confused, I don't quite understand how to create a clone or an item with ownership from x: ArrayBase<ViewRepr<&f64>, Dim<[usize; 2]>>(If called directly, either will prompt that the method does not exist). Also, during learning of the documentation I found that seems we couldn't use slices directly like python on a ndarray object, but needs to call the .slice() method, however the examples in the documentation do not provide how to use the exact sliced value to participate in calculation.

I would like to try to describe my needs directly, could you please give a sample minimal implementation.

I need to perform some slightly complex computational logic on a two-dimensional numpy array which are usually difficult to accomplish directly using numpy's high-level APIs. To avoid the inefficiency problem of operating in python, I tried to calling numpy's c-api with rust. For example, if I have the following (5, 4) array (use i and j to denote their row and column index numbers respectively)

import numpy as np 

arr = np.arange(20, dtype=np.float64).reshape((5,4))
# >>> print(arr)
# array([[ 0.,  1.,  2.,  3.],
#        [ 4.,  5.,  6.,  7.],
#        [ 8.,  9., 10., 11.],
#        [12., 13., 14., 15.],
#        [16., 17., 18., 19.]])

If I want to operate on the first column only, and let each item equal to the sum of its previous two items (starting from row 3 and onwards only), expect the results as follow,

arr2 = operate(arr)
# >>> print(arr2)
# array([[ 0.,  1.,  2.,  3.],
#        [ 4.,  5.,  6.,  7.],
#        [ 4.,  9., 10., 11.],
#        [ 8., 13., 14., 15.],
#        [12., 17., 18., 19.]])

( If dealing with python lists, the above operation can be expressed as arr[i][0] = arr[i-1][0] + arr[i-2][0])

I would like to do the operation in rust, preferably by modifying the original memory directly, or by doing a clone, which I don't think would incur too much overhead. The expected high level usage in python is to have the effect of inputing a two-dimensional array and returning an processed array of the same size. I'd like to know what the recommended best practice is for doing it in rust-numpy's way, frequent use of .slice(s![i, 0]) doesn't seem to be an efficient way from my speculation.

[adamreichold]

I'd like to know what the recommended best practice is for doing it in rust-numpy's way

The recommended way to do this is have your function accept a PyReadonlyArray<T, D>, then use .as_array() to obtain an ArrayView<T, D> of the data stored in the NumPy array. The actual computations are then performed using the API provided by the ndarray crate. This should eventually produce an Array<T, D> which can be transformed (without copying) into a &'py PyArray<T, D> by calling .into_pyarray(py) which is then becomes the return value of your function. (As in the examples presented in this repository, this typically split so that the actual logic is a separate function taking ArrayView<T,D> and returning Array<T,D> with a wrapper that does nothing but the conversions.)

In your case, the index-based definition seems workable for the implementation, even though it might incur bounds checking overhead. You would probably start by creating an empty array using the zeros method and filling its entries using a loop. You do not need to call .slice(...) to access a single element. Doing array[(i, 0)] should work.

https://docs.rs/ndarray/latest/ndarray/doc/ndarray_for_numpy_users/index.html might be a good starting point if you have used NumPy so far and want to learn the differences when using ndarray.

[adamreichold]

(Doing the operation in-place is also possible but more complicated: At the moment the unsafe as_array_mut needs to be used or when 0.17.0 is released, PyReadwriteArray which provides a safe variant of as_array_mut.)

[GoodManWEN]

Can you provide a simple implementation that returns the input PyReadonlyArray2 data as-is to &PyArray2<f64> form? I've checked type.ArrayView but it doesn't look like it has an impl to let it to be converted into ndarray::Array.

Also at your suggestion I can return a numpy.array object to python indeed if I create a new ndarray::array (zeros), and according to my tests it has a very efficient indexing performance, slightly better than Vec<[f64; n]> in huge 2d arrays, but avoids the pyo3 conversion of vec to list, which is really encouraging!

[adamreichold]

Can you provide a simple implementation that returns the input PyReadonlyArray2 data as-is to &PyArray2 form? I've checked type.ArrayView but it doesn't look like it has an impl to let it to be converted into ndarray::Array.

fn just_a_clone<'py>(array: &'py PyReadonlyArray2<f64>) -> &'py PyArray2<f64> {
    let py = array.py();
    let array_view = array.as_array();
    let owned_array = array_view.to_owned();
    owned_array.into_pyarray(py)
}

[messense]

I'd recommend read the cargo documentation about dependency management if you are new to Rust.

[adamreichold]

One is that, since it is not explicitly stated in the documentation, I would like to know if this project avoids type conversion and type copy overhead between py and rust. As we know, if you pass a two-dimensional array to rust with pyo3 way, you may need to go through a type conversion from numpy.int to python.int then to rust.usize, which will consume a lot of resources when there is a lot of data, and I think it would be great if this was avoided in this project.

This crate only wraps NumPy ndarray instances, it does not copy or convert them. This also implies that if you Rust-side function expects e.g. PyReadonlyArray2<f64>, but you pass in a one-dimensional array or one storing numpy.int, then this will fail due to a type mismatch.

All operations on these arrays are then typically performed in terms of ArrayView providing by the ndarray crate which still only wrap the existing data without any copying or conversion, i.e. the PyReadonlyArray::as_array method.

Another point is that due to some kind of missing clarity I'm not quite sure what the examples/parallel project is aimed at. Since its project name is 'parallel', does it mean that this project can break the GIL limit of python and get multiple cores used at the same time in one process for simultaneous computation (which is necessary in production environments). I' m interested in the par_map_collect function called in the example, which seems to serve such a role, but unfortunately I didn't find the relevant search results in the documentation.

As written above, operating on the array contents is usually done by operating on ArrayView instances from the ndarray crate, which provides the par_map_collect method. This way, while the GIL is still held, the actual numerical operations are dispatched into Rayon's thread pool and thereby utilise multiple hardware threads.