Thread Safe Decorator <Help Wanted> #2598
-
Building upon the example in the repo, I tried calling the decorator from a Here's the Rust: use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
#[pyclass(name = "Counter")]
pub struct Counter{
#[pyo3(get)]
count: usize,
wraps: Py<PyAny>,
}
#[pymethods]
impl Counter{
#[new]
fn __new__(wraps: Py<PyAny>) -> Self {
Counter {
count: 0,
wraps: wraps,
}
}
#[args(args = "*", kwargs = "**")]
fn __call__(
&mut self,
py: Python<'_>,
args: &PyTuple,
kwargs: Option<&PyDict>,
) -> PyResult<Py<PyAny>> {
self.count += 1;
return self.wraps.call(py, args, kwargs);
}
}
#[pymodule]
pub fn rust_counter(_py: Python<'_>, module: &PyModule) -> PyResult<()> {
module.add_class::<Counter>()?;
Ok(())
} Calling from from concurrent.futures import ThreadPoolExecutor
from rust_counter import Counter
@Counter
def hello():
print("hello")
with ThreadPoolExecutor(4) as ex:
futures = []
for _ in range(4):
futures.append(ex.submit(hello)) Results in me getting: I've tried a few methods:
I was just wondering if anyone knows how to make this kind of pattern thread safe. I thought the Based on similar issues, this may be a problem w/ the |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 7 replies
-
Okay I will leave this question up for posteriority, but I was able to get a solution working. Rust: use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
use std::sync::{Arc, Mutex};
#[pyclass(name = "Counter")]
pub struct Counter{
// #[pyo3(get)]
count: Arc<Mutex<usize>>,
wraps: Py<PyAny>,
}
#[pymethods]
impl Counter{
#[new]
fn __new__(wraps: Py<PyAny>) -> Self {
Counter {
count: Arc::new(Mutex::new(0)),
wraps: wraps,
}
}
fn count(&self) -> usize{
return *self.count.lock().unwrap()
}
#[args(args = "*", kwargs = "**")]
fn __call__(
&self,
py: Python<'_>,
args: &PyTuple,
kwargs: Option<&PyDict>,
) -> PyResult<Py<PyAny>> {
{
let mut count = self.count.lock().unwrap();
*count += 1;
}
return self.wraps.call(py, args, kwargs);
}
}
#[pymodule]
pub fn rust_counter(_py: Python<'_>, module: &PyModule) -> PyResult<()> {
module.add_class::<Counter>()?;
Ok(())
} I'm unsure why, but the additional scope around the count incrementing is crucial. If not, the process deadlocks and I believe this comes from the Caveats:
Testing code: from concurrent.futures import ThreadPoolExecutor
from rust_counter import Counter
@Counter
def hello():
return "hello"
num_threads, num_runs = 10, 100_000
with ThreadPoolExecutor(num_threads) as ex:
futures = []
for _ in range(num_runs):
futures.append(ex.submit(hello))
assert len([f for f in futures if f.exception()]) == 0
assert hello.count() == num_runs Feedback welcome! |
Beta Was this translation helpful? Give feedback.
-
I think that this is the case, and that the executor is yielding in the middle of the |
Beta Was this translation helpful? Give feedback.
-
@mejrs and @davidhewitt thanks for all the help. I do have a follow-up question... when trying to switch to the Here's the Rust: use pyo3::prelude::*;
use pyo3::types::{PyDict, PyTuple};
use std::sync::atomic::AtomicUsize;
#[pyclass(name = "Counter")]
pub struct Counter{
count: AtomicUsize,
wraps: Py<PyAny>,
}
#[pymethods]
impl Counter{
#[new]
fn __new__(wraps: Py<PyAny>) -> Self {
Counter {
count: AtomicUsize::new(0),
wraps: wraps,
}
}
fn count(&mut self) -> usize{
// Also how the heck can I make this a read-only function
// .into_inner() consumes the atomic
return self.count.get_mut().clone()
}
#[args(args = "*", kwargs = "**")]
fn __call__(
&mut self,
py: Python<'_>,
args: &PyTuple,
kwargs: Option<&PyDict>,
) -> PyResult<Py<PyAny>> {
// how in here can I mutate an AtomicUsize
// w/o the interior mutability and allowing for
// &self to avoid `Already Borrowed`
py.allow_threads(|| {
*self.count.get_mut() += 1;
});
return self.wraps.call(py, args, kwargs);
}
}
#[pymodule]
pub fn rust_counter(_py: Python<'_>, module: &PyModule) -> PyResult<()> {
module.add_class::<Counter>()?;
Ok(())
} |
Beta Was this translation helpful? Give feedback.
I think that this is the case, and that the executor is yielding in the middle of the
hello
function. If another function then goes through__call__
and attempts to mutably borrow the decorator, it detects the outstanding borrow. This is not a problem with the changes you have made, because having multiple&self
borrows is fine.