New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using async iterator-like SQLX fetch with Rayon #1090
Comments
You may have an easier time if you just use an async threadpool, like tokio provides. What sort of results come out of your parallel process? The shape of that will affect how you might shoehorn this into rayon. |
Thx @cuviper , each operation is essentially a bool -- in a way, I am trying to parallel-ize validation of a large dataset, searching for any row that do not pass validation. So ordering is not important, but would be good to abort early if any validation fails. |
One way you could try is with tokio channels, using async std::iter::from_fn(move || rx.blocking_recv()).par_bridge() //... |
If you are using |
Crossposting with stackoverflow:
Rust SQLX lib provides an iterator-like interface
fetch(...)
usually used withwhile let Some(row) = rows.try_next().await? {...}
construct. In my case, each row may take some time to process, so I would like to use Rayon'spar_iter()
, but that requires a real iterator. Usingfetch_all
is not an option because all rows may simply not fit into memory.How can I use Rayon's
par_iter
to process millions of rows produced by the sqlx'sfetch
stream?The text was updated successfully, but these errors were encountered: