Task with native C++ modules wrapped in pybind11 and running in Distributed Dask

[tiborkiss]

I tried to find some example with Dask arrays where the task graph contains calls to modules implemented in C++ and exposed through pybind11. If you have seen somewhere, please don't hesitate to add a comment.

I expect there are a few challenges when mixing distributed Dask with native binding. I am thinking about the following:

• reduce the copying of large parameters when passing to a native code and when the result is returned from native to python. Ideally some sort of zero-copy approach I would prefer.
• memory ownership in which side (python or native side) to be held.

There are different array implementations, such as zarr, geopandas etc. How they solve the above challenges?
Now I have a plan for a very simple proof of concept, which would showcase the cooperation between C++ modules and Dask arrays in very general form. I guess later I will have to integrate to GIS-friendly data structures to be able to reuse some of the existing libraries which are parallelizable in Dask.