Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add an example to create a virtual dataset with FancySelection #1772

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

payno
Copy link
Contributor

@payno payno commented Dec 2, 2020

A simple example on how we can create FancySelection for VirtualSource (and as a consequence virtual dataset).
It would have been useful to me so maybe it could help someone else in the future.

@codecov
Copy link

codecov bot commented Dec 2, 2020

Codecov Report

Merging #1772 (2cff583) into master (95e3497) will increase coverage by 1.40%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1772      +/-   ##
==========================================
+ Coverage   88.51%   89.91%   +1.40%     
==========================================
  Files          17       17              
  Lines        2273     2311      +38     
==========================================
+ Hits         2012     2078      +66     
+ Misses        261      233      -28     
Impacted Files Coverage Δ
h5py/_hl/files.py 90.15% <0.00%> (-2.58%) ⬇️
h5py/_hl/vds.py 96.55% <0.00%> (-0.82%) ⬇️
h5py/_hl/selections.py 87.50% <0.00%> (-0.50%) ⬇️
h5py/h5py_warnings.py 100.00% <0.00%> (ø)
h5py/ipy_completer.py 0.00% <0.00%> (ø)
h5py/_hl/filters.py 92.85% <0.00%> (+0.07%) ⬆️
h5py/_hl/group.py 96.83% <0.00%> (+0.08%) ⬆️
h5py/_hl/base.py 96.18% <0.00%> (+0.55%) ⬆️
h5py/_hl/dataset.py 93.14% <0.00%> (+0.64%) ⬆️
h5py/_hl/attrs.py 96.89% <0.00%> (+0.71%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95e3497...2cff583. Read the comment docs.

@payno
Copy link
Contributor Author

payno commented Dec 2, 2020

In fact I am not sure this is the best way to go. In this example I need to acces FancySelection by importing h5py._hl.selections as selectionand so importing protected '_hl'. But I don't see an other way to accomplish this. Would you have idea ? Is this feature foreseen ?

@takluyver
Copy link
Member

Does v_source[[1, 2, 5]] work for doing fancy selection?

What's the performance like with this? I recall that making many small separate mappings for a virtual dataset caused poor performance when reading from it, and I'd be somewhat concerned that fancy indexing might be similar, as it's lots of small hyperslabs in the source selection.

@payno
Copy link
Contributor Author

payno commented Dec 3, 2020

Does v_source[[1, 2, 5]] work for doing fancy selection?

In fact it does. It call also 'select'. The same operation could be done by (for the first selection)

v_source_1 = h5py.VirtualSource(file_path, "original_data/d1", shape=(10, 10, 10))
v_source_1 = v_source_1[[2, 3]]

which is in fact more convenient thank you. I will update the example.

What's the performance like with this? I recall that making many small separate mappings for a virtual dataset caused poor performance when reading from it, and I'd be somewhat concerned that fancy indexing might be similar, as it's lots of small hyperslabs in the source selection.

Make sense. I didn't had a look at performances yet.

@takluyver takluyver added the vds Virtual datasets label Jun 30, 2021
@aragilar
Copy link
Member

aragilar commented Nov 2, 2021

I think the remaining bit before this can be merged is the removal of import h5py._hl.selections as selection (which isn't needed any more), but I'm not up to date with the state of VDS.

@payno
Copy link
Contributor Author

payno commented Nov 3, 2021

Thanks. In fact I was no more using the 'h5py._hl.selections' directly. I removed the import and improved file formatting (black - flake). Let me know if there is more improvements to be done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
vds Virtual datasets
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants