open_datatree performance improvement on NetCDF, H5, and Zarr files

[aladinor]

open_datatree performance improvement on NetCDF files

• Closes Improving performance of open_datatree #8994 (NetCDF + Zarr datatree)
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[welcome]

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

[flamingbear]

I had thoughts about the legacyhdf5 api and how it might be incorporated.

[flamingbear]

This looks great. Thanks for working through the dual library stuff with me.

[TomNicholas]

Yes very excited by this! Two final things:

• This deserves a whats-new.rst entry!
• Would you be willing to add an benchmark test? You can see here how we benchmark opening and loading a single netCDF file

xarray/asv_bench/benchmarks/dataset_io.py

Line 125 in 447e5a3

def time_load_dataset_netcdf4(self):

Alternatively we could leave adding that benchmark to a separate PR?

[kmuehlbauer]

Sorry for being late to the party, but do we really want to have mode in open_datatree? The open_* functions are for read access only (imho).

Update: It looks like there are already several keyword arguments in the open_*-functions at least for netcdf4/h5netcdf backends which are only needed for write access (mode, format, invalid_netcdf, clobber, diskless, persist and maybe even more).

[kmuehlbauer]

Beside my other comment on the open_*-function kwargs there is not much to add to @TomNicholas's comment.

We might want to further deduplicate code in the backends (by moving this into backend/store.py). There has already quite some work been done by @jthielen in #7437.

Should open_datatree be added to the API with this PR too?

Those issues can be handled in subsequent PR.

Great work @aladinor 🚀.

[keewis]

In the meeting yesterday we decided to not bother with deduplication for now (and anyways it is nice to have backends somewhat self-contained).

The API will be extended in #9033, the idea is to have one single PR that marks the entire DataTree API as public.

Edit: and I agree, since open_dataset in particular already has these parameters, it's fine if open_datatree also has them (before marking BackendEntrypoint.open_datatree as public and documenting it we should probably clean it up a bit more)