You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reading very large files with the h5wasm provider is not possible, for several reasons:
maximum size of ArrayBuffer and also "file" in emscripten is often < about 2GB
maximum size of memory in the browser is a limitiation for in-memory file representation
unreasonable demands on network/infrastructure to download entire huge files.
Requested solution or feature
For web file servers with HDF5/NeXus files that support range requests, on-demand loading could enable access to very large NeXus files that would be infeasible to read as a whole, using emscripten's lazyFilefunctionality
Alternatives you've considered
HSDS and grove providers already allow this type of random access to parts of a NeXus file.
Additional context
Because sync file access is required, this might require refactoring the h5wasm provider to operate from a worker.
Note that it could potentially be refactored to a service worker that uses the same API as a grove server, if that simplifies things.
The text was updated successfully, but these errors were encountered:
Note that for local files, the emscripten WORKERFS interface could be used to get random access to huge local files from a worker without copying the whole file into memory, which is another benefit of moving the provider to a worker.
Note that it could potentially be refactored to a service worker that uses the same API as a grove server, if that simplifies things.
This would be brilliant! However, it seems that synchronous XHR requests inside Service Workers are currently not supported in Chrome and Safari—only in Firefox.
Is your feature request related to a problem?
Reading very large files with the h5wasm provider is not possible, for several reasons:
Requested solution or feature
For web file servers with HDF5/NeXus files that support range requests, on-demand loading could enable access to very large NeXus files that would be infeasible to read as a whole, using emscripten's
lazyFile
functionalityAlternatives you've considered
HSDS and grove providers already allow this type of random access to parts of a NeXus file.
Additional context
Because sync file access is required, this might require refactoring the h5wasm provider to operate from a worker.
Note that it could potentially be refactored to a service worker that uses the same API as a grove server, if that simplifies things.
The text was updated successfully, but these errors were encountered: