How does loading very large files in the browser work?

[billyc]

Hi, I'm amazed at how well this performs with large multi-gigabyte local files. I'm trying to understand the code, specifically whether or not the entire ArrayBuffer is loaded when a file is drag/dropped in.

It appears that only the HDF5 header is read -- just enough to populate the list of datasets in the file; and then a specific dataset is loaded into memory only when the user clicks on it. This makes sense!

But in the code it very clearly in h5wasm-api.ts is passing a full ArrayBuffer into initFIle(), calling arrayBuffer() on the File object. Doesn't that load everything? A blob can be sliced but an arraybuffer is fully in memory... or am I mistaken?

Perhaps I am I looking in the wrong place in the code?

I'd love to know how this works because I am trying to integrate this with a specific use case, and our files are all far too big to load all at once.

Thanks for any hints! And what a great tool, thank you for this gift.

[billyc]

Is this all encapsulated in the React "suspense-fetch" library? 🤔

[axelboc]

Hi @billyc, this has to do with the new H5WasmLocalFileProvider (and corresponding API: local/h5wasm-local-file-api.ts), which uses h5wasm inside a web worker together with the Emscripten WORKERFS virtual filesystem (instead of MEMFS filesystem). Contrary to H5WasmProvider, this new provider accepts a File object from an HTML file input and passes it on to the WORKERFS file system, so the file no longer has to be loaded into memory. Check out #1582 and #1604 for more info.

Now of course, this all works also because the viewer (i.e. @h5web/app) requests data and metadata lazily from the data provider, as the user explores the file. react-suspense-fetch is the library that enables us to do this, yes. It "suspends" React components while data/metadata is being requested, which allows showing loading messages and spinners, and caches the results in memory to avoid requesting the same data/metadata twice. Note that there's currently no expiry mechanism, so with very large files, there's still a risk of running out of memory.

Hope it makes sense, let me know 😉

[billyc]

@axelboc A-ha, there it is! I was searching through the master branch and did not find the LocalFileProvider. Thought I was going mad.

I'll take a look, thanks so much!

(edit: found it, it was there all along ✅ )