If directory reader should be keep

[Czaki]

During today's plugin working group meeting, I ask a question if the possibility of defining directory reader should be kept.

The idea of a directory reader is for one of two scenarios:

1. Read a whole stack from separated files
2. Read some "project" saved as multiple files.

My main doubt is if it will be easy to debug why the output of such a reader is different than we expect.

For the first option, the equivalent will be a plugin that gets a list of files. The situation where the user needs to mark all files in a directory give him a possibility to check if there are no additional or missed file but may introduce some problem if there are multiple files with the same name but different extensions.

For the second option, the user could simply point to some starting files and the plugin could check if other needed files are present in the same directory. I suggest this solution if such a "project" has some main file with a description.

Both alternatives force the user to more explicit point what he expects to load so may simplify understanding where is the problem, but removing the option to define directory reader may introduce some inconveniences to end-users.

[tlambert03]

Fully agree with the assessment here. I think allowing plugins to declare for support was probably an ill-conceived attempt to "leave things flexible", mostly for your scenario number 2. But it's probably almost never used, and if it was, leaves a tremendous amount of responsibility to the plugin to do things correctly. And it also becomes very hard to determine which plugin should be used (unlike when we have a file extension).

Additionally, plugins who do want to support some sort of project archive could simply consume/create zip files and then change the extension. So there's definitely a workaround.

I'm +1

[DragaDoncila]

I actually love directory readers, and have used them frequently, mostly in scenario-2-like situations. While I agree that you could just zip the files/use a "header" file to point you to the rest of the needed data, I do think being able to drag in a directory is a very natural interaction. Would this remove our built-in "read directory as a stack" behavior? I would definitely not be down for that. Also, how would this affect zarr readers, since zarrs are ultimately just directories?

The other issue is that while it's easy for plugin developers to take a zip or a "header file" and read around it, it may be less obvious to users that they should zip up their directory/add extra metadata for it to be readable - when before they didn't have to. Of course this can be addressed with documentation, but it might be a tough argument.

Having said that, I'm in full agreement on how tricky they are to associate e.g. for the reader preference dialog, we don't even save an association for directories, because it's just so hard to associate anything besides the choice for that individual directory. And I've tried to think of more sensible associations but haven't really come up with any.

I guess I'm not voicing strong dissent here, except for the "directory as a stack" behavior which I absolutely think we should try to keep.

[Czaki]

Maybe allow directories, but only one with extension? Like zarr?

[tlambert03]

except for the "directory as a stack" behavior which I absolutely think we should try to keep.

agree completely. though that could essentially be handled on the napari side, not the plugin side. Where, if a user opens a directory as a stack, napari passes the files to plugins using the existing list of strings API (as opposed to passing the directory itself to plugins and saying "do something interesting with this")

The other issue is that while it's easy for plugin developers to take a zip or a "header file" and read around it, it may be less obvious to users that they should zip up their directory/add extra metadata for it to be readable - when before they didn't have to. Of course this can be addressed with documentation, but it might be a tough argument.

yep, agree here too.

it's perhaps worth pointing out that one of my original use cases for the accepts_directories case was my own ndtiffs plugin, which has this logic:

@napari_hook_implementation
def napari_get_reader(path: PathLike) -> Optional[ReaderFunction]:
    if isinstance(path, str) and has_lls_data(path):
        return reader_function

that has_lls_data was a pretty powerful function that could detect whether a folder of files followed a specific convention (essentially a number of tiff files with a settings.txt file). So, now that I think about it, it would be nice to know how I could accomplish such a thing if we didn't pass the full directory to plugins :)

[DragaDoncila]

Well, I think the idea is that maybe you would drag and drop the "settings.txt", for example, and then crawl through the rest of the directory looking for exactly the same things as you otherwise would've. Or even just get parent of "settings.txt" and pass that to has_lls_data? Of course I don't know exactly how that function works but as a developer, you know the structure you need, so you could just walk that structure back to get to the directory.

But if you're doing that, then it just seems like more work for the plugin developer, just so that the user can drag in "settings.txt" instead of the whole directory (less useful for the user imo). The other issue is that now the association between file extension and reader is even vaguer - what if I drag in one tiff as my header file? Doesn't mean I want all tiffs to be read with this directory reader...

[tlambert03]

The problem there though is that I can still support a folder that lacks settings.txt, in which case I would look at the filename patterns. So saying “just drop the settings.txt file” would still restrict what that plugin can do compared to the current model.

It’s just a question of: is that power worth it given the open endedness of the spec … and how do we reliably pick a single plugin if a user drops a directory. Your extension association wouldn’t work anymore, so we’d definitely need to run all of the get reader functions each time (which could be fine… but again, it’s a question of is it worth the benefit, and could we not achieve something similarly powerful without accepts_directories in the plugin schema)

[DragaDoncila]

Well, to be fair I don't think there's a problem with just showing all available directory readers and letting the user pick. I agree that it might be annoying not to associate them, but we could also just associate individual directories on first read. Not ideal, but a good way of balancing ease of workflow with explicit choices on behalf of the user.

I definitely take your point though, having to run all the get_readers is cumbersome if you're doing it for every directory (rather than just a few times per file extension as you would be now if the user is saving associations).

[sofroniewn]

I also liked drag and drop folder and get all files loaded and "concatenated", so just 1 layer instead of 100 layers for 100 images. If we can preserve the behavior that we like then I'm open to changing things to simplify things overall

[tlambert03]

Right, that behavior can definitely be preserved. What this really comes down to is whether we maintain the accepts_directories flag in the plugin schema. And whether we pass the full directory to a plugin and say "here, do something with this". We can always go into a directory and pass all the files to plugins, either individually or with "stack=True".

[DragaDoncila]

We can always go into a directory and pass all the files to plugins, either individually or with "stack=True".

I guess I don't see what the difference is here. If we drop accepts_directores but just pass all the files in the directory when a user drags them in, that doesn't seem any different to me, except that the developer gets to be less specific in their schema and we rely more on the get_reader validation?

[tlambert03]

the difference is whether we (napari) traverse the directory ourselves, looking for files and extensions, and matching them with plugins that expressly support those files, for example, "one text file and one or more tiff files" (using readers.filename_patterns). Which is more declarative, but less powerful... vs we (napari) completely staying out of the business of looking inside of a directory, but leaving it entirely up to plugins to "behave" when passed a full directory with who-knows-what inside of it.

Ultimately, this comes down to safety & repeatability for the end-user (that we napari can control) vs power and expressiveness for plugins (that we can't control). It's a non-obvious choice either way. but, like @jni said, it's an easy decision to punt on

[DragaDoncila]

Hmm ok I guess I see this as two different pieces of functionality, and I almost feel like we should support both. The first one is like a "file by file" reader, and the other one is the "directory" reader that might look for specific file structure/naming conventions/subfolders etc.

Almost like we should split accepts_directories into two different options. I don't know what they'd be called. I guess like accepts_directories and accepts_multiple_files or something. Except those multiple files could be in a directory...

[Czaki]

The general problem is that if we follow approach with has_lls_data is that we do not get information what file is missed.

[DragaDoncila]

I'm not sure what you mean by this sorry. If it's a matter of not providing information to the user on why the reader failed, I feel like this could be resolved by providing a mechanism for feedback in the viewer from a plugin, rather than not accepting directories at all?

[tlambert03]

Yeah, I agree with @DragaDoncila… receiving “error feedback” from a plugin is a different issue

[jni]

I still haven't seen someone explain how zarr folders will work if we remove support for directories?

[jni]

(thanks @DragaDoncila for raising the issue)

[Czaki]

Zarr directory has "extension".

[tlambert03]

Right, a plugin needn't use "accepts_directories" for zarr... But rather add zarr to their list of supported extensions.

(Though: now that you mention it, we need to check that the code actually does rhat! 😬)

[jni]

Ah. So should this issue be renamed "accepts arbitrarily named directories?

I actually agree with Draga that using a directory as a file format is great and super useful, so I don't want to remove that capability. However, I don't much mind asking developers of such formats to specify an extension...

But still, generally, unless this is causing major implementation issues/bugs, I'd err on the side of keeping this capability...

[tlambert03]

unless this is causing major implementation issues/bugs, I'd err on the side of keeping this capability...

This is enough for me to put this on hold. It's definitely not causing either of those at this point. Let's ice it for now

[brisvag]

I wanted to add my use case to the list (call it "case 3"). I usually have volume images and particle definitions (points and orientations, which I display as points and vectors). Due to the quirks of tomography processing, these are usually spread out in different directories and their naming scheme (and extensions) may differ significantly between use cases. What's more, I need an ensemble view, cause sometimes one file contains metadata (such as pixel size) that will be needed to correctly display data from a different file. Zipping all of this would be prohibitive.

This is currently pretty much impossible (I need multiple directories and/or lists of files to be passed at once, since I need to know all of the data that's getting passed). If I had to suggest a direction here, I would rather support a system where a plugin can accept even more than one directory at once. However, I realixe that this complicates things even more :p

Currently (and for the foreseeable future) I'm resorting to wrapping napari, rather than using the plugin system. Bonus point: I generate a "dataset" object that allows interacting on the whole dataset, which I currently have no way to expose to the user through the plugin system.

[Czaki]

file contains metadata (such as pixel size) that will be needed to correctly display data from a different file

I do not understand why you cannot pass this single file to the reader and calculate all others relative to this? This is how I handle such cases.

I generate a "dataset" object that allows interacting on the whole dataset, which I currently have no way to expose to the user through the plugin system.

All layers have a metadata dictionary that could store such an object and make it discoverable by the plugin widget.

[brisvag]

I do not understand why you cannot pass this single file to the reader and calculate all others relative to this? This is how I handle such cases.

It's not a single file with the metadata, it other files in the dataset. For example, the header of each volume file woudl contain its own pixel size info, but particle files might have no info about this. Each particle file is associated to each volume based on something else in the header; then I retrieve for each particle file the correct pixel size from their volume companion, and scale it accordingly. It's just quirks of how these files are, unfortunately.

All layers have a metadata dictionary that could store such an object and make it discoverable by the plugin widget.

Right, I admit I forgot abotu the metadata... I'll certainly use that now :P

[Czaki]

It's not a single file with the metadata, it other files in the dataset. For example, the header of each volume file woudl contain its own pixel size info, but particle files might have no info about this. Each particle file is associated to each volume based on something else in the header; then I retrieve for each particle file the correct pixel size from their volume companion, and scale it accordingly. It's just quirks of how these files are, unfortunately.

I personally still will prefer one file with (maybe relative) paths to all other files/directories to simply manage what will be loaded and got an error message if something is missed. Which allows creating multiple "projects" reusing the same files (for example one density map and different models/interpretation). But this is my personal preference.