Layer data model and separating data source/views

[andy-sweet]

In the architecture working group meetings we've been discussing napari's layer data model over the last few weeks. By taking a look through our issues, that's not a new topic of discussion and it covers a wide range of topics/problems. I found the following issues, but I'm sure I missed some.

• Layer models #1353
• Potential approach to layer slicing #1775
• proposal: reducing complexity of models #1325
• Refactor Points data model #858
• Resimplify the image slice/view logic #2156
• [discussion] architecture for clipping/embedded planes and other "geometrical masks" #3148
• points.data should be the original data #2519
• Specifying physical coordinates for axis in add_ methods #2246
• Drop channel_axis parameter #3019
• the "order" layer-creation parameter is not parsed as expected #3344
• Discussion about MultiscaleData API #2803
• #2484
• Please explicitly specify the coordinate system. #1904
• Dims sliders should be sorted by dimension order #2770
• Allow scale and translation parameters to have units #1701
• Rename ImageSlice/ImageView objects #1574

I'm opened this so that others can contribute to that discussion, provide examples of problems and desired usage, and come up with ideas that could help.

[andy-sweet]

However, given the breadth of topics here, we also want to initially focus on a more specific issue around separating a layer's data source and views/slices of that data, so that we can make some tangible progress. We chose this issue for a few reasons.

• It comes up fairly frequently in the issues above.
• Some of us have been blocked/confused by it while working on other issues.
• It seems like a lot of the related behavior is currently implemented through private attributes and methods, so we should be able to avoid making breaking changes.
• It helps unblock work on multiple/ortho canvases/views.

This more specific issue is likely not going to focus on the data structures used to store data and the potential APIs around dimension names, order, types, and units (i.e. these will likely be the same as they are now in napari). But any design decisions should aim to keep those ideas in mind.

The initial plan is to start a design document to form some coherent ideas around the following.

• The current state of layer data/slicing in napari.
• Desired example usages of layer data/slicing both internally within napari and by plugins.
• Goals, non-goals, and maybe requirements of the work.

That document will be completely open for collaboration, so we'll share it here when it's created and has taken some initial form.

[sofroniewn]

However, given the breadth of topics here, we also want to initially focus on a more specific issue around separating a layer's data source and views/slices of that data, so that we can make some tangible progress. We chose this issue for a few reasons.

Yes, agreed - i think this is one of those foundational things that we'll need to improve the API and enable the other work. Makes a lot of sense to me

[brisvag]

I'll pitch my vision for the future, so we can see what people agree on and maybe have a clearer picture of where our choices will lead us :) What I imagine interacting with a datasource/layer model is something like this:

• We have two separate widgets in the viewer, one listing all the data sources with their types and shapes, and one listing the layers. Imagine something like the video editing software KDEnlive, loading clips into the project versus working with them in the timeline.
• When you open a new dataset, the datasource gets populated with it, and a first appropriate layer gets spawned.
• Selecting a layer highlights its datasource(s), and viceversa, so you never lose track of what's what.
• One can easily duplicate a layer, which will then link to the same datasource as the original, or create a new layer from a datasource with a contextual menu.
• One can easily split the canvas in multiple ViewBoxes. This either [personally, I lean towards the first option]
  • creates new layers for the new canvas by duplicating existing ones (or does nothing, and layer have to be added manually); this way each canvas is independent, has its own layerlist and can show arbtrary data, or
  • provides a view on the same layerlist but with a different camera and/or sliced dims, or
  • a hybrid, where we get "Layer views", which inherit everything from their parent but allow for specific changes relative to that canvas
• Eventually, when we manage to get/keep our data on the gpu with cupy or something similar, one could click on a datasource and say "load onto GPU", which would keep it loaded for quicker processing and visualisation

[andy-sweet]

Thanks @brisvag - I really like the ideas and the detail here!

One important thing you touched on is explicit definitions of what terms like data, data source, layer, view, slice mean, and I think that's a really important topic here. Especially since the current definition of Layer in the code encapsulates a lot of those and getting away from that feels like one of the big goals of this work.

I personally think about a layer as something that will be composited/merged into a single visualization. So for multi-canvas (where each canvas presents a separate visualization), I think of each canvas maintaining its own layer list (separate from the data source list) like in your first idea. I think this addresses the two big usages I think of when it comes to multi-canvas, which are the following.

1. Each canvas presents a different camera view (and thus likely slice) of the same data source(s). One example would be multi ortho views.
2. Each canvas presents the same camera view (in world coordinates) of different data sources. We can do something like this with our tile/grid visualization right now, but it has plenty of limitations and implementation details.

After writing that, another variation of your idea came to mind which is maintaining the same list across different canvases, but only showing those that are enabled for that canvas.

But I'm almost certainly missing some other usage here.

[NeonSpork]

Edit: I realized after posting this, that this should perhaps be in an entirely new discussion? If that's the case, let me know and I'll remove this post and create a new discussion. 🙂

Allow me to preface this by saying I literally only discovered napari this morning, so I'm still familiarizing myself with it, but the project is very intriguing! I apologize in advance if this has been addressed elsewhere. I was a bit unsure of exactly where to post this, but ultimately I decided this discussion was perhaps the most appropriate.

I was wondering about using napari as an interface for viewing and annotating DICOM images. Specifically CT and/or MRI images, with annotations primarily to use the images as training sets for machine learning.

Having a tool integrated in a jupyter script that could run natively in the process would be an insane improvement to the current workflow (or lack thereof), and the user interface of napari seems pretty sound.

• Is using napari to view and annotate DICOM images remotely within the scope of what is possible?
• Is this something that would need a plugin to work?
• Or have I completely misunderstood something fundamental about how napari works that would render this whole idea unfeasible?

[tlambert03]

I was wondering about using napari as an interface for viewing and annotating DICOM images. Specifically CT and/or MRI images, with annotations primarily to use the images as training sets for machine learning. Is using napari to view and annotate DICOM images remotely within the scope of what is possible?

Yep! If I understand the question correctly, there are a number of napari plugins that should help you load DICOM (after which you'd be free to create a new "labels" layer and start annotating. don't work with DICOM myself, so can't vouch for one over the other, but you might try any of these:

• https://github.com/InsightSoftwareConsortium/napari-itk-io
• https://github.com/AllenCellModeling/napari-aicsimageio (requires the bioformats backend)
• https://github.com/nipy/napari-nibabel (support for DICOM still pending I think?)

also CC @grlee77, who I believe works a lot with these sorts of images and may be able to answer questions more directly.

Edit: I realized after posting this, that this should perhaps be in an entirely new discussion? If that's the case, let me know and I'll remove this post and create a new discussion. 🙂

no worries :) but yeah, if you have more followup questions, let's create a new issue to discuss. Or, feel free to stop by our zulip channel to chat with us: https://napari.zulipchat.com/

[NeonSpork]

That sounds pretty much like what I was picturing! I will look at those links, thank you!