Light mode, Dark mode, LUTs, and blending.

[psobolewskiPhD]

Based on the discussion at the Community Meeting (05.01.2022, napari/meeting-notes@eadff94) I wanted to get some thoughts down.

First aspect, the Light theme canvas

Napari offers two themes by default, one Light and one Dark. The light theme has a white canvas (255, 255, 255), which makes additive blending fail, because adding to 255 just clips to 255. This was first, I think, documented by @Czaki here:
#2140
and again by me here—at the time I didn't realize the nature of the problem:
#3374

As can be seen, in Light theme, the canvas combined with default additive blending mode ends up problematic because it can make the user think there is a bug or issue with the file if they use default opening parameters: don't specify a non-additive blending mode. Worse yet, many users, especially new ones, may not be fully aware of blending modes and blending in general. I didn't get it until @kevinyamauchi prompted me to change blending modes and I started thinking about it. Plus, this occurs even with 1 layer, making the blending issue even less intuitive, the fact that napari blends with canvas is not obvious.

In my opinion—and as was noted in the Community Meeting—the behavior in Light and Dark themes would be the same, so the user doesn't note a change. This would require both themes to have the same canvas, for example, or to not blend with canvas at all. The simplest work around at the moment, in this spirit, is to make the bottom most layer have blending opaque and then use additive for the rest.

Second aspect, inverted LUTs

But! the above fix still only works for conventional LUTS: color on black. At the moment, the use of multiple inverted LUTs in napari is problematic, regardless of Light vs Dark mode. The default napari colormaps only include one such LUT: gray r but a number of such LUTs are becoming commonly used, popularized to a degree by @cleterrier see: https://twitter.com/search?q=%23invertedLUT and
https://github.com/cleterrier/ChrisLUTs
(Note: napari by defaults ships with 3 BOP LUTS from ChrisLUTs)

Personally, I find the inverted LUTs very pleasing when working during the day, with light mode OS, but the behavior in napari is problematic, again due to blending, regardless of mode, because of additive blending between layers. In Dark mode only rears its head with multiple inverted LUTs blended.
Some code to try (should be able to paste into the console)

import napari
from skimage import data
from vispy.color import Colormap

I_Forest = Colormap([[1, 1, 1], [0, 0.6, 0]])
I_Blue = Colormap([[1, 1, 1], [0, 0, 1]])
I_Bordeaux = Colormap([[1, 1, 1], [0.8, 0, 0.2]])

Forest = Colormap([[0, 0, 0], [0, 0.6, 0]])
Blue = Colormap([[0, 0, 0], [0, 0, 1]])
Bordeaux = Colormap([[0, 0, 0], [0.8, 0, 0.2]])


viewer = napari.view_image(data.lily(), 
                            colormap = [("I_Blue", I_Blue), ("I_Bordeaux", I_Bordeaux), ("I_Forest", I_Forest), "gray_r"], 
                            channel_axis=2,
                            rgb=False)

In Dark mode you should see just a hint of the image in pink. If you toggle vis from top to bottom, you will slowly reveal the bottom layer. Again, additive blending here is making virtually everything white. At best you can try to set blending to translucent and play with opacity but it's not ideal.
If you use the conventional LUTs:

viewer = napari.view_image(data.lily(), 
                            colormap = [("Blue", Blue), ("Bordeaux", Bordeaux), ("Forest", Forest), "gray"], 
                            channel_axis=2,
                            rgb=False)

You will see the expected behavior where the composite is properly formed and everything works as expected.

If you then try to use Light mode napari with the inverted LUTs, the current behavior of blending to the canvas 255 again crops up, but setting opaque doesn't really help.

So for working with inverted LUTs a different blending mode between layers is needed. There are a few options, such as subtractive blending or minimum. Extensive discussion in the context of ImageJ can be found here:
https://forum.image.sc/t/multi-channel-composite-view-with-inverted-luts-in-imagej-fiji/61163/49
For subtractive blending, @kevinyamauchi implemented one option here:
#3910
Alas, again option only works for conventional LUTs and only in Light mode, because with inverted LUTs you just subtract down to zero and get a black square. Also, this mode of subtraction inverts the colors—see the PR above for examples.

An alternative is to invert before subtracting:

subtractive': dict(
        depth_test=False,
        cull_face=False,
        blend=True,
        blend_func=('ONE_MINUS_SRC_COLOR', 'src_alpha'),
        blend_equation=('func_reverse_subtract'),
),

This works, but results in an artifact that I think is due to clipping. Here's a zoom of I_Blue layer:

Versus opaque which should be "ground truth" here:

(opacity also doesn't work as expected, but that may be fixable with a bit of play.)

A slightly different option:

'subtractive': dict(
        depth_test=False,
        cull_face=False,
        blend=True,
        blend_func=('one_minus_src_color', 'src_color'),
        blend_equation=('func_reverse_subtract'),
    ),

It's a bit saturated, but better than clipped I would say:

An alternative option is to use minimum blending. That requires a vispy PR (being brave!): vispy/vispy#2282
to add the GL_MIN and GL_MAX modes.
The flaw with this one it selects the MIN of R, G, B independently, if I am understanding the docs right (https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glBlendEquation.xhtml) so you tend to just get black where it's "blending" multiple channels:

[psobolewskiPhD]

Update: the original vispy PR while functional, was totally misimplemented by modifying auto-generated files. I've not been able to figure out the proper path—nor had much time to do so—so it's DOA at the moment.

[sofroniewn]

Hi @psobolewskiPhD, so sorry this post hasn't gotten a lot of engagement yet! I'm definitely very supportive of getting the light mode and inverse lookup tables working well in napari. It looks like vispy/vispy#2320 just got merged which may be an important step!!

Can you give an update here on how you think we should best move forward this (or point to another more recent convo, which I may have missed). Thanks!!

[psobolewskiPhD]

FYI vispy 0.10.0 is live: https://github.com/vispy/vispy/releases/tag/v0.10.0

[psobolewskiPhD]

So I started to go ahead with implementing minimum blending. Easy peasy.
Except, an issue.
napari at present uses vispy GL_PRESETS for translucent and additive but in vispy these don't specify the blend_equation. Meanwhile, the settings are like toggles (see vispy set_state) so if you explicitly set min as your blend_equation while the preset doesn't set it, then if you use the preset it will inherit the new equation. If everything is using add it's not apparent, but if you use something totally different it's...less than ideal.
So either you drop use of those in napari, like @kevinyamauchi did here:
https://github.com/napari/napari/pull/3910/files#
Or PR to vispy and see? I'm going to do the later, since it seems like a bug to me.
But napari could still choose to not use the presets...
Any thoughts @kevinyamauchi @sofroniewn ?

[jni]

@brisvag might also have thoughts. 😂

[psobolewskiPhD]

Well, I made a PR to vispy, because I really do think the behavior is a bug.
vispy/vispy#2324
I have an example in the PR that shows the bug.
So to update: currently for min blending in napari either this bug needs to be fixed (PR merged, new release of vispy) or drop using the vispy presets in napari like @kevinyamauchi did.
(For the record: I didn't observe this bug previously because my min blending was implemented in @kevinyamauchi branch where he overwrote the vispy defaults!)

[brisvag]

Yup, agreed that that's a bug. Let's keep the vispy PR train rolling! :)

[psobolewskiPhD]

Going to bump this, because the GL_PRESETS bug in vispy fix has been merged (❤️ @brisvag ) so that is likely to be in a vispy release in the near future.
So looking at the checklist above:

So to sum up:
[x] Fix Light mode multichannel images—don't blend with canvas
[ ] Add minimum blending mode for inverted LUTs (easy.)
[ ] Add some inverted LUTs to the GUI (easy?)
[ ] Implement subtractive blending for inverted LUTs (hard)
[ ] Add a way to manage colormaps/LUTs, perhaps via plugin (TBD)
[ ] Add a way to make RGBA composites, perhaps via contextual menu (medium?)

Now a PR to implement nr 2 on the list above is easy, because i've been using it for some time. I figure I would just need to write some tests.
Should that PR also include some inverted LUTs (beyond gray r) so people can use this new blending mode (point 3 above)?
Or is that a 2nd PR?
Regarding nr. 4 above (true subtractive blending), I've spent a lot of time with the GL docs and various forums, etc. I'm pretty confident this is not possible using GL blending. Maybe there is a texture/shader strategy, but that's way beyond what I can comprehend.
The other points (plus maximum blending) are obviously not blocking, but nice to have things.

[sofroniewn]

Thanks for bumping this @psobolewskiPhD - I'd say start with the simple easy PR with some test, let's get that in and then we can make a second PR with more LUTs. I'm flagging for @potating-potato in case he knows of anyone on the CZI side that might have so cycles to help review/ advise here.

[psobolewskiPhD]

I've put in a PR to implement just minimum (and fix translucent_no_depth lacking a blend_eq):
#4875

So looking at the checklist above the current status is:

• Fix Light mode multichannel images—don't blend with canvas
• Add minimum blending mode for inverted LUTs (easy.)
• Add some inverted LUTs to the GUI (easy?)
• Implement subtractive blending for inverted LUTs (hard)
• Add a way to manage colormaps/LUTs, perhaps via plugin (TBD)
• Add a way to make RGBA composites, perhaps via contextual menu (medium?)

[liu-ziyang]

Unfortunately, we (CZI) currently don't have much bandwidth to commit to providing additional help on this.
On the review part, @perlman can you take a look and see if you can give some input on 4875? I am hoping once you are freed with the shader work you can help a little here

[psobolewskiPhD]

Regarding No. 2 above and colormaps, I've made a colormap manager widget:

1. that gives access to the inverted ChrisLUTs from @cleterrier (https://github.com/cleterrier/ChrisLUTs)
2. permits easy creation of simple vispy colormaps, conventional and inverted, but just a single end-point at the moment
3. permits importing and applying ImageJ .lut files, both binary and ascii formatted
   You can find the widget here:
   https://github.com/psobolewskiPhD/napari_scripts
   It should run with python napari_colormaps_widget.py

I have some thoughts to add a way to make a colormap from an RGB image, ala this twitter thread:
https://twitter.com/henriqueslab/status/1436618944845070336?s=21&t=YRgE-u6HzBX6nZKEOqVCQA
(there I used the macOS color picker).
Also, I think it could be useful to be able to save the imported LUTs or perhaps save a set of colormaps for easy reuse.

[psobolewskiPhD]

This topic was discussed at the Aug 17 2022 community meeting:
https://hackmd.io/BXWDZ3i8Q6OAEASrkaSNIQ
CC: @kevinyamauchi @andy-sweet @Czaki

Aspects that seemed to have some agreement:

1. minimum blending being generally useful and something that should be in with appropriate docs
2. End-user curation/creation of colormaps/LUTs would be a good thing to be core—at some point
3. End-user colormaps being persistent in the napari colormap list could be considered a bug
4. Importing ImageJ/Fiji LUT is best for a plugin

In the short term, perhaps adding an example of minimum blending with a set of colormaps (ChrisLUTs?) is a good compromise? Perhaps the kidney example as it's in the sample data and 3Ch?

[Czaki]

3. End-user colormaps being persistent in the napari colormap list could be considered a bug

maybe not fully as a bug, but as a feature that behavior may change in future

Importing ImageJ/Fiji LUT is best for a plugin

At this moment Yes. In future I would love to see a feature that will allow persisting such loaded colormaps between sessions (maybe only API, maybe full GUI support, where such plugins become obsolete).

[andy-sweet]

In the short term, perhaps adding an example of minimum blending with a set of colormaps (ChrisLUTs?) is a good compromise? Perhaps the kidney example as it's in the sample data and 3Ch?

Yes, this sounds like a good compromise to me. I think it's OK to add a few more colormaps that work well with minimum blending + light theme. Then we can improve the general approach (e.g. colormaps as a user preference/setting) in the long term.

[psobolewskiPhD]

I added an example to the PR:
3c5e546
I ended up using Cells3D sample, because it's pretty familiar, so the effect is easier to "get".

I think this part of the website also needs amending in the PR:
https://napari.org/stable/guides/layers.html#blending-layers
But I'm not sure where/how to do that. Is that part of the other repo?

[andy-sweet]

Looks great!

The markdown source for the page you mentioned is in napari/napari:

napari/docs/guides/layers.md

Line 53 in 29cd6fe

## Blending layers

So you should just need to edit that, which you can should/do in the same PR you add minimum blending, the colormaps, and the example.

[psobolewskiPhD]

Updating the status of this Discussion. Here's the checklist from above:

• Fix Light mode multichannel images—don't blend with canvas
• Add minimum blending mode for inverted LUTs (easy.)
• Add some inverted LUTs to the GUI (easy?)
• Implement subtractive blending for inverted LUTs (hard)
• Add a way to manage colormaps/LUTs, perhaps via plugin (TBD)
• Add a way to make RGBA composites, perhaps via contextual menu (medium?)

Minimum blending along with a handful of inverted colormaps (ChrisLUTs) are in 0.4.17 #4875
As mentioned above, proper subtractive blending via OpenGL/vispy is not trivial because none of the default GL blend equations support it (it's essentially 1-sum(1-value) for each of R, G, B, with appropriate clipping). I think it might be possible to do using shaders/textures?

I think the rest of the points regarding colormap management and/or importing ImageJ colormaps are sort of a new discussion/issue. Likewise better saving of rendered composites at full resolution via make composite contextual menu, or such is probably beyond the scope of this discussion.
One last point: is there any interest in maximum blending a conventional colormap analog of minimum? it has the advantage that it doesn't saturate, but the same issue as minimum where it doesn't use alpha.

[brisvag]

I think it doesn't hurt to have an extra way to do things, so +1 from me for the maximum blending!

[psobolewskiPhD]

I forgot about this, but it's on my to-do list!

[psobolewskiPhD]

So thinking out-loud here:
the canvas color plays a crucial role in the way additive and minimum blending modes work:

• Additive blending of the bottom-most visible layer only works with a black canvas, because you can't add to white aka (1, 1, 1).
• Minimum blending of the bottom-most visible layer only works with a white canvas, because black (0, 0, 0) will always be the minimum.

For a multi-channel image, this can be addressed by making the bottom-most layer translucent instead. This is the default for split channels at present after this PR: #4394
Open the 3D cells example, switch to Light mode. It's fine. But if you toggle the bottom layer to invisible then you get a white square, because the top one was additive and now it's blending with canvas. The same happens if you use a inverted LUT, minimum blending, and Dark theme.
So this can be addressed by changing the canvas color to match the blending mode instead of as currently by theme (there is an override). But that's a bit annoying because you have to keep swapping the canvas depending on your blending mode.

So, what if instead we just make the bottom-most visible layer always use translucent blending? For a single channel there is no difference between additive and transparent (on black canvas). Likewise, minimum and transparent are identical on a white canvas.
It would be a bit magic, because we'd need to override the setting in blending menu—is this even possible? But it would be the most elegant fix to the issue I think, from the user point of view anyways.

[psobolewskiPhD]

OK, I've got a proof-of-concept script:
https://gist.github.com/psobolewskiPhD/480cb15f6424f0eabc2dee5524b20ef2
Regardless of your theme (you can toggle it) this minimum blending example should JustWork regardless which layers you toggle. Rearranging and removing should also work.
(napari events are pretty powerful!)
Really eager to hear some feedback.

[brisvag]

Instead of changing the blending altogether, it seems like we could handle this on the vispy side and simply change the gl context as appropriate. This would also make it possible/easier to allow both translucent and translucent_no_depth at the bottom. Do you have a hunch on what exactly inside the translucent preset is responsible of this fix?

[psobolewskiPhD]

The most elegant way I think would be to blend with the color of the initial point of the colormap, because that is the background for that layer.
So a conventional colormap, say Green [Black -> Green] would blend to black, regardless of canvas.
For a different colormap, it would use whatever the 0 point is, so White for one of the inverted colormaps [White -> Green].

[brisvag]

This basically sounds like changing the contrast limits though... maybe we can simply reroute the opacity slider to that?

[psobolewskiPhD]

I think you are right, reducing opacity is functionally the same as moving the upper range of contrast limits—not sure it's mathematically the same though. However, alpha still works to dim stuff that is saturated, while contrast limits won't? Edit: try binary blobs.

[brisvag]

Yes but what does "dim stuff" actually mean? I think it's just an informal way of saying "blend with the canvas with the following function":

blend_func=('src_alpha', 'one_minus_src_alpha')

So we're back to square one :P

[brisvag]

I opened a PR over at #5487 with a possible solution to part of this problem. Please test it out and tell me how it feels!

[psobolewskiPhD]

Status update:
Merging #5487 was huge!! Hurrah @brisvag 🎉
It basically fixed the issue of blending with the canvas. For additive and minimum blending there is now custom overrides to ensure proper behavior when at the bottom, regardless of the menu blending status.
So I think with that and the minimum blending, largely all of the goals of this discussion thread have been met!

On knockoff effect: for splitting multichannel and our sample images, we have been using translucent for the base and additive for the rest—this was implemented to guard against blending with canvas. This is slightly annoying if you choose to reorder the layers, but no longer needed because of the new behavior of overriding the bottom visible layer blending mode.
I think it could be worth to go through and drop the translucent + additive pattern for split channel and samples to pure additive. It would make sorting layers easier and maybe be more intuitive? Anyone have thoughts?

Remaining things and tangents:

• update docs PR Add a note about how blending interacts with canvas docs#69
• maximum blending, it's on my TODO.
• colormaps plugin

[brisvag]

I think it could be worth to go through and drop the translucent + additive pattern for split channel and samples to pure additive. It would make sorting layers easier and maybe be more intuitive? Anyone have thoughts?

Yup, totally agree!