Add (interactive) slice explorer in plotly.

[mkcor]

Description

Hello @sciunto and @emmanuelle !

This PR is an addition to the 3D image processing introduction tutorial, which landed with #4850.
It relates to #4850 (comment) and #4762.
I've made a slice explorer with Plotly so it renders even in the static HTML page.
In future iterations, I'll add a 3D visualization showing the position of the selected slice within the dataset.

I've chosen the density_heatmap function so I could readily use its animation_frame argument for the slider interaction (selection). This function is made for aggregation, which I'm bypassing by setting the number of bins to the data size...

Ideally, I wanted to use imshow to view the image slices (planes) but this wouldn't offer such a convenient abstraction in terms of slider interaction.

Checklist

• Docstrings for all functions
• Gallery example in ./doc/examples (new features only)
• Benchmark in ./benchmarks, if your changes aren't covered by an
  existing benchmark
• Unit tests
• Clean style in the spirit of PEP8

For reviewers

• Check that the PR title is short, concise, and will make sense 1 year
  later.
• Check that new functions are imported in corresponding __init__.py.
• Check that new features, API changes, and deprecations are mentioned in
  doc/release/release_dev.rst.

[pep8speaks]

Hello @mkcor! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-02-14 18:18:19 UTC

[emmanuelle]

Thanks @mkcor. It's a bit hacky but it works beautifully. I've pushed a commit to your branch because you need to set a specific plotly renderer for sphinx-gallery to integrate correctly the figure.

Ideally we would use go.Image or px.imshow but indeed the animation is less straightforward. Maybe I can find a way to use imshow though, let me think about it a little bit...

[emmanuelle]

One caveat of using density_heatmap is that the resulting html page weighs more than 100 Mo! This observation led me to start plotly/plotly.py#2746
(px.imshow has a new mode where you can pass the image data as a binary string like a png string, therefore the figure is much smaller). Thanks for triggering me to work on this, it's been on my todo list for a long time :-).

[emmanuelle]

@mkcor I made good progress in plotly/plotly.py#2746, so you'll be able to write

fig = px.imshow(img, animation_frame=0, binary_string=True)

(binary_string=True is to make sure the size of the figure stays small)

This feature should be available in plotly 4.11 so in a month I believe. In the meantime I'm interested in your feedback if you have the time to do some tests with the corresponding plotly branch!

[mkcor]

@emmanuelle this is great news! :)

In the meantime I'm interested in your feedback if you have the time to do some tests with the corresponding plotly branch!

For sure! I will update this example, try out the new feature with other datasets as well, and let you know.

[ianhi]

Hiya,

I just stumbled across this when searching for a multi-dim microscopy image dataset to use as an example. This looks awesome, and I'm super stoked about all the microscopy data that will be available in version 0.18!

Would you be interested in an example showing how to slice image stacks using ipywidgets+matplotlib? As an example of whats possible, below is an example of the hyperslicer function that @jrussell25 just contributed to mpl-interactions. hyperslicer will automatically create sliders (ipywidgets sliders or matplotlib sliders - and potentially potentially animated) that can be used to explore slices of N-dim (hyper)stacks. The full docs for it are at: https://mpl-interactions.readthedocs.io/en/latest/examples/hyperslicer.html
(mpl-interactions is a library I've been working on to smooth over the process of making interactive matplotlib plots)

For example using hyperslicer with the skimage cells dataset:

from mpl_interactions import hyperslicer
fig, ax = plt.subplots()
z_step = 0.0650000
N = cells.shape[0]
z_values = np.linspace(-N / 2, N / 2, cells.shape[0]) * z_step # assume the center is the focus
controls = hyperslicer(cells, axes=(("z", z_values),), play_buttons=True, play_button_pos="left")

Suggested example

I'm nott necessarily suggesting including mpl_interactions in the skimage examples as it's probably too fresh and I'm really the only maintainer. But would you be interested in a sibling PR to this that shows how to accomplish stack slicing using matplotlib and ipywidgets? I'm imagining we could make something based off the code in mpl_interactions (all BSD 3 clause licensed) https://github.com/ianhi/mpl-interactions/blob/0c6508cafee0c7e709d1d8ef9f12a3dadef70558/mpl_interactions/generic.py#L489

Happy to hear your thoughts and thanks so much for skimage!

[ianhi]

Got a bit carried away here and found another potential example 😬. Similarl to above question, would you be interested in an example using ipywidgets or matplotlib widgets for varying the values of some transform? e..g adjusting the expsoure (https://scikit-image.org/docs/dev/auto_examples/applications/plot_3d_image_processing.html#adjust-exposure)

Like with hyperslicer below example uses mpl_interactions.imshow as it's slightly this easier, but it shouldn't be too complex with just matplotlib + ipywidgets.

plt.figure()
plt.imshow(exposure.adjust_gamma(cells[30], 1))
def f(gamma):
    return exposure.adjust_gamma(cells[30], gamma)
controls = iplt.imshow(f, gamma = (.5, 4), title="gamma = {gamma:.2f}", play_button_pos='left', play_buttons=True)
plt.tight_layout()

[mkcor]

Hello @ianhi!

Thank you for your interest, your contributions, and your patience.

Following your inputs, I think I'll split the 'Explore 3D images' tutorial into its main part and the other part about adjusting exposure. Then, we will be more at ease to expand each part with alternative ways in terms of interactive exploration (whether of data slices or parameter values). Otherwise, the example will get too cluttered.

As you have seen and read in the example, the downside of Jupyter widgets is that, unlike Plotly figures, they don't render in our docs (static HTML pages). Having said that, I think it's still valuable to include relevant code meant to run with a Jupyter kernel.

[mkcor]

Hi again @ianhi,

Got a bit carried away here and found another potential example grimacing. Similarl to above question, would you be interested in an example using ipywidgets or matplotlib widgets for varying the values of some transform? e..g adjusting the expsoure (https://scikit-image.org/docs/dev/auto_examples/applications/plot_3d_image_processing.html#adjust-exposure)

Like with hyperslicer below example uses mpl_interactions.imshow as it's slightly this easier, but it shouldn't be too complex with just matplotlib + ipywidgets.

plt.figure()
plt.imshow(exposure.adjust_gamma(cells[30], 1))
def f(gamma):
    return exposure.adjust_gamma(cells[30], gamma)
controls = iplt.imshow(f, gamma = (.5, 4), title="gamma = {gamma:.2f}", play_button_pos='left', play_buttons=True)
plt.tight_layout()

Actually, there is already a section dedicated to exposure adjustment in the narrative docs. So how about adding interactive exposure adjustment to this existing example, whether for gamma correction or logarithmic correction?

[emmanuelle]

@mkcor please checkout the new release of plotly :-).

fig = px.imshow(img, animation_frame=0, binary_string=True)

and other examples https://plotly.com/python/imshow/#exploring-3d-images-and-timeseries-with-animationframe

[mkcor]

@emmanuelle for sure! I'll update this PR later today :-)

[emmanuelle]

Hi @mkcor could you please merge master into your branch? The recurring errors in the circleCI doc build are (hopefully) fixed in master, and this way reviewers can look at the generated html pages without fetching your branch.

[mkcor]

@emmanuelle for sure! I forgot that this PR was initially submitted before we added the "Update branch" button... ;-)
Here is the rendered tutorial, with the new interactive slider at the very bottom of the page.

[jni]

The interactive slice explorer is brilliant! 🎉 As is the fact that we can view rendered docs in PRs! Life is good. =)

I have a couple of suggestions for the tutorial:

• the broken bits are really jarring in the online documentation, and I think there should be stronger warnings that things are not expected to work. e.g. "let us try to visualise the data with io.imshow", followed by a broken plot, feels broken even though it is broken by design. So I would change that to read something like, "Unfortunately, many image viewers, such as matplotlib's imshow, are only capable of displaying 2D data. We can see that they raise an error when we try to add 3D data:"... Then "IPython has built-in widgets that let us interact with our data. However, they require both a browser and an active Python kernel to be able to display the widget in the browser and execute Python code when the widget is activated. In the online version of this example, the following code will actually not display anything. Similarly for running the code in pure Python."
• additionally, we should use fig, ax = plt.subplots(); ax.imshow instead of using io.imshow, which I think we are aiming to deprecate.

Thank you!

[grlee77]

Another possibility similar to the display function shown in that demo, but that may be easier for users, is to use the skimage.util.montage function to tile the 2D slices for display.

For example, to make a montage by tiling every second slice of the data:

import skimage.util
data_montage = skimage.util.montage(data[::2])
_, ax = plt.subplots(1)
ax.imshow(data_montage, cmap="gray")
ax.set_axis_off()

Specify padding_width and fill can give a boundary between the tiled slices (fill=np.nan is another possibility that displays as white)

data_montage = skimage.util.montage(data[::2], padding_width=2, fill=data.max())
_, ax = plt.subplots(1)
ax.imshow(data_montage, cmap="gray")
ax.set_axis_off()

[mkcor]

Thanks, @jni! See 8aa2d2a to make sure I included your wording suggestions correctly.

[mkcor]

Super nice, @grlee77! Thanks for the suggestion. Included in dda908c. I wasn't sure I should keep util.montage wrapped under display but decided to keep the helper function in the end.

[mkcor]

cf6bfdd as per #5120 (review) @emmanuelle 😉

[grlee77]

@mkcor: Would this setting have helped with the fig.show() usage in #4946?

(adding fig.show() to get figures in browser windows when running from the command line seemed to have broken rendering in the sphinx gallery)

[mkcor]

In the same file, there is also:

scikit-image/doc/source/conf.py

Lines 153 to 157 in 044ac80

	# set plotly renderer to capture _repr_html_ for sphinx-gallery
	import plotly.io as pio
	pio.renderers.default = 'sphinx_gallery_png'
	from plotly.io._sg_scraper import plotly_sg_scraper
	image_scrapers = ('matplotlib', plotly_sg_scraper,)

and from looking at the Git blame, pio.renderers.default wasn't meant to be set in two different places... I tried removing either instance, thinking there might be some unintended conflict, rebuilding the docs with the plotly.io.show(fig) call, but the figure still wouldn't render in the gallery.

[mkcor]

Falling back on #5346 then... 😞 @emmanuelle

[mkcor]

@grlee77 @emmanuelle sorry for the false alarm! I must have mixed up something in my local testing... We're all good with #5347 now! ☺️
Will update this PR accordingly as well.

[jni]

@mkcor it works! 🎉

No thumbnail though... Is that expected?

Also, the 3D dataset gallery page is broken, not sure whether this is specific to this PR or also on master...

[mkcor]

Will update this PR accordingly as well.

Done dfd9954. Confirming that we can spare import plotly and a few characters -- fig.show() instead of plotly.io.show(fig). In #5347 somehow I copied the existing example; I can edit later. Ah, function vs method...

[grlee77]

Thanks, @mkcor. This looks good now (I didn't realize this one was still pending!).

One issue I did have when trying it out is that occasionally the Plotly plot would behave strangely when dragging the animation slider (the slider itself sometimes moved to a lower position on the screen mid-drag and the image would quit updating). Hitting the "home" button at top to reset the axes would restore it to a working state again. It might be somehow related to dragging too quickly as I never saw the issue if I just clicked around on different frames, only when dragging the slider. In any case, that seems more likely to be a Plotly issue than an issue with this demo, so I have approved here. I was just curious if you have observed anything similar? I did try in both firefox and chrome and saw the same thing in both.

[mkcor]

Thank you for your review, @grlee77! Looking now at the issue you pointed out...

Sorry, it's taken me a little while because my local docs build had been broken since #5472; I had to run

$ pip install -e . --upgrade --force-reinstall

for the changes in skimage/feature/setup.py to take effect. I wonder if this should be documented under https://scikit-image.org/docs/stable/install.html#updating-the-installation?

[mkcor]

Without 4cba76b, the Plotly figure wouldn't display anymore, so I have made the script consistent (cf. 2ed4cf9).

One issue I did have when trying it out is that occasionally the Plotly plot would behave strangely when dragging the animation slider (the slider itself sometimes moved to a lower position on the screen mid-drag and the image would quit updating). Hitting the "home" button at top to reset the axes would restore it to a working state again. It might be somehow related to dragging too quickly as I never saw the issue if I just clicked around on different frames, only when dragging the slider.

Yes! I just reproduced the issue (in Firefox, with my local file). It does seem related to dragging the slider too quickly, indeed, and I was able to reset the expected display/animation by hitting the "home" button as well.

[lagru]

I think this has waited long enough. 😄 Thanks everyone!