Register pre/post save hooks, call them sequentially #696

davidbrochart · 2022-02-08T20:32:01Z

Closes #695

codecov-commenter · 2022-02-08T20:35:06Z

Codecov Report

Merging #696 (8bef774) into main (5d962d7) will decrease coverage by 0.19%.
The diff coverage is 48.43%.

@@            Coverage Diff             @@
##             main     #696      +/-   ##
==========================================
- Coverage   70.95%   70.76%   -0.20%     
==========================================
  Files          62       62              
  Lines        7515     7555      +40     
  Branches     1190     1199       +9     
==========================================
+ Hits         5332     5346      +14     
- Misses       1826     1851      +25     
- Partials      357      358       +1

Impacted Files	Coverage Δ
jupyter_server/services/contents/manager.py	`81.77% <44.06%> (-6.42%)`	⬇️
jupyter_server/services/contents/filemanager.py	`72.24% <100.00%> (+1.51%)`	⬆️
...pyter_server/services/contents/largefilemanager.py	`83.83% <100.00%> (ø)`
jupyter_server/services/kernels/kernelmanager.py	`80.69% <0.00%> (-1.90%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d962d7...8bef774. Read the comment docs.

rahul26goyal

dont have much context on the functionality.. added some minor comments.

jupyter_server/services/contents/filemanager.py

davidbrochart · 2022-02-15T12:53:58Z

Should we go ahead and merge it? This will allow to move forward with jupyterlab/jupyterlab#11599.

davidbrochart · 2022-02-15T13:15:17Z

I just saw the comments in the team compass:

Should probably have different ways to add/remove hooks as needed

I'm not sure, as this would allow e.g. a server extension to remove the hooks added by another extension?

Order is probably important

It will probably be impossible to enforce a particular order. If e.g. multiple server extensions register hooks, then it is going to be the order in which the extensions are loaded.
I think we should mention that the order in which hooks are called is not known.

Uniqueness should probably be enforced in case the validator triggers multiple times?

Maybe it should also be possible e.g. or different server extensions to register the same hook?

Maybe expose a new trait exposing the collection of hooks for manipulation.

I thinks this could be part of another PR, as it seems to require more discussion and specification.
The goal of this PR is to keep the same API, and just allow for multiple hooks instead of overriding an unique hook. I don't want to block jupyterlab/jupyterlab#11599 if a more substantial work is needed.

vidartf · 2022-02-22T12:39:02Z

Uniqueness should probably be enforced in case the validator triggers multiple times?

Maybe it should also be possible e.g. or different server extensions to register the same hook?

I'm not sure if we are talking about the same thing. I think the original meaning here was "the same function should not be called twice by the same event". Two different server extensions can install a hook, but if they both register the same function, then the hook should only be called once. For example by making the collection of hooks a set.

It will probably be impossible to enforce a particular order. If e.g. multiple server extensions register hooks, then it is going to be the order in which the extensions are loaded.

If this is true, then maybe the RTC handler's use of this hook needs to be rethought? AFAICT, order will be very significant for that to function as expected.

davidbrochart · 2022-02-22T13:16:47Z

If this is true, then maybe the RTC handler's use of this hook needs to be rethought? AFAICT, order will be very significant for that to function as expected.

But if the RTC handler won't be able to use this hook, then will any server extension? How can we enforce hook calling order?

davidbrochart · 2022-03-03T14:16:17Z

I think it is urgent to move on with jupyterlab/jupyterlab#11599, after what was reported in jupyterlab/jupyterlab#12154, since it will likely solve the issue, but this PR is blocking. I suggest that either:

we merge this PR as-is,
or we drop this PR while we work on a better solution, and we show a warning each time a file hook overwrites an existing one.

I am not aware of anyone using the file hooks (did some GitHub search, but I agree it's not enough), so I would be in favor of merging this PR.
Thoughts?

Zsailer · 2022-03-03T16:01:59Z

What is self.post_save_hook after it has been set multiple times? It's not a list—it's the value of the last time it was called, correct? The list is stored under self._post_save_hooks.

Here's an problem I see using an example:

If anyone has configured a post_save_hook, which likely lives in a deployment's custom config file (and likely not shared on Github if it's a company), they will expect the value of post_save_hook to be the value coming from their config file. However, since extensions would set this trait after user config is loaded in the server's start-up sequence (init_configurables happens before load_server_extension), the value of this trait would be the post_save_hook provided by the last extension to set it, not user's configuration.

davidbrochart · 2022-03-03T21:56:59Z

Which is not different from the current situation: an extension configures a post_save_hook and then another one configures a new post_save_hook. The first extension expects the value to be the one it has configured, but it's going to be the one from the second extension. Worse, the post_save_hook the first extension configured will never be called.

davidbrochart · 2022-03-08T16:10:27Z

I think this PR only improves on the current situation, and I'm planning to merge it soon if there is no feedback.

Zsailer · 2022-03-08T16:42:06Z

Which is not different from the current situation: an extension configures a post_save_hook and then another one configures a new post_save_hook. The first extension expects the value to be the one it has configured, but it's going to be the one from the second extension. Worse, the post_save_hook the first extension configured will never be called.

Yes, but this is a bug. That's my point. We shouldn't perpetuate this issue by capitalizing on it here.

This trait wasn't intended to be owned by extensions. As a user—you would expect that your custom configuration supersedes any config set by an extension. In jupyter_server land, where extensions are more prevalent (i.e. everything is an extension), traits like this become an issue. I think we need to fix/deprecate/etc this trait rather than exploit it here.

I would suggest we create a new trait for JupyterLab to use that's a List and deprecate the old. We can iterate on the new trait (e.g. enforcing order, removing duplicates, etc.) to make it more robust over time.

davidbrochart · 2022-03-08T16:53:38Z

Now I'm confused, what is the audience that this feature targets? It seems to me that it's not usable by anybody, and that it should even be removed from the documentation. In my case, it would have saved me a lot of time.

EDIT: sorry for ranting here 😄

davidbrochart · 2022-03-08T17:18:40Z

I can create a new trait that is a List, but it could be dangerous to expose this list and allow anybody to change it. Ideally, one would just append to it, but it is easy to override it. Also, I don't know if appending is supported when the trait is set through a config file?

Zsailer · 2022-03-08T17:21:55Z

As a user—you would expect that your custom configuration supersedes any config set by an extension

Sorry for the confusion... "user" is a misleading term to use here.

The intended us is: a person who runs Jupyter Server or JupyterLab can write a custom function and pass it to the contents manager through the config system using the ContentsManager.pre_save_hook in a jupyter config file, e.g.:

# jupyter_server_config.py file 
c.FileContentsManager.pre_save_hook = my_custom_function

That's what the documentation is (attempting) to communicate 😅.

I think a good rule of thumb is, configurable traits (i.e. traits with config=True) are intended to be configured by user config (from config files or CLI), not extensions. If we're going to break this rule, we should carefully consider the consequences.

If JupyterLab overrides this trait after the user config is loaded, pre_save_hook will not return my_custom_function but JupyterLab's hook. IMO, that is a bug in its current form.

If we change this trait to be a List and allow multiple sources, including extensions, to provide hooks to that list—that seems okay to me. We probably want to add some extra logic to this List trait to address the issues that the comments above mention.

Zsailer · 2022-03-08T17:24:29Z

I can create a new trait that is a List, but it could be dangerous to expose this list and allow anybody to change it. Ideally, one would just append to it, but it is easy to override it. Also, I don't know if appending is supported when the trait is set through a config file?

Then, let's not make this a configurable trait?

Let's add a new API to contents managers to register hooks through the API instead of the config system?

Zsailer · 2022-03-08T17:27:04Z

I think we're just debating the "trait-ness" or "configurability" of this API. 😆 Let's just add a new, separate API for registering hooks from extensions? It doesn't have to collide with or go through this trait?

davidbrochart · 2022-03-08T17:29:43Z

Yes you're right, a new "non-trait" API sounds good 👍

davidbrochart · 2022-03-08T20:35:58Z

Not sure what's happening in the CI:

File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/zmq/sugar/context.py", line 190, in term
    return super(Context, self).term()

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
tests/services/sessions/test_api.py::test_restart_kernel[jp_server_config0]

blink1073 · 2022-03-08T20:59:51Z

Not sure what's happening in the CI

I think this is related to jupyter/jupyter_client#751

jupyter_server/services/contents/filemanager.py

jupyter_server/services/contents/manager.py

jupyter_server/services/contents/filemanager.py

kevin-bates · 2022-03-10T15:07:40Z

jupyter_server/services/contents/largefilemanager.py

@@ -18,7 +18,7 @@ def save(self, model, path=""):
        if chunk is not None:
            path = path.strip("/")

-            self.run_pre_save_hook(model=model, path=path)
+            self.run_pre_save_hooks(model=model, path=path)


In looking at the history of this issue, including jupyterlab/jupyterlab#11599, it led me to PR #643, where this call was within the chunk == 1 block. As a result, that PR winds up calling the pre-save hook on every chunk, although the post-save hook is (still) only called on the last chunk.

I suspect a bug was introduced there but wanted to bring it up here.

Indeed, thanks for pointing that out Kevin. I think it's just a matter of adding the if chunk == 1 back, right? I can open a new PR for that.

Yes, I agree.

I think this particular hook needs to still be constrained to chunk 1 even though chunks don't appear to be in use (and handled in the else block).

Yes, but it's part of #716. How does it sound to merge this PR and then #716?

Wouldn't the sequence be the other way around? #716 is fixing a bug in the existing (pre-this-PR) code base, then this PR is moving to a multi-valued hook trait. As it stands right now, this PR has a bug - it calls the pre-save hooks on every chunk (when chunking is used). I'd rather see the issue fixed as part of merging #716 and the merge that would need to happen to update this PR be applied, thereby correcting the chunking/pre-save-hooks bug. I thought the end game was to have the multi-valued hooks in place, merging this PR prior to #716 will re-introduce the single-hook trait.

I'm fine with merging #716 first 👍

davidbrochart · 2022-03-10T16:57:58Z

I added a test for register_pre_save_hook, since that's the one I will use in jupyterlab/jupyterlab#11599.
Post-save hooks seems a bit more complicated to test, since we would need to access the return value (the model) of the save method.

davidbrochart · 2022-03-10T17:03:44Z

Actually this should be the response to the request, I'll try to add a test for post-hooks then.

davidbrochart · 2022-03-11T09:15:34Z

I think this is ready to be merged, if someone could hit the button 😄

Zsailer · 2022-03-11T16:14:51Z

Great work here @davidbrochart! I'm happy to merge this.

We picked up some merge conflicts here after merging #716—do you mind rebasing here?

I don't think we can release until #717 is merged, because we learned from #713 that #165 introduced some backwards incompatible changes.

I propose that we wait until after the weekend to cut a new release with these additions, since releasing before/over a weekend can be dangerous 😅

davidbrochart · 2022-03-11T16:29:47Z

Sounds good @Zsailer, I'll rebase and we'll release next week hopefully!

…d hook override warnings

davidbrochart · 2022-03-14T08:47:45Z

I just rebased, I think the CI failures are unrelated.

blink1073 · 2022-03-14T10:26:58Z

Kicking CI to pick up fixes

blink1073 · 2022-03-14T10:29:25Z

Once this passes, I'll merge and make a minor release

blink1073 · 2022-03-14T10:30:20Z

Actually, I kicked #724 as well. Assuming both pass we'll get them both in

blink1073 · 2022-03-14T10:34:05Z

Ditto with #717

blink1073 · 2022-03-14T10:49:17Z

Also #725

vidartf · 2022-03-31T16:14:53Z

@davidbrochart Do you want to do a follow up to update the documentation on this? https://jupyter-server.readthedocs.io/en/latest/developers/savehooks.html?highlight=pre_save_hook

The change would be to this file: https://github.com/jupyter-server/jupyter_server/blob/main/docs/source/developers/savehooks.rst

davidbrochart · 2022-03-31T16:22:28Z

Will do 👍

davidbrochart added the enhancement label Feb 8, 2022

rahul26goyal reviewed Feb 10, 2022

View reviewed changes

jupyter_server/services/contents/filemanager.py Outdated Show resolved Hide resolved

jupyter_server/services/contents/filemanager.py Outdated Show resolved Hide resolved

davidbrochart force-pushed the save_hooks branch 3 times, most recently from f78c480 to 5f0f6d2 Compare February 14, 2022 12:55

Zsailer mentioned this pull request Feb 23, 2022

Meeting Notes 2022 jupyter-server/team-compass#15

Closed

davidbrochart force-pushed the save_hooks branch 2 times, most recently from 98ca951 to adfe0d0 Compare March 8, 2022 20:15

Zsailer reviewed Mar 8, 2022

View reviewed changes

jupyter_server/services/contents/filemanager.py Show resolved Hide resolved

Zsailer reviewed Mar 8, 2022

View reviewed changes

jupyter_server/services/contents/manager.py Show resolved Hide resolved

Zsailer reviewed Mar 8, 2022

View reviewed changes

jupyter_server/services/contents/filemanager.py Outdated Show resolved Hide resolved

davidbrochart force-pushed the save_hooks branch from 18147cf to c474a89 Compare March 9, 2022 01:09

kevin-bates reviewed Mar 10, 2022

View reviewed changes

davidbrochart mentioned this pull request Mar 10, 2022

Call pre_save_hook only on first chunk of large files #716

Merged

davidbrochart force-pushed the save_hooks branch from 5355683 to d0b0813 Compare March 10, 2022 16:52

Zsailer mentioned this pull request Mar 11, 2022

allow handlers to work without an authorizer in the Tornado settings #717

Merged

davidbrochart added 7 commits March 14, 2022 09:13

Register pre/post save hooks, call them sequentially

88b03ff

Changes according to review

e3a3d5e

Update documentation

4eabb5f

Restore pre/post_save_hook traits, add register_pre/post_save_hook()

eadd7d4

Move post-save hooks to ContentsManager, add deprecation warnings, ad…

1f4e371

…d hook override warnings

Add pre-save hook test

87a36be

Add post-save hook test

8bef774

davidbrochart force-pushed the save_hooks branch from b1ecd2e to 8bef774 Compare March 14, 2022 08:16

blink1073 closed this Mar 14, 2022

blink1073 reopened this Mar 14, 2022

blink1073 merged commit 32b2f74 into jupyter-server:main Mar 14, 2022

davidbrochart deleted the save_hooks branch March 14, 2022 10:53

davidbrochart mentioned this pull request Mar 31, 2022

Update documentation about registering file save hooks #770

Merged

Register pre/post save hooks, call them sequentially #696

Register pre/post save hooks, call them sequentially #696

Conversation

davidbrochart commented Feb 8, 2022

codecov-commenter commented Feb 8, 2022 • edited

Codecov Report

rahul26goyal left a comment

Choose a reason for hiding this comment

davidbrochart commented Feb 15, 2022

davidbrochart commented Feb 15, 2022

vidartf commented Feb 22, 2022 • edited

davidbrochart commented Feb 22, 2022

davidbrochart commented Mar 3, 2022

Zsailer commented Mar 3, 2022

davidbrochart commented Mar 3, 2022 • edited

davidbrochart commented Mar 8, 2022

Zsailer commented Mar 8, 2022

davidbrochart commented Mar 8, 2022 • edited

davidbrochart commented Mar 8, 2022

Zsailer commented Mar 8, 2022

Zsailer commented Mar 8, 2022

Zsailer commented Mar 8, 2022

davidbrochart commented Mar 8, 2022

davidbrochart commented Mar 8, 2022

blink1073 commented Mar 8, 2022

kevin-bates Mar 10, 2022 • edited

Choose a reason for hiding this comment

davidbrochart Mar 10, 2022

Choose a reason for hiding this comment

kevin-bates Mar 10, 2022

Choose a reason for hiding this comment

kevin-bates Mar 11, 2022

Choose a reason for hiding this comment

davidbrochart Mar 11, 2022

Choose a reason for hiding this comment

kevin-bates Mar 11, 2022

Choose a reason for hiding this comment

davidbrochart Mar 11, 2022

Choose a reason for hiding this comment

davidbrochart commented Mar 10, 2022

davidbrochart commented Mar 10, 2022

davidbrochart commented Mar 11, 2022

Zsailer commented Mar 11, 2022

davidbrochart commented Mar 11, 2022

davidbrochart commented Mar 14, 2022

blink1073 commented Mar 14, 2022

blink1073 commented Mar 14, 2022

blink1073 commented Mar 14, 2022

blink1073 commented Mar 14, 2022

blink1073 commented Mar 14, 2022

vidartf commented Mar 31, 2022

davidbrochart commented Mar 31, 2022

codecov-commenter commented Feb 8, 2022 •

edited

vidartf commented Feb 22, 2022 •

edited

davidbrochart commented Mar 3, 2022 •

edited

davidbrochart commented Mar 8, 2022 •

edited

kevin-bates Mar 10, 2022 •

edited