Support timeout trigger skip instead of fail

[yasirroni]

This is an initial PR for support timeout that trigger skip instead of fail. If this concept is accepted, I'm willing to add pytest and doc. Thank you.

[RonnyPfannschmidt]

Why skip?

[yasirroni]

The rationale is because I use pytest with --nbdime that run notebooks and pytest at the same time. Fails due to timeout in this directory containing various notebooks example is deemed as not crucial. Instead of not testing those notebooks at all, I would be rather knowing if those notebooks did fails due to timeout and not other error. Ideally, I should try and catch timeout, but changing how nbdime and pytest works seems too complicated.

Currently, what is possible in nbdime is to allow notebook to fail (I already use it), but sadly timeout error is not fail in the cell but on the side of pytest (can't be catch by nbdime as far I know).

Thus, I need pytest that allow timeout error.

[flub]

Thanks for giving more rationale. I'm currently leaning against this I'm afraid. The current philosophy of pytest-timeout is that timeouts are fatal and need to be debugged. You seem to use it as some kind of "i don't care enough about the results to wait longer" if I understand correctly (I might not have understood correctly!), which I'm not really keen to add more features to pytest-timeout for.

[yasirroni]

You seem to use it as some kind of "i don't care enough about the results to wait longer" if I understand correctly (I might not have understood correctly!), which I'm not really keen to add more features to pytest-timeout for.

It's more like that "let skip that for now". Skip is different from success, it works like a warning.
_
Without skip support, I'm in a position of "skip the whole example notebook" or "workflow will always fail due to single example require 24 hours of simulation running".

Or, I need to change the notebook itself, like comments all cells below the long run cell and tell user to uncomment those cells to run the example.

[RonnyPfannschmidt]

From my pov having it xfail would be a stretch, but Skip ain't acceptable

Even then, xfail should not available as global configuration by default, but rather as context manager