WIP: Fix handling of bad magics

[ceball]

Fixes #147

[ceball]

Before I can get to the actual fix, I'm having some problems with the test framework :)

1. Where is --nbval-lax tested? I.e. I wasn't sure where to put my tests. In the end, I copy/pasted test_timeouts.py.

2. I'm having an issue running both new tests together:

traitlets.traitlets.TraitError: The 'session' trait of a KernelManager instance must be a Session, but a value of class 'jupyter_client.session.Session' (i.e. <jupyter_client.session.Session object at 0x7f5507c33c10>) was specified.

full version:

$ pytest -v tests/test_magics.py
=============================== test session starts ================================
platform linux -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.12.0 -- /home/sefkw/mc3/envs/celltestsui/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.7.6', 'Platform': 'Linux-5.4.0-7634-generic-x86_64-with-debian-bullseye-sid', 'Packages': {'pytest': '5.4.1', 'py': '1.8.1', 'pluggy': '0.12.0'}, 'Plugins': {'xdist': '1.32.0', 'html': '2.1.1', 'metadata': '1.8.0', 'cov': '2.8.1', 'forked': '1.1.2', 'nbval': '0.9.5'}}
rootdir: /home/sefkw/code/external/nbval
plugins: xdist-1.32.0, html-2.1.1, metadata-1.8.0, cov-2.8.1, forked-1.1.2, nbval-0.9.5
collected 2 items                                                                  
tests/test_magics.py::test_magics[%dirs-expected_passes0] PASSED             [ 50%]

tests/test_magics.py::test_magics[%this_magic_does_not_exist-expected_passes1] FAILED [100%]
===================================== FAILURES =====================================

_____________ test_magics[%this_magic_does_not_exist-expected_passes1] _____________
testdir = <Testdir local('/tmp/pytest-of-sefkw/pytest-36/test_magics1')>

magic = '%this_magic_does_not_exist', expected_passes = [False]
@pytest.mark.parametrize("magic, expected_passes", [
    (r"%dirs", [True]),
    (r"%this_magic_does_not_exist", [False])
])
def test_magics(testdir, magic, expected_passes):
    # Setup notebook to test:
    sources = [
        # In [1]:
        magic,
    ]
    nb = build_nb(sources)

    # Write notebook to test dir
    nbformat.write(nb, os.path.join(
        str(testdir.tmpdir), 'test_magics.ipynb'))

    # Run tests
    result = testdir.inline_run('--nbval-lax', '--current-env', '-s')
    reports = result.getreports('pytest_runtest_logreport')

    # Setup and teardown of cells should have no issues:
    setup_teardown = [r for r in reports if r.when != 'call']
    for r in setup_teardown:


      assert r.passed


E           AssertionError: assert False

E            +  where False = <TestReport 'test_magics.ipynb::Cell 0' when='setup' outcome='failed'>.passed
/home/sefkw/code/external/nbval/tests/test_magics.py:35: AssertionError

------------------------------- Captured stdout call -------------------------------

=============================== test session starts ================================

platform linux -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.12.0

rootdir: /tmp/pytest-of-sefkw/pytest-36/test_magics1

plugins: xdist-1.32.0, html-2.1.1, metadata-1.8.0, cov-2.8.1, forked-1.1.2, nbval-0.9.5

collected 1 item
test_magics.ipynb E
====================================== ERRORS ======================================

___________________ ERROR at setup of test_magics.ipynb::Cell 0 ____________________
self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

cls = <class 'jupyter_client.manager.KernelManager'>
def get(self, obj, cls=None):
    try:


      value = obj._trait_values[self.name]


E           KeyError: 'session'
/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:528: KeyError
During handling of the above exception, another exception occurred:
cls = <class '_pytest.runner.CallInfo'>

func = <function call_runtest_hook.. at 0x7f5507bede60>

when = 'setup'

reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(cls, func, when, reraise=None) -> "CallInfo":
    #: context of invocation: one of "setup", "call",
    #: "teardown", "memocollect"
    start = time()
    excinfo = None
    try:


      result = func()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/runner.py:244:


  lambda: ihook(item=item, **kwds), when=when, reraise=reraise


)

/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/runner.py:217:

self = <_HookCaller 'pytest_runtest_setup'>, args = ()

kwargs = {'item': <IPyNbCell Cell 0>}, notincall = set()
def __call__(self, *args, **kwargs):
    if args:
        raise TypeError("hook calling supports only keyword arguments")
    assert not self.is_historic()
    if self.spec and self.spec.argnames:
        notincall = (
            set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
        )
        if notincall:
            warnings.warn(
                "Argument(s) {} which are declared in the hookspec "
                "can not be found in this hook call".format(tuple(notincall)),
                stacklevel=2,
            )


  return self._hookexec(self, self.get_hookimpls(), kwargs)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/hooks.py:289:

self = <_pytest.config.PytestPluginManager object at 0x7f5504ba3a90>

hook = <_HookCaller 'pytest_runtest_setup'>

methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f5504b52b50>>]

kwargs = {'item': <IPyNbCell Cell 0>}
def _hookexec(self, hook, methods, kwargs):
    # called from all hookcaller instances.
    # enable_tracing will set its own wrapping function at self._inner_hookexec


  return self._inner_hookexec(hook, methods, kwargs)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/manager.py:87:

self = <pluggy._tracing._TracedHookExecution object at 0x7f5504ba3050>

hook = <_HookCaller 'pytest_runtest_setup'>

hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f5504b52b50>>]

kwargs = {'item': <IPyNbCell Cell 0>}
def __call__(self, hook, hook_impls, kwargs):
    self.before(hook.name, hook_impls, kwargs)
    outcome = _Result.from_call(lambda: self.oldcall(hook, hook_impls, kwargs))
    self.after(outcome, hook.name, hook_impls, kwargs)


  return outcome.get_result()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/_tracing.py:80:

self = <pluggy.callers._Result object at 0x7f5504b5d390>
def get_result(self):
    """Get the result(s) for this hook call.

    If the hook was marked as a ``firstresult`` only a single value
    will be returned otherwise a list of results.
    """
    __tracebackhide__ = True
    if self._excinfo is None:
        return self._result
    else:
        ex = self._excinfo
        if _py3:


          raise ex[1].with_traceback(ex[2])


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/callers.py:80:

cls = <class 'pluggy.callers._Result'>

func = <function _TracedHookExecution.call.. at 0x7f5507bf1440>
@classmethod
def from_call(cls, func):
    __tracebackhide__ = True
    result = excinfo = None
    try:


      result = func()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/callers.py:52:


outcome = _Result.from_call(lambda: self.oldcall(hook, hook_impls, kwargs))

/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/_tracing.py:78:

hook = <_HookCaller 'pytest_runtest_setup'>

methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f5504b52b50>>]

kwargs = {'item': <IPyNbCell Cell 0>}
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    methods,
    kwargs,


  firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,


)

/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/manager.py:81:

hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f5504b52b50>>]

caller_kwargs = {'item': <IPyNbCell Cell 0>}, firstresult = False
def _multicall(hook_impls, caller_kwargs, firstresult=False):
    """Execute a call into multiple python functions/methods and return the
    result(s).

    ``caller_kwargs`` comes from _HookCaller.__call__().
    """
    __tracebackhide__ = True
    results = []
    excinfo = None
    try:  # run impl and wrapper setup functions in a loop
        teardowns = []
        try:
            for hook_impl in reversed(hook_impls):
                try:
                    args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                except KeyError:
                    for argname in hook_impl.argnames:
                        if argname not in caller_kwargs:
                            raise HookCallError(
                                "hook call must provide argument %r" % (argname,)
                            )

                if hook_impl.hookwrapper:
                    try:
                        gen = hook_impl.function(*args)
                        next(gen)  # first yield
                        teardowns.append(gen)
                    except StopIteration:
                        _raise_wrapfail(gen, "did not yield")
                else:
                    res = hook_impl.function(*args)
                    if res is not None:
                        results.append(res)
                        if firstresult:  # halt further impl calls
                            break
        except BaseException:
            excinfo = sys.exc_info()
    finally:
        if firstresult:  # first result hooks return a single value
            outcome = _Result(results[0] if results else None, excinfo)
        else:
            outcome = _Result(results, excinfo)

        # run all wrapper post-yield blocks
        for gen in reversed(teardowns):
            try:
                gen.send(outcome)
                _raise_wrapfail(gen, "has second yield")
            except StopIteration:
                pass


      return outcome.get_result()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/callers.py:208:

self = <pluggy.callers._Result object at 0x7f5504b5d210>
def get_result(self):
    """Get the result(s) for this hook call.

    If the hook was marked as a ``firstresult`` only a single value
    will be returned otherwise a list of results.
    """
    __tracebackhide__ = True
    if self._excinfo is None:
        return self._result
    else:
        ex = self._excinfo
        if _py3:


          raise ex[1].with_traceback(ex[2])


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/callers.py:80:

hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f5504b52b50>>]

caller_kwargs = {'item': <IPyNbCell Cell 0>}, firstresult = False
def _multicall(hook_impls, caller_kwargs, firstresult=False):
    """Execute a call into multiple python functions/methods and return the
    result(s).

    ``caller_kwargs`` comes from _HookCaller.__call__().
    """
    __tracebackhide__ = True
    results = []
    excinfo = None
    try:  # run impl and wrapper setup functions in a loop
        teardowns = []
        try:
            for hook_impl in reversed(hook_impls):
                try:
                    args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                except KeyError:
                    for argname in hook_impl.argnames:
                        if argname not in caller_kwargs:
                            raise HookCallError(
                                "hook call must provide argument %r" % (argname,)
                            )

                if hook_impl.hookwrapper:
                    try:
                        gen = hook_impl.function(*args)
                        next(gen)  # first yield
                        teardowns.append(gen)
                    except StopIteration:
                        _raise_wrapfail(gen, "did not yield")
                else:


                  res = hook_impl.function(*args)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/pluggy/callers.py:187:

item = <IPyNbCell Cell 0>
def pytest_runtest_setup(item):
    _update_current_test_var(item, "setup")


  item.session._setupstate.prepare(item)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/runner.py:123:

self = <_pytest.runner.SetupState object at 0x7f5504b64450>

colitem = <IPyNbCell Cell 0>
def prepare(self, colitem):
    """ setup objects along the collector chain to the test-method
        and teardown previously setup objects."""
    needed_collectors = colitem.listchain()
    self._teardown_towards(needed_collectors)

    # check if the last collection node has raised an error
    for col in self.stack:
        if hasattr(col, "_prepare_exc"):
            exc = col._prepare_exc
            raise exc
    for col in needed_collectors[len(self.stack) :]:
        self.stack.append(col)
        try:
            col.setup()
        except TEST_OUTCOME as e:
            col._prepare_exc = e


          raise e


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/runner.py:376:

self = <_pytest.runner.SetupState object at 0x7f5504b64450>

colitem = <IPyNbCell Cell 0>
def prepare(self, colitem):
    """ setup objects along the collector chain to the test-method
        and teardown previously setup objects."""
    needed_collectors = colitem.listchain()
    self._teardown_towards(needed_collectors)

    # check if the last collection node has raised an error
    for col in self.stack:
        if hasattr(col, "_prepare_exc"):
            exc = col._prepare_exc
            raise exc
    for col in needed_collectors[len(self.stack) :]:
        self.stack.append(col)
        try:


          col.setup()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/runner.py:373:

self = 
def setup(self):
    """
    Called by pytest to setup the collector cells in .
    Here we start a kernel and setup the sanitize patterns.
    """

    if self.parent.config.option.current_env:
        kernel_name = CURRENT_ENV_KERNEL_NAME
    else:
        kernel_name = self.nb.metadata.get(
            'kernelspec', {}).get('name', 'python')


  self.kernel = RunningKernel(kernel_name, str(self.fspath.dirname))


/home/sefkw/code/external/nbval/nbval/plugin.py:233:

self = <nbval.kernel.RunningKernel object at 0x7f5504b5d250>

kernel_name = ':nbval-parent-env'

cwd = '/tmp/pytest-of-sefkw/pytest-36/test_magics1'
def __init__(self, kernel_name, cwd=None):
    """
    Initialise a new kernel
    specify that matplotlib is inline and connect the stderr.
    Stores the active kernel process and its manager.
    """

    self.km, self.kc = start_new_kernel(
        kernel_name=kernel_name,
        stderr=open(os.devnull, 'w'),


      cwd=cwd,


    )

/home/sefkw/code/external/nbval/nbval/kernel.py:88:

startup_timeout = 60, kernel_name = ':nbval-parent-env'

kwargs = {'cwd': '/tmp/pytest-of-sefkw/pytest-36/test_magics1', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}

km = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>
def start_new_kernel(startup_timeout=60, kernel_name='python', **kwargs):
    """Start a new kernel, and return its Manager and Client"""
    logger.debug('Starting new kernel: "%s"' % kernel_name)
    km = KernelManager(kernel_name=kernel_name,
                       kernel_spec_manager=NbvalKernelspecManager())


  km.start_kernel(**kwargs)


/home/sefkw/code/external/nbval/nbval/kernel.py:53:

self = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

kw = {'cwd': '/tmp/pytest-of-sefkw/pytest-36/test_magics1', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}
def start_kernel(self, **kw):
    """Starts a kernel on this host in a separate process.

    If random ports (port=0) are being used, this method must be called
    before the channels are created.

    Parameters
    ----------
    `**kw` : optional
         keyword arguments that are passed down to build the kernel_cmd
         and launching the kernel (e.g. Popen kwargs).
    """


  kernel_cmd, kw = self.pre_start_kernel(**kw)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/jupyter_client/manager.py:301:

self = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

kw = {'cwd': '/tmp/pytest-of-sefkw/pytest-36/test_magics1', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}
def pre_start_kernel(self, **kw):
    """Prepares a kernel for startup in a separate process.

    If random ports (port=0) are being used, this method must be called
    before the channels are created.

    Parameters
    ----------
    `**kw` : optional
         keyword arguments that are passed down to build the kernel_cmd
         and launching the kernel (e.g. Popen kwargs).
    """
    if self.transport == 'tcp' and not is_local_ip(self.ip):
        raise RuntimeError("Can only launch a kernel on a local interface. "
                           "This one is not: %s."
                           "Make sure that the '*_address' attributes are "
                           "configured properly. "
                           "Currently valid addresses are: %s" % (self.ip, local_ips())
                           )

    # write connection file / get default ports


  self.write_connection_file()


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/jupyter_client/manager.py:248:

self = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>
def write_connection_file(self):
    """Write connection info to JSON dict in self.connection_file."""
    if self._connection_file_written and os.path.exists(self.connection_file):
        return

    self.connection_file, cfg = write_connection_file(self.connection_file,


      transport=self.transport, ip=self.ip, key=self.session.key,


        stdin_port=self.stdin_port, iopub_port=self.iopub_port,
        shell_port=self.shell_port, hb_port=self.hb_port,
        control_port=self.control_port,
        signature_scheme=self.session.signature_scheme,
        kernel_name=self.kernel_name
    )

/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/jupyter_client/connect.py:469:

self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

cls = <class 'jupyter_client.manager.KernelManager'>
def __get__(self, obj, cls=None):
    """Get the value of the trait by self.name for the instance.

    Default values are instantiated when :meth:`HasTraits.__new__`
    is called.  Thus by the time this method gets called either the
    default value or a user defined value (they called :meth:`__set__`)
    is in the :class:`HasTraits` instance.
    """
    if obj is None:
        return self
    else:


      return self.get(obj, cls)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:556:

self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

cls = <class 'jupyter_client.manager.KernelManager'>
def get(self, obj, cls=None):
    try:
        value = obj._trait_values[self.name]
    except KeyError:
        # Check for a dynamic initializer.
        dynamic_default = self._dynamic_default_callable(obj)
        if dynamic_default is None:
            raise TraitError("No default value found for %s trait of %r"
                             % (self.name, obj))


      value = self._validate(obj, dynamic_default())


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:535:

self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

value = <jupyter_client.session.Session object at 0x7f5507c33c10>
def _validate(self, obj, value):
    if value is None and self.allow_none:
        return value
    if hasattr(self, 'validate'):


      value = self.validate(obj, value)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:591:

self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

value = <jupyter_client.session.Session object at 0x7f5507c33c10>
def validate(self, obj, value):
    if isinstance(value, self.klass):
        return value
    else:


      self.error(obj, value)


/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:1677:

self = <traitlets.traitlets.Instance object at 0x7f5509cf8650>

obj = <jupyter_client.manager.KernelManager object at 0x7f5507dd5f10>

value = <jupyter_client.session.Session object at 0x7f5507c33c10>
def error(self, obj, value):
    kind = type(value)
    if six.PY2 and kind is InstanceType:
        msg = 'class %s' % value.__class__.__name__
    else:
        msg = '%s (i.e. %s)' % ( str( kind )[1:-1], repr( value ) )

    if obj is not None:
        e = "The '%s' trait of %s instance must be %s, but a value of %s was specified." \
            % (self.name, class_of(obj),
               self.info(), msg)
    else:
        e = "The '%s' trait must be %s, but a value of %r was specified." \
            % (self.name, self.info(), msg)


  raise TraitError(e)


E       traitlets.traitlets.TraitError: The 'session' trait of a KernelManager instance must be a Session, but a value of class 'jupyter_client.session.Session' (i.e. <jupyter_client.session.Session object at 0x7f5507c33c10>) was specified.
/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/traitlets/traitlets.py:1524: TraitError

================================= warnings summary =================================

/home/sefkw/code/external/nbval/nbval/plugin.py:115

/home/sefkw/code/external/nbval/nbval/plugin.py:115: PytestDeprecationWarning: direct construction of IPyNbFile has been deprecated, please use IPyNbFile.from_parent

return IPyNbFile(path, parent)
/home/sefkw/code/external/nbval/nbval/plugin.py:312

/home/sefkw/code/external/nbval/nbval/plugin.py:312: PytestDeprecationWarning: direct construction of IPyNbCell has been deprecated, please use IPyNbCell.from_parent

cell, options)
-- Docs: https://docs.pytest.org/en/latest/warnings.html

============================= short test summary info ==============================

ERROR test_magics.ipynb::Cell 0 - traitlets.traitlets.TraitError: The 'session' t...

=========================== 2 warnings, 1 error in 0.21s ===========================

================================= warnings summary =================================

tests/test_magics.py::test_magics[%dirs-expected_passes0]

tests/test_magics.py::test_magics[%this_magic_does_not_exist-expected_passes1]

/home/sefkw/mc3/envs/celltestsui/lib/python3.7/site-packages/_pytest/terminal.py:289: PytestDeprecationWarning: TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.

See https://docs.pytest.org/en/latest/deprecations.html#terminalreporter-writer for more information.

"TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.\n"
-- Docs: https://docs.pytest.org/en/latest/warnings.html

============================= short test summary info ==============================

FAILED tests/test_magics.py::test_magics[%this_magic_does_not_exist-expected_passes1]

===================== 1 failed, 1 passed, 2 warnings in 1.17s =====================

[ceball]

Hmm, re. problem 2 above: travis does not have the problem. So I guess it's related to my environment; I'll try a new environment and see what happens.

[ceball]

I had been using conda. I tried in a newly created python virtualenv (using my system python 3) - same problem :(

I suppose I should compare the packages here to the ones on travis:

(nbv) $ python --version
Python 3.8.2

(nbv) $ pip freeze
attrs==19.3.0
backcall==0.2.0
coverage==5.1
cycler==0.10.0
decorator==4.4.2
ipykernel==5.3.0
ipython==7.15.0
ipython-genutils==0.2.0
jedi==0.17.0
jsonschema==3.2.0
jupyter-client==6.1.3
jupyter-core==4.6.3
kiwisolver==1.2.0
matplotlib==3.2.1
more-itertools==8.3.0
mpmath==1.1.0
nbformat==5.0.7
-e git+git@github.com:ceball/nbval.git@ab7f9e8129848eabda3ce2633f5305ccfa81131c#egg=nbval
numpy==1.18.5
packaging==20.4
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
prompt-toolkit==3.0.5
ptyprocess==0.6.0
py==1.8.1
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytest-cov==2.9.0
python-dateutil==2.8.1
pyzmq==19.0.1
six==1.15.0
sympy==1.6
tornado==6.0.4
traitlets==4.3.3
wcwidth==0.2.4

[ceball]

It's not the dependencies. Locally I get

• Same correct result as on travis if I run py.test -v tests --nbval --current-env --sanitize-with tests/sanitize_defaults.cfg --ignore tests/ipynb-test-samples
• The weird traitlets.traitlets.TraitError: The 'session' trait of a KernelManager instance must be a Session, but a value of class 'jupyter_client.session.Session' (i.e. <jupyter_client.session.Session object at 0x7f353ffdea00>) was specified. if I append -k test_magics to the above command.

[ceball]

I could try to narrow down, but I don't suppose you want to take a guess @vidartf ?

[ceball]

Ok, the test results are now as I expect, i.e. they reflect the bug I'll fix in this PR: %this_magic_does_not_exist passes silently (which is wrong, hence causing the tests added to this PR to fail).

=================================== FAILURES ===================================
________________ test_magics[%this_magic_does_not_exist-False] _________________
testdir = <Testdir local('/tmp/pytest-of-travis/pytest-0/test_magics2')>
magic = '%this_magic_does_not_exist', expected_pass = False
    @pytest.mark.parametrize("magic, expected_pass", [
        (r"%dirs", True),
        (r"%precision bad", False),
        (r"%this_magic_does_not_exist", False)
    ])
    def test_magics(testdir, magic, expected_pass):
        nb = build_nb([
            # In [1]:
            magic,
        ])
        nb_name = 'test_magics'
        nbformat.write(nb, os.path.join(
            str(testdir.tmpdir), nb_name+".ipynb"))
    
        # using subprocess because otherwise second and subsequent tests always fail (some state left over somewhere in the jupyter stack)
        result = testdir.runpytest_subprocess('--nbval-lax', '--current-env', '-s', '-v')
    
>       assert result.ret == (not expected_pass)
E       assert <ExitCode.OK: 0> == True
E         +<ExitCode.OK: 0>
E         -True