raise NotImplementedError("unreachable") when a falsifying case is found
#15
Comments
|
Ah, this is likely due to https://docs.python.org/3/library/traceback.html#traceback.format_exception changing signatures:
|
pradyunsg
changed the title
raise NotImplementedError("unreachable") error when a falsifying case is foundraise NotImplementedError("unreachable") when a falsifying case is found
pradyunsg
changed the title
raise NotImplementedError("unreachable") when a falsifying case is foundraise NotImplementedError("unreachable") when a falsifying case is found
|
A different error happens on 3.9. ❯ hypothesis fuzz -- tests/test_requirements_tokeniser.py
using up to 1 processes to fuzz:
tests/test_requirements_tokeniser.py::test_names
Now serving dashboard at http://localhost:9999/
* Serving Flask app 'hypofuzz.dashboard'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://localhost:9999
Press CTRL+C to quit
127.0.0.1 - - [03/Dec/2022 12:30:11] "POST / HTTP/1.1" 200 -
found failing example for tests/test_requirements_tokeniser.py::test_names
Process Process-2:
Traceback (most recent call last):
File "/Users/pradyunsg/.asdf/installs/python/3.9.8/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Users/pradyunsg/.asdf/installs/python/3.9.8/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/hypofuzz/interface.py", line 91, in _fuzz_several
fuzz_several(*tests)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/hypofuzz/hy.py", line 389, in fuzz_several
raise Exception("Found failures for all tests!")
Exception: Found failures for all tests!
Traceback (most recent call last):
File "/Users/pradyunsg/Developer/github/packaging/.venv39/bin/hypothesis", line 8, in <module>
sys.exit(main())
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/pradyunsg/Developer/github/packaging/.venv39/lib/python3.9/site-packages/hypofuzz/entrypoint.py", line 93, in fuzz
raise NotImplementedError("unreachable")
NotImplementedError: unreachable |
|
🙏 Thanks so much for the report! I've been flat-out with EOY work projects and travelling home to see family over the break, but I'm aiming to fix this in the next few weeks. Unfortunately HypoFuzz is more of an MVP / "spike" than a production-ready system, and then I joined @anthropics instead of having 6-12 months of PhD to harden it 😅... and Hypothesis itself soaks up most of the time I carve out for OSS. Hypofuzz is suffering somewhat from having too many big improvements on the table combined with few enough users that I'm more motivated to work on quicker wins elsewhere. |
|
Thanks @Zac-HD! I'm looking forward to the fixes but please don't feel any sort of push for urgency on this from my end. :) |
|
First, @Zac-HD , this looks amazing and plan on taking full advantage of this tool. I just ran into this same issue (ignoring the 3.10 error), but after inspecting
I may look into customizing |
|
Kudos @jgbos, that's it! The solution will be to print "all tests fail" and then sys.exit(1), so pretty similar tbh 😁
It's going to take a slightly deeper refactor I think, including a pile of new dashboard code (😬) - if you just restart for that test it'll promptly pick the same failing example up from the database. But more generally I'd strongly advise against continuing to run on failure for more than the few seconds we do in Hypothesis itself (in case e.g. the minimal and every non-minimal example fail with different errors). It's a common request, but IMO kinda misguided because either the code or the test will have to change, and the pressure to keep a 'clean' build is actually an important benefit from the fuzzer. |
|
@Zac-HD thanks for the feedback, glad I understood the error. As for continuing to run, I realized the best approach is to create a new target with the same test function and strategy with a new random seed. I'm currently customizing to support testing black box systems and trying to understand the regions of success or failures. |
For that I'd suggest a test that never fails, but just logs and then discards whatever exception it raises 🙂 |
|
Thanks for the fix @Zac-HD! ^.^ |
|
FWIW, I ended up not using hypothesis and ended up with heavily-parameterised tests instead in pypa/packaging#624 :) |
|
We'll get you someday, I'm sure 😁 (e.g. when you need to test non-ascii strings...) |
👋🏽
I was trying to use hypothesis + hypofuzz to fuzz a hand-written parser in https://github.com/pypa/packaging/, after we got a report of a parser regression in pypa/packaging#618. In my first attempt to do so, I seem to have successfully hit a
NotImplementedError, details below. :)Steps to reproduce
pip install hypofuzzpip install git+https://github.com/pypa/packaging.git@606c71acce93d04e778cdbdea16231b36d6b870f(get the current main)hypothesis fuzz -- {the-test-file-above}.Output
The text was updated successfully, but these errors were encountered: