New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[test] add a line matcher object #12219
base: master
Are you sure you want to change the base?
Conversation
FYI: some stuff from opened PRs is included inside, so the diff will be smaller once they are merged. In addition, with the current implementation, it's easy for me to make some "shortcut" functions (the main logic is essentially done, except for the nice diff with regex patterns). |
So, after some investigation, it's quite hard to have a "nice" diff because of regexes. I'll probably make another PR for that one because it's much more complex than what I thought at first (I think I'll take some inspiration from I'll wait for some feedback before coming back to that PR. EDIT: I've found various typos in the documentation so I'll update it tomorrow. |
After considering #12216, I'll update this PR as well. |
f654399
to
5752a51
Compare
Sure! I just pushed some docs because there multiple typos and refactored a bit things. By the way, I also observed that we had a bug in autodoc thanks to that... so I fixed it (I've got 3 PRs that need to be merged before that just for the docs to be correctly rendered). |
Checking some of the design reasoning for the method signature:
|
I don't know which flavor to have by default... I would like to say "yay, it's maybe better to have a pure string flavor" because you usually want to have an exact match (I think most of our tests have exact matching and I think people don't want to escape possible meta-characters...). One possibility that I had in mind is just to have other methods:
But I'm not very happy with the
I think |
I think I'd probably prefer a developer experience of:
That is: binding the flavour to the method name. That's partly because it could help me to think about and write / read code, but also there are some second-order downstream static analysis/dependency benefits (tooling/analysts can infer to some degree that |
I am probably missing some context here -- can someone point me to a brief overview of what this PR is for? The body text of the description simply says here is the matcher! I'm aware Bénédikt has his thesis coming up soon, so no rush, just would be interested in the background given it's a 3000 line PR. A |
Oh yes, I think I never created an issue for that and just rushed to the implementation as if it were a hackathon. So, here's the context: we were doing some cleanups about colorization and co and I observed that many tests always have the same logic, namely:
and then, they match the lines one by one. All those lines could be replaced by a single function, let's say Technically speaking, the original issue could be solved by just adding two methods to I also think that it could be useful for matching autodoc outputs. I created a class for formatting expected RST output for enumeration test cases and I thought "how nice it would be to be able to do that but for other autodoc outputs as well" so I think the matcher object can be used in conjunction with those factory classes in a more user-friendly API. At least, tests would be easier to write (and probably shorter). Also, while the pytest output is fine, sometimes it's hard to detect exactly where the error happenned in the huge diff (the line matcher object would "highlight" the erroneous blocks; for now the logic is simple enough but I intended to do it using diff-like algorithms).
Yes, I'm currently writing it ! |
The argument is sound. I can make the flavor-agnostic method private and expose dispatcher methods. |
Here's the line matching feature.
TL;DR: Usage is
for the warnings for instance. It automatically removes the colors at the beginning and searches for a line matching the above string. The 'flavor' can maybe changed to 're' by default or have a
assert_re_match
method but that's how you would use the object in general.There are some options with their default values but I'm not sure which one should be the default. For now, the line matcher object for test application uses the default options but maybe I can make it with better defaults.
@chrisjsewell, I'd suggest you have a look at the tests (
tests/test_testing/test_matcher.py
) to see how it would be used. It's still WIP but I would like some feedback and what to expose (actually, with the current API you can more or less match whatever you want more or less however you want but it's better if you don't need to add options or if you have dedicated methods so as an extension's writer I'd be glad if you could give me some feedback on what would the functionalities that you want for sure).In the tests I used the most precise match information (namely the Line + offset) but you can ignore the offset and simply match against strings (see the tests for Line and Block objects).
I'm still wondering how to have a "nice" error context when the match fails... well, if you don't need the regex support you can just write something like
app.stderr().lines() == [...]
. Maybe I can also expose a simple interface for simple matching.