New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support fail_fast on check level #1143
Comments
Also happy to add it if there is a will to accept the issue. |
how is this different from |
The current fail_fast can only be specified at the whole "yaml" level. What I want to do is to specify it differently for different checks so that failure of only some checks is "fails fast". Example: You can specify fail_fast today only at the top level (which means that both :
In my scenario I would like the pre-commit fail fast only when
|
why though, isn't the current behavior good enough? |
Nope. The first check - "build" is special because other checks depend on it. It's really a prerequisite. If it fails, it can have some undesired side effects. In my case, it might be that the other checks attempt to start builds the image in all 8 parallel processes run by pre-commit on my 8 processor machine (and fail) which might take a long time (and precious resources). The other steps on the other hand, are not special and I am very happy if they do not fail fast. I have several checks there - for example we have pylint, mypy, flake8, and others. I want to see all of them failing and producing output rather than fail fast when first of them fail. In the current solution it is "all-or-nothing". I cannot specify different behaviour for some of the tasks only. Or maybe I am missing something and my case can be implemented currently ? |
presumably the other steps are under your control as well and they could short circuit if the image is missing? I don't think the proposal is unreasonable, I just want to make sure it's necessary before building something that needs to be supported indefinitely :) |
I understand. Short circuiting is what I try to do there -> I already protected in most cases against this, but it's really "workaround". The only way I found for short-circuitting it is to store results of the first step in specified folder and check that folder for failure. Variables are not passed between pre-commit checks, nor you cannot check status of previous checks. But this is clumsy and cumbersome way (and prone to errors). There are multiple problems with it - for example what happens if I still want to run just one pre-commit using previously built images and the "build" step just failed and left this "marker file" in ? The logic to manage it becomes quickly a problem. Simple "fail_fast" when build step fails is really simple comparing to that. |
does |
Nope. It is more complex. We mount our sources during build in order to avoid full rebuild (and the time it takes). So we only trigger rebuild of docker when our dependencies change. In our case (in order to save precious 10s of seconds it takes to run This check takes few ms but then we also implemented interactive check - if the build needs to be done, we ask the developer whether to run the build - answer 'yes' - (this might take up to few minutes) or whether to use already built image - answer 'no' - (this in many cases is enough) or whether to quit the build entirely ('quit'). The last option makes the check to fail. And when it fails I would like it to "fail_fast" (i.e. quit). So that we do not run any more checks. |
won't the image be absent in that case? I don't understand |
This is all for handling the case of continuously evolving project with big codebase and multiple dependencies (https://github.com/apache/airflow). And developer pulling in other's changes and rebasing the work on top of the other changes, Imaging there is already an image build previously (say few days ago). The case I am going to handle is to check if we need to run Running So we only want to attempt to make the build when one of the important files change (setup.py, Dockerfile, npm-packages.json etc.). If none of those change, we know that it is OK to do And we want the user to only check it once per Imagine this developer pulling and rebasing on top of a change where someone added dependency to setup.py. This dependency requires to rebuild the Docker image. But it's not needed by the developer because the developer works on different part of the codebase and pre-commit checks for those changes will run just fine on the previously build image. Then if we find that one of those files changed, the user might decide if they want to wait few minutes more and build the new image, or try to still use the old image hoping that the pre-commits on old image will run fine (which in most cases is perfectly OK and is super fast). I hope it explains the case. |
Presumably you already have something in place to know whether the image is up to date or not right? either through building a deterministic tag based on your inputs (and checking existence) or through other means it's very unclear to me why you can't use the same heuristic inside your dependent hooks to fail fast I completely understand that branches change dependencies, etc. but I really think there's a way to solve this very-very-very-very special use case without adding complexity to the framework |
Yeah. I actually have something in place and it works fine . But when I choose "quit" I would love to quit the whole pre-commit check immediately - and currently I cannot do it :(. Another - maybe simpler - use case might be as follows.
Currently you can either fail fast all of them, or none. But I would like to fail fast only when pycompile failes, not when mypy, pylint, flake8 fails |
I mean if you want to submit a PR go for it, I already changed the tags to I still don't think it has much of an advantage over just using |
I will go for PR indeed. There are some valid cases for --all-files to run locally. For example when you do a heavy refactor with multiple files involved, you might want to run --all-files. We do it quite often in Apache Airflow . Airflow is now heading towards 2.0 release - it includes a number of refactors that are being implemented by different people in parallel and backwards-incompatible changes that might have some undesirable side effects. And quite often when you work locally you want to do rebase on top of someone else's changes and use |
@potiuk any updates on this :D |
Not yet. I am currently at the ApacheCon Berlin conference (and talking here also praising pre-commit framework at my talk!). I had very little time to take a look at that - likely I will take a look when I am back from the conference (next week). |
no problem! good luck :D |
@potiuk did you have a chance to circle back on this? |
Not yet :(. We kind of can live without this feature for now and I am now deep into migrating whole Apache Airflow to Gihub Actions. I am finishing this up as we speak (we rolled it out yesterday and we have some fixes on the way) so I might come back to pre-commit improvements. I have a few things that I miss that we might want to use:
I haven't looked it up yet in existing issues (but I guess somebody already raised such issues) but those are the three most-needed things for us. WDYT ? |
there is an open issue for 3. already and I've thought a lot about 2. but haven't decided on ergonomics or opened an issue yet |
I promise I come back to it soon. We love pre-commit and as you might see we are a heavy-user https://github.com/apache/airflow/blob/master/.pre-commit-config.yaml: 51 checks and counting |
This feature would also be great so that if the Sometimes hooks that come after other hooks are not safe unless they pass some sort of safety mechanism first, but at the same time for anything else, I'd like it to run all fixers automatically (think end-of-file/mixed-line-ending/trailing-white-space) since doing a git add + git commit cycle for each individual fixer to run due to fail_fast is a tad annoying. |
tbf I'd consider it a bug that a formatter tries to touch a file that has syntax errors in it -- probably should report that to yamlfmt |
Hi @asottile |
yep feel free to tackle this |
Cool, just one quick check - is the idea to have the fail_fast within the configuration on "repo" level or for each individual hook on "hooks" level? |
it would be per-hook |
I would love to have fail_fast configuration option on check level not only at the level of the whole configuration file.
The use case I have in mind - i have one specific pre-commit check that performs docker builds of images that are used by later checks. This is fairly complex build and I have some optimisations in place to build it once and not re-run Docker build . again (because just running Docker build . later on will introduce some delays). I would like to mark that single first check only as "fail_fast". If it fails, none of the other checks should run. But if any other check fails, It should be perfectly OK to run all other tasks without failing fast.
The text was updated successfully, but these errors were encountered: