fix cancellation error not being detected and erroneously cached #2926
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
tonistiigi
force-pushed
the
eof-cancel-fix
branch
from
e727469
to
6644f16
Compare
AkihiroSuda
reviewed
Jun 23, 2022
| // grpc does not set cancel correctly when stream gets cancelled and then Recv is called | ||
| if err != nil && ctx.Err() == context.Canceled { | ||
| // when this error comes from containerd it is not typed at all, just concatenated string | ||
| if strings.Contains(err.Error(), "EOF") { |
There was a problem hiding this comment.
Doesn’t seem robust.
Where is the source of this EOF
There was a problem hiding this comment.
At least one of the places should be https://github.com/containerd/containerd/blob/main/content/proxy/content_writer.go#L44 but there might be more.
Should we ever match an error that is not a real cancellation error here it isn't a big issue. We have already verified that our own context was canceled so at least it was the user's intent. Matching non-cancel error in here means that the error does not get cached as we don't consider it a real error and another parallel process requesting the same result would need to try to recompute the result again.
AkihiroSuda
approved these changes
Jul 21, 2022
charles-dyfis-net
added a commit
to charles-dyfis-net/oci-build-task
that referenced
this pull request
Sep 27, 2022
Motivating change is moby/buildkit#2926, whereby errors were not correctly propagated.
charles-dyfis-net
added a commit
to charles-dyfis-net/oci-build-task
that referenced
this pull request
Sep 27, 2022
Motivating change is moby/buildkit#2926, whereby errors were not correctly propagated. Signed-off-by: Charles Duffy <charles@dyfis.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should fix the
EOFerrors happening more frequently in the CI.The problem is that when context is canceled in the middle of a fetch stream(grpc stream pushing to containerd), in certain conditions it ends with
EOFerrors instead of cancel error what actually triggered the case. This is deep in containerd libraries so can't be fixed here so instead we need to have more exception cases on checking the errors.The errors are cached both in solver and in the image resolver so I combined them both to use same helper function.
Signed-off-by: Tonis Tiigi tonistiigi@gmail.com