Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: randomly failed e2e test and continuous prefetch warnings #555

Open
hangvane opened this issue Nov 3, 2023 · 1 comment
Open

CI: randomly failed e2e test and continuous prefetch warnings #555

hangvane opened this issue Nov 3, 2023 · 1 comment

Comments

@hangvane
Copy link
Contributor

hangvane commented Nov 3, 2023

https://github.com/containerd/nydus-snapshotter/actions/runs/6739821947/job/18321972302

time="2023-11-03T00:41:46Z" level=fatal msg="connection error: desc = \"transport: Error while dialing dial unix:///run/containerd-nydus/containerd-nydus-grpc.sock: timeout\": unavailable"
[2023-11-03 00:41:46.650093 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob e4d6efbd0a0d130541b68d7fe421474184657b37eff8fef1d6e3048454b0c671, offset 0, size 6424, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.650321 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob b194a89b4d14dfe137e08bc984b7491b079e169597fab03c6e92d1c7e76fbc97, offset 0, size 5697, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.650374 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 35abf43dce465d40f45a4445af0c570dcd4dc36faa6a225928aeb64c122c3a08, offset 10485760, size 146731, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.651170 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob a8f84a5d40140aa0586b41668a28f9579d0a6c9dc0fdc8abb09197cef00a2cef, offset 0, size 9415, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend

Seems that the prefetch warnings in e2e test exist for at least 2 months, and may randomly fail e2e test.

Additionally, the warning infomation is covered by Invalid argument (os error 22), making it hard to trace the error.

@imeoer
Copy link
Collaborator

imeoer commented Nov 6, 2023

It seems this error happened earlier:

time="2023-11-03T00:41:42.464732357Z" level=warning msg="snapshot garbage collection failed" error="error reading from server: EOF: unavailable" snapshotter=nydus

It looks like the nydusd HTTP API server is broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants