fs: allow pre/background-fetch to be called before layer verification

[ktock]

Fixes #461

Currently, pre/background fetch cannot be called before layer verification so are not started simultaneously. This is because each of them needs to be invoked after the layer verification (*VerifiableReader.SkipVerify() or *VerifiableReader.VerifyTOC()) to make sure the fetched contents are expected. However, this behaviour slows down prefetch start in some cases.

This commit fixes this issue by allowing pre/background fetch to called before the layer verification. Any verification error before the layer verification is recorded and reported when the verification is performed.

The filesystem calls pre/background fetch immediately after the resolving completes. If the verification is required and an error is reported, it doesn't use that layer and discards it.

This starts the prefetch of the last layer in an image (ghcr.io/stargz-containers/drupal:8.7.6-esgz) 2x faster.

PR

PREFETCH sha256:bcd0cd035a844ea1c05b7bd6442e325bcb322b2175057e27c7725742717979fc 0s
PREFETCH sha256:41f172cfe3112ae7e06ee505d98cf5118946887384de73ebb332a505a828216f 1.733332ms
PREFETCH sha256:097aa006e05a3c66fa49de639f1169651a96e0f5bf62e96a00c482a20d1544b4 4.633834ms
PREFETCH sha256:781944a73b833c77c4897b69b30c9d9cf056688474273ea8bed56804d476fd6f 7.104947ms
PREFETCH sha256:17baeb2460dba726c73efb8979853252dc259e663cfc5b0f4b025ff4f9a1f4f0 9.628442ms
PREFETCH sha256:98a916e9956f38306ac5a72e17d222dbdd7af396dd44888e65d13666ecb4a075 13.996185ms
PREFETCH sha256:a5a2fa154196646f6746d3a94a1a3823d7e48379c0de77f0d19131cb9e92b7fe 14.313651ms
PREFETCH sha256:42ae14a59a279933d351ca5ccf60dc2002421ebcb9b5ce82d1fdf475f47bb937 23.871516ms
PREFETCH sha256:045b220d1bf3f49df96d650cd2deb4214603ad3bacac6e13677b4e7fc493f39b 31.140094ms
PREFETCH sha256:3f1b7a32a660c8b6a3111d4f919e4d1505e65b46bde1371ec24573a55b14aeaa 60.366947ms
PREFETCH sha256:1df86156aba8a40e20e18a63a62984164bbd7844e2135dda7c8bf61c6cec22e1 81.684658ms
PREFETCH sha256:e079b3bc111d3dd1f4dd94e82ce8774704888e1ce0836c3528258b33dbb2afa7 167.528988ms
PREFETCH sha256:258ba052917ec5d46baf307bad8d37aacda593e0ac03ae103f0cd1569368c5b7 291.824681ms
PREFETCH sha256:39802d797a07cd2e6dde521669242e16cd1b09efe45c2c6890d77a815950d398 576.582316ms
PREFETCH sha256:18edab1eab19448135e04ee73fdcfd7e25c9c8e68a19c0f3b1d6ce7274fe9e10 898.654479ms
PREFETCH sha256:8d59096a159c46e92c07ac3d29216eca4ec5abeb38eb3c060392ef5669118d89 902.781711ms

main

PREFETCH sha256:1df86156aba8a40e20e18a63a62984164bbd7844e2135dda7c8bf61c6cec22e1 0s
PREFETCH sha256:097aa006e05a3c66fa49de639f1169651a96e0f5bf62e96a00c482a20d1544b4 70.479252ms
PREFETCH sha256:e079b3bc111d3dd1f4dd94e82ce8774704888e1ce0836c3528258b33dbb2afa7 1.721875568s
PREFETCH sha256:39802d797a07cd2e6dde521669242e16cd1b09efe45c2c6890d77a815950d398 1.799494305s
PREFETCH sha256:8d59096a159c46e92c07ac3d29216eca4ec5abeb38eb3c060392ef5669118d89 1.814123932s
PREFETCH sha256:41f172cfe3112ae7e06ee505d98cf5118946887384de73ebb332a505a828216f 1.835857163s
PREFETCH sha256:bcd0cd035a844ea1c05b7bd6442e325bcb322b2175057e27c7725742717979fc 1.85022436s
PREFETCH sha256:a5a2fa154196646f6746d3a94a1a3823d7e48379c0de77f0d19131cb9e92b7fe 1.864604425s
PREFETCH sha256:17baeb2460dba726c73efb8979853252dc259e663cfc5b0f4b025ff4f9a1f4f0 1.880930087s
PREFETCH sha256:18edab1eab19448135e04ee73fdcfd7e25c9c8e68a19c0f3b1d6ce7274fe9e10 1.894974876s
PREFETCH sha256:98a916e9956f38306ac5a72e17d222dbdd7af396dd44888e65d13666ecb4a075 1.924277123s
PREFETCH sha256:3f1b7a32a660c8b6a3111d4f919e4d1505e65b46bde1371ec24573a55b14aeaa 1.939283183s
PREFETCH sha256:045b220d1bf3f49df96d650cd2deb4214603ad3bacac6e13677b4e7fc493f39b 1.953945586s
PREFETCH sha256:781944a73b833c77c4897b69b30c9d9cf056688474273ea8bed56804d476fd6f 1.971640613s
PREFETCH sha256:42ae14a59a279933d351ca5ccf60dc2002421ebcb9b5ce82d1fdf475f47bb937 1.990116989s
PREFETCH sha256:258ba052917ec5d46baf307bad8d37aacda593e0ac03ae103f0cd1569368c5b7 2.00428146s

[rdpsin]

Thank you very much! I tested out the patch and prefetch pull times reduced by upto 3x for some images. 😄

I'm curious about why there is still latency during prefetch and if we can do something to bring it down even further. Thanks again!

[ktock]

Thank you very much! I tested out the patch and prefetch pull times reduced by upto 3x for some images. smile

Thank you for testing this patch!

I'm curious about why there is still latency during prefetch and if we can do something to bring it down even further.

Prefetch starts after layer.Resolve(). Though layer.Resolve() of layers start simultaneously, the time to take for layer.Resolve() seems different among layers and this seems to cause the difference of start latency of prefetch.

[AkihiroSuda]

Can we have a new release with this soon?