Lower layer prefetches take longer to start than previous layer prefetch

[rdpsin]

Hello,

I was doing some profiling and found that lower layers prefetches take more time to start than the previous layer prefetch. I would like to know if this the expected behavior and why? (I thought all prefetches are started simultaneously)

By prefetch start, I mean the time layer.Prefetch is invoked (calculated using time.Now().UnixNano()):

stargz-snapshotter/fs/layer/layer.go

Line 439 in 25d16aa

func (l *layer) Prefetch(prefetchSize int64) error {

The image I tested was drupal_latest: https://hub.docker.com/_/drupal, although this behavior was exhibited for every image I tested. (Image was optimized to eStargz with gzip compression level 9)

stargz-snapshotter commit id: 94f9987
/etc/containerd-stargz-grpc/config.toml:

max_concurrency = 10
[blob]
chunk_size = 524288

[directory_cache]
direct = true

Prefetch starts relative to first prefetch

Layer 0: 0.0ms
Layer 1: 148.14306ms
Layer 2: 182.61523ms
Layer 3: 673.055408ms
Layer 4: 693.450489ms
Layer 5: 745.22816ms
Layer 6: 766.971316ms
Layer 7: 790.142632ms
Layer 8: 812.166566ms
Layer 9: 827.747773ms
Layer 10: 872.461101ms
Layer 11: 896.243876ms
Layer 12: 923.447124ms
Layer 13: 1122.284268ms
Layer 14: 1175.396076ms
Layer 15: 1231.160189ms
Layer 16: 1323.520349ms
Layer 17: 1496.857682ms

The total time to prefetch all layers was 1.49686s

Would love some clarity on this. Thank you.

[ktock]

This is because the prefetch of each layer is called after verification which is not performed simultaneously.
I've created a patch #467 to fix this issue.