A flexible way to specify prefetch files to pull Nydus images

[billie60]

Users defined prefetch files list, which can be set to --prefetch files when starting the nydus daemon.

Nydus has implemented the addition of the -- prefetch-patterns parameter to the nydusify convert command. With this parameter, we can specify a prefetch list when converting nydus format images and pull files based on this prefetch list when starting a container. However, due to different services based on one image need to access different files, this method is not flexible enough. Here is another idea, users can customize prefetch lists and pull according to the required prefetch files when pulling images.

Requirements

• NRI 2.0: Which has been integrated into containerd since v1.7.0-beta.1.

WorkFlow

• Create a pod.yaml file. Define the user-defined prefetch list file path in the labels or annotation field.

sudo tee pod.yaml <<- EOF
metadata:
  name: busybox-sandbox
  namespace: default
  attempt: 1
  uid: hdishd83djaidwnduwk28bcsb
log_directory: /tmp
labels:
  prefetch_list: /path/to/prefetchList.txt
linux: {}
EOF

• Pod creation
  During the pod creation period, complete the following process.

  1. Start an NRI plugin and subscribe to pod creation event. Note that NRI plugin can only be called from containerd/CRI. So creation a pod using crictl as below.

   crictl runp pod.yaml

  2. NRI plugin invokes the RunPodSandbox method to obtain image name and prefetch list file path when monitoring the pod creation command.
  3. NRI plugin sends image name + prefetch list file path as a massage to snapshotter through the specified socket API.
  4. After receiving the massage sent by NRI, the snapshotter will add a handler to parse the message and temporarily store the result as a global variable.

• Pull image
  When pulling an image, containerd will notify snapshot to start a new daemon. When starting nydusd, obtain the prefetch list file path from the global variable and add it to the -- prefetch files parameter.
  After completing the startup of nydusd, identify the corresponding image name when pulling the image, and then pull according to the prefetch list.

[imeoer]

That's a reasonable extension, what do you think? cc @changweige

[changweige]

SGTM. My only suggestion is to add the designated KV to annotations rather than POD label since the label is used for node selection and affinity.

[sctb512]

LGTM. I think there are three things we should consider.

1. How to implement a nydus snapshotter API to receive image+prepfetch list file path?
2. How to store the image+prepfetch list file path? In snapshotter database or memory cache?
3. May we also need API to UPDATE/REMOVE image+prepfetch list file path?

[sctb512]

Another question is how do we upload the prefetch list file to every machine in the cluster?