unawareness runtime prefetch #572

billie60 · 2024-01-05T09:56:25Z

add prefetchlist store prefetchlist storage service.
Modify the optimizer to publish the access file list as a prefetchlist to the storage service when obtaining it.

imeoer · 2024-01-08T01:29:38Z

tools/prefetchlistStoreService/main.go

@@ -0,0 +1,96 @@
+/*


How about naming it tools/prefetch-distribution?

imeoer · 2024-01-08T01:31:20Z

tools/prefetchlistStoreService/main.go

+func main() {
+	serverCache = Cache{Items: make(map[string]*CacheItem)}
+
+	http.HandleFunc("/api/v1/prefetch/upload", uploadHandler)


Should only accept POST for upload and GET for download, we'd better use the echo web framework.

imeoer · 2024-01-08T01:36:57Z

cmd/optimizer-nri-plugin/main.go

@@ -104,6 +109,16 @@ func buildFlags(args *PluginArgs) []cli.Flag {
 			Usage:       "whether to overwrite the existed persistent files",
 			Destination: &args.Config.Overwrite,
 		},
+		&cli.StringFlag{


&cli.StringFlag{ Name: "prefetch-distribution-endpoint", Usage: "The service endpoint of prefetch distribution, for example: http://localhost:1323/api/v1/prefetch/upload", Destination: &args.Config.PrefetchDistributionEndpoint, },

Any updates?

imeoer · 2024-01-08T01:39:40Z

cmd/optimizer-nri-plugin/main.go

 func (p *plugin) StartContainer(_ *api.PodSandbox, container *api.Container) error {
 	dir, imageName, err := GetImageName(container.Annotations)
 	if err != nil {
 		return err
 	}
+	containerName := container.Annotations[containerNameLabel]


Does the annotation exist?

Yes, even without adding annotations in container.yaml, it can still use container.Annotations [containerNameLabel] to get container name

If you only need container name, container.Name is enough.

imeoer · 2024-01-08T01:41:00Z

cmd/optimizer-nri-plugin/main.go

+			log.WithError(err).Error("failed to send prefetch to http server")
+		}
+	}()
+
 	globalFanotifyServer[imageName] = fanotifyServer


We should protect the map using mutex.

Any updates?

imeoer · 2024-01-08T01:41:33Z

cmd/optimizer-nri-plugin/main.go

+
+	err = postRequest(item, url)
+	if err != nil {
+		return fmt.Errorf("error uploading to server: %w", err)


Use errors.Wrap.

imeoer · 2024-01-08T01:41:36Z

cmd/optimizer-nri-plugin/main.go

+func sendToServer(imageName, prefetchlistPath, containerName, serverURL string) error {
+	data, err := os.ReadFile(prefetchlistPath)
+	if err != nil {
+		return fmt.Errorf("error reading file: %w", err)


Use errors.Wrap.

imeoer · 2024-01-08T01:43:19Z

cmd/optimizer-nri-plugin/main.go

+		return fmt.Errorf("failed to read response body: %w", err)
+	}
+
+	fmt.Println("Server Response:", string(body))


Should check the HTTP status code and return an error if the status code is invalid.

sctb512 · 2024-01-12T02:08:13Z

cmd/optimizer-nri-plugin/main.go

 func (p *plugin) StartContainer(_ *api.PodSandbox, container *api.Container) error {
 	dir, imageName, err := GetImageName(container.Annotations)
 	if err != nil {
 		return err
 	}
+	containerName := container.Annotations[containerNameLabel]


If you only need container name, container.Name is enough.

sctb512 · 2024-01-12T02:14:34Z

cmd/optimizer-nri-plugin/main.go

+		}
+	}
+
+	item := CacheItem{


Can we define the generic API for different storage service? Sometimes, we may need to send more information to the server.

Also, does the API provide a query endpoint?

sctb512 · 2024-01-12T02:23:10Z

tools/prefetch-distribution/main.go

+		return c.String(http.StatusBadRequest, "Invalid request payload")
+	}
+
+	serverCache.Set(item)


Can we define an interface for difference storage backend?

sctb512 · 2024-01-12T02:29:16Z

cmd/optimizer-nri-plugin/main.go

I guess can we move the fanotify server to an separate component. Because for containerd 1.6.x, the NRI runs in a different construct. If we do this, the NRI for both containerd versions can be simplified to send a request to server.

codecov · 2024-01-22T14:21:07Z

Codecov Report

Attention: Patch coverage is 5.88235% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 34.66%. Comparing base (02b3776) to head (394a502).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #572      +/-   ##
==========================================
+ Coverage   34.64%   34.66%   +0.01%     
==========================================
  Files          65       65              
  Lines        6552     6552              
==========================================
+ Hits         2270     2271       +1     
+ Misses       3967     3966       -1     
  Partials      315      315

Files	Coverage Δ
pkg/daemon/daemon.go	`0.00% <ø> (ø)`
pkg/system/system.go	`6.46% <ø> (+0.30%)`	⬆️
config/global.go	`19.80% <33.33%> (+0.41%)`	⬆️
pkg/manager/daemon_adaptor.go	`0.00% <0.00%> (ø)`
pkg/manager/manager.go	`0.00% <0.00%> (ø)`
pkg/daemon/config.go	`0.00% <0.00%> (ø)`

imeoer · 2024-02-27T03:14:34Z

.github/workflows/optimizer.yml

+            sudo systemctl restart nydus-snapshotter
+          fi
+          sleep 5
+          NYDUS_VER=v$(curl --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' -s "https://api.github.com/repos/dragonflyoss/nydus/releases/latest" | jq -r .tag_name | sed 's/^v//')


The --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' can be removed.

imeoer · 2024-02-27T03:15:37Z

.github/workflows/optimizer.yml

+              echo "Error: --prefetch-files not found in nydus-snapshotter.log"
+              exit 1
+          fi
+          # output=$(sudo cat /etc/nydus/logs/nydus-snapshotter.log | grep "nydusd command")


Should be removed?

imeoer · 2024-02-27T03:16:42Z

cmd/optimizer-nri-plugin/main.go

@@ -104,6 +109,16 @@ func buildFlags(args *PluginArgs) []cli.Flag {
 			Usage:       "whether to overwrite the existed persistent files",
 			Destination: &args.Config.Overwrite,
 		},
+		&cli.StringFlag{


Any updates?

imeoer · 2024-02-27T03:17:16Z

cmd/optimizer-nri-plugin/main.go

+			log.WithError(err).Error("failed to send prefetch to http server")
+		}
+	}()
+
 	globalFanotifyServer[imageName] = fanotifyServer


Any updates?

imeoer · 2024-02-27T03:19:21Z

cmd/optimizer-nri-plugin/main.go

+	router *mux.Router
+}
+
+const endpointImageName string = "/api/v1/imagename"


/api/v1/prefetch?reference=$image

imeoer · 2024-02-27T03:20:13Z

cmd/optimizer-nri-plugin/main.go

+		}
+		defer resp.Body.Close()
+
+		responseBody, err := io.ReadAll(resp.Body)


Should check if resp.StatusCode != http.StatusOK { before this.

imeoer · 2024-02-27T03:20:41Z

go.mod

@@ -55,6 +56,12 @@ require (
 	k8s.io/utils v0.0.0-20230726121419-3b25d923346b
 )

+require (


Merge these into below.

imeoer · 2024-02-27T03:22:07Z

misc/optimizer/snapshotter-config.toml

@@ -0,0 +1,135 @@
+version = 1


Can we reuse misc/snapshotter/config.toml instead of creating a new one?

1. add prefetchlist store prefetchlist storage service. 2. Modify the optimizer to publish the access file list as a prefetchlist to the storage service when obtaining it. 3. modify http server add lru algo 4. use echo web framework 5. modify based on comments 6. get and post prefetchlist in optimizer

1. send post request to optimizer 2. store prefetchlist 3. add prefetchlist in nydusd

billie60 force-pushed the unawareness-prefetch branch 3 times, most recently from 059d595 to 2994ad5 Compare January 5, 2024 10:40

imeoer reviewed Jan 8, 2024

View reviewed changes

billie60 force-pushed the unawareness-prefetch branch 6 times, most recently from bed33a8 to 01d5e46 Compare January 11, 2024 04:53

sctb512 reviewed Jan 12, 2024

View reviewed changes

billie60 changed the title ~~anawareness runtime prefetch~~ unanawareness runtime prefetch Jan 22, 2024

billie60 force-pushed the unawareness-prefetch branch 4 times, most recently from d0582fa to b96cf48 Compare January 22, 2024 14:14

billie60 force-pushed the unawareness-prefetch branch 13 times, most recently from 4111e29 to dcd0f2f Compare January 29, 2024 12:46

billie60 force-pushed the unawareness-prefetch branch 7 times, most recently from bf97c9a to 1fd5320 Compare January 30, 2024 07:06

billie60 force-pushed the unawareness-prefetch branch 4 times, most recently from 546b4d2 to c2fa245 Compare February 6, 2024 07:58

imeoer changed the title ~~unanawareness runtime prefetch~~ unawareness runtime prefetch Feb 27, 2024

imeoer reviewed Feb 27, 2024

View reviewed changes

billie60 force-pushed the unawareness-prefetch branch 12 times, most recently from dde1a0c to ce605e9 Compare March 4, 2024 10:30

billie60 added 2 commits March 4, 2024 18:28

unawareness prefetch implementation on snapshotter side

34bd531

1. send post request to optimizer 2. store prefetchlist 3. add prefetchlist in nydusd

billie60 force-pushed the unawareness-prefetch branch from ce605e9 to 394a502 Compare March 11, 2024 08:47

modify optimizer test

394a502

imeoer mentioned this pull request Apr 24, 2024

Nydus 2024 Roadmap dragonflyoss/nydus#1575

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unawareness runtime prefetch #572

unawareness runtime prefetch #572

billie60 commented Jan 5, 2024

imeoer Jan 8, 2024

imeoer Jan 8, 2024

imeoer Jan 8, 2024

imeoer Feb 27, 2024

imeoer Jan 8, 2024

billie60 Jan 8, 2024

sctb512 Jan 12, 2024

imeoer Jan 8, 2024

imeoer Feb 27, 2024

imeoer Jan 8, 2024

imeoer Jan 8, 2024

imeoer Jan 8, 2024

sctb512 Jan 12, 2024

sctb512 Jan 12, 2024

sctb512 Jan 12, 2024

sctb512 Jan 12, 2024

sctb512 Jan 12, 2024

codecov bot commented Jan 22, 2024 •

edited

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

imeoer Feb 27, 2024

unawareness runtime prefetch #572

Are you sure you want to change the base?

unawareness runtime prefetch #572

Conversation

billie60 commented Jan 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 22, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 22, 2024 •

edited