Tansfer only application's manifests to cmp-server for plugin-based applications

[jsolana]

Summary

Reduce the volume of files transferred to the cmp-server during GenerateManifests op to optimize the manifest generation process for plugin-based applications.

Motivation

In monorepos, excessive data transfer during GenerateManifests can create bottlenecks by sending more information than the plugin actually requires to generate the manifest for an application. This inefficiency significantly impacts performance, especially with repeated operations.

The full repository (excluding globs) is being sent in each gRPC request to the cmp-server. For each call to the cmp-server, the repo-server creates a tar.gz of the entire repo and sends it, then the cmp-server has to decompress it and create all the files on the disk, run the generate command, and delete it. Most of the time is spent waiting for command executions, creating and deleting directories and files.

Proposal

Transmit only the necessary information to the cmp-server.

As a quick win, we can use appPath instead of repoPath in generateManifestsCMP for plugin-based applications.

reposerver/repository/repository.go:

// generateManifestsCMP will send the appPath files to the cmp-server over a gRPC stream.
// The cmp-server will generate the manifests. Returns a response object with the generated
// manifests.
func generateManifestsCMP(ctx context.Context, appPath string, env []string, cmpClient pluginclient.ConfigManagementPluginServiceClient, tarDoneCh chan<- bool, tarExcludedGlobs []string) (*pluginclient.ManifestResponse, error) {
	generateManifestStream, err := cmpClient.GenerateManifest(ctx, grpc_retry.Disable())
	if err != nil {
		return nil, fmt.Errorf("error getting generateManifestStream: %w", err)
	}
	opts := []cmp.SenderOption{
		cmp.WithTarDoneChan(tarDoneCh),
	}

	// In this case, we using use appPath instead of repoPath to only send app resources
	err = cmp.SendRepoStream(generateManifestStream.Context(), appPath, appPath, generateManifestStream, env, tarExcludedGlobs, opts...)
	if err != nil {
		return nil, fmt.Errorf("error sending file to cmp-server: %s", err)
	}

	return generateManifestStream.CloseAndRecv()
}

The screenshot below represents time duration of GenerataeManifests op before and after apply this quick win:

Note: A more general solution could involve leveraging the value described in the annotation argocd.argoproj.io/manifest-generate-paths. This approach can be extended to other solutions as kustomize or helm (to discuss maybe in another issue).

[jsolana]

If it is ok, I can create the MR :)

[Oded-B]

I'd use this.
But even if argocd.argoproj.io/manifest-generate-paths is set, and we use that value to copy files we would want to have this behavior off by default as some people might "include/import" out-of-app-folder files during manifest rendering, even if they didn't configure ArgoCD to trigger sync on those file changes