Saving bundles as an intermediate ocitar #612

ashpect · 2023-11-24T10:34:06Z

Implements this proposal.
Implemented except writing the tests, waiting for initial review, after which e2e tests can be written.

Signed-off-by: ashpect <ashishndiitr@gmail.com>

joaopapereira

Thanks for the PR, made some comments in the code.
On a more generic note:

We need to add some tests to this new feature
Can we use ggcr to read/write the tarballs for us, since this is a regular OCI tar we should be able to use something like https://github.com/google/go-containerregistry/tree/main/pkg/v1/tarball to do this work.
In terms of form, I think I would prefer us to provide variables like ociTarPath to the Push functions instead of providing them to the constructors since that has been the pattern we have been using. The idea here is that when you call the function you choose the Repository/location where you want to store the result. Another option that might make more sense here is instead of calling NewContents().Push maybe we should have a new function function called NewContents().PushToTar() since we do not need the majority of the code in Push because we are not building tags or writing them

joaopapereira · 2024-05-24T14:28:25Z

pkg/imgpkg/cmd/copy_repo_src.go

+			processedImages, err = c.tarImageSet.Import(tempDir, importRepo, c.registry, true)
+			if err != nil {
+				return nil, fmt.Errorf("Importing OCI tar: %s", err)
+			}


Since this is similar to the else case, maybe we can move this outside the if statement

The else case is calling the import function with false as a parameter while here its true which in turns calls the readoci tar different to read tar depending on the bool in the import function.

joaopapereira · 2024-05-24T14:29:44Z

pkg/imgpkg/cmd/copy_repo_src.go

+			if err != nil {
+				return nil, err
+			}
+			err = image.ExtractOciTarGz(c.OciFlags.OcitoReg, tempDir)


Let us make sure we clean up after ourselves

Suggested change

err = image.ExtractOciTarGz(c.OciFlags.OcitoReg, tempDir)

defer os.RemoveAll(tempDir)

err = image.ExtractOciTarGz(c.OciFlags.OcitoReg, tempDir)

joaopapereira · 2024-05-24T14:36:47Z

pkg/imgpkg/cmd/copy_repo_src.go

-			if processedImage.ImageIndex != nil {
-				continue
-			}
+		// This is added to not read the lockfile and change the ref for oci-flag. Will be removed once we add an inflate option to copy the refs.


Is this commented out because if we copy from oci-tar to the repository, we do not have all the images? If that is the case, we should change the comment to highlight something like:

// When copying images from an oci-tar to a repository, imgpkg will not try to access the origin repositories and will only copy the OCI Image to the registry, similar to the behavior we currently have on the `imgpkg push` command.

joaopapereira · 2024-05-24T14:37:58Z

pkg/imgpkg/cmd/copy_repo_src.go

+	err = os.RemoveAll(tempDir)
+	if err != nil {
+		fmt.Println("Error cleaning up temporary directory:", err)
+	}
+


Added a prior comment that I think it would be better to use than this part of the code. Because if anything fails in the meanwhile we will leave the folder back

Suggested change

err = os.RemoveAll(tempDir)

if err != nil {

fmt.Println("Error cleaning up temporary directory:", err)

}

joaopapereira · 2024-05-24T14:38:17Z

pkg/imgpkg/cmd/oci_flags.go

@@ -0,0 +1,23 @@
+// Copyright 2023 The Carvel Authors.


Suggested change

// Copyright 2023 The Carvel Authors.

// Copyright 2024 The Carvel Authors.

joaopapereira · 2024-05-24T14:39:25Z

pkg/imgpkg/image/tar_image.go

+
+// CreateOciTarFileAndDeleteFolder creates a oci tar file from the source folder and deletes the source folder. Used while pushing the oci-tar
+func CreateOciTarFileAndDeleteFolder(source, target string) error {
+


Suggested change

joaopapereira · 2024-05-24T14:43:04Z

pkg/imgpkg/image/tar_image.go

@@ -180,3 +181,119 @@ func (i *TarImage) isExcluded(relPath string) bool {
 	}
 	return false
 }
+
+// CreateOciTarFileAndDeleteFolder creates a oci tar file from the source folder and deletes the source folder. Used while pushing the oci-tar


Why are we deleting the source folder?
If I do imgpkg push -b repo/app1-config -f config/ --to-oci-tar /path/to/file.tar is my config folder going to be deleted? We should not do that

joaopapereira · 2024-05-24T14:43:24Z

pkg/imgpkg/image/tar_image.go

+
+// ExtractOciTarGz extracts the oci tar file to the extractDir
+func ExtractOciTarGz(inputDir, extractDir string) error {
+


Suggested change

joaopapereira · 2024-05-24T14:46:26Z

pkg/imgpkg/imagedesc/image_intermediate.go

@@ -0,0 +1,156 @@
+// Copyright 2023 The Carvel Authors.


Suggested change

// Copyright 2023 The Carvel Authors.

// Copyright 2024 The Carvel Authors.

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect temporarily deployed to TanzuNet Registry Dev e2e November 24, 2023 10:34 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e November 24, 2023 10:34 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e November 28, 2023 23:29 — with GitHub Actions Inactive

ashpect temporarily deployed to TanzuNet Registry Dev e2e November 28, 2023 23:29 — with GitHub Actions Inactive

ashpect had a problem deploying to GCR e2e November 29, 2023 02:17 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e November 29, 2023 02:17 — with GitHub Actions Failure

ashpect had a problem deploying to GCR e2e December 8, 2023 03:42 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e December 8, 2023 03:42 — with GitHub Actions Failure

ashpect force-pushed the copy branch from deadd87 to 5b3d191 Compare March 3, 2024 15:19

ashpect had a problem deploying to GCR e2e March 3, 2024 15:19 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e March 3, 2024 15:19 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e March 3, 2024 15:23 — with GitHub Actions Failure

ashpect had a problem deploying to GCR e2e March 3, 2024 15:23 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e March 3, 2024 15:23 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e March 20, 2024 12:25 — with GitHub Actions Failure

ashpect had a problem deploying to GCR e2e March 20, 2024 12:25 — with GitHub Actions Failure

ashpect requested a deployment to TanzuNet Registry Dev e2e May 22, 2024 07:08 — with GitHub Actions Waiting

ashpect requested a deployment to GCR e2e May 22, 2024 07:08 — with GitHub Actions Waiting

ashpect force-pushed the copy branch from 8909900 to 5df051f Compare May 22, 2024 07:09

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 22, 2024 07:09 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e May 22, 2024 07:09 — with GitHub Actions Inactive

ashpect requested a deployment to TanzuNet Registry Dev e2e May 22, 2024 07:50 — with GitHub Actions Waiting

ashpect requested a deployment to GCR e2e May 22, 2024 07:50 — with GitHub Actions Waiting

ashpect had a problem deploying to GCR e2e May 22, 2024 07:54 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e May 22, 2024 07:54 — with GitHub Actions Failure

ashpect added 4 commits May 22, 2024 14:57

Saving images to disk and adding flag --to-oci-tar

1b452bd

Signed-off-by: ashpect <ashishndiitr@gmail.com>

WIP

a0ee159

Signed-off-by: ashpect <ashishndiitr@gmail.com>

wip

5e81147

Signed-off-by: ashpect <ashishndiitr@gmail.com>

updates

7bf3e60

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect had a problem deploying to GCR e2e May 22, 2024 09:42 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e May 22, 2024 09:42 — with GitHub Actions Failure

update headers

77d8930

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect force-pushed the copy branch from f446a42 to 77d8930 Compare May 22, 2024 09:45

ashpect had a problem deploying to GCR e2e May 22, 2024 09:45 — with GitHub Actions Failure

ashpect had a problem deploying to TanzuNet Registry Dev e2e May 22, 2024 09:45 — with GitHub Actions Failure

ashpect marked this pull request as ready for review May 22, 2024 10:47

ashpect temporarily deployed to GCR e2e May 23, 2024 07:43 — with GitHub Actions Inactive

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 23, 2024 07:43 — with GitHub Actions Inactive

fixing error bug importing tar

187106d

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect force-pushed the copy branch from 615cb12 to 187106d Compare May 23, 2024 07:47

ashpect temporarily deployed to GCR e2e May 23, 2024 07:47 — with GitHub Actions Inactive

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 23, 2024 07:47 — with GitHub Actions Inactive

ashpect requested a deployment to TanzuNet Registry Dev e2e May 23, 2024 16:45 — with GitHub Actions Waiting

ashpect requested a deployment to GCR e2e May 23, 2024 16:45 — with GitHub Actions Waiting

ashpect temporarily deployed to GCR e2e May 23, 2024 16:45 — with GitHub Actions Inactive

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 23, 2024 16:45 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e May 23, 2024 17:04 — with GitHub Actions Inactive

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 23, 2024 17:04 — with GitHub Actions Inactive

ashpect added 2 commits May 23, 2024 22:35

add support for index

387e4a6

Signed-off-by: ashpect <ashishndiitr@gmail.com>

update vendor

733a2a0

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect force-pushed the copy branch from 08c9137 to 733a2a0 Compare May 23, 2024 17:06

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 23, 2024 17:06 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e May 23, 2024 17:06 — with GitHub Actions Inactive

renuy requested review from praveenrewar and joaopapereira May 24, 2024 06:02

joaopapereira requested changes May 24, 2024

View reviewed changes

resolving reviews

b1defd2

Signed-off-by: ashpect <ashishndiitr@gmail.com>

ashpect temporarily deployed to TanzuNet Registry Dev e2e May 25, 2024 11:07 — with GitHub Actions Inactive

ashpect temporarily deployed to GCR e2e May 25, 2024 11:07 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Saving bundles as an intermediate ocitar #612

Saving bundles as an intermediate ocitar #612

ashpect commented Nov 24, 2023 •

edited

joaopapereira left a comment

joaopapereira May 24, 2024

ashpect May 25, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

joaopapereira May 24, 2024

	err = image.ExtractOciTarGz(c.OciFlags.OcitoReg, tempDir)
	defer os.RemoveAll(tempDir)
	err = image.ExtractOciTarGz(c.OciFlags.OcitoReg, tempDir)

	// Copyright 2023 The Carvel Authors.
	// Copyright 2024 The Carvel Authors.


		// CreateOciTarFileAndDeleteFolder creates a oci tar file from the source folder and deletes the source folder. Used while pushing the oci-tar
		func CreateOciTarFileAndDeleteFolder(source, target string) error {


		// ExtractOciTarGz extracts the oci tar file to the extractDir
		func ExtractOciTarGz(inputDir, extractDir string) error {

Saving bundles as an intermediate ocitar #612

Are you sure you want to change the base?

Saving bundles as an intermediate ocitar #612

Conversation

ashpect commented Nov 24, 2023 • edited

joaopapereira left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashpect commented Nov 24, 2023 •

edited