Merge Porch into main branch #2815

martinmaly · 2022-02-18T20:11:00Z

Simple package orchestration (Simple package orchestration #2663)
oci-support feature branch (oci-support feature branch #2666)
Remove Placeholder kptlib (Remove Placeholder kptlib #2665)
Fix Porch Build (Fix Porch Build #2672)
Correct Apache License Text (Merge main branch into porch #2675)
Implement Direct Package Fetch (Implement Direct Package Fetch #2667)
Initial fn Interface (Initial fn Interface #2676)
Upgrade go-containerregistry to v0.8.0 (Upgrade go-containerregistry to v0.8.0 #2674)
git: handle resolution of per-package versions (git: handle resolution of per-package versions #2681)
PackageDraft interface should not inherit from Package (PackageDraft interface should not inherit from Package #2678)
Minor fixups to better surface errors (Minor fixups to better surface errors #2680)
Clean up Makefile (Clean up Makefile #2685)
Render and Eval via Simple Memory FS (Render and Eval via Simple Memory FS #2684)
Working towards applying packages from a repository (porch: Working towards applying packages from a repository #2686)
Add priority and fairness to RBAC roles (Add priority and fairness to RBAC roles #2692)
Applying: publish MVP status (Applying: publish MVP status #2695)
Fix Docker Build (Fix Docker Build #2696)
Tidy Go Modules (Tidy Go Modules #2699)
Add Roundtrip Tests (Add Roundtrip Tests #2703)
Expose Renderer api with FileSystem abstraction (Expose Renderer api with FileSystem abstraction #2683)
Create fix-all and verify-fix-all (Create fix-all and verify-fix-all #2702)
Fix few file headers (Fix few file headers #2707)
Define a FunctionEvaluator gRPC Interface (Define a FunctionEvaluator gRPC Interface #2700)
Integrate porch with renderer interface (Integrate porch with renderer interface #2710)
Introduce applyset abstraction (Introduce applyset abstraction #2697)
Git repository test (Git repository test #2706)
Create Kptfile in config/deploy (Create Kptfile in config/deploy #2708)
Update FunctionRunner Interface (Update FunctionRunner Interface #2714)
Simple gRPC kpt function runner service (Simple gRPC kpt function runner service #2711)
Use simple evaluator in tests (Use simple evaluator in tests #2717)
Fix os calls (Fix os calls #2718)
git: create ApprovePackageRevision method, remove draft hack (git: create ApprovePackageRevision method, remove draft hack #2712)
Kpt location parsing (Kpt location parsing #2688)
Use apply-setters directly (Use apply-setters directly #2719)
Upgrade kpt dependencies (Upgrade kpt dependencies #2727)
Use Lower-Level Function Runner Interface (Use Lower-Level Function Runner Interface #2730)
Use kyaml In-Memory Filesystem (Use kyaml In-Memory Filesystem #2731)
Upgrade porch dependencies (Upgrade porch dependencies #2726)
Build Function Runner Docker Image (Build Function Runner Docker Image #2729)
tests: Fix test flake from timestamp sanitization (tests: Fix test flake from timestamp sanitization #2713) (tests: Fix test flake from timestamp sanitization (#2713) #2742)
Create function-runner Deployment (Create function-runner Deployment #2733)
Use kpt Renderer in Porch (Use kpt Renderer in Porch #2735)
RemoteRootSyncSet: use specified OCI image (RemoteRootSyncSet: use specified OCI image #2722)
RemoteRootSyncSet: apply changes with force (RemoteRootSyncSet: apply changes with force #2723)
RemoteRootSyncSet: Remove hack where we set namespace (RemoteRootSyncSet: Remove hack where we set namespace #2724)
Create push-images target to push all our images (Create push-images target to push all our images #2721)
Create dockerfile / makefile / manifest for pushing controllers to cluster (Create dockerfile / makefile / manifest for pushing controllers to cluster #2694)
Introduce mapTaskToMutation function (Introduce mapTaskToMutation function #2744)
Fix kpt file system interactions (Fix kpt file system interactions #2746)
Pass Function Config to Eval (Pass Function Config to Eval #2747)
Connect to gRPC function runner (Connect to gRPC function runner #2745)
Fix crash (Fix crash #2748)
Fix clone directory rewriting (Fix clone directory rewriting #2749)
client generation: fix plural of packagerevisionresources (client generation: fix plural of packagerevisionresources #2751)
Update kpt dependencies (Update kpt dependencies #2743)
Streamline Docker Image Building (Streamline Docker Image Building #2752)
Run Porch With Consistent Working Dir (Run Porch With Consistent Working Dir #2753)
Add subpkgs test for porch (Add subpkgs test for porch #2754)
Basic e2e Test (Basic e2e Test #2755)
Refactor Server Startup (Refactor Server Startup #2770)
Tidy in 1.17 compatible mode (Tidy in 1.17 compatible mode #2769)
Create Engine using Options (Create Engine Using Options #2771)
Support lazy credential resolution (Support lazy credential resolution #2772)
Pass Context to OpenRepository (Pass Context to OpenRepository #2773)
Use Context to Drive Server Shutdown (Use Context to Drive Server Shutdown #2774)
Clean up Required/Optional API Fields (Clean up Required/Optional API Fields #2778)
Easier deployment onto GKE (Easier deployment onto GKE #2776)
Support Creating Porch Deployment Config (Kustomize Porch Deployment Config #2777)
Update Porch Deployment and Instructions (Update Porch Deployment and Instructions #2782)
Use controller-gen v0.8.0 (Use controller-gen v0.8.0 #2780)
Set renderer when building CaDEngine (Set renderer when building CaDEngine #2787)
Fix missing error handling (Fix missing error handling #2784)
Fix small typo (Fix small typo #2793)
Don't call into kpt render if we don't have a package (Don't call into kpt render if we don't have a package #2788)
Split git_test To Use Git Server (Split git_test To Use Git Server #2797)
Add CreatePackageRevision Test (Add CreatePackageRevision Test #2800)
Cache apply-setters:v0.1 function (Cache apply-setters:v0.1 function #2799)
Return Git PackageRevisionResources Correctly (Return Git PackageRevisionResources Correctly #2801)
Make Git Server hostable in cluster (Make Git Server hostable in cluster #2798)
Add PackageRevision Approval API (Add PackageRevision Approval API #2810)
Add Clone Package Test (Add Clone Package Test #2807)
Revert OCI and Location (Revert OCI and Location #2814)

Rebased on main and squashed Co-authored-by: Louis DeJardin <dejardin@google.com>

* Fix Makefile Formatting * Remove dependency on kptlib * Tidy modules * Delete kptlib * Remove kptlib replace Clause

* Fix Porch Build Location of ClonerUsingGitExec changed in the OCI branch. * Add Porch Workflow

LICENSE file is supposed to be an exact copy of https://www.apache.org/licenses/LICENSE-2.0.txt.

* Update VSCode Workspace * Add launch.json * Move Clone Logic Into Own File * Implement Git pacakge fetch

Initial version of the function interfaces.

* Tidy Porch Modules * Ignore Modules In .build Directory

We can have a git tag that looks like `packageDir/version`, this allows for independent versioning of packages that share a git repo. We need to try both ref forms when resolving git packages.

In particular, some of the methods such as GetResources are ambiguous until we finish the draft (do we want the in-progress resources or the upstream resources?)

In general, we should always return the error, unless we actually want to mask it e.g. for security reasons (and in that case, we would normally expect to log the underlying error instead of wrapping).

* Remove init and deinit (no longer useful) * Fix module discovery * Clean up directory references

* Render and Eval via Simple Memory FS For now the filesystem is inserted at lower level; In future changes the mutation Apply will accept the filesystem abstraction to avoid back-and-forth between filesystem and PackageResources. * Clean up Function Evaluation to use new interface * Remove unnecessary kpt module

First steps (with lots of hard-coding and hacks) towards applying from a repository to a cluster.

aggregated-apiservers seem to query these resources, and spam the logs (at least) if they don't have permission.

We start to expose the status of the sync operation.

Remove kpt directory from Docker build.

* Fix Go Modules * Run Tidy in Porch Workflow

* Expose Renderer api with FileSystem abstraction * Merge Fs methods in pkg.go * Update name * Merge ReadKptfile and address changes * Update porch go.sum * Update results write * Suggested changes

* Create fix-all and verify-fix-all We run all the formatters / header makers etc. In CI, we run them and make sure that the git checkout is still clean. Ideally we would also run generate, but it is very slow, and it also requires GOPATH to be set. We can do that in CI later. * Add missing headers

* Add install-protoc.sh script * Define a FunctionEvaluator interface * Create an empty server implementation

* Integrate porch with renderer interview * Suggested changes * Add implementation

Replacing our very primitive apply logic with something better using server-side apply.

We create a simple git server, and we try round-tripping a package. Some hacks/todos uncovered that we need to tackle.

Our package logic requires a Kptfile, so adding one (for now).

And make names more consistent.

Several functions are built into the server Docker image and are exec'd with input. Added simple client which can be used (in ./func directory): ``` go run ./client \ --address=192.168.8.202:9445 \ --package=./config/ \ --image=gcr.io/kpt-fn/set-namespace:v0.2.0 \ -- \ namespace=foo ```

Some blueprints we encounter in tests use the older version of the function.

Newly created Git package returned package resources at its old commit SHA where no package contents existed, thus returning empty resources. As package changes, advance the reference to return up-to-date package revision contents.

* Make Git Server hostable in cluster * Add makefile targets to build and push its images

* Add PackageRevision .../approval subresource * Generate client

* Add Clone Package Test * Fix issues with git server's reporting of symbolic references

Preparing for merge of porch branch back into main, * Revert "Kpt location parsing (kptdev#2688)" * Revert "oci-support feature branch (kptdev#2666)"

* Tidy go modules * Fix year in license header * Fixed few more issues that crept in during merges from main and preparing for merge back to main.

droot

I reviewed the non-porch portion and seems good to me. Minor clarification on pre-submits. Might ping you offline.

droot · 2022-02-22T18:09:53Z

.github/workflows/porch.yml

+      - "docs/**"
+      - "site/**"
+
+jobs:


Two things that will help us to analyze if we need to adjust the presubmit workflow (locally and in CI)

How long does the porch specific tests take ?

What dependencies do a kpt contributor needs to install to run tests for porch

Porch tests are about 5 mins (https://github.com/GoogleContainerTools/kpt/actions/runs/1867953046)

no dependencies needed.

The e2e tests will have mode where they run against a k8s cluster where they'd install Porch but that part is not yet implemented.

go.mod

droot · 2022-02-22T21:10:43Z

.github/workflows/porch.yml

+      - "docs/**"
+      - "site/**"
+
+jobs:


go.mod

martinmaly and others added 30 commits January 24, 2022 16:59

Simple package orchestration (kptdev#2663)

f00cc03

oci-support feature branch (kptdev#2666)

9f7687f

Rebased on main and squashed Co-authored-by: Louis DeJardin <dejardin@google.com>

Remove Placeholder kptlib (kptdev#2665)

b5de73f

* Fix Makefile Formatting * Remove dependency on kptlib * Tidy modules * Delete kptlib * Remove kptlib replace Clause

Fix Porch Build (kptdev#2672)

2a3d1da

* Fix Porch Build Location of ClonerUsingGitExec changed in the OCI branch. * Add Porch Workflow

Correct Apache License Text (kptdev#2675)

a747137

LICENSE file is supposed to be an exact copy of https://www.apache.org/licenses/LICENSE-2.0.txt.

Implement Direct Package Fetch (kptdev#2667)

1512349

* Update VSCode Workspace * Add launch.json * Move Clone Logic Into Own File * Implement Git pacakge fetch

Initial fn Interface (kptdev#2676)

66adf78

Initial version of the function interfaces.

Upgrade go-containerregistry to v0.8.0 (kptdev#2674)

81260a4

* Tidy Porch Modules * Ignore Modules In .build Directory

git: handle resolution of per-package versions (kptdev#2681)

1d5d77a

We can have a git tag that looks like `packageDir/version`, this allows for independent versioning of packages that share a git repo. We need to try both ref forms when resolving git packages.

PackageDraft interface should not inherit from Package (kptdev#2678)

b10084c

In particular, some of the methods such as GetResources are ambiguous until we finish the draft (do we want the in-progress resources or the upstream resources?)

Minor fixups to better surface errors (kptdev#2680)

3878005

In general, we should always return the error, unless we actually want to mask it e.g. for security reasons (and in that case, we would normally expect to log the underlying error instead of wrapping).

Clean up Makefile (kptdev#2685)

437bd07

* Remove init and deinit (no longer useful) * Fix module discovery * Clean up directory references

Working towards applying packages from a repository (kptdev#2686)

d420794

First steps (with lots of hard-coding and hacks) towards applying from a repository to a cluster.

Add priority and fairness to RBAC roles (kptdev#2692)

4da503c

aggregated-apiservers seem to query these resources, and spam the logs (at least) if they don't have permission.

Applying: publish MVP status (kptdev#2695)

f1e7c9f

We start to expose the status of the sync operation.

Fix Docker Build (kptdev#2696)

f8cb5f7

Remove kpt directory from Docker build.

Tidy Go Modules (kptdev#2699)

7f4fd85

* Fix Go Modules * Run Tidy in Porch Workflow

Add Roundtrip Tests (kptdev#2703)

7672d2f

Expose Renderer api with FileSystem abstraction (kptdev#2683)

30a99b1

* Expose Renderer api with FileSystem abstraction * Merge Fs methods in pkg.go * Update name * Merge ReadKptfile and address changes * Update porch go.sum * Update results write * Suggested changes

Fix few file headers (kptdev#2707)

2f25f7f

Define a FunctionEvaluator gRPC Interface (kptdev#2700)

3fa3838

* Add install-protoc.sh script * Define a FunctionEvaluator interface * Create an empty server implementation

Integrate porch with renderer interface (kptdev#2710)

7dde871

* Integrate porch with renderer interview * Suggested changes * Add implementation

Introduce applyset abstraction (kptdev#2697)

111cd99

Replacing our very primitive apply logic with something better using server-side apply.

Git repository test (kptdev#2706)

dfed272

We create a simple git server, and we try round-tripping a package. Some hacks/todos uncovered that we need to tackle.

Create Kptfile in config/deploy (kptdev#2708)

432de49

Our package logic requires a Kptfile, so adding one (for now).

Update FunctionRunner Interface (kptdev#2714)

b1bcae5

And make names more consistent.

Merge main branch into porch.

74f4ee9

martinmaly added 6 commits February 17, 2022 09:18

Cache apply-setters:v0.1 function (kptdev#2799)

64400a5

Some blueprints we encounter in tests use the older version of the function.

Make Git Server hostable in cluster (kptdev#2798)

80d3877

* Make Git Server hostable in cluster * Add makefile targets to build and push its images

Add PackageRevision Approval API (kptdev#2810)

132abb5

* Add PackageRevision .../approval subresource * Generate client

Add Clone Package Test (kptdev#2807)

e0ed44f

* Add Clone Package Test * Fix issues with git server's reporting of symbolic references

Revert OCI and Location (kptdev#2814)

2307648

Preparing for merge of porch branch back into main, * Revert "Kpt location parsing (kptdev#2688)" * Revert "oci-support feature branch (kptdev#2666)"

martinmaly requested review from droot, mortent, mengqiy, frankfarzan, yuwenma and natasha41575 as code owners February 18, 2022 20:11

martinmaly force-pushed the merge branch 2 times, most recently from 40b3e76 to b047e0d Compare February 19, 2022 05:57

martinmaly removed request for frankfarzan, yuwenma and natasha41575 February 19, 2022 06:45

Pre-merge Cleanup (kptdev#2816)

881b7f0

* Tidy go modules * Fix year in license header * Fixed few more issues that crept in during merges from main and preparing for merge back to main.

droot reviewed Feb 22, 2022

View reviewed changes

Cleanup Code (kptdev#2811)

f1dbc0b

martinmaly force-pushed the merge branch from b047e0d to a9e8379 Compare February 22, 2022 18:42

martinmaly added 3 commits February 22, 2022 10:57

Merge main into porch

18a9945

Use k8s Dependencies v0.23.2

0aec520

Merge porch into main

0fa4a0a

martinmaly force-pushed the merge branch from a9e8379 to 0fa4a0a Compare February 22, 2022 19:32

droot approved these changes Feb 22, 2022

View reviewed changes

.github/workflows/porch.yml

- "docs/**"

- "site/**"

jobs:

Copy link

Contributor

droot Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok.

go.mod Show resolved Hide resolved

martinmaly merged commit 6e783a8 into kptdev:main Feb 22, 2022

martinmaly deleted the merge branch February 22, 2022 21:24

martinmaly restored the merge branch February 22, 2022 21:46

martinmaly deleted the merge branch February 22, 2022 22:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge Porch into main branch #2815

Merge Porch into main branch #2815

martinmaly commented Feb 18, 2022

droot left a comment

droot Feb 22, 2022

martinmaly Feb 22, 2022

droot Feb 22, 2022

droot Feb 22, 2022

Merge Porch into main branch #2815

Merge Porch into main branch #2815

Conversation

martinmaly commented Feb 18, 2022

droot left a comment

Choose a reason for hiding this comment

droot Feb 22, 2022

Choose a reason for hiding this comment

martinmaly Feb 22, 2022

Choose a reason for hiding this comment

droot Feb 22, 2022

Choose a reason for hiding this comment

droot Feb 22, 2022

Choose a reason for hiding this comment