From e86c479690d22a4e8e21adb736c4d6641af61c35 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Fri, 25 Mar 2022 14:29:27 -0400 Subject: [PATCH 1/3] Converge local and cloud gorelease scripts, improve release notes --- .gitignore | 3 + releasing/README.md | 291 ++---------------- releasing/cloudbuild-local.sh | 2 +- releasing/cloudbuild.sh | 131 +------- releasing/cloudbuild.yaml | 5 +- releasing/compile-changelog.sh | 53 ++++ .../{localbuild.sh => run-goreleaser.sh} | 90 ++++-- releasing/vendor_kustomize.diff | 281 ----------------- releasing/vendor_kustomize.sh | 118 ------- 9 files changed, 159 insertions(+), 815 deletions(-) create mode 100755 releasing/compile-changelog.sh rename releasing/{localbuild.sh => run-goreleaser.sh} (57%) delete mode 100755 releasing/vendor_kustomize.diff delete mode 100755 releasing/vendor_kustomize.sh diff --git a/.gitignore b/.gitignore index d9a5081d25..3307a3850f 100644 --- a/.gitignore +++ b/.gitignore @@ -27,3 +27,6 @@ site/public/ site/resources/ site/.hugo_build.lock **/node_modules/ + +# goreleaser artifacts +**/dist/ diff --git a/releasing/README.md b/releasing/README.md index 38418f674a..bcd9f51e18 100644 --- a/releasing/README.md +++ b/releasing/README.md @@ -16,6 +16,25 @@ This document describes how to perform a [semver release] of one of the several [Go modules] in this repository. +In this repo, releasing a module is accomplished by applying +a tag to the repo and pushing it upstream. A minor release +branch is also created as necessary to track patch releases. + +A properly formatted tag (described below) contains +the module name and version. + +Pushing the tag upstream will trigger [Google Cloud Build] to build a release +and make it available on the [release page]. + +Cloud build reads its instructions from the +[`cloudbuild.yaml`] file in this directory. + +We use a Go program to make the tagging and branch +creation process less error prone. + +See this [multi-module repo] tagging discussion +for an explanation of the path-like portion of these tags. + #### semver review Go's [semver]-compatible version tags take the form `v{major}.{minor}.{patch}`: @@ -141,7 +160,7 @@ go mod edit -require=sigs.k8s.io/kustomize/kyaml@$versionKyaml plugin/builtin/pa Create the PR: ``` -createBranch pinToKyaml "Pin to kyaml $versionKyaml" +createBranch pinToKyaml "Update kyaml to $versionKyaml" ``` ``` createPr @@ -201,7 +220,7 @@ gorepomod pin cmd/config --doIt Create the PR: ``` -createBranch pinToCmdConfig "Pin to cmd/config $versionCmdConfig" && +createBranch pinToCmdConfig "Update cmd/config to $versionCmdConfig" && createPr ``` @@ -256,7 +275,7 @@ gorepomod pin api --doIt Create the PR: ``` -createBranch pinToApi "Pin to api $versionApi" && +createBranch pinToApi "Update api to $versionApi" && createPr ``` @@ -383,7 +402,7 @@ Image sha256 can be found in the image registry in the GCP project [k8s-staging-kustomize]. Commit and push your changes. Then create a PR to [k8s.io] to promote -new images. Assign the PR to @monopole and @Shell32-natsu. +the new image. ## Update kustomize-in-kubectl @@ -410,267 +429,7 @@ https://github.com/kubernetes/kubernetes/pull/103419 https://github.com/kubernetes/kubernetes/pull/106389 ----- -Older notes follow: - -## Public Modules - -[`sigs.k8s.io/kustomize/api`]: #sigsk8siokustomizeapi -[`sigs.k8s.io/kustomize/kustomize`]: #sigsk8siokustomizekustomize -[`sigs.k8s.io/kustomize/kyaml`]: #sigsk8siokustomizekyaml -[`sigs.k8s.io/kustomize/cmd/config`]: #sigsk8siokustomizecmdconfig - -[kustomize/v3.2.1]: /../../releases/tag/kustomize%2Fv3.2.1 - -| Module Name | Module Description | Example Tag | Example Branch Name | -| ------ | --- | --- | --- | -| [`sigs.k8s.io/kustomize/kustomize`] | kustomize executable | _kustomize/v3.2.2_ | _release-kustomize-v3.2.2_ | -| [`sigs.k8s.io/kustomize/api`] | kustomize API | _api/v0.1.0_ | _release-api-v0.1_ | -| [`sigs.k8s.io/kustomize/kyaml`] | k8s-specific yaml editting | _kyaml/v0.3.3_ | _release-kyaml-v0.2_ | -| [`sigs.k8s.io/kustomize/cmd/config`] | kyaml related commands | _cmd/config/v0.3.3_ | _release-cmd/config-v0.3_ | - -### sigs.k8s.io/kustomize/kustomize - -The `kustomize` program, a CLI for performing -structured edits to Kubernetes Resource Model (KRM) YAML. - -There's only one public package in this module. -It's called `main`, it's therefore unimportable. -It holds the `kustomize` executable. - -See the [installation instructions](../docs/INSTALL.md) -to perform an install. - - -### sigs.k8s.io/kustomize/api - -The [kustomize Go API](https://github.com/kubernetes-sigs/kustomize/tree/master/api). - -The packages in this module are used by API clients, -like the `kustomize` program itself, and programs in -other repositories, e.g. `kubectl`. They include -methods to read, edit and emit modified YAML. - -Release notes should appear on the [release page]. - -The package has a toy executable called `api`, which, -if run, prints the API release version, but has no -other use. - -### sigs.k8s.io/kustomize/kyaml - -The [kyaml module](https://github.com/kubernetes-sigs/kustomize/tree/master/kyaml). - -Low level packages for transforming YAML associated -with KRM objects. - -These are used to build _kyaml filters_, computational units -that accept and emit KRM YAML, editing or simply validating it. - -The kustomize `api` and the `cmd/config` packages are built on this. - -### sigs.k8s.io/kustomize/cmd/config - -The [cmd/config module](https://github.com/kubernetes-sigs/kustomize/tree/master/cmd/config). - -A collection od CLI commands that correspond to -kyaml filters. - -## Manual process - -In this repo, releasing a module is accomplished by applying -a tag to the repo and pushing it upstream. A minor release -branch is also created as necessary to track patch releases. - -A properly formatted tag (described below) contains -the module name and version. - -Pushing the tag upstream will trigger [Google Cloud Build] to build a release -and make it available on the [release page]. - -Cloud build reads its instructions from the -[`cloudbuild.yaml`] file in this directory. - -We use a Go program to make the tagging and branch -creation process less error prone. - -See this [multi-module repo] tagging discussion -for an explanation of the path-like portion of these tags. - -### Get up to date - -It's assumed that whatever is already commited to the latest -commit is passing all tests and appropriate for release. - - -``` -git fetch upstream -git checkout master -git rebase upstream/master -make prow-presubmit-check -``` - -### Select a module to release - -E.g. -``` -module=kustomize # The kustomize executable -module=api # The API -``` - -### Review tags to help determine new tag - -Local: -``` -git tag -l | grep $module -``` - -Remote: -``` -git ls-remote --tags upstream | grep $module -``` - -Set the version you want: - -``` -major=0; minor=1; patch=0 -``` - - -### Create the release branch - -Every module release occurs on it's own git branch. - -The branch name doesn't include the patch number, -since the branch accumulates patch releases. - -> TODO: define procedure for doing a cherrypick (committing a patch) to a -> release branch that already exists. - -Name the branch: - -``` -branch="release-${module}-v${major}.${minor}" -echo "branch=$branch" -``` - -Create it: -``` -git checkout -b $branch -``` - -### Define the release tag - -``` -tag="${module}/v${major}.${minor}.${patch}" -echo "tag=$tag" -``` - -### Pin modules to their dependencies. - -This is achieved via a `replace` directive -in a module's `go.mod` file. - -``` -# Update the following as needed, obviously. - -# git checkout -b pinTheRelease -# go mod edit -dropreplace=sigs.k8s.io/kustomize/api $module/go.mod -# go mod edit -require=sigs.k8s.io/kustomize/api@v0.1.1 $module/go.mod -# git commit -a -m "Drop API module replacement" - -``` - -### Push the release branch - -``` -git push -f upstream $branch -``` - -#### if replacing a release... - -Must delete the tag before re-pushing it. -Dangerous - only do this if you're sure nothing -has already pulled the release. - -Delete the tag locally: - -``` -git tag --delete $tag -``` - -Delete it upstream: -``` -# Disable push protection: -git remote set-url --push upstream git@github.com:kubernetes-sigs/kustomize.git - -# The empty space before the colon effectively means delete the tag. -git push upstream :refs/tags/$tag - -# Enable push protection: -git remote set-url --push upstream no_push -``` - -Optionally visit the [release page] and delete -(what has now become) the _draft_ release for that -version. - -### Tag the local repository - -``` -git tag -a $tag -m "Release $tag on branch $branch" -``` - -Move the `latest_kustomize` tag: -``` -git tag -d latest_kustomize -git push upstream :latest_kustomize -git tag -a latest_kustomize -``` - -### Optionally build locally - -[localbuild.sh]: localbuild.sh - -Load the same version of `goreleaser` referenced in `cloudbuild.yaml` via docker and run [localbuild.sh] from the container's command line: - -``` -# Get goreleaser image from cloudbuild.yaml -export GORELEASER_IMAGE=goreleaser/goreleaser:v0.172.1 - -# Drop into a shell -docker run -it --entrypoint=/bin/bash -v $(pwd):/go/src/github.com/kubernetes-sigs/kustomize -w /go/src/github.com/kubernetes-sigs/kustomize $GORELEASER_IMAGE - -# Run build -./releasing/localbuild.sh TAG [--snapshot] -``` - - -### Optionally build and release locally - -[cloudbuild-local.sh]: cloudbuild-local.sh - -Install [`cloud-build-local`], then run [cloudbuild-local.sh]: - -``` -./releasing/cloudbuild-local.sh $module -``` - -This should create release artifacts in a local directory. - -### Trigger the cloud build by pushing the tag - -Push the tag: - -``` -git remote set-url --push upstream git@github.com:kubernetes-sigs/kustomize.git -git push upstream $tag -git push upstream latest_kustomize -git remote set-url --push upstream no_push -``` - -This triggers [Google Cloud Build]. -### Update release notes +# Testing changes to the release pipeline -Visit the [release page] and edit the release notes as desired. +You can test the release script locally by running [cloudbuild.sh](cloudbuild.sh) in a container or by installing Cloud Build Local and running [cloudbuild-local.sh](cloudbuild-local.sh). See each of those files for more details on their usage. diff --git a/releasing/cloudbuild-local.sh b/releasing/cloudbuild-local.sh index 9e00c18d85..6773135ced 100755 --- a/releasing/cloudbuild-local.sh +++ b/releasing/cloudbuild-local.sh @@ -27,7 +27,7 @@ cp releasing/cloudbuild.yaml $config # Add the --snapshot flag to suppress the # github release and leave the build output # in the kustomize/dist directory. -sed -i "s|# - '--snapshot|- '--snapshot|" $config +sed -i "" "s|# - '--snapshot|- '--snapshot|" $config echo "Executing cloud-build-local with config file $config :" echo "=========================" diff --git a/releasing/cloudbuild.sh b/releasing/cloudbuild.sh index ce73e6b45a..60c647cb59 100755 --- a/releasing/cloudbuild.sh +++ b/releasing/cloudbuild.sh @@ -1,131 +1,32 @@ #!/bin/bash # -# Usage (from top of repo): +# This script is called by Kustomize's Cloud Build release pipeline. +# It installs jq (required for release note construction) +# and then runs goreleaser (http://goreleaser.com). # -# releasing/cloudbuild.sh TAG [--snapshot] +# To test it locally, run it in a goreleaser container: # -# Where TAG is in the form +# # Get goreleaser image from cloudbuild.yaml +# export GORELEASER_IMAGE=goreleaser/goreleaser:v0.179.0 # -# api/v1.2.3 -# kustomize/v1.2.3 -# cmd/config/v1.2.3 -# ... etc. +# # Drop into a shell +# docker run -it --entrypoint=/bin/bash -v $(pwd):/go/src/github.com/kubernetes-sigs/kustomize -w /go/src/github.com/kubernetes-sigs/kustomize $GORELEASER_IMAGE # -# Cloud build should be configured to trigger on tags -# matching: +# # Run this script in the container, where $TAG is the tag to "release" (e.g. kyaml/v0.13.4) +# ./releasing/cloudbuild.sh $TAG --snapshot # -# [\w/]+/v\d+\.\d+\.\d+ -# -# This script runs goreleaser (http://goreleaser.com), -# presumably from a cloudbuild.yaml step that installed it. set -e set -x fullTag=$1 shift -echo "fullTag=$fullTag" - -remainingArgs="$@" -echo "Remaining args: $remainingArgs" - -# Take everything before the last slash. -# This is expected to match $module. -module=${fullTag%/*} -echo "module=$module" - -# Find previous tag that matches the tags module -prevTag=$(git tag -l "$module*" --sort=-version:refname --no-contains=$fullTag | head -n 1) - -# Generate the changelog for this release -# using the last two tags for the module -changeLogFile=$(mktemp) -git log $prevTag..$fullTag \ - --pretty=oneline \ - --abbrev-commit --no-decorate --no-color --no-merges \ - -- $module > $changeLogFile -echo "Release notes:" -cat $changeLogFile - -# Take everything after the last slash. -# This should be something like "v1.2.3". -semVer=`echo $fullTag | sed "s|$module/||"` -echo "semVer=$semVer" -# This is probably a directory called /workspace -echo "pwd = $PWD" - -# Sanity check -echo "### ls -las . ################################" -ls -las . -echo "###################################" - - -# CD into the module directory. -# This directory expected to contain a main.go, so there's -# no need for extra details in the `build` stanza below. -cd $module - -skipBuild=true -if [[ "$module" == "kustomize" || "$module" == "pluginator" ]]; then - # If releasing a main program, don't skip the build. - skipBuild=false +if ! command -v jq &> /dev/null +then + # This assumes we are in an alpine container (which is the case for goreleaser images) + echo "Installing jq." + apk add jq --no-cache fi -goReleaserConfigFile=$(mktemp) - -cat <$goReleaserConfigFile -project_name: $module - -archives: -- name_template: "${module}_${semVer}_{{ .Os }}_{{ .Arch }}" - -builds: -- skip: $skipBuild - - ldflags: > - -s - -X sigs.k8s.io/kustomize/api/provenance.version={{.Version}} - -X sigs.k8s.io/kustomize/api/provenance.gitCommit={{.Commit}} - -X sigs.k8s.io/kustomize/api/provenance.buildDate={{.Date}} - - goos: - - linux - - darwin - - windows - - goarch: - - amd64 - - arm64 - - s390x - - ppc64le - -checksum: - name_template: 'checksums.txt' - -env: -- CGO_ENABLED=0 -- GO111MODULE=on - -release: - github: - owner: kubernetes-sigs - name: kustomize - draft: true - -EOF - -cat $goReleaserConfigFile - -date - -time /usr/local/bin/goreleaser release \ - --debug \ - --timeout 10m \ - --parallelism 7 \ - --config=$goReleaserConfigFile \ - --release-notes=$changeLogFile \ - --rm-dist \ - --skip-validate $remainingArgs - -date +./releasing/run-goreleaser.sh "$fullTag" release "$@" diff --git a/releasing/cloudbuild.yaml b/releasing/cloudbuild.yaml index f22d9b18a2..3349ecc6cb 100644 --- a/releasing/cloudbuild.yaml +++ b/releasing/cloudbuild.yaml @@ -1,3 +1,6 @@ +# Cloud build should be configured to trigger with this configuration on tags matching: +# [\w/]+/v\d+\.\d+\.\d+ +# steps: - name: 'bash' args: @@ -36,7 +39,7 @@ steps: # Run goreleaser indirectly via a shell script # to configure it properly. -- name: goreleaser/goreleaser:v0.172.1 +- name: goreleaser/goreleaser:v0.179.0 timeout: 12m entrypoint: /bin/sh dir: myClone diff --git a/releasing/compile-changelog.sh b/releasing/compile-changelog.sh new file mode 100755 index 0000000000..e79b926968 --- /dev/null +++ b/releasing/compile-changelog.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# +# Builds a PR-oriented changelog from the git history for the given module. +# +# Usage (from top of repo): +# +# releasing/compile-changelog.sh MODULE TAG CHANGE_LOG_FILE +# +# Where TAG is in the form +# +# api/v1.2.3 +# kustomize/v1.2.3 +# cmd/config/v1.2.3 +# ... etc. +# + +set -o errexit +set -o nounset +set -o pipefail + +if [[ -z "${1-}" ]] || [[ -z "${2-}" ]] || [[ -z "${3-}" ]]; then + echo "Usage: $0 " + echo "Example: $0 kyaml kyaml/v0.13.4 changelog.txt" + exit 1 +fi + +module=$1 +fullTag=$2 +changeLogFile=$3 + +# Find previous tag that matches the tags module +prevTag=$(git tag -l "$module*" --sort=-version:refname --no-contains="$fullTag" | head -n 1) + +commits=( $(git log "$prevTag".."$fullTag" \ + --pretty=format:'%H' \ + --abbrev-commit --no-decorate --no-color --no-merges \ + -- "$module") ) + +# There is a 256 character limit on the query parameter for the GitHub API, so split into batches then deduplicate results +batchSize=5 +results="" +for((i=0; i < ${#commits[@]}; i+=batchSize)) +do + commitList=$(IFS="+"; echo "${commits[@]:i:batchSize}" | sed 's/ /+/g') + if newResults=$(curl -sSL "https://api.github.com/search/issues?q=$commitList+repo%3Akubernetes-sigs%2Fkustomize" | jq -r '[ .items[] | { number, title } ]'); then + results=$(echo "$results" "$newResults" | jq -s '.[0] + .[1] | unique') + else + echo "Failed to fetch results for commits: $commitList" + exit 1 + fi +done + +echo "${results}" | jq -r '.[] | select( .title | startswith("Back to development mode") | not) | "#\(.number): \(.title)" ' > "$changeLogFile" diff --git a/releasing/localbuild.sh b/releasing/run-goreleaser.sh similarity index 57% rename from releasing/localbuild.sh rename to releasing/run-goreleaser.sh index 02bb7bbf1b..e4b4ddd455 100755 --- a/releasing/localbuild.sh +++ b/releasing/run-goreleaser.sh @@ -1,10 +1,10 @@ #!/bin/bash # -# Works exactly like cloudbuild.sh but doesn't perform a release. +# Builds and optionally releases the specified module # # Usage (from top of repo): # -# releasing/localbuild.sh TAG [--snapshot] +# releasing/run-goreleaser.sh TAG MODE[build|release] [--snapshot] # # Where TAG is in the form # @@ -13,56 +13,70 @@ # cmd/config/v1.2.3 # ... etc. # -# This script runs a build through goreleaser (http://goreleaser.com) but nothing else. -# -set -e -set -x +set -o errexit +set -o nounset +set -o pipefail + +if [[ -z "${1-}" || -z "${2-}" ]]; then + echo "Usage: $0 TAG MODE [goreleaser flags]" + echo " TAG: the tag to build or release, e.g. api/v1.2.3" + echo " MODE: build or release" + exit 1 +fi fullTag=$1 shift echo "fullTag=$fullTag" +if [[ $1 == "release" || $1 == "build" ]]; then + mode=$1 + shift +else + echo "Error: mode must be build or release" + exit 1 +fi + remainingArgs="$@" -echo "Remaining args: $remainingArgs" +echo "Remaining args: $remainingArgs" # Take everything before the last slash. # This is expected to match $module. module=${fullTag%/*} echo "module=$module" -# Find previous tag that matches the tags module -prevTag=$(git tag -l "$module*" --sort=-version:refname --no-contains=$fullTag | head -n 1) +# Take everything after the last slash. +# This should be something like "v1.2.3". +semVer=${fullTag#$module/} +echo "semVer=$semVer" # Generate the changelog for this release # using the last two tags for the module changeLogFile=$(mktemp) -git log $prevTag..$fullTag \ - --pretty=oneline \ - --abbrev-commit --no-decorate --no-color --no-merges \ - -- $module > $changeLogFile -echo "Release notes:" -cat $changeLogFile - -# Take everything after the last slash. -# This should be something like "v1.2.3". -semVer=`echo $fullTag | sed "s|$module/||"` -echo "semVer=$semVer" +./releasing/compile-changelog.sh "$module" "$fullTag" "$changeLogFile" +echo +echo "######### Release notes: ##########" +cat "$changeLogFile" +echo "###################################" +echo # This is probably a directory called /workspace -echo "pwd = $PWD" # Sanity check -echo "### ls -las . ################################" +echo +echo "############ DEBUG ##############" +echo "pwd = $PWD" +echo "ls -las ." ls -las . echo "###################################" - +echo # CD into the module directory. # This directory expected to contain a main.go, so there's # no need for extra details in the `build` stanza below. cd $module +# This is used in goreleaser.yaml skipBuild=true if [[ "$module" == "kustomize" || "$module" == "pluginator" ]]; then # If releasing a main program, don't skip the build. @@ -112,16 +126,26 @@ release: EOF -cat $goReleaserConfigFile +echo +echo "############# CONFIG ##############" +cat "$goReleaserConfigFile" +echo "###################################" +echo + +args=( + --debug + --timeout 10m + --parallelism 7 + --config="$goReleaserConfigFile" + --rm-dist + --skip-validate +) +if [[ $mode == "release" ]]; then + args+=(--release-notes="$changeLogFile") +fi date - -time /usr/local/bin/goreleaser build \ - --debug \ - --timeout 10m \ - --parallelism 7 \ - --config=$goReleaserConfigFile \ - --rm-dist \ - --skip-validate $remainingArgs - +export PATH="/usr/local/bin:$PATH" +set -x +time goreleaser "$mode" "${args[@]}" $remainingArgs date diff --git a/releasing/vendor_kustomize.diff b/releasing/vendor_kustomize.diff deleted file mode 100755 index b116f0f3f3..0000000000 --- a/releasing/vendor_kustomize.diff +++ /dev/null @@ -1,281 +0,0 @@ -commit 1b893558aa83ac6491e5ba416b493170a9045fec -Author: Jingfang Liu -Date: Mon Nov 12 10:26:12 2018 -0800 - - last change - -diff --git a/staging/src/k8s.io/cli-runtime/artifacts/kustomization/configMap.yaml b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/configMap.yaml -new file mode 100644 -index 0000000000..0008853094 ---- /dev/null -+++ b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/configMap.yaml -@@ -0,0 +1,8 @@ -+ -+apiVersion: v1 -+kind: ConfigMap -+metadata: -+ name: the-map -+data: -+ altGreeting: "Good Morning!" -+ enableRisky: "false" -diff --git a/staging/src/k8s.io/cli-runtime/artifacts/kustomization/deployment.yaml b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/deployment.yaml -new file mode 100644 -index 0000000000..6e79409080 ---- /dev/null -+++ b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/deployment.yaml -@@ -0,0 +1,30 @@ -+apiVersion: apps/v1 -+kind: Deployment -+metadata: -+ name: the-deployment -+spec: -+ replicas: 3 -+ template: -+ metadata: -+ labels: -+ deployment: hello -+ spec: -+ containers: -+ - name: the-container -+ image: monopole/hello:1 -+ command: ["/hello", -+ "--port=8080", -+ "--enableRiskyFeature=$(ENABLE_RISKY)"] -+ ports: -+ - containerPort: 8080 -+ env: -+ - name: ALT_GREETING -+ valueFrom: -+ configMapKeyRef: -+ name: the-map -+ key: altGreeting -+ - name: ENABLE_RISKY -+ valueFrom: -+ configMapKeyRef: -+ name: the-map -+ key: enableRisky -diff --git a/staging/src/k8s.io/cli-runtime/artifacts/kustomization/kustomization.yaml b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/kustomization.yaml -new file mode 100644 -index 0000000000..6e1e3202d5 ---- /dev/null -+++ b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/kustomization.yaml -@@ -0,0 +1,5 @@ -+nameprefix: test- -+ resources: -+- deployment.yaml -+- service.yaml -+- configMap.yaml -diff --git a/staging/src/k8s.io/cli-runtime/artifacts/kustomization/service.yaml b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/service.yaml -new file mode 100644 -index 0000000000..2942cdb7df ---- /dev/null -+++ b/staging/src/k8s.io/cli-runtime/artifacts/kustomization/service.yaml -@@ -0,0 +1,13 @@ -+kind: Service -+apiVersion: v1 -+metadata: -+ name: the-service -+spec: -+ selector: -+ deployment: hello -+ type: LoadBalancer -+ ports: -+ - protocol: TCP -+ port: 8666 -+ targetPort: 8080 -+ -\ No newline at end of file -diff --git a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/BUILD b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/BUILD -index 22b34de008..b91d1c0130 100644 ---- a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/BUILD -+++ b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/BUILD -@@ -35,12 +35,15 @@ go_library( - "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", - "//staging/src/k8s.io/apimachinery/pkg/util/yaml:go_default_library", - "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", -+ "//staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps:go_default_library", - "//staging/src/k8s.io/client-go/discovery:go_default_library", - "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", - "//staging/src/k8s.io/client-go/rest:go_default_library", - "//staging/src/k8s.io/client-go/restmapper:go_default_library", - "//vendor/golang.org/x/text/encoding/unicode:go_default_library", - "//vendor/golang.org/x/text/transform:go_default_library", -+ "//vendor/sigs.k8s.io/kustomize/pkg/commands/build:go_default_library", -+ "//vendor/sigs.k8s.io/kustomize/pkg/fs:go_default_library", - ], - ) - -diff --git a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/builder_test.go b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/builder_test.go -index 7fd526b33c..801f13f772 100644 ---- a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/builder_test.go -+++ b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/builder_test.go -@@ -465,27 +465,48 @@ func TestPathBuilderWithMultipleInvalid(t *testing.T) { - } - - func TestDirectoryBuilder(t *testing.T) { -- b := newDefaultBuilder(). -- FilenameParam(false, &FilenameOptions{Recursive: false, Filenames: []string{"../../../artifacts/guestbook"}}). -- NamespaceParam("test").DefaultNamespace() -+ tests := []struct { -+ directories []string -+ singleItem bool -+ number int -+ expectedNames []string -+ }{ -+ {[]string{"../../../artifacts/guestbook"}, false, 3, []string{"redis-master"}}, -+ {[]string{"../../../artifacts/kustomization"}, true, 3, []string{"test-the-deployment"}}, -+ {[]string{"../../../artifacts/guestbook", "../../../artifacts/kustomization"}, false, 6, []string{"redis-master", "test-the-deployment"}}, -+ } - -- test := &testVisitor{} -- singleItemImplied := false -+ for _, tt := range tests { -+ b := newDefaultBuilder(). -+ FilenameParam(false, &FilenameOptions{Recursive: false, Filenames: tt.directories}). -+ NamespaceParam("test").DefaultNamespace() - -- err := b.Do().IntoSingleItemImplied(&singleItemImplied).Visit(test.Handle) -- if err != nil || singleItemImplied || len(test.Infos) < 3 { -- t.Fatalf("unexpected response: %v %t %#v", err, singleItemImplied, test.Infos) -- } -+ test := &testVisitor{} -+ singleItemImplied := false - -- found := false -- for _, info := range test.Infos { -- if info.Name == "redis-master" && info.Namespace == "test" && info.Object != nil { -- found = true -- break -+ err := b.Do().IntoSingleItemImplied(&singleItemImplied).Visit(test.Handle) -+ if err != nil || singleItemImplied != tt.singleItem || len(test.Infos) < tt.number { -+ t.Fatalf("unexpected response: %v %t %#v", err, singleItemImplied, test.Infos) -+ } -+ -+ contained := func(name string) bool { -+ for _, info := range test.Infos { -+ if info.Name == name && info.Namespace == "test" && info.Object != nil { -+ return true -+ } -+ } -+ return false -+ } -+ -+ allFound := true -+ for _, name := range tt.expectedNames { -+ if !contained(name) { -+ allFound = false -+ } -+ } -+ if !allFound { -+ t.Errorf("unexpected responses: %#v", test.Infos) - } -- } -- if !found { -- t.Errorf("unexpected responses: %#v", test.Infos) - } - } - -diff --git a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/visitor.go b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/visitor.go -index 32c1a691a5..d7a37e1cde 100644 ---- a/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/visitor.go -+++ b/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/resource/visitor.go -@@ -20,10 +20,12 @@ import ( - "bytes" - "fmt" - "io" -+ "io/ioutil" - "net/http" - "net/url" - "os" - "path/filepath" -+ "strings" - "time" - - "golang.org/x/text/encoding/unicode" -@@ -38,6 +40,9 @@ import ( - utilerrors "k8s.io/apimachinery/pkg/util/errors" - "k8s.io/apimachinery/pkg/util/yaml" - "k8s.io/apimachinery/pkg/watch" -+ "k8s.io/cli-runtime/pkg/kustomize/k8sdeps" -+ "sigs.k8s.io/kustomize/pkg/commands/build" -+ "sigs.k8s.io/kustomize/pkg/fs" - ) - - const ( -@@ -452,7 +457,10 @@ func ExpandPathsToFileVisitors(mapper *mapper, paths string, recursive bool, ext - if err != nil { - return err - } -- -+ if isKustomizationDir(path) { -+ visitors = append(visitors, NewKustomizationVisitor(mapper, path, schema)) -+ return filepath.SkipDir -+ } - if fi.IsDir() { - if path != paths && !recursive { - return filepath.SkipDir -@@ -463,7 +471,10 @@ func ExpandPathsToFileVisitors(mapper *mapper, paths string, recursive bool, ext - if path != paths && ignoreFile(path, extensions) { - return nil - } -- -+ if strings.HasSuffix(path, "kustomization.yaml") { -+ visitors = append(visitors, NewKustomizationVisitor(mapper, filepath.Dir(path), schema)) -+ return nil -+ } - visitor := &FileVisitor{ - Path: path, - StreamVisitor: NewStreamVisitor(nil, mapper, path, schema), -@@ -479,6 +490,14 @@ func ExpandPathsToFileVisitors(mapper *mapper, paths string, recursive bool, ext - return visitors, nil - } - -+func isKustomizationDir(path string) bool { -+ if _, err := os.Stat(filepath.Join(path, "kustomization.yaml")); err == nil { -+ return true -+ } -+ return false -+} -+ -+ - // FileVisitor is wrapping around a StreamVisitor, to handle open/close files - type FileVisitor struct { - Path string -@@ -507,6 +526,37 @@ func (v *FileVisitor) Visit(fn VisitorFunc) error { - return v.StreamVisitor.Visit(fn) - } - -+// KustomizationVisitor prorvides the output of kustomization build -+type KustomizationVisitor struct { -+ Path string -+ *StreamVisitor -+} -+ -+// Visit in a KustomizationVisitor build the kustomization output -+func (v *KustomizationVisitor) Visit(fn VisitorFunc) error { -+ fSys := fs.MakeRealFS() -+ f := k8sdeps.NewFactory() -+ var out bytes.Buffer -+ cmd := build.NewCmdBuild(&out, fSys, f.ResmapF, f.TransformerF) -+ cmd.SetArgs([]string{v.Path}) -+ // we want to silence usage, error output, and any future output from cobra -+ // we will get error output as a golang error from execute -+ cmd.SetOutput(ioutil.Discard) -+ _, err := cmd.ExecuteC() -+ if err != nil { -+ return err -+ } -+ v.StreamVisitor.Reader = bytes.NewReader(out.Bytes()) -+ return v.StreamVisitor.Visit(fn) -+} -+ -+func NewKustomizationVisitor(mapper *mapper, path string, schema ContentValidator) *KustomizationVisitor { -+ return &KustomizationVisitor{ -+ Path: path, -+ StreamVisitor: NewStreamVisitor(nil, mapper, path, schema), -+ } -+} -+ - // StreamVisitor reads objects from an io.Reader and walks them. A stream visitor can only be - // visited once. - // TODO: depends on objects being in JSON format before being passed to decode - need to implement diff --git a/releasing/vendor_kustomize.sh b/releasing/vendor_kustomize.sh deleted file mode 100755 index 71b9ac54cc..0000000000 --- a/releasing/vendor_kustomize.sh +++ /dev/null @@ -1,118 +0,0 @@ -#!/bin/bash -# -# Copyright 2018 The Kubernetes Authors. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -e -set -x - -DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" - -# vendor_kustomize.sh creates the change in kubernetes repo for vendoring kustomize - -function setUpWorkspace { - KPATH=~/kustomize_vendor - mkdir $KPATH - GOPATH=$KPATH -} - -function cloneK8s { - mkdir -p $KPATH/src/k8s.io - cd $KPATH/src/k8s.io - - git clone git@github.com:kubernetes/kubernetes.git -} - -function godepRestore { - cd $KPATH/src/k8s.io/kubernetes - - # restore dependencies - hack/run-in-gopath.sh hack/godep-restore.sh -} - -function getKustomizeDeps { - # get Kustomize and Kustomize dependencies - hack/run-in-gopath.sh godep get sigs.k8s.io/kustomize/pkg/commands - hack/run-in-gopath.sh godep get github.com/bgentry/go-netrc/netrc - hack/run-in-gopath.sh godep get github.com/hashicorp/go-cleanhttp - hack/run-in-gopath.sh godep get github.com/hashicorp/go-getter - hack/run-in-gopath.sh godep get github.com/hashicorp/go-safetemp - hack/run-in-gopath.sh godep get github.com/hashicorp/go-version - - # The hashes below passed bin/pre-commit.sh with kustomize HEAD at time of merger. - DEPS=( - "hashicorp/go-getter 4bda8fa99001c61db3cad96b421d4c12a81f256d" - "hashicorp/go-cleanhttp d5fe4b57a186c716b0e00b8c301cbd9b4182694d" - "hashicorp/go-safetemp b1a1dbde6fdc11e3ae79efd9039009e22d4ae240" - "hashicorp/go-version 270f2f71b1ee587f3b609f00f422b76a6b28f348" - "bgentry/go-netrc 9fd32a8b3d3d3f9d43c341bfe098430e07609480" - "mitchellh/go-homedir 58046073cbffe2f25d425fe1331102f55cf719de" - "mitchellh/go-testing-interface a61a99592b77c9ba629d254a693acffaeb4b7e28" - "ulikunitz/xz v0.5.4" - ) - - function foo { - cd $KPATH/src/k8s.io/kubernetes/_output/local/go/src/github.com/$1 - git checkout $2 - } - for i in "${DEPS[@]}"; do - foo $i - done -} - -function updateK8s { - # Copy k8sdeps from Kustomize to cli-runtime in staging - mkdir -p $KPATH/src/k8s.io/kubernetes/staging/src/k8s.io/cli-runtime/pkg/kustomize - cp -r $KPATH/src/k8s.io/kubernetes/_output/local/go/src/sigs.k8s.io/kustomize/k8sdeps \ - $KPATH/src/k8s.io/kubernetes/staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps - - # Change import path of k8sdeps - find $KPATH/src/k8s.io/kubernetes/staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps \ - -type f -name "*.go" | \ - xargs sed -i \ - 's!sigs.k8s.io/kustomize/k8sdeps!k8s.io/cli-runtime/pkg/kustomize/k8sdeps!' - - - # Add kustomize command to kubectl - cp $DIR/vendor_kustomize.diff $KPATH/vendor_kustomize.diff - - cd $GOPATH/src/k8s.io/kubernetes - git apply --ignore-space-change --ignore-whitespace $KPATH/vendor_kustomize.diff -} - -function godepSave { - # Save all dependencies into k8s.io/kubernetes/vendor by running - # hack/godep-save.sh - hack/run-in-gopath.sh hack/godep-save.sh -} - -function verify { - # make sure in k8s.io/kubernetes/vendor/sigs.k8s.io/kustomize - # there is no internal package - test 0 == $(ls $KPATH/src/k8s.io/kubernetes/vendor/sigs.k8s.io/kustomize | grep “internal” | wc -l) - - # Make sure it compiles. - test 0 == $(bazel build cmd/kubectl:kubectl) - - # next step, open a PR - echo "The change for vendoring kustomize is ready in $GOPATH/src/k8s.io/kubernetes.\n Next step, open a PR for it.\n" -} - -setUpWorkspace -cloneK8s -godepRestore -getKustomizeDeps -updateK8s -godepSave -verify From 5833f4ca800d22b1d5fca869c9d26289f9f36e18 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Mon, 28 Mar 2022 14:37:29 -0400 Subject: [PATCH 2/3] Make Goreleaser use the tag passed to the script --- releasing/run-goreleaser.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/releasing/run-goreleaser.sh b/releasing/run-goreleaser.sh index e4b4ddd455..f377ceef9d 100755 --- a/releasing/run-goreleaser.sh +++ b/releasing/run-goreleaser.sh @@ -28,6 +28,7 @@ fi fullTag=$1 shift echo "fullTag=$fullTag" +export GORELEASER_CURRENT_TAG=$fullTag if [[ $1 == "release" || $1 == "build" ]]; then mode=$1 From 9e57ab72fca760eccb4efbab190cb8240ca1329d Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Mon, 28 Mar 2022 15:12:12 -0400 Subject: [PATCH 3/3] Copy newer Go into goreleaser image --- releasing/cloudbuild.yaml | 10 ++++++++++ releasing/run-goreleaser.sh | 12 ++++++++++++ 2 files changed, 22 insertions(+) diff --git a/releasing/cloudbuild.yaml b/releasing/cloudbuild.yaml index 3349ecc6cb..8ed4621c9a 100644 --- a/releasing/cloudbuild.yaml +++ b/releasing/cloudbuild.yaml @@ -37,6 +37,14 @@ steps: - checkout - $TAG_NAME +# Copy a newer version of Go into the goreleaser workspace. +# Use the same source image as the builder in kustomize.Dockerfile +- name: golang:alpine + entrypoint: /bin/sh + args: + - '-c' + - 'mkdir -p /workspace/bin && cp $(go env GOROOT)/bin/go /workspace/bin' + # Run goreleaser indirectly via a shell script # to configure it properly. - name: goreleaser/goreleaser:v0.179.0 @@ -44,6 +52,8 @@ steps: entrypoint: /bin/sh dir: myClone secretEnv: ['GITHUB_TOKEN'] + env: + - 'GO_BINARY_PATH=/workspace/bin/go' args: - releasing/cloudbuild.sh - $TAG_NAME diff --git a/releasing/run-goreleaser.sh b/releasing/run-goreleaser.sh index f377ceef9d..7f51728de4 100755 --- a/releasing/run-goreleaser.sh +++ b/releasing/run-goreleaser.sh @@ -51,6 +51,16 @@ echo "module=$module" semVer=${fullTag#$module/} echo "semVer=$semVer" +# Because of https://github.com/kubernetes-sigs/kustomize/issues/4542 +# we need to manually install a newer version of Go into an older goreleaser image. +# This points goreleaser to that version of Go, or the version discovered from PATH if unspecified. +goBinary="go" +if [[ -n "${GO_BINARY_PATH:-}" ]]; then + echo "GO_BINARY_PATH is set, using go version from $GO_BINARY_PATH" + goBinary="$GO_BINARY_PATH" +fi +sh -c "$goBinary version" + # Generate the changelog for this release # using the last two tags for the module changeLogFile=$(mktemp) @@ -101,6 +111,8 @@ builds: -X sigs.k8s.io/kustomize/api/provenance.gitCommit={{.Commit}} -X sigs.k8s.io/kustomize/api/provenance.buildDate={{.Date}} + gobinary: ${goBinary} + goos: - linux - darwin