Feat/4982 unwanted dependency checks #5509

antoooks · 2024-01-08T15:08:52Z

Addresses #4982

Add unwanted dependencies checking referring to kubernetes/kubernetes unwanted-dependencies.json file
Add unwanted-dependencies-check Github workflow

k8s-ci-robot · 2024-01-08T15:08:54Z

This PR has multiple commits, and the default merge method is: merge.
You can request commits to be squashed using the label: tide/merge-method-squash

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2024-01-08T15:09:01Z

Hi @antoooks. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

charles-chenzz

/ok-to-test

charles-chenzz · 2024-01-10T01:10:18Z

.github/workflows/unwanted-dependencies-check.yml

@@ -0,0 +1,42 @@
+name: unwanted-dependencies-check
+


can this yaml run on prow job? as we have an issue which want to migrate some actions workflows to prow
#5486

There's a requirement to make the job optional, is it possible in prow to always run it but not cancelling the other pipeline?

I'm not quite sure, but prow can run multiple jobs and choose to make some jobs optional. we might need to investigate prow( but prow is quite a huge project)

I have confirmed with @natasha41575, it is decided that for this PR we'll still stick to GH action, and if your PR for prow is already merged we can migrate it to prow

my PR for prow is still in pending review, we should merge this first

varshaprasad96

The changes look good to me. Looks like this test is failing in the CI for this PR. Can it be fixed?

antoooks · 2024-02-07T18:56:48Z

The changes look good to me. Looks like this test is failing in the CI for this PR. Can it be fixed?

Hi @varshaprasad96, It is expected to be failed on purpose and will be treated as an optional test, so it won't affect approval. (please refer to the referenced issue page)

I think since it checks on different parts of kustomize, we should have a separate PR with respective domain owners doing the fix.

varshaprasad96 · 2024-02-07T21:40:04Z

Hi @varshaprasad96, It is expected to be failed on purpose and will be treated as an optional test, so it won't affect approval. (please refer to the referenced issue page)

Got it! Thanks!

varshaprasad96

/lgtm
/approve

k8s-ci-robot · 2024-02-07T21:40:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: antoooks, varshaprasad96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [varshaprasad96]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

koba1t · 2024-02-08T05:40:17Z

Hi!
I think Kubernetes only imports below three packages. So we need to check only the below packages.
https://github.com/kubernetes/kubernetes/blob/f07b47c3d1e596f53640285afb0d53a81fe6f259/go.mod#L232-L234

antoooks · 2024-02-08T16:03:32Z

Hi! I think Kubernetes only imports below three packages. So we need to check only the below packages. https://github.com/kubernetes/kubernetes/blob/f07b47c3d1e596f53640285afb0d53a81fe6f259/go.mod#L232-L234

Noted, thanks for pointing it out

antoooks · 2024-02-08T16:40:12Z

@koba1t feedback added

* Add buildMetadata task and ref Move build metadata tasks, draft buildMetadata reference Clean up buildMetadata ref Add managed by label task Add local non-generated task Add local generated resource Add remote generator task Clean up tasks and ref Add local transformer annotation example Add local and remote transformer example * Address PR feedback and general cleanup cherrypick updates from feature branch fix script fix release script

add unwanted dependency check script

remove json file add extra condition to ignore docs changes revert unintended changes revert unintended changes

charles-chenzz

/lgtm
/assign @koba1t

antoooks · 2024-02-26T16:43:51Z

FYI, the check meant to be fail because we do have unwanted dependencies :)

koba1t · 2024-03-09T14:10:50Z

Hi @antoooks
Sorry for the delayed response. It's many times to take investigate.
I think there is something wrong with this check step.

For Example

in github.com/spf13/viper module.

in CI result

Looks like this dependency were found in the result.

...
Error: unwanted dependencies found. (github.com/spf13/viper at /home/runner/work/kustomize/kustomize/kyaml/go.sum)
Error: unwanted dependencies found. (github.com/spf13/viper at /home/runner/work/kustomize/kustomize/kustomize/go.sum)
Error: unwanted dependencies found. (github.com/spf13/viper at /home/runner/work/kustomize/kustomize/api/go.sum)
...

run `go mod graph`

github.com/spf13/viper was required only on the sigs.k8s.io/kustomize/cmd/gorepomod module. But this module appears in the go mod graph result.

$ pwd
${HOME}/kustomize/kustomize
$ go mod graph | grep viper           
sigs.k8s.io/kustomize/cmd/gorepomod github.com/spf13/viper@v1.17.0
github.com/spf13/viper@v1.17.0 github.com/fsnotify/fsnotify@v1.6.0
github.com/spf13/viper@v1.17.0 github.com/hashicorp/hcl@v1.0.0
github.com/spf13/viper@v1.17.0 github.com/magiconair/properties@v1.8.7
...

I run after disabling to gowork.

$ GOWORK=off go mod graph | grep viper
$

Maybe this result was mixed in the go.work results.

Could you investigate this more?
If you need my help, please feel free to send a mention to me!

antoooks · 2024-03-17T17:01:28Z

GOWORK=off go mod graph | grep viper

Hi @koba1t, thank you for checking and finding this, I have found a logic error on line 83, which supposed to detect if unwanted deps are exist inside go.sum. But instead I write it as -z $(cat $file | fgrep $dep) which means if it does not exist.

Same error also happened on line 92 with go mod graph.

k8s-ci-robot · 2024-03-17T17:01:55Z

New changes are detected. LGTM label has been removed.

koba1t · 2024-03-18T18:54:58Z

Thanks for your great work, @antoooks!

But I'm still concerned that the unwanted-dependencies-check is still down...
Maybe you need to look at "status.unwantedReferences" in the hack/unwanted-dependencies.json` file.
https://github.com/kubernetes/kubernetes/blob/aa73f3163a52e9a99df01b86f60eeed31abd54d9/cmd/dependencyverifier/dependencyverifier.go#L46-L48

Because I think this entry for the exception to the unwanted dependencies list.
https://github.com/kubernetes/kubernetes/pull/116598/files#diff-09f70f26cff6c34dc0063a45ca43cf5025d653058b06f7966fc8e6f3b9cecd3eR102

antoooks · 2024-03-19T17:01:32Z

Thanks for your great work, @antoooks!

But I'm still concerned that the unwanted-dependencies-check is still down... Maybe you need to look at "status.unwantedReferences" in the hack/unwanted-dependencies.json` file. https://github.com/kubernetes/kubernetes/blob/aa73f3163a52e9a99df01b86f60eeed31abd54d9/cmd/dependencyverifier/dependencyverifier.go#L46-L48

Because I think this entry for the exception to the unwanted dependencies list. https://github.com/kubernetes/kubernetes/pull/116598/files#diff-09f70f26cff6c34dc0063a45ca43cf5025d653058b06f7966fc8e6f3b9cecd3eR102

Hey @koba1t, thanks for the feedback. Do you mean we need to whitelist dependencies in the PR commit you are referring on your comment?

koba1t · 2024-03-20T15:54:25Z

.github/workflows/unwanted-dependencies-check.yml

+          with:
+              title: 🐛 unwanted-dependencies-check failed for ${{ github.sha }}
+              token: ${{ secrets.GITHUB_TOKEN }}
+              labels: kind/bug


Maybe kind/dependencies is the better label for that issue.

koba1t · 2024-03-20T16:03:50Z

hack/unwanted-dependencies-check.sh

+        READ_PATH=$(realpath ${JSON_PATH_LOCAL})
+    else
+        # Default behavior: pull unwanted-dependencies.json from kubernetes/kubernetes upstream repo
+        JSON_PATH_URL='https://raw.githubusercontent.com/kubernetes/kubernetes/e51fe4a61cca7f4a0875630da433f280b52c138a/hack/unwanted-dependencies.json'


I think that the JSON file may be old.
So, I think maybe better to use the latest file.

koba1t · 2024-03-20T19:06:54Z

Hey @koba1t, thanks for the feedback. Do you mean we need to whitelist dependencies in the PR commit you are referring on your comment?

I think that is better because we have a few unwanted dependencies that we can't delete.
For example,
github.com/pkg/error was required evanphx/json-patch package, and we can't delete this dependency.
https://github.com/evanphx/json-patch/blob/master/v5/go.mod#L7

antoooks · 2024-04-16T14:16:03Z

hi @koba1t , I have integrated all the changes we discussed. However it seems that the unwanted deps list on k/k has changed. Could you check for deps I need to put into whitelist?

koba1t · 2024-04-17T15:44:26Z

 2024-03-27 15:06:09 (33.8 MB/s) - ‘/home/runner/work/kustomize/kustomize/hack/unwanted-dependencies.json’ saved [11055/11055]

Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/kustomize/go.sum)
Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/api/go.sum)
Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/kyaml/go.sum)
Error: unwanted dependencies found. (github.com/pkg/errors at /home/runner/work/kustomize/kustomize/kustomize/go.sum)
Error: unwanted dependencies found. (github.com/pkg/errors at /home/runner/work/kustomize/kustomize/api/go.sum)

The above dependencies are listed on the whitelist.
https://github.com/kubernetes/;kubernetes/blob/d831b13e6f6fb5efb566100286190fedca6dd340/hack/unwanted-dependencies.json#L162-L183

Maybe that is better to find the whitelist from the unwanted-dependencies.json files on k/k.

antoooks · 2024-05-08T10:57:59Z

 2024-03-27 15:06:09 (33.8 MB/s) - ‘/home/runner/work/kustomize/kustomize/hack/unwanted-dependencies.json’ saved [11055/11055]

Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/kustomize/go.sum)
Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/api/go.sum)
Error: unwanted dependencies found. (github.com/mailru/easyjson at /home/runner/work/kustomize/kustomize/kyaml/go.sum)
Error: unwanted dependencies found. (github.com/pkg/errors at /home/runner/work/kustomize/kustomize/kustomize/go.sum)
Error: unwanted dependencies found. (github.com/pkg/errors at /home/runner/work/kustomize/kustomize/api/go.sum)

The above dependencies are listed on the whitelist. https://github.com/kubernetes/;kubernetes/blob/d831b13e6f6fb5efb566100286190fedca6dd340/hack/unwanted-dependencies.json#L162-L183

Maybe that is better to find the whitelist from the unwanted-dependencies.json files on k/k.

Thanks for the note @koba1t , fixed on the subsequent commit

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 8, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 8, 2024

k8s-ci-robot requested review from annasong20 and yuwenma January 8, 2024 15:09

antoooks marked this pull request as draft January 8, 2024 15:14

antoooks marked this pull request as ready for review January 8, 2024 15:44

k8s-ci-robot requested review from koba1t and natasha41575 January 8, 2024 15:45

antoooks changed the title ~~[WIP] Feat/4982 unwanted dependency checks~~ Feat/4982 unwanted dependency checks Jan 8, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 8, 2024

charles-chenzz reviewed Jan 10, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 10, 2024

varshaprasad96 reviewed Feb 7, 2024

View reviewed changes

varshaprasad96 approved these changes Feb 7, 2024

View reviewed changes

k8s-ci-robot assigned varshaprasad96 Feb 7, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 7, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 7, 2024

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 8, 2024

antoooks force-pushed the feat/4982-unwanted-dependency-checks branch from dd82d0c to 629f8a9 Compare February 21, 2024 15:39

ncapps and others added 6 commits February 22, 2024 00:01

add unwanted-dependencies check script

7fa6665

add unwanted dependency check script

add unwanted-dependencies-check workflow

fc19557

add sudo on run script check

6ee295a

add license

2df6455

fix: remove -f predicate from find command inside script

551b0c6

antoooks force-pushed the feat/4982-unwanted-dependency-checks branch from d7cc869 to 9f518fe Compare February 21, 2024 16:01

antoooks added 2 commits February 22, 2024 00:08

only check in kyaml, api, and kustomize

af0c556

remove json file add extra condition to ignore docs changes revert unintended changes revert unintended changes

rever unintended changes

35fa808

antoooks force-pushed the feat/4982-unwanted-dependency-checks branch from a95165a to 35fa808 Compare February 21, 2024 16:09

charles-chenzz reviewed Feb 26, 2024

View reviewed changes

k8s-ci-robot assigned koba1t and charles-chenzz Feb 26, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 26, 2024

fix checking logic

0fa4698

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2024

koba1t reviewed Mar 20, 2024

View reviewed changes

add whitelist, make use of k/k master unwanted-dependencies.json

b710c13

enable whitelist compilation from unwanted dependency list from k/k

30c53e8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/4982 unwanted dependency checks #5509

Feat/4982 unwanted dependency checks #5509

antoooks commented Jan 8, 2024 •

edited

k8s-ci-robot commented Jan 8, 2024

k8s-ci-robot commented Jan 8, 2024

charles-chenzz left a comment

charles-chenzz Jan 10, 2024 •

edited

antoooks Jan 10, 2024

charles-chenzz Jan 11, 2024

antoooks Jan 13, 2024

charles-chenzz Jan 13, 2024

varshaprasad96 left a comment

antoooks commented Feb 7, 2024

varshaprasad96 commented Feb 7, 2024

varshaprasad96 left a comment

k8s-ci-robot commented Feb 7, 2024

koba1t commented Feb 8, 2024

antoooks commented Feb 8, 2024

antoooks commented Feb 8, 2024

charles-chenzz left a comment

antoooks commented Feb 26, 2024

koba1t commented Mar 9, 2024 •

edited

antoooks commented Mar 17, 2024

k8s-ci-robot commented Mar 17, 2024

koba1t commented Mar 18, 2024

antoooks commented Mar 19, 2024

koba1t Mar 20, 2024 •

edited

koba1t Mar 20, 2024

koba1t commented Mar 20, 2024

antoooks commented Apr 16, 2024

koba1t commented Apr 17, 2024

antoooks commented May 8, 2024 •

edited

Feat/4982 unwanted dependency checks #5509

Are you sure you want to change the base?

Feat/4982 unwanted dependency checks #5509

Conversation

antoooks commented Jan 8, 2024 • edited

k8s-ci-robot commented Jan 8, 2024

k8s-ci-robot commented Jan 8, 2024

charles-chenzz left a comment

Choose a reason for hiding this comment

charles-chenzz Jan 10, 2024 • edited

Choose a reason for hiding this comment

antoooks Jan 10, 2024

Choose a reason for hiding this comment

charles-chenzz Jan 11, 2024

Choose a reason for hiding this comment

antoooks Jan 13, 2024

Choose a reason for hiding this comment

charles-chenzz Jan 13, 2024

Choose a reason for hiding this comment

varshaprasad96 left a comment

Choose a reason for hiding this comment

antoooks commented Feb 7, 2024

varshaprasad96 commented Feb 7, 2024

varshaprasad96 left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 7, 2024

koba1t commented Feb 8, 2024

antoooks commented Feb 8, 2024

antoooks commented Feb 8, 2024

charles-chenzz left a comment

Choose a reason for hiding this comment

antoooks commented Feb 26, 2024

koba1t commented Mar 9, 2024 • edited

For Example

in CI result

run go mod graph

antoooks commented Mar 17, 2024

k8s-ci-robot commented Mar 17, 2024

koba1t commented Mar 18, 2024

antoooks commented Mar 19, 2024

koba1t Mar 20, 2024 • edited

Choose a reason for hiding this comment

koba1t Mar 20, 2024

Choose a reason for hiding this comment

koba1t commented Mar 20, 2024

antoooks commented Apr 16, 2024

koba1t commented Apr 17, 2024

antoooks commented May 8, 2024 • edited

antoooks commented Jan 8, 2024 •

edited

charles-chenzz Jan 10, 2024 •

edited

koba1t commented Mar 9, 2024 •

edited

run `go mod graph`

koba1t Mar 20, 2024 •

edited

antoooks commented May 8, 2024 •

edited