Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent excessive repo-server disk usage for large repos (#8845) #8897

Merged
merged 8 commits into from Mar 29, 2022

Conversation

crenshaw-dev
Copy link
Collaborator

@crenshaw-dev crenshaw-dev commented Mar 25, 2022

Fixes #8845

Fetching specific commits allowed us to get commits pointing to non-standard refspecs (like refs/pulls/*/head).

But it costs a lot more disk space (demonstrated in the issue) and slows down app refreshes after commits.

This change defaults to the old behavior (fetch standard refspecs, checkout commit) and then falls back to fetching a specific commit if the checkout fails.

So if someone has an Application pointing to HEAD of a repo with 100k commits, this will efficiently fetch each commit without filling up the disk.

And if someone has an application pointing to refs/pulls/123/head, this will fail on the initial fetch and then fall back to fetching the specific commit. It's higher latency, but I think it's justified to avoid filling up the disk.

I've also confirmed there is no regression by testing this app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-cd
  namespace: argocd
spec:
  destination:
    name: ""
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: manifests/core-install
    repoURL: https://github.com/argoproj/argo-cd.git
    targetRevision: refs/pull/8897/head

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev crenshaw-dev changed the title fix: prevent excessive repo-server disk usage for large repos fix: prevent excessive repo-server disk usage for large repos (#8845) Mar 25, 2022
@codecov
Copy link

codecov bot commented Mar 25, 2022

Codecov Report

Merging #8897 (a595331) into master (b1ff9db) will increase coverage by 0.03%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##           master    #8897      +/-   ##
==========================================
+ Coverage   43.10%   43.13%   +0.03%     
==========================================
  Files         186      186              
  Lines       23359    23359              
==========================================
+ Hits        10069    10077       +8     
+ Misses      11849    11841       -8     
  Partials     1441     1441              
Impacted Files Coverage Δ
reposerver/repository/repository.go 60.08% <69.23%> (+0.85%) ⬆️
util/settings/settings.go 48.10% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1ff9db...a595331. Read the comment docs.

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev crenshaw-dev force-pushed the fetch-without-origin-by-default branch from 9515384 to 03d93c0 Compare March 25, 2022 14:55
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev crenshaw-dev force-pushed the fetch-without-origin-by-default branch from 0d0208b to 8ea0cba Compare March 25, 2022 15:01
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev crenshaw-dev marked this pull request as ready for review March 25, 2022 17:03
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev
Copy link
Collaborator Author

Note: planning to run this internally for a week to see if it solves an internal problem we faced. This is sensitive code with a lot of edge cases, so we should be careful.

@crenshaw-dev crenshaw-dev added the cherry-pick/2.3 Candidate for cherry picking into the 2.3 release branch label Mar 25, 2022
Copy link
Collaborator

@alexmt alexmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @crenshaw-dev ! Internal testing was successful. LGTM

@alexmt alexmt merged commit 3f51d92 into argoproj:master Mar 29, 2022
alexmt pushed a commit that referenced this pull request Mar 29, 2022
…#8897)

fix: prevent excessive repo-server disk usage for large repos (#8845) (#8897)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
@crenshaw-dev crenshaw-dev deleted the fetch-without-origin-by-default branch March 29, 2022 22:59
wojtekidd pushed a commit to wojtekidd/argo-cd that referenced this pull request Apr 25, 2022
…oj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: wojtekidd <wojtek.cichon@protonmail.com>
sujeilyfonseca added a commit to sujeilyfonseca/argo-cd that referenced this pull request Jun 24, 2022
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <kykchong@redhat.com>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Fix unit tests

Signed-off-by: jannfis <jann@mistrust.net>

Co-authored-by: jannfis <jann@mistrust.net>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Bump version to 2.3.2

* Bump version to 2.3.2

* fix: Set QPS and burst rate for resource ops client (argoproj#8915)

* fix: Set QPS and burst rate for resource ops client

Signed-off-by: jannfis <jann@mistrust.net>

* fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: bump gitops engine version to v0.6.2

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* docs: update v2.4+ roadmap items (argoproj#8593)

Signed-off-by: ishitasequeira <isequeir@redhat.com>

* docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Bump version to 2.3.3

* Bump version to 2.3.3

* Add manifest for OnePipeline

(cherry picked from commit 15aa211080ef020e6a2ceaee9b845eb3259db237)

* Load additional resource overrides from dedicated ConfigMap

* Run unit tests

(cherry picked from commit 7605d5b0e2e816bb1cf9a29c5910c0fd511900c2)

* Install and config Git for unit tests

(cherry picked from commit 05dda11f6adf3191712b4598c8d55fe8ca1647a6)

* Add doc for changes

* feat: Argo CD CI pipeline changes (argoproj#4)

* updated cicd image

* upadted registry region

* updated one.pipeline.yaml to use the latest scripts

* updated makefile to add required targets

* feat: Argo CD v2.3.2 (argoproj#5)

* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <kykchong@redhat.com>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Fix unit tests

Signed-off-by: jannfis <jann@mistrust.net>

Co-authored-by: jannfis <jann@mistrust.net>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Bump version to 2.3.2

* Bump version to 2.3.2

* feat: Updated CHANGES.md

Co-authored-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Co-authored-by: Keith Chong <kykchong@redhat.com>
Co-authored-by: argo-bot <argoproj@gmail.com>
Co-authored-by: jannfis <jann@mistrust.net>
Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>

* feat: Add .whitesource configuration file (argoproj#6)

Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com>

* docs: CHANGES.md

Updated the CHANGES.md file to include updated
information about changes made.

Contributes to: automation-saas/native-AWS#1413

Signed-off-by: Sujeily Fonseca <sujeily.fonseca@ibm.com>

Co-authored-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Co-authored-by: Keith Chong <kykchong@redhat.com>
Co-authored-by: argo-bot <argoproj@gmail.com>
Co-authored-by: jannfis <jann@mistrust.net>
Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com>
Co-authored-by: Nikolas McGovern <Nikolas.McGovern@ibm.com>
Co-authored-by: Rahul Mourya <Rahul.Mourya2@ibm.com>
Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com>
sujeilyfonseca added a commit to sujeilyfonseca/argo-cd that referenced this pull request Jul 7, 2022
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <kykchong@redhat.com>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Fix unit tests

Signed-off-by: jannfis <jann@mistrust.net>

Co-authored-by: jannfis <jann@mistrust.net>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Bump version to 2.3.2

* Bump version to 2.3.2

* fix: Set QPS and burst rate for resource ops client (argoproj#8915)

* fix: Set QPS and burst rate for resource ops client

Signed-off-by: jannfis <jann@mistrust.net>

* fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: bump gitops engine version to v0.6.2

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* docs: update v2.4+ roadmap items (argoproj#8593)

Signed-off-by: ishitasequeira <isequeir@redhat.com>

* docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* Bump version to 2.3.3

* Bump version to 2.3.3

* fix: Fix docs build error (argoproj#8895)

* work with specific jinja version

Signed-off-by: pashavictorovich <pavel@codefresh.io>

* fix: fix broken monaco editor collapse icons (argoproj#8709)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: upgrade to go 1.17.8 (argoproj#8866) (argoproj#9004)

* chore: upgrade to go 1.17.8

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: use 1.17 so it's always latest in the series

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: allow cli/ui to follow logs (argoproj#8987) (argoproj#9065)

Signed-off-by: Daniel Helfand <helfand.4@gmail.com>

* Merge pull request from GHSA-xmg8-99r8-jc2j

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-6gcg-hp2x-q54h

* fix: do not allow symlinks from directory-type applications

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: add new util file

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: use t.TempDir for simpler tests

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* address comments

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-r642-gv9p-2wjj

Signed-off-by: jannfis <jann@mistrust.net>

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>

* Bump version to 2.3.4

* Bump version to 2.3.4

* test: fix ErrorContains (argoproj#9445)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: missing Helm params (argoproj#9565) (argoproj#9566)

* fix: missing Helm params

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* use absolute paths, fix tests

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix race in test

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: upgrade golangci-lint to v1.46.2 (argoproj#9448)

* chore: upgrade golangci-lint to v1.46.2

Because:

* Installation of golangci-lint v1.45.2 is currently broken and fails
  silently due to a redacted dependency
  (blizzy78/varnamelen#13)

This commit:

* Upgrades golangci-lint to v1.46.2

Signed-off-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com>

* fix: lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: lint

Signed-off-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com>

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix: test race (argoproj#9469)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: lint issues

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: update golangci-lint (argoproj#8988)

* chore: update golangci-lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: remove obsolete repo-server unit test (argoproj#9559)

Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>

* chore: Make unit tests run on platforms other than amd64 (argoproj#8995)

Signed-off-by: jannfis <jann@mistrust.net>

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: eliminate go-mpatch dependency (argoproj#9045)

* chore: eliminate go-mpatch dependency

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: abstract out resource list function

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: don't exit the program in anything but the main function

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: better error messages

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: better error messages

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* test: directory app manifest generation (argoproj#9503)

* test: directory app manifest generation

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* git doesn't support empty dirs

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-h4w9-6x78-8vrj

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-2m7h-86qq-fp4v

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix references

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

use long enough state param for oauth2

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

typo

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

more entropy

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix test

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-q4w5-4gq2-98vm

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Merge pull request from GHSA-jhqp-vf4w-rpwq

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

defer instead of multiple close calls

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

oops

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

don't count jsonnet against max

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix codegen

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

add caveat about 300x ratio

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix versions

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests/lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* chore: fix docs gen

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* Bump version to 2.3.5

* Bump version to 2.3.5

* docs: Changes for v2.3.5

Documented key decision factors to use Argo CD v2.3.5.

Contributes to: automation-saas/automation-saas/native-AWS#1972

Signed-off-by: Sujeily Fonseca <sujeily.fonseca@ibm.com>

Co-authored-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Co-authored-by: Keith Chong <kykchong@redhat.com>
Co-authored-by: argo-bot <argoproj@gmail.com>
Co-authored-by: jannfis <jann@mistrust.net>
Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com>
Co-authored-by: pasha-codefresh <pavel@codefresh.io>
Co-authored-by: Daniel Helfand <helfand.4@gmail.com>
Co-authored-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick/2.3 Candidate for cherry picking into the 2.3 release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

repo-server duplicates git packfiles, filling up disk
2 participants