We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

+1

{
    "Version": "v2.10.4+f5d63a5",
    "BuildDate": "2024-03-18T08:09:23Z",
    "GitCommit": "f5d63a5c77d2e804e51ef94bee3db441e0789d00",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

My understanding is that this isn't an issue with golang's crypto library. Rather it has to do with the hash algorithms ssh accepts in Ubuntu 22.04 (the version Argo CD images are built on).

It looks like Ubuntu 22.04 does support the listed algorithms (based on this guide), but maybe neither algorithm is enabled by default in the handshake. You might have to explicitly enable that host key algorithm.

$ ssh -Q sig 
ssh-ed25519
sk-ssh-ed25519@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com

I'm not too familiar with all the internal workings of SSH, but would it be a solution to change the known_hosts entry for azure devops to: ssh.dev.azure.com rsa-sha2-512 instead of ssh.dev.azure.com ssh-rsa?

I've changed this within my setup using configs.ssh.knownHosts. I'm not sure if this helps during the brownout, but it seems to be working at the moment.

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

argocd repo add with --insecure-skip-server-verification flag will bypass the verification which causes the issue.

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

Do you encounter issue pulling git repositories directly on the host machine?

Do you encounter issue pulling git repositories directly on the host machine?

I haven't tested this, but I can at least say that the config is used with a regular ssh ssh.dev.azure.com call; if I input invalid config (e.g. foobar as HostkeyAlgorithms) then the ssh command fails.

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd.
I had to

1. Generate new public private key ssh-keygen -t rsa-sha2-512
2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

I can confirm what is being mentioned here. During the microsoft brownouts, we are able to work locally with Azure Devops GIT using SSH keys, however all of our argo instances fail to connect citing the error mentioned above.

We just got into this issue right now.
We did the following steps to stop having the "You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally." error:

• Created a new pair of SSH keys with SHA2 512 with:

ssh-keygen -f key_file_name -t rsa-sha2-512

• Replaced the keys in ArgoCD and Azure DevOps.

• Changed known hosts values in ArgoCD chart values. The value is configs.ssh.knownHosts. We replaced the default Azure DevOps known hosts with the following ones:

ssh.dev.azure.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Hope this helps as a temporal workaround.

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

1. Generate new public private key ssh-keygen -t rsa-sha2-512
2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

1. Generate new public private key ssh-keygen -t rsa-sha2-512
2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

We have this config since a few weeks ago.
Nevertheless, we observed the clone errors today.

I would expect you will see the errors appearing again during the next brownout period. :-/

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set:

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:

- mountPath: /app/config/ssh
     name: ssh-known-hosts

So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

@rouke-broersma
About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)
We are also very hesitant to add the --insecure flag.
This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.
We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

Probably the same happened to us and that's why we saw it solved...

@rouke-broersma
About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)
We are also very hesitant to add the --insecure flag.
This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.
We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set:

argo-cd/common/common.go

Line 75 in 4d527f8

DefaultPathSSHConfig = "/app/config/ssh"

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:

- mountPath: /app/config/ssh
     name: ssh-known-hosts

So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

We had a syntax error in the ssh-config mounted at /home/argocd/.ssh/config and that error showed up in the Argo UI, so I'd assume it is being used.

Very possible that we tested after the brownout however.

I did some testing yesterday and believe the issue might be in Argo.

It seems like the git implementation used in argo does some parts with the Go user agent and some with the OpenSSH user agent.
Both processes probably need to use the rsa-sha2-256/512 signature to be compliant with Azure DevOps. It might also explain why changing git config does not solve the issue, maybe not both processes use the same git config location. (maybe if both /home/argocd/.ssh/config and /app/config/ssh are updated it might work?)

To check which of the 2 processes is using the wrong signature I used the reposerver container to manually do a git fetch (uses OpenSSH user agent) with extra verbosity. (GIT_SSH_COMMAND="ssh -vvv -i ./../key -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts" git fetch).

This results in the following (removed a lot of content that is not ):

...
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256
...
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: KEX algorithms: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
debug2: ciphers ctos: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: ciphers stoc: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug3: record_hostkey: found key type RSA in file /app/config/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /app/config/ssh/ssh_known_hosts:13
debug1: Will attempt key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: Next authentication method: publickey
debug1: Offering public key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
Authenticated to ssh.dev.azure.com ([40.74.28.31]:22) using "publickey".
...

It shows that it uses rsa-sha2-512, and thus should work with Azure DevOps during the brownouts and after the removal of ssh-rsa, but since we have not seen any other brownouts I could not run this during brownout.
Since openSSH uses the correct signature, it probably is the go-git client that does not use the correct signature thus breaking ArgoCD with Azure DevOps.

While for Github the issue was to use an elliptic key, this should be fixed in argo or the go-git client.
It also looks like Flux users also had problems with host key algoritms. They added a flag for this to the project (fluxcd/source-controller#711).

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.
@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

1. Generate new public private key ssh-keygen -t rsa-sha2-512
2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

We are having issues again today, so it seems like this did not infact resolve

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

We are having issues again today so it seems like it just worked for us because the brownout had ended

Looks like the brownout is over again...

Joining this thread as we also just recently regenerated our SSH key as Microsoft advised but then during the next brownout observed the issue. Reading through this I'm not certain we've identified a solution or if we need one (aka once ADO kill ssh-rsa it just works)...

Anyone considered using HTTPS? Any pros or cons I should be aware of over SSH? I'm guess SSH security is considered better asymmetric crypto versus a PAT based http basic auth? Anything else?

https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/

According to their schedule the brownout times and lengths will increase to 8 and 12 hours per day in the next week(s). That will be very annoying to sit out and hope it will take the correct algorightm afterwards

I just took a look at the code and have no idea how everything is wired together, but these parts look interesting:

https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267-L279 where the git ssh client is constructed
+
https://github.com/argoproj/argo-cd/blob/master/common/common.go#L74-L77 where the default paths are defined

I took a look in our reposerver with cat /app/config/ssh/ssh_known_hosts and the result is:

ssh.dev.azure.com rsa-sha2-512 ....

which is what we defined as custom config.

Nevertheless, we are getting the errors. Maybe we need a way to configure custom algorithms here? https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267

@patst we are coming to the same conclusions by digging through the code and we hope to test this to confirm and hopefully submit a patch.

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

Hi, I'm using ArgoCD v2.7.11

Thanks to @bartoszpyrek I managed to get this working by :

• generating a new RSA key with rsa-sha2-512 algorithm : ssh-keygen -t rsa-sha2-512
• adding generated public key to my Azure DevOps profile

Public key is of form :

ssh-rsa AAAAB3NzaC1y..... user@host

• doing a simple test on ArgoCD Ui

Note the new line in yellow after -----END OPENSSH PRIVATE KEY-----

This worked fine 👌

Also, it seems there's no need to update know-hosts configuration, nor skip server verification

Finally, I updated my secret which contains my SSH private key with the new one, also with the newline at the end, and restarted argocd-repo-server pods

I manually deleted repositories which uses the old SSH key and let ArgoCD re-create them (ArgoCD manages itself its configuration, using App of App pattern)

Now everything is fine, let's see if this works during next days...

Good luck 🤞

@aurel4oxand let me guess you tested this somewhere in the last 40 minutes? The brownout is over :)

@rouke-broersma I wrote a PR that implements the workaround: #18007

You wrote a PR implementing key exchange options based on comments by @zamedic who is now saying that this may actually not be the fix because they still have issues. They were most likely testing their workaround after a brownout had already ended. If you follow the linked comment you can see that we believe the issue is with the host key algorithms not key exchange algorithms. They are two separate configurations. See: #17634 (comment)

Your change is still a needed change in the grand scheme of things, but unless you can confirm that your fix actually works during a brownout, I don't believe this helps in this case.

In out stack, we use Argo CD to manage Argo CD. When I implemented the fix, Argo CD was failing with the main error, so I could not bump the version. I had to log onto k8s and manually bump the repo server version, after which, stuff appeared to work. I do however see today, that this appears to have been purely coincidental. I am continuing to try and resolve this, however today the brownouts appeared to be even more inconsistent with some fetches working and others not.

Also, I am a complete noob when it comes to how GIT/SSH works when it comes to the transport, so learning alot.

How do I reproduce the command that ArgoCD fires at Azure DevOps? Is git-upload-pack easily executed from the commandline?

I tried to reproduce a git operation that uses the ssh-rsa signature but I failed. It keeps defaulting to rsa-sha2-512.

$env:GIT_SSH_COMMAND="ssh -vvv -i keyfile -o IdentitiesOnly=yes"
git pull

results in a verbose ssh log:

...
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
...
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
...
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
...

Additionally I can confirm that any RSA key works irrespective of the -t parameter. The key generated with ssh-keygen -t ssh-rsa -f ssh-rsa-keyfile also just works. At least assuming the brown out is active 😅

@cveld enabling ssh-rsa in the ssh user config file should be more or less similar to what argocd does. Ie the inverse of this doc: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops#q-ssh-cannot-establish-a-connection-what-should-i-do

You can't reproduce it by default with an ssh or git client because they disabled ssh-rsa in openssh 1-2 years go.

It's not an exact reproduction because argocd does this in go code instead of using the native client but the approximation is close enough I think.

@rouke-broersma thanks! I was able to reproduce the brown out with the following in ~/.ssh/config:

Host ssh.dev.azure.com vs-ssh.visualstudio.com
  HostKeyAlgorithms ssh-rsa

I have multiple Argo CD clusters, on 1 cluster I am running some small changes, that 1 appears to not be experiencing brownouts while the other clusters are experiencing brownouts.

The difference is the go-get version. I ran
go get -u github.com/go-git/go-git/v5 which bumped the module up to v5.12.0

Will need to see whats in that change and deploy my fix to all environments to confirm...

So this is a no-go. So the only fix, I think, would be to modify

argo-cd/util/git/ssh.go

Line 54 in 0f11dfb

kexAlgos = DefaultSSHKeyExchangeAlgorithms

to either have a hardcoded list of supported algorithms or to read the supported algorithms from ~/.ssh/config.

@rouke-broersma IMO I think initializing SupportedSSHKeyExchangeAlgorithms from an environment variable would be the way to go. The default would be the same as today. Perhaps it could also be worthwhile having a new configmap to set the environment variable from.

Ok, I believe I have traced down the root cause...
By bumping the go-git, it also bumped https://github.com/skeema/knownhosts to version 1.2.2
this version contains the changes for the new algorithms: skeema/knownhosts@bd8e67e

Ok, I believe I have traced down the root cause...

By bumping the go-git, it also bumped https://github.com/skeema/knownhosts to version 1.2.2

this version contains the changes for the new algorithms: skeema/knownhosts@bd8e67e

Seems solid. It would break old implementations though but i assume that is ok since ssh-rsa is deprecated and dropped to start with.

But only the ArgoCD maintainers can decide this if it not hits a BC or if they allow it and so on

So this is a no-go. So the only fix, I think, would be to modify

argo-cd/util/git/ssh.go

Line 54 in 0f11dfb

kexAlgos = DefaultSSHKeyExchangeAlgorithms

to either have a hardcoded list of supported algorithms or to read the supported algorithms from ~/.ssh/config.

@rouke-broersma IMO I think initializing SupportedSSHKeyExchangeAlgorithms from an environment variable would be the way to go. The default would be the same as today. Perhaps it could also be worthwhile having a new configmap to set the environment variable from.

The issue is with HostKeyAlgorithms not KeyExchangeAlgoritms.

The issue with an environment variable is that the nativeclient which is used for 90% of argocd git interaction would read from ~/.ssh/config so you would need to add the config to both the ssh client config and the environment variable. Ideally both clients would read from the same source, so this difference between the two clients is removed.

See confusion here for example: #17634 (comment)

I think short term the update of the knownhosts library will fix this issue, but long term the differences between go-git and nativeclient need to be resolved because it seems like argocd maintainers also forget about these differences (understandable, but dangerous since it involves crypto and authentication).

Thanks @zamedic - I've merged your PR.

Before cherry-picking this change into supported release branches and putting out a fix release, can somebody confirm that this actually fixes the issue? For example, by running an image that's been built off the latest code (once the image has been built)? I do not have access to an Azure DevOps Git repository, unfortunately.

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

We're preparing our dev environment for testing this change.

Unfortunately there is no brownout at the moment, so we will have to wait until Microsoft randomly turns on the brownout somewhere within the next 20 minutes to 2 hours.

is this image also available as a helm chart? i could test it but my clusters have argo via helm chart installed

is this image also available as a helm chart? i could test it but my clusters have argo via helm chart installed

You can overwrite the image in the helm chart: https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L56

@jannfis I can confirm the fix addresses the issue for my reproduction case.

I built the image from the PR branch, and was able to test it just before the most recent brown out ended. I had 2 separate namespace-scoped Argo CD instances targeting the same private Azure repo...

• one Argo CD instance based on v2.10 which was consistently failing w/ this error
• one Argo CD instance based on @zamedic's PR branch, which was consistently passing
  during the same time period.

So presuming the brown out period is shared between them (no reason to think otherwise), I can no longer reproduce.

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

We're preparing our dev environment for testing this change.

Unfortunately there is no brownout at the moment, so we will have to wait until Microsoft randomly turns on the brownout somewhere within the next 20 minutes to 2 hours.

The brownout is active... and we are no longer impacted 🎉

We're looking forward to the next Argo release!

Thanks for testing, folks. I'll cut releases today.

2.10.9 is out, 2.9.14 and 2.8.18 are being built and on their way.

Please let us know if those new releases do not fix the issue for you.

We are waiting on a brownout to test in our dev environment (2.10.9) then move forward to qa and prod... I'll post an update as soon as I know if it works!

We appreciate the hard work!

We appreciate the hard work!

All credits actually go to the people in this thread discussing their observations and potential causes, @zamedic who finally found the root cause and contributed a fix, and all those fine folks who tested it.

The Argo community is just awesome ❤️

That's amazing, GG !

Any chance to get this fix applied to previous releases ?
Currently I'm using Argocd v2.7 which is impacted by the brownout
Upgrading ArgoCD to more recent version is planned but not immediately actually..

Thanks for this 🙏

That's amazing, GG !

Any chance to get this fix applied to previous releases ? Currently I'm using Argocd v2.7 which is impacted by the brownout Upgrading ArgoCD to more recent version is planned but not immediately actually..

Thanks for this 🙏

You're running an EOL version of ArgoCD and will not receive this patch, you should change your upgrade processes to reflect this: https://argo-cd.readthedocs.io/en/stable/developer-guide/release-process-and-cadence/#patch-releases-eg-25x

That's amazing, GG !
Any chance to get this fix applied to previous releases ? Currently I'm using Argocd v2.7 which is impacted by the brownout Upgrading ArgoCD to more recent version is planned but not immediately actually..
Thanks for this 🙏

You're running an EOL version of ArgoCD and will not receive this patch, you should change your upgrade processes to reflect this: https://argo-cd.readthedocs.io/en/stable/developer-guide/release-process-and-cadence/#patch-releases-eg-25x

Thx, I should've check that 👍

Since I've not heard from folks still affected after upgrading, I'm going to close this issue for now.

If anybody running 2.10.9, 2.9.14 or 2.8.18 (and higher) still encounter RSA-related issues with Azure DevOps, feel free to ping on this issue to have it re-opened.