Ingest https://api.github.com/meta

[jsoref]

Summary

GitHub has added their ssh keys to https://api.github.com/meta

Motivation

A number of us suffered a major GitHub driven CI outage as covered by #7723 and temporarily fixed by #7722

Proposal

How do you think this should be implemented?

Parse https://api.github.com/meta to extract the ssh_keys array (probably at start of the proper module).

At present, it has this content:

  "ssh_keys": [
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl",
    "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=",
    "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
  ],

This would allow ArgoCD to automatically ingest the latest version of the SSH keys in a secure manner (as long as ArgoCD properly validates the TLS cert for the domain).

[jannfis]

While it may sound tempting, I think we shouldn't do this.

What if people decide to remove github.com SSH keys from their configuration, because they don't use GitHub? Will they automatically be re-added? Or what if they are in a disconnect environment, and don't have access to github.com (or the Internet) at all? Or what if GitHub delivers "garbage" by accident from that endpoint for a while?

I can imagine this can be provided as a script that can be used to to retrieve the keys from GitHub API and feed it into Argo CD's certificate store via argocd cert add-ssh ... to update the keys - but important - on the user's request.

We could also integrate a proper check in our CI system to verify the SSH host keys we ship against the ones shipped officially by GitHub to detect any configuration drift between them, to enable us to evaluate & possibly act on those.

[jsoref]

There should be a flag to turn it on/off.

The alternative of adding a check to CI seems like a good first step.

[OmerKahani]

We could also integrate a proper check in our CI system to verify the SSH host keys we ship against the ones shipped officially by GitHub to detect any configuration drift between them, to enable us to evaluate & possibly act on those.

I think there is an assumption here that everyone is running the latest ArgoCD.

There should be a flag to turn it on/off.

This solution would support all ArgoCD from version X, no matter if they run the latest version of not

[jannfis]

I was giving this a little more thought, and I'm still opposing having this feature in the core as automated means to update the Argo CD configuration.

For one (and this is a weaker concern), this would (presumably) only solve an issue for one very vendor specific use-case (i.e. for github.com). We do ship keys for other vendor's public offerings (i.e. gitlab.com, bitbucket.com) as well.

Second, I believe that this pretty much has some security implications. We'd put some trust in an API endpoint returning data to configure some security relevant piece of information, and the data gathered from that endpoint is not even authenticated (as in, it does not provide proper message authentication or an externally verifiable signature).

We have all the pieces in the puzzle to help people automate this, if they really want to (e.g. user can set up a recurring job to fetch data and re-configure Argo CD using the CLI). But I don't think we should automate this as part of core Argo CD.

I would even go as far as to propose a radical change and say we should stop shipping the SSH host keys provided with our manifests, because this somehow implies control or authority over this data where we clearly don't have that.

[jsoref]

I'm sure that BitBucket and GitLab can be convinced to offer similar APIs.

The data is validated by TLS which is definitely a validated source.

I think a job that does this instead of burning in the values might be better. Clearly burning in the values had bad results.

[jannfis]

The data is validated by TLS which is definitely a validated source.

No, it is not. I don't want to nit on that, but simplified speaking, TLS only validates whether the peer you are talking to is indeed the one you think it is, and it makes some promises about the integrity of the data that the peer sent (i.e. that it was not modified on the way from system a to system b) - hence the name "Transport Layer Security", as it secures the transport layer between two systems. However, it makes no assumption about the integrity or validity of the actual meaning of the data that is being sent or received. Also, with TLS you usually delegate trust to a third-party (the Certificate Authority that issued the server's certificate), and there are many of such authorities trusted by default these days. You can workaround this by pinning your trust to a specific CA, or even to a specific certificate, but well - for this special use-case we'd then have a chicken-egg problem.

A good analogy to that would be the actual connection to a Git repository: Just because you are talking through a valid TLS (or also SSH for that matter) connection to the Git repository's server does not mean that your repository wasn't compromised or could contain commits by rogue actors, possibly resulting in malicious data being returned for your consumption. That's why you would ensure that the data being returned (e.g. the commits to the repository) is actually what they are intended to be, and originate from the person or entity you'd assume it would, by using strict enforcement of commit signatures (using PGP or a similar strong protocol that supports message authentication).

[jsoref]

Ok, you're right, this is pointless.

PGP keys only assert that someone either had access to keys, access to enough computing power to forge a signature, access to your computer to hack its processing path, or access to the supply chain for the software running on your computer.

The same applies for TLS. If your supply chain is compromised that you can't properly validate a TLS connection, then, sure, you aren't doing what you think you're doing.
If the DNS zone isn't signed (it currently isn't, but I expect that it will be eventually -- possibly before the next rollout -- com adoption has hit 3.5%), then you could claim that someone could hijack DNS for it. Current DNS has a CAA record for github.com, limiting issuers to digicert.com and globalsign.com. A partial DNS hijack shouldn't result in a CA other than one of those two issuing certs, as they should use observation points to recognize a discrepancy -- and risk losing their CA issuing bit. A full DNS take-over OTOH is the same as someone compromising the PGP key. It proves someone either had access to the key to make a change, or had computing power to forge them. Similarly, contractual agreements should result in digicert and globalsign not issuing incorrect certs.