Regression in 2.9.x - authentication with signed ssh key fails #1963

forsberg · 2022-01-03T09:25:36Z

I'm using signed ssh keys to authenticate to a host. This has worked flawlessly up until Paramiko 2.8.1, but with 2.9.0 and 2.9.1 it fails.

Example program:

import os
import sys
import paramiko
import logging

script_dir = os.path.dirname(os.path.abspath(__file__))

def try_connect(hostname):
    logging.basicConfig(level=logging.DEBUG, stream=sys.stderr)

    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    proxy_command = os.path.join(script_dir, "../../../ssh/proxy-command")

    sock = paramiko.ProxyCommand(f"{proxy_command} {hostname}")

    client.connect(hostname, 22, username="root", sock=sock, key_filename="/home/forsberg/.ssh/robot-test-cert.pub")

if __name__ == "__main__":
    try_connect(sys.argv[1])

The proxy command is just a netcat forwarding the TCP packets, and is probably not important anyway in this context.
robot-test-cert.pub is signed via ssh-keygen, and has the following metadata:

ssh-keygen -Lf ~/.ssh/robot-test-cert.pub 
/home/forsberg/.ssh/robot-test-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:JtAqrYvjJBLxkXDeSkXEuGyTR9uykjzaqlGIExjFi/g
        Signing CA: RSA SHA256:E5U81luuNJQIojmQVtDNfnKa4s89Q1gMTOIDqhk1jJE (using ssh-rsa)
        Key ID: "erik.forsberg"
        Serial: 1
        Valid: from 2022-01-02T09:36:22 to 2022-01-10T09:36:22
        Principals: 
                root
                ssh_gate
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Running this program with paramiko 2.8.1 gives the following output:

python ssh_via_paramiko hostname.example.com
DEBUG:paramiko.transport:starting thread (client mode): 0xb8e398b0
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.8.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_8.2
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.2)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256', 'curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256'] server key:['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa'] client encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] server encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] client mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: curve25519-sha256@libssh.org
DEBUG:paramiko.transport:HostKey agreed: ssh-rsa
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-rsa host key for hostname.example.com: b'4a16f769992de3607928b51c5d1decee'
DEBUG:paramiko.transport:Trying discovered key b'87a2c32a46a2be4f644fb8cc69f47707' in /home/forsberg/.ssh/robot-test
DEBUG:paramiko.transport:Adding public certificate /home/forsberg/.ssh/robot-test-cert.pub
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (publickey) successful!

However, with paramiko 2.9.1 I instead get the following:

python ssh_via_paramiko hostname.example.com
DEBUG:paramiko.transport:starting thread (client mode): 0x1fbd4c10
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.9.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_8.2
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.2)
DEBUG:paramiko.transport:=== Key exchange possibilities ===
DEBUG:paramiko.transport:kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group16-sha512, diffie-hellman-group18-sha512, diffie-hellman-group14-sha256
DEBUG:paramiko.transport:server key: rsa-sha2-512, rsa-sha2-256, ssh-rsa
DEBUG:paramiko.transport:client encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
DEBUG:paramiko.transport:server encrypt: chacha20-poly1305@openssh.com, aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com
DEBUG:paramiko.transport:client mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
DEBUG:paramiko.transport:server mac: umac-64-etm@openssh.com, umac-128-etm@openssh.com, hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, umac-64@openssh.com, umac-128@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
DEBUG:paramiko.transport:client compress: none
DEBUG:paramiko.transport:server compress: none
DEBUG:paramiko.transport:client lang: <none>
DEBUG:paramiko.transport:server lang: <none>
DEBUG:paramiko.transport:kex follows: False
DEBUG:paramiko.transport:=== Key exchange agreements ===
DEBUG:paramiko.transport:Kex: curve25519-sha256@libssh.org
DEBUG:paramiko.transport:HostKey: rsa-sha2-512
DEBUG:paramiko.transport:Cipher: aes128-ctr
DEBUG:paramiko.transport:MAC: hmac-sha2-256
DEBUG:paramiko.transport:Compression: none
DEBUG:paramiko.transport:=== End of kex handshake ===
DEBUG:paramiko.transport:kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Got EXT_INFO: {'server-sig-algs': b'ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com'}
DEBUG:paramiko.transport:Adding ssh-rsa host key for hostname.example.com: b'4a16f769992de3607928b51c5d1decee'
DEBUG:paramiko.transport:Trying discovered key b'87a2c32a46a2be4f644fb8cc69f47707' in /home/forsberg/.ssh/robot-test
DEBUG:paramiko.transport:Adding public certificate /home/forsberg/.ssh/robot-test-cert.pub
DEBUG:paramiko.transport:userauth is OK
DEBUG:paramiko.transport:Finalizing pubkey algorithm for key of type 'ssh-rsa-cert-v01@openssh.com'
DEBUG:paramiko.transport:Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
DEBUG:paramiko.transport:Server-side algorithm list: ['ssh-ed25519', 'sk-ssh-ed25519@openssh.com', 'ssh-rsa', 'rsa-sha2-256', 'rsa-sha2-512', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'sk-ecdsa-sha2-nistp256@openssh.com']
DEBUG:paramiko.transport:Agreed upon 'rsa-sha2-512' pubkey algorithm
INFO:paramiko.transport:Authentication (publickey) failed.
DEBUG:paramiko.transport:Trying discovered key b'd491164ce81ead130e0bc2843ca59ad4' in /home/forsberg/.ssh/robot-test
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/forsberg/ferro/ferroamp-os-continuous-deploy/ehub-target-tests/src/ehub_target_tests/ssh_via_paramiko.py", line 21, in <module>
    try_connect("hostname.example.com")
  File "/home/forsberg/ferro/ferroamp-os-continuous-deploy/ehub-target-tests/src/ehub_target_tests/ssh_via_paramiko.py", line 18, in try_connect
    client.connect(hostname, 22, username="root", sock=sock, key_filename="/home/forsberg/.ssh/robot-test-cert.pub")
  File "/home/forsberg/.virtualenvs/feos-cont-deploy/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/home/forsberg/.virtualenvs/feos-cont-deploy/lib/python3.8/site-packages/paramiko/client.py", line 678, in _auth
    key = self._key_from_filepath(
  File "/home/forsberg/.virtualenvs/feos-cont-deploy/lib/python3.8/site-packages/paramiko/client.py", line 598, in _key_from_filepath
    key.load_certificate(cert_path)
  File "/home/forsberg/.virtualenvs/feos-cont-deploy/lib/python3.8/site-packages/paramiko/pkey.py", line 652, in load_certificate
    raise ValueError(err.format(blob.key_type, self.get_name()))
ValueError: PublicBlob type ssh-rsa-cert-v01@openssh.com incompatible with key type ssh-dss

The text was updated successfully, but these errors were encountered:

powo · 2022-01-04T08:22:02Z

Have a similar issue with 2.9, with a ssh-rsa key and a legacy SSH Server, that does not send a list of supported algorithms:

...
DEBUG transport:1873 Finalizing pubkey algorithm for key of type 'ssh-rsa'
DEBUG transport:1873 Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
DEBUG transport:1873 Server-side algorithm list: [u'']
DEBUG transport:1873 Agreed upon 'rsa-sha2-512' pubkey algorithm
INFO transport:1873 Authentication (publickey) failed.
...

forsberg · 2022-01-04T08:27:09Z

@powo - I think you issue is #1961 rather than the problem in this ticket.

jun66j5 · 2022-01-05T11:34:18Z

I configured OpenSSH 8.4p1 with CA public keys and tested with paramiko 2.8.x and 2.9.x. Authentication with paramiko 2.8.x succeeded, however with 2.9.x failed, even if ssh serer isn't legacy.

OpenSSH auth.log

Jan  5 11:11:45 5c588f07c36e sshd[4192]: error: userauth_pubkey: type mismatch for decoded key (received 4, expected 0) [preauth]

jun66j5 · 2022-01-06T09:20:56Z

Root cause is that public key algorithm name while sending userauth request with signed RSA key should be rsa-sha2-512-cert-v01@openssh.com, however rsa-sha2-512 is used.

Tried to fix it: jun66j5/paramiko@3784df1ee.

RoniKish · 2022-01-20T11:16:18Z

This was already disccused on #1961
Using disabled_algorithms fixes the issue, look at the above for more info

jun66j5 · 2022-01-20T12:19:08Z

This was already disccused on #1961 Using disabled_algorithms fixes the issue, look at the above for more info

This issue occurs on OpenSSH 8.4p1 (not legacy) too. It is not caused by deprecated ssh-rsa algorithm.

11chri · 2022-02-09T13:22:22Z

I have the same issue and can confirm that the proposed fix #1963 (comment) would solve it. With this fix, I could establish an SSH connection.

Root cause is that public key algorithm name while sending userauth request with signed RSA key should be rsa-sha2-512-cert-v01@openssh.com, however rsa-sha2-512 is used.

Tried to fix it: jun66j5/paramiko@3784df1.

Just specifying disabled_algorithms as proposed in #1963 (comment) did not fix the issue.

This was already disccused on #1961 Using disabled_algorithms fixes the issue, look at the above for more info

jun66j5 mentioned this issue Jan 5, 2022

paramiko ssh server (2.9.0) always rejects authentication with rsa-sha2-256 and ssh-rsa publickey algorithms #1955

Closed

bendubuisson mentioned this issue Jan 17, 2022

Paramiko 2.9.x breaks auth with signed key ryhsiao/pysftp#5

Open

jun66j5 mentioned this issue Mar 8, 2022

raskey.py is hardcoded with "ssh-rsa" in places where we need support for rsa-sha2-256 and other versions #1997

Closed

11chri mentioned this issue Mar 11, 2022

Fix publickey authentication with signed RSA key (#1963) #1977

Closed

bitprophet closed this as completed in d25e5f3 Mar 18, 2022

jk464 mentioned this issue Aug 23, 2023

Fix Cert-based SSH Auth Issues in Paramiko StackStorm/st2#6017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression in 2.9.x - authentication with signed ssh key fails #1963

Regression in 2.9.x - authentication with signed ssh key fails #1963

forsberg commented Jan 3, 2022 •

edited

powo commented Jan 4, 2022

forsberg commented Jan 4, 2022

jun66j5 commented Jan 5, 2022

jun66j5 commented Jan 6, 2022

RoniKish commented Jan 20, 2022

jun66j5 commented Jan 20, 2022

11chri commented Feb 9, 2022

Regression in 2.9.x - authentication with signed ssh key fails #1963

Regression in 2.9.x - authentication with signed ssh key fails #1963

Comments

forsberg commented Jan 3, 2022 • edited

powo commented Jan 4, 2022

forsberg commented Jan 4, 2022

jun66j5 commented Jan 5, 2022

OpenSSH auth.log

jun66j5 commented Jan 6, 2022

RoniKish commented Jan 20, 2022

jun66j5 commented Jan 20, 2022

11chri commented Feb 9, 2022

forsberg commented Jan 3, 2022 •

edited