paramiko ssh server (2.9.0) always rejects authentication with rsa-sha2-256 and ssh-rsa publickey algorithms

[jun66j5]

I tried paramiko 2.9.0 for server purpose, Client with rsa-sha2-512 publickey algorithm works fine, however the server always rejects authentication with rsa-sha2-256 and ssh-rsa publickey algorithms.

Testing using the following command and demo_server.py:

$ ssh -i /tmp/user_rsa_key -o 'PubkeyAcceptedKeyTypes rsa-sha2-256' -o 'PreferredAuthentications publickey' -p 2200 robey@127.0.0.1

The server mode wrongly ignores the given rsa-sha2-256 in USERAUTH_REQUEST from the client and verity the signature with rsa-sha2-512.

DEB [20211224-10:52:46.504] thr=1   paramiko.transport: Compression: none
DEB [20211224-10:52:46.504] thr=1   paramiko.transport: === End of kex handshake ===
DEB [20211224-10:52:46.507] thr=1   paramiko.transport: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
DEB [20211224-10:52:46.510] thr=1   paramiko.transport: Switch to new keys ...
DEB [20211224-10:52:46.510] thr=1   paramiko.transport: Auth request (type=none) service=ssh-connection, username=robey
INF [20211224-10:52:46.510] thr=1   paramiko.transport: Auth rejected (none).
DEB [20211224-10:52:46.511] thr=1   paramiko.transport: Auth request (type=publickey) service=ssh-connection, username=robey
DEB [20211224-10:52:46.511] thr=1   paramiko.transport: Finalizing pubkey algorithm for key of type u'rsa-sha2-256'
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Server-side algorithm list: [u'']
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Agreed upon 'rsa-sha2-512' pubkey algorithm
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Finalizing pubkey algorithm for key of type 'ssh-rsa'
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Server-side algorithm list: [u'']
DEB [20211224-10:52:46.512] thr=1   paramiko.transport: Agreed upon 'rsa-sha2-512' pubkey algorithm
INF [20211224-10:52:46.512] thr=1   paramiko.transport: Auth rejected: invalid signature
INF [20211224-10:52:46.512] thr=1   paramiko.transport: Auth rejected (publickey).
DEB [20211224-10:52:46.513] thr=1   paramiko.transport: EOF in transport thread

I tried to temporally fix it: jun66j5/paramiko@518e02026.

[bitprophet]

Weird that this didn't show up in the tests, likely an edge case that got overlooked given the usual focus on client side mode. Thanks for the report!

[bitprophet]

Found a test that proves this, will dig into it based on your patch and get 2.9.1 out.

[bitprophet]

2.9.1 released for this. Thanks again!

[jun66j5]

Thanks for the quick response and fix! Confirmed the issue goes away with paramiko 2.9.1.

[ylangisc]

Can you please verify if the bug is partially fixed only as we still have issues to connect with Cyberduck and ssh-rsa to version 2.9.1? Refer to iterate-ch/cyberduck#12733 (comment).

[powo]

@ylangisc looks identical to #1961

[teunis90]

I can confirm that I still hit the same/similar bug on version 2.9.1 using RSA keys. If we switch to password login it does work.

Here's the debug trace:

Fails:

Versions:
Python 3.8.7
paramiko==2.9.1

root@00146b78fd9d:/app# ./test-ssh.py
start
DEBUG:paramiko.transport:starting thread (client mode): 0x167ff280
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.9.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-Cisco-1.25
INFO:paramiko.transport:Connected (version 2.0, client Cisco-1.25)
DEBUG:paramiko.transport:=== Key exchange possibilities ===
DEBUG:paramiko.transport:kex algos: diffie-hellman-group-exchange-sha1, diffie-hellman-group14-sha1
DEBUG:paramiko.transport:server key: ssh-rsa
DEBUG:paramiko.transport:client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-cbc, 3des-cbc, aes192-cbc, aes256-cbc
DEBUG:paramiko.transport:server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-cbc, 3des-cbc, aes192-cbc, aes256-cbc
DEBUG:paramiko.transport:client mac: hmac-sha1, hmac-sha1-96
DEBUG:paramiko.transport:server mac: hmac-sha1, hmac-sha1-96
DEBUG:paramiko.transport:client compress: none
DEBUG:paramiko.transport:server compress: none
DEBUG:paramiko.transport:client lang: <none>
DEBUG:paramiko.transport:server lang: <none>
DEBUG:paramiko.transport:kex follows: False
DEBUG:paramiko.transport:=== Key exchange agreements ===
DEBUG:paramiko.transport:Kex: diffie-hellman-group-exchange-sha1
DEBUG:paramiko.transport:HostKey: ssh-rsa
DEBUG:paramiko.transport:Cipher: aes128-ctr
DEBUG:paramiko.transport:MAC: hmac-sha1
DEBUG:paramiko.transport:Compression: none
DEBUG:paramiko.transport:=== End of kex handshake ===
DEBUG:paramiko.transport:Got server p (2048 bits)
DEBUG:paramiko.transport:kex engine KexGex specified hash_algo <built-in function openssl_sha1>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-rsa host key for 1.2.3.4: b'0ef9h412d03a152f934d1b06e84c41ba' # changed for security
DEBUG:paramiko.transport:Trying SSH key b'1df1aee321a9af1d34fc6f97bfa24fc9' # changed for security
DEBUG:paramiko.transport:userauth is OK
DEBUG:paramiko.transport:Finalizing pubkey algorithm for key of type 'ssh-rsa'
DEBUG:paramiko.transport:Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']
DEBUG:paramiko.transport:Server-side algorithm list: ['']
DEBUG:paramiko.transport:Agreed upon 'rsa-sha2-512' pubkey algorithm
INFO:paramiko.transport:Authentication (publickey) failed.
Traceback (most recent call last):
  File "./testje.py", line 50, in <module>
    ssh.connect('1.2.3.4', username='imp_ro', pkey=ssh_key, timeout=3)
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 766, in _auth
    raise saved_exception
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 666, in _auth
    self._transport.auth_publickey(username, pkey)
  File "/usr/local/lib/python3.8/site-packages/paramiko/transport.py", line 1634, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)
  File "/usr/local/lib/python3.8/site-packages/paramiko/auth_handler.py", line 258, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

Works:

Versions:
Python 3.8.3
paramiko==2.7.2

root@0f75ba5c03b8:/app# ./test-ssh.py
start
DEBUG:paramiko.transport:starting thread (client mode): 0x29cccdc0
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.7.2
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-Cisco-1.25
INFO:paramiko.transport:Connected (version 2.0, client Cisco-1.25)
DEBUG:paramiko.transport:kex algos:['diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-cbc', '3des-cbc', 'aes192-cbc', 'aes256-cbc'] client mac:['hmac-sha1', 'hmac-sha1-96'] server mac:['hmac-sha1', 'hmac-sha1-96'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group-exchange-sha1
DEBUG:paramiko.transport:HostKey agreed: ssh-rsa
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha1
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:Got server p (2048 bits)
DEBUG:paramiko.transport:kex engine KexGex specified hash_algo <built-in function openssl_sha1>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-rsa host key for 1.2.3.4: b'0ef9h412d03a152f934d1b06e84c41ba' # changed for security
DEBUG:paramiko.transport:Trying SSH key b'1df1aee321a9af1d34fc6f97bfa24fc9' # changed for security
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (publickey) successful!
command
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 0] Max packet out: 4096 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
end

Also tried with settings:

paramiko.Transport._preferred_kex = ('diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1')
paramiko.Transport._preferred_ciphers = ('aes128-ctr', 'hmac-sha1')

Which does change the Key exchange agreements, but I still get the same error:

...

DEBUG:paramiko.transport:=== Key exchange agreements ===
DEBUG:paramiko.transport:Kex: diffie-hellman-group-exchange-sha1
DEBUG:paramiko.transport:HostKey: ssh-rsa
DEBUG:paramiko.transport:Cipher: aes128-ctr
DEBUG:paramiko.transport:MAC: hmac-sha1
DEBUG:paramiko.transport:Compression: none
DEBUG:paramiko.transport:=== End of kex handshake ===

...

INFO:paramiko.transport:Authentication (publickey) failed.
Traceback (most recent call last):
  File "./testje.py", line 52, in <module>
    ssh.connect('195.238.86.6', username='imp_ro', pkey=ssh_key, timeout=3)
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 766, in _auth
    raise saved_exception
  File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 666, in _auth
    self._transport.auth_publickey(username, pkey)
  File "/usr/local/lib/python3.8/site-packages/paramiko/transport.py", line 1634, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)
  File "/usr/local/lib/python3.8/site-packages/paramiko/auth_handler.py", line 258, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

Code in test-ssh.py:

#!/usr/local/bin/python
import sys
import logging

from paramiko import AutoAddPolicy, RSAKey
from paramiko.client import SSHClient
from io import StringIO
import paramiko

root = logging.getLogger()
root.setLevel(logging.DEBUG)
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger("paramiko").setLevel(logging.DEBUG)

ssh_client = SSHClient()

ssh_client.load_system_host_keys()
ssh_client.set_missing_host_key_policy(AutoAddPolicy())
ssh_key_as_string = """-----BEGIN OPENSSH PRIVATE KEY-----
... secret ...
-----END OPENSSH PRIVATE KEY-----"""
ssh_key = RSAKey.from_private_key(StringIO(ssh_key_as_string))

with ssh_client as ssh:
    print('start')
    ssh.connect('1.2.3.4', username='myuser', pkey=ssh_key, timeout=3)
    print('command')
    stdin, stdout, stderr = ssh.exec_command('my ssh command')
    print('end')

Cisco device does not support ssh-dss, and password login is prohibited on production. Can I add anymore information to support resolving this issue?

[jun66j5]

I can confirm that I still hit the same/similar bug on version 2.9.1 using RSA keys. If we switch to password login it does work.

No. This issue is when using paramiko for server purpose. You've used as a client. Visit #1961.

[teunis90]

@jun66j5 thanks, missed that part. Will repost my message in #1961.