When I run git push from a Jenkins-initiated shell, I get an error 'SSH-2.0-OpenSSH_9.6' is not recognized as an internal or external command

[pdcaren]

• [X ] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

64 bit
$ git --version --build-options

git version 2.44.0.windows.1
cpu: x86_64
built from commit: ad0bbff
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows 2022 server
$ cmd.exe /c ver

Microsoft Windows [Version 10.0.20348.2340]

 - What options did you set as part of the installation? Or did you choose the
   defaults?

One of the following:

type "C:\Program Files\Git\etc\install-options.txt"
type "C:\Program Files (x86)\Git\etc\install-options.txt"
type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
type "$env:USERPROFILE\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

It works when running directly as a user, but fails when running from a jenkins ssh agent

Details

CMD (also fails in powershell)

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git push -v -u origin --all 

 - What did you expect to occur after running these commands?

I expected a push to happen using the SSH remote

 - What actually happened instead?

F:\Jenkins\workspace\migrate\DevSecOps\RandoMaker>git push -v -u origin --all 
15:25:59  Pushing to <sshurl>
15:26:00  'SSH-2.0-OpenSSH_9.6' is not recognized as an internal or external command,
15:26:00  operable program or batch file.

 - If the problem was occurring with a specific repository, can you provide the
   URL to that repository to help us with testing?

it's a private repo on ADO

[dscho]

It's hard to deduce from the provided information what is going on.

The problem might be a configuration setting. Look at the output of git config -l, in particular watching out for things like core.sshCommand.

Another possible way to find out more is to call set GIT_TRACE=1 and/or set GIT_TRACE2_PERF=1 before re-running the push.

[pdcaren]

So I did the GIT_TRACE commands.

10:08:09  bcbst\si_jenkins_t@WN002473 F:\Jenkins\workspace\migrate\DevSecOps\RandoMaker>git config -l 
10:08:10  10:08:09.954806 exec-cmd.c:244          trace: resolved executable dir: F:/ProgramFiles/Git/mingw64/bin
10:08:10  10:08:10.037822 common-main.c:55             | d0 | main                     | version      |     |           |           |              | 2.42.0.windows.2
10:08:10  10:08:10.037822 common-main.c:56             | d0 | main                     | start        |     |  0.083687 |           |              | git.exe config -l
10:08:10  10:08:10.074813 ..._win32_process_info.c:120 | d0 | main                     | data_json    | r0  |  0.120679 |  0.120679 | process      | windows/ancestry:["git.exe","cmd.exe","cmd.exe","java.exe","cmd.exe","sshd.exe","sshd.exe","sshd.exe","services.exe","wininit.exe","(truncated)"]
10:08:10  10:08:10.083814 repository.c:143             | d0 | main                     | def_repo     | r1  |           |           |              | worktree:F:/Jenkins/workspace/migrate/DevSecOps/RandoMaker
10:08:10  10:08:10.083814 git.c:463               trace: built-in: git config -l
10:08:10  10:08:10.083814 git.c:464                    | d0 | main                     | cmd_name     |     |           |           |              | config (config)
10:08:10  diff.astextplain.textconv=astextplain
10:08:10  filter.lfs.clean=git-lfs clean -- %f
10:08:10  filter.lfs.smudge=git-lfs smudge -- %f
10:08:10  filter.lfs.process=git-lfs filter-process
10:08:10  filter.lfs.required=true
10:08:10  http.sslbackend=openssl
10:08:10  http.sslcainfo=F:/ProgramFiles/Git/mingw64/etc/ssl/certs/ca-bundle.crt
10:08:10  core.autocrlf=true
10:08:10  core.fscache=true
10:08:10  core.symlinks=false
10:08:10  pull.rebase=false
10:08:10  init.defaultbranch=main
10:08:10  credential.helper=manager
10:08:10  credential.https://dev.azure.com.usehttppath=true/
10:08:10  credential.credentialstore=dpapi
10:08:10  http.proxy=[http://webgateway.bcbst.com:80](http://webgateway.bcbst.com/)
10:08:10  core.repositoryformatversion=0
10:08:10  core.filemode=false
10:08:10  core.bare=false
10:08:10  core.logallrefupdates=true
10:08:10  core.symlinks=false
10:08:10  core.ignorecase=true
10:08:10  remote.origin.url=git@ssh.dev.azure.com:v3/BCBST-EIT/DevSecOps/RandoMaker
10:08:10  remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
10:08:10  10:08:10.089813 ..._win32_process_info.c:168 | d0 | main                     | data_json    | r1  |  0.136102 |  0.136102 | process      | windows/memory:{"PageFaultCount":9429,"PeakWorkingSetSize":7217152,"PeakPagefileUsage":35684352}
10:08:10  10:08:10.089813 git.c:723                    | d0 | main                     | exit         |     |  0.136197 |           |              | code:0
10:08:10  10:08:10.090813 trace2/tr2_tgt_perf.c:217    | d0 | main                     | atexit       |     |  0.136292 |           |              | code:0
10:08:10  [Pipeline] bat
10:08:10  
10:08:10  bcbst\si_jenkins_t@WN002473 F:\Jenkins\workspace\migrate\DevSecOps\RandoMaker>git push -v -u origin --all 
10:08:10  10:08:10.527835 exec-cmd.c:244          trace: resolved executable dir: F:/ProgramFiles/Git/mingw64/bin
10:08:10  10:08:10.604832 common-main.c:55             | d0 | main                     | version      |     |           |           |              | 2.42.0.windows.2
10:08:10  10:08:10.604832 common-main.c:56             | d0 | main                     | start        |     |  0.077527 |           |              | git.exe push -v -u origin --all
10:08:10  10:08:10.637831 ..._win32_process_info.c:120 | d0 | main                     | data_json    | r0  |  0.110027 |  0.110027 | process      | windows/ancestry:["git.exe","cmd.exe","cmd.exe","java.exe","cmd.exe","sshd.exe","sshd.exe","sshd.exe","services.exe","wininit.exe","(truncated)"]
10:08:10  10:08:10.643833 repository.c:143             | d0 | main                     | def_repo     | r1  |           |           |              | worktree:F:/Jenkins/workspace/migrate/DevSecOps/RandoMaker
10:08:10  10:08:10.645834 git.c:463               trace: built-in: git push -v -u origin --all
10:08:10  10:08:10.645834 git.c:464                    | d0 | main                     | cmd_name     |     |           |           |              | push (push)
10:08:10  10:08:10.648839 builtin/push.c:401           | d0 | main                     | region_enter | r1  |  0.121032 |           | push         | label:transport_push
10:08:10  10:08:10.650833 transport.c:1360             | d0 | main                     | region_enter | r1  |  0.122968 |           | transport_pu | ..label:get_refs_list
10:08:10  10:08:10.650833 run-command.c:726            | d0 | main                     | child_start  |     |  0.123654 |           |              | ....[ch0] class:transport/ssh argv:[ssh git@ssh.dev.azure.com 'git-receive-pack '\''v3/BCBST-EIT/DevSecOps/RandoMaker'\''']
10:08:10  10:08:10.650833 run-command.c:659       trace: run_command: unset GIT_PREFIX; ssh git@ssh.dev.azure.com 'git-receive-pack '\''v3/BCBST-EIT/DevSecOps/RandoMaker'\'''
10:08:10  Pushing to ssh.dev.azure.com:v3/BCBST-EIT/DevSecOps/RandoMaker
10:08:10  'SSH-2.0-OpenSSH_9.4' is not recognized as an internal or external command,
10:08:10  operable program or batch file.

There's still nothing remotely pointing to why it's using that as a command from Jenkins.
Also should there be a core.sshCommand?

[dscho]

10:08:10 10:08:10.650833 run-command.c:659 trace: run_command: unset GIT_PREFIX; ssh git@ssh.dev.azure.com 'git-receive-pack '''v3/BCBST-EIT/DevSecOps/RandoMaker''''

Okay, so that means ssh is called.

10:08:10 Pushing to ssh.dev.azure.com:v3/BCBST-EIT/DevSecOps/RandoMaker
10:08:10 'SSH-2.0-OpenSSH_9.4' is not recognized as an internal or external command,
10:08:10 operable program or batch file.

And this makes it look as if it is not Git's ssh.exe that is picked up, maybe it is not there, or it is not in the PATH at that point.

There's still nothing remotely pointing to why it's using that as a command from Jenkins. Also should there be a core.sshCommand?

It might help to configure it, hard-coding the path to Git for Windows' ssh.exe. But that might not work as desired, either, because SSH needs access to some private key to be able to authenticate.

But you could at least write a shell script that outputs all kinds of interesting information such as the contents of the $PATH variable, and/or the output of type -p ssh, before calling exec ssh "$@", and then configure core.sshCommand to point to that script. That would not fix it, but it could lead the path to help you diagnose better what is going wrong in your setup.