pause_before_connecting does not pause before connecting

[tbpower]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

I was deploying a simple template to build a VM on a VMware systems and ran into a problem with the build timing out before the VM was fully set up and running. When this occurs the SSH daemon isn't running yet so trying to connect to it too early cannot succeed. We expected to use the pause_before_connecting as a sleep period to wait until the SSH daemon was started and listening on its port. It never gets to the pause timer because it timed out trying to do its initial (and unneeded) connection.

This is really a duplicate of #7430 that was submitted a couple of years ago and the "solution" was to update the documentation to describe how the option actually behaves. I believe that if it works incorrectly to begin with, simply documenting that behavior is not a solution at all. The option states "pause BEFORE connecting", not Connect -> log off -> pause -> and then reconnect.

The way it currently functions makes the option completely useless. I cannot imagine any scenario where I would ever want to connect via SSH, wait, and then reconnect. If you're going to have an option called pause_before_connecting then please make it actually pause before connecting.

Reproduction Steps

Steps to reproduce this issue:
Produce on a system that takes longer than 3 minutes to download Ubuntu ISO and build the VM.

Packer version

Packer v1.8.2

Simplified Packer Template

{

  "builders": [
    {
      "type": "vsphere-iso",
      "convert_to_template": "true",
      "host": "mysys.i.mydomain.com",
      "username": "administrator@i.mydomain.com",
      "password":"mypwd",
      "vcenter_server": "vcenter7.i.mydomain.com",
      "insecure_connection": "true",

      "iso_checksum": "none",
      "iso_url" : "https://releases.ubuntu.com/20.04/ubuntu-20.04.4-live-server-amd64.iso",

      "vm_name": "testsys",
      "cpus": "4",
      "ram": "4096",
      "datastore" : "datastore1",
      "datacenter" : "Testing",
      "storage": [
        {
          "disk_size": "40000"
        }
      ],
      "network_adapters": [
        {
          "network": "VLAN 111"
        }
      ],

      "pause_before_connecting": "5m",
      "ssh_password": "mypwd",
      "ssh_username": "mypwd",
      "ssh_timeout": "20m",

      "boot_command": [
      "<wait40><enter><>wait><enter><wait><enter><wait><enter><wait><enter><wait><enter>",
      "<wait><down><wait><down><wait><down><wait><down><wait><down><wait><enter><wait><enter><wait><down><enter><wait>",
      "admin <enter> admin <enter> admin <enter> admin <enter> admin <enter><enter><wait><enter>",
      "<enter><wait><down><wait><down><wait><enter><pageDown><wait><down><wait><enter>"
    ]

    }
  ]
}

Operating system and Environment details

Client: Ubuntu 20.04 workstation
VMware: vcenter 7.0.0

Log Fragments and crash.log files

==> vsphere-iso: File /home/myuser/.cache/packer/f56bff3df62a8576625321cf140b318c53705f7f.iso already uploaded; continuing
==> vsphere-iso: File [datastore1] packer_cache//f56bff3df62a8576625321cf140b318c53705f7f.iso already exists; skipping upload.
==> vsphere-iso: Creating VM...
==> vsphere-iso: Customizing hardware...
==> vsphere-iso: Mounting ISO images...
==> vsphere-iso: Adding configuration parameters...
==> vsphere-iso: Set boot order temporary...
==> vsphere-iso: Power on VM...
==> vsphere-iso: Waiting 10s for boot...
==> vsphere-iso: Typing boot command...
==> vsphere-iso: Waiting for IP...
==> vsphere-iso: IP address: 192.168.121.25
==> vsphere-iso: Using SSH communicator to connect: 192.168.121.25
==> vsphere-iso: Waiting for SSH to become available...
==> vsphere-iso: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> vsphere-iso: Clear boot order...
==> vsphere-iso: Power off VM...
==> vsphere-iso: Destroying VM...
Build 'vsphere-iso' errored after 2 minutes 57 seconds: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

Set the env var PACKER_LOG=1 for maximum log detail.

[lbajolet-hashicorp]

Hi @tbpower,

I understand the option in question may be misleading, but it is actually working as intended: the problem it attempts to solve is that some distributions will boot sshd, and then perform the rest of the system boot sequence, which can be long in some cases (if for example the source image has to boot databases, services, mount points, etc.).
This can prevent the provisioner from working afterwards as it may need some of these long-to-boot services, in which case this option lets practitioners express this need to wait before provisioning, although it is already possible to connect.

In your case, this is a matter of the behaviour of the plugin SDK, where upon running a build, the ssh communication is established directly before any provisioning occurs (with a few exceptions I will agree, but generally this is expected).
As such, since ssh is primordial for any action to occur, your best bet would be to use the ssh_timeout or the ssh_handshake_attempts options to control for how long the connection attempts will be performed.

Please do note that in the current implementation, both options have a default value set, and the shorter of the two will have precedence over the other, as such you may want to define both at the same time until this behaviour changes (PR: hashicorp/packer-plugin-sdk#116).

Now to get back to the pause_before_connecting, I will concur with you that this behaviour is surprising considering the name of the option, and maybe there's a better way to convey the expected behaviour. I don't think we will remove it altogether just yet as people may be using it and we'd be breaking their templates without warning, but I guess we should discuss what possibilities we have here, so thanks for bringing that up to our attention.

[github-actions]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-749