The Path to Packer Plugin SDK v1.0.0

[nywilken]

The Path to Packer Plugin SDK v1.0.0

The plugin architecture for Packer has proven successful since its inception but has introduced hurdles while working with HCL2 support.

In a recent change to github.com/zclconf/go-cty, a critical package for working with HCL2 types, a breaking change presented issues for Packer HCL2 development - refer to https://github.com/hashicorp/packer-plugin-sdk#135 for details.
The change to remove encoding/gob from zclconf/go-cty, which was an experiment for Terraform, made it impossible to upgrade Packer because of Packer's reliance on the encoding/gob serialization. The use of gob for HCL2 is a decision made early on to avoid major refactors to the Packer plugin architecture.

Looking into other plugin system implementations used within HashiCorp products such as Terraform, Nomad, and Waypoint, we found the standard based on hashicorp/go-plugin, gRPC, and msgpack - a native serialization format for zclconf/go-cty.

This series of proposals below list out several sequential changes for migrating from the bespoke plugin system in Packer from RPC with encoding/gob to using hashicorp/go-plugin with gRPC and msgpack.

The solution space for changing Packer's wire protocol involves temporarily patching the breaking change preventing the use of zclconf/go-cty v1.11.0+, augmenting the Packer SDC command to help plugin developers automate the transition,
the migration to hashicorp/go-plugin, and finally the implementation of a gRPC based plugin client/server to be used by Packer core and its plugins.

Logical workstreams are divided into development phases. Discussion of these topics can take place inside each pull-request.

Phase 1: Packer Plugin SDK 0.5.0 ✅ (Release Announcement v0.5.1)

The first phase resolves the encoding/gob issue blocking Packer and the Packer SDK from using updated versions of github.com/zclconf/go-cty and github.com/hashicorp/hcl/v2.

1. Fork github.com/zclconf/go-cty for Packer
   1. Add Packer SDK changes required for using the nywilken/go-cty fork.
   2. Add Packer core and plugins changes required for using the nywilken/go-cty fork.
2. Introduce packer-sdc fix sub-command for automating go.mod file changes to add/remove SDK dependencies.

Phase 2: Packer Plugin SDK 0.6.0

In the second phase the focus is on creating an adapter around Packer's dependency on gob to slowly migrate Packer's plugin system architecture to a more modern implementation.

1. Update Packer Plugin SDK to use go-pluginfor serving plugins over RPC.
2. Add gRPC plugin server to Packer Plugin SDK
   Use go-plugin to create a gRPC implementation of Packer's plugin server and client.
3. Add support for Packer client protocol negotiation to allow for both RPC and gRPC plugin communication within the SDK.

Phase 3: Packer Plugin SDK 1.0.0

The exact timeline for this phase will depend on feedback and Packer telemetry data from users. By this point, all plugins would be working with an updated SDK supporting RPC and gRPC.

Phase 4: Deprecate RPC communication for Packer plugins

Update Packer Plugin SDK to default to gRPC for all plugin communication.

[NorseGaud]

Thanks! If you can post updates here when the phases are reached, that would be helpful.

[nywilken]

Thanks! If you can post updates here when the phases are reached, that would be helpful.

@NorseGaud absolutely! We are working to get an announcement out to the discuss forum, as well. Hoping that between GitHub and Discuss folks can follow along throughout the process. Cheers.

[nywilken]

@NorseGaud we ran into a bit of snag with the v0.5.0 release so I refrained from posting on Discuss and the forum. However, we did update the release notes to indicate we have an issue and that folks should hold on updating.

[nywilken]

The Packer plugin SDK has been released as v0.5.1 and is ready for use. We created a discuss post for plugin developers with details on the fix.

With 0.5.1 out the door we will begin working on the go-plugin support.