Add packer_test package for blackbox tests
#12983
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The mini plugin implemented here is a minimal Packer plugin that we use for acceptance testing Packer core. There's a few components exposed so we can write templates using it, and make sure Packer interprets it all as it should, and runs/errors as we expect it.
Acceptance testing, i.e. running Packer core commands in a controlled environment and ensuring the behaviour is consistent to what we expect/document, is not something we have a robust and usable framework for at the moment. This commit is a proposal for a base testing framework of the sort, that is meant to be shipped with packer core, and which will eventually host most of the tests we currently do in command where we mock an environment.
Since compiling plugins is quick, but each invocation still takes a bit of time, we run those compilation jobs in parallel to shave of a couple seconds from a test run.
Creating the Name() function for every gadget we have is superfluous a bit, as we're essentially parroting the name of the test itself as implementation for the function. So instead of requiring every checker implements `Name', we now default to returning the type name, but if the Name function exists for the checker, we invoke it and return the value for that function. This allows us to only define the function where needed, and not systematically.
When creating a temporary plugin directory, we had to build a cleanup function, which could be as simple as `os.Mkdirall` without any kind of warning that a directory failed to be cleaned-up, or we could do some more work in order to report the possible issues around this. That code would quickly be redundant, as there's not a ton of variability in the code that can be written for this step, so we abstract it through a pre-defined cancellation function which can be safely defer invoked.
The loading_test had been added to the repository at first, but were not versioned at that time, making those tests impossible to run on CI.
The CopyFile function is essentially a go recreation of the `cp' command, which copies one file from a source path to a destination directory or file. This can be used for several tests in the future.
The PluginVersionConfig structure was first introduced when building the early versions of the test package, but it was an unnecessary abstraction over go-version.Version. So we remove that structure definition, and instead we directly use the version for building those temporary plugins.
Since the MakePluginDir function takes the TestSuite as receiver, we don't need to additionally pass in a reference to testing.T, since the test suite already contains one instance, and offers a function to get it from.
The NewPluginBuildConfig function was essentially a shortcut to `version.Must(version.NewSemver(v))', which is superfluous at this point, we can directly pass the version string to BuildSimplePlugin and let that function do the creation/check.
When running a PackerCommand for acceptance tests, we generally run the test on a temporary plugin directory, populated by test plugins. Setting that temporary directory means we need to set the environment variable, which while it could be easier with a constant for the name for example, isn't too straightforward. Therefore for those tests we add a new function for PackerCommand so that it automatically sets that envvar for the current command.
When running "Assert" on a packer command, we run a series of checkers on the command's output/error code, which provoke test failures if they fail. Panic checking used to be part of Run, but in the end this would make more sense to have that as a regular checker if asserting the results of a packer command, so that's the approach we adopt with this commit.
Calling BuildSimplePlugin for a particular version used to mean that we had to build it regardless of if it was previously done or not. This commit changes this behaviour so that it checks first that the plugin wasn't pre-compiled, and if it was, we immediately return.
MakePluginDir used to only load plugins that were precompiled at the start of the tests, but now when invoked with any list of plugins, this will attempt to compile plugins one-by-one, so we don't need to modify the tests in several places when running tests. There's still value in compiling the plugins in advance though: as they run in parallel, they all get compiled at once, so we shave off a few seconds from the test run.
When building the temporary plugin directory for a test, we didn't check that the LoadPluginVersion call succeeded and returned a path, which may cause errors down the line when attempting to install the plugin. To avoid this problem, we do the check at that time, and immediately fail if a plugin isn't found.
This commit adds a few scenarios of plugin installations to the test suite, in order to document and ensure we behave appropriately when installing pre-releases/metadata.
Since sometimes we want to check for matches, and sometimes we want to check for a lack of match, we add one more option for the Grep gadget: inverse. This essentially replicates `grep -v`, and will succeed only if the regex provided did NOT match on the requested streams.
The ExpectedInstalledName function returns the expected full name of a plugin binary after it's installed. This is used for tests that need to copy the binary to some place before they can run commands and ensure the logic for managing plugins conforms to the docs/specs.
The WriteFile function creates a new file to the specified location, and writes some contents to it.
When writing tests, one may need to write a one-off checker for a packer command that ran, without having to completely implement the Checker interface. This commit introduces a generic CustomChecker implementation (i.e. a function) that can be one-off implemented by developers if their test doesn't fit the existing gadgets, and the need is not generic/reusable enough to justify introducing a new gadget for other users.
The pipe checker is an attempt at replicating how one would go to write commands on a CLI and piping them together, coupled with a `test`.
When running a pipeline on a command's output, a simple check is making sure the pipeline returned something empty or not. This is the goal of those two implementations, basically either the input is empty as expected, or it errors, and the reverse.
As a common use case in console-oriented pipelines, we check that a specific command returned a certain number of lines. With the combination of LineCount and Compare, we can do exactly this.
By default PackerCommands are run with PACKER_LOG=1. If for any reason we don't want that, we can remove it from the environment so we only see the user-facing logs.
As we've introduced pipelines, we can use those to compose a version of grep that doesn't have specific logic. Besides, this refactor allows us to expose grep as a function with variadic options, so this makes it more concise and clear to assert an input with Grep.
The name and semantics were a bit unclear with how they were previously named, so this commit changes the name of those functions so it's clearer they are expecting something, and what they're expecting.
For consistency with other gadgets like Grep, we make the MustSucceed and MustFail gadgets private with a function to return an instance of it.
When troubleshooting a pipeline for a test, it can be useful to print the input out without necessarily preventing the pipeline to work. The Tee gadget is exactly made for this purpose, the input of the Tee is printed out through `t.Log`, and the input is forwarded to the next step in the pipeline.
Instead of manually creating and populating a PipeChecker, we add functions to do this.
Since the command object is created using the TestSuite as argument, we can keep a reference to the test suite's scoped T() instance for the command's lifecycle, and reuse it later during `Assert`, instead of needing to pass it as argument when invoking the function.
Since the tester plugin used for blackbox testing is called packer-plugin-tester, we rename the directory it's stored in to plugin_tester.
If for some reason we only want to run a test on either stream without doing some manipulation beforehand, we can run a PipeChecker, however these would error if no pipe gadget was defined, preventing this use-case. Instead of errorring then, this commit just ignores if no pipe is present, as none is required for the test to run.
The ExpectedInstalledName function used to compute the expected name of a plugin binary for a given version, based on the invoker's environment, used to cleanup the version string passed in parameter of the function, which could be problematic. Besides the logic applied would produce some invalid binary names as the prerelease plugin would not have a `-` separator, so the resulting plugin would be ignored, and we couldn't test metadata rejection with this logic. This commit therefore changes how the function works: the version string is still parsed to account for manipulation errors, but the string is left as-is for the final binary name.
Installing a plugin manually to a directory is something needed for some tests, especially those not relying on packer commands to install plugins as they reject/correct the path/version. Therefore this function is introduced so we have an easy way to install a binary as a plugin somewhere on the provided plugin directory.
If a plugin is installed with a non-canonical version in its name (e.g. 01.01.01), Packer rejects it with a message to that effect in stderr, so we add a test for this use-case.
Plugins with metadata information in their file name (i.e. v1.0.0+metadata) should be ignored by Packer as they could introduce ambiguity since the metadata is free-form, so we add that test to make sure Packer behaves coherently.
As we're introducing a --ignore-prerelease-plugins flag to both the validate and build subcommands, we need to make sure they work as we expect it to, so we add a test case for that.
Since on Windows extensions are mandatory in order to have something executable, we compile Packer with a `.exe` suffix during tests, so we can use the executable afterwards to run tests with.
Windows relies on the `TMP` (or alternatives) being set in the environment in order to be able to create temporary directories and files. If this is not set, the `os.TempDir` function defaults on the windows installation root directory (typically C:\Windows), leading to permission errors when running Packer in the context of a test, as we're installing plugins in a temporary directory. To avoid this problem, we get the current setting from the test's invocation environment, and forward it to the subcommand we execute for our tests.
When manually installing a plugin to the plugin directory, we compute a SHA256SUM file from the plugin binary, and install it alongside it so we can test the loading process for Packer. In the introduction of the function, we added a check that if we were running on Windows, we'd remove the extension of the sumfile's name before writing it. This is actually not necessary (and breaks the loading logic) as Packer looks for the name of the plugin with extension, followed by _SHA256SUM in order to compare the effective digest of the file to the one written to this file. Since this prevents the tests that use this function from succeeding in a Windows environment, we remove this extra step.
When building a pipeline to count the number of lines returned by Packer, it can be a bit cumbersome to have to chain the calls to MkPipeCheck to do that check, so we add one convenience function for the simplest case: counting the number of lines on stdout, without any kind of filtering.
The installation with a metadata part in the version for a plugin had one test that relied on the plugin directories being populated with packer plugins install --path. This could change in the future, while the command should remain functional, so we explicitely call it in the test instead of through the function that creates/populates a temp plugin dir.
To make sure we do scrub the metadata in the plugin name when installing it from a local binary, we add a test that does that installation with both alternatives: 1.0.0-dev and 1.0.0-dev+metadata, which should result in only one alternative being installed (the last one that succeeded).
These changes include a series of test cases for validating packer init
using the force and upgrade flag. Include in this test is a test case
for validating the plugin installation error when init encounters a
plugin whose reported version does not match the version within the
plugin name.
```
--- PASS: Test_PackerCoreSuite (18.66s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce (4.91s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce/installs_any_missing_plugins (2.76s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce/reinstalls_plugins_matching_version_constraints (2.14s)
--- PASS: Test_PackerCoreSuite/TestPackerInitUpgrade (3.70s)
--- PASS: Test_PackerCoreSuite/TestPackerInitUpgrade/upgrades_a_plugin_to_the_latest_matching_version_constraints (2.02s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithMixedVersions (1.96s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithMixedVersions/skips_the_plugin_installation_with_mixed_versions_before_exiting_with_an_error (1.96s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource (1.22s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/try_installing_from_a_non-github_source,_should_fail (0.07s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/manually_install_plugin_to_the_expected_source (0.59s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/re-run_packer_init_on_same_template,_should_succeed_silently (0.55s)
```
| required_plugins { | ||
| tester = { | ||
| source = "github.com/sylviamoss/comment" | ||
| version = ">= 0.2.22, <0.2.25" # plugin describe reports 1.0.0 so init must fail |
There was a problem hiding this comment.
This one may be problematic, the 0.2.24 release has different versions depending on the OS, not sure how it was built, but as it stands on linux amd64 the version reports 0.2.24 so it succeeds.
We could either make another borked release with a consistent behaviour on all OSes, or we could manually patch the release with the mismatch so the test behaves consistently on all platforms
Since legacy config files may declare single plugin components, we need to warn that they're not supported anymore. This is in process of being PR'd into main, but to ensure the config works as intended and we do get the error, we add some tests for that.
'%d' gets output as-is in the temporary workdir we create. This is unnecessary and could even be problematic in some cases, so we scrub it from the MkdirTemp call.
lbajolet-hashicorp
force-pushed
the
acc_test_logic
branch
from
db4ddae
to
2ab83b5
Compare
Remotely installing plugins with a pre-release as part of the constraint is unsupported by Packer, and should error if that happens. This test makes sure that this gets treated as an error if that's the case, even before attempting to connect to the source.
lbajolet-hashicorp
force-pushed
the
acc_test_logic
branch
from
2ab83b5
to
e5f70ae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new model for running blackbox tests with Packer.
By blackbox testing we mean invoking the executable in a realistic but controlled environment, so we can run tests based on a user's perspective, and run shell-like pipelines on the command's outputs.
This allows us to ensure Packer behaves consistently between versions, and makes sure if there's a change in a test, that it is covered and conscious.
Since we're changing how plugins are loaded with Packer 1.11, the tests included in this branch are essentially covering those changes, with eventually more and more tests that can/will be migrated to this architecture.