cli: allow separate test suites with separate local-validators (when testing on localnet)

[paul-schaaf]

No description provided.

[paul-schaaf]

@tomlinton

re: #1261 (comment)

The list of test validator settings is here. I don't really see anything there that needs to change per test outside of account, clone, and maybe the slots_per_epoch and warp_slot option (and even then, you can probably get away with just loading the data you need under different addresses). The CLI should control all the port, path, etc options to allow for test/ledger isolation and parallelism.

account, clone, and maybe the slots_per_epoch and warp_slot

those alone justify a separate config for me.

(and even then, you can probably get away with just loading the data you need under different addresses)

cause this isn't clean. one of the benefits you get by splitting up the tests into separate suites is that you get truly separate validators that are not polluted with unintended state. this would make that separation less useful.

So maybe for all other options, we have a tests/Test.toml that specifies those and then if you really want to adjust those options you can override them in your tests/my-test/Test.toml?

[tomlinton]

cause this isn't clean. one of the benefits you get by splitting up the tests into separate suites is that you get truly separate validators that are not polluted with unintended state. this would make that separation less useful.

It is a tradeoff, but I prefer that to the configuration complexity. I can't really think of many scenarios where I'm worried about having extra accounts loaded in my test validator. Acknowledging --clone will slow everything down significantly for every separate validator, I think we should discourage use of --clone in favour of --account anyway.

So maybe for all other options, we have a tests/Test.toml that specifies those and then if you really want to adjust those options you can override them in your tests/my-test/Test.toml?

Yea that works. It'd be nice if this was non breaking, and we could just keep the test settings in the root Anchor.toml and then walk each test subdirectory and look for overrides. If we can do that then I see no downside to this, as it is completely optional.

If users define [[test.validator.accounts]] at the root they won't have away to remove that state via a subsequent inherited configuration. That is OK, as it can be avoided by putting the test files in the specific nested directories.

[paul-schaaf]

Acknowledging --clone will slow everything down significantly for every separate validator, I think we should discourage use of --clone in favour of --account anyway.

Assuming youre using the mainnet url, using --clone ensures your test is as close to the real environment though whereas --account does not. I'm wondering whether there is a way to make anchor cache cloned accounts and use --account from then on. Then you could have faster CI while youre developing and a one time --clone at the end of whatever PR youre making to your project to make sure it really all still works. But this is probably something for a different issue.

Armani and I discussed this offline and up with the following design for the dir structure. We basically do what tsconfig.json does which is extending a base file. So when youre creating a new repo you have the following structure:

.
Anchor.toml
tests
--- suite1
------ Test.toml
------ test.ts

where the Test.toml inside the suite1 dir just looks like this:

extends = ["../../Anchor.toml"]

that way you can pull your config from anywhere you like, whether it's the Anchor.toml or a separate Test.toml or even multiple .toml files or you dont pull any other config at all and specify it in the Test.toml.

this is nice cause you can upgrade easily without moving config, and it's very transparent what youre importing vs overriding which is not as transparent.

Thinking ahead, this also allows for options like accounts = ["./accounts/"] where the accounts can be both the already "compiled" .json files or anchor yml/toml account files

[tomlinton]

Sounds good 👍

[paul-schaaf]

clsoed by #1681