compiletest, a general purpose completion tester

[epage]

I saw compiletester and thought that'd be handy, so I went and created completest. Its not published yet because I've not yet verified if my changes work yet.

Besides sharing completion testing between parsers like clap and bpaf, this can help with

• people testing hand-written completions (like in cargo)
• people doing end-to-end testing of dynamic completions (e.g. Parser::complete)

I'm also going to experiment with plugging this into trycmd so you can write completion tests in markdown file which I hope will make adding new completion tests lighter weight and make maintaining existing ones have lower overhead. For example, I can see having all tests for one shell in a single markdown file and then you can diff the different shell implementations.

[pacak]

I don't mind making it into a separate project but I was planning to change some things:

• some controlled amount of concurrency and reuse of the shell if one configuration is tested with multiple inputs
• ANSI sequences - currently they are stripped but it should be possible to handle them while rendering HTML. Colorizing completions is PITA but possible. Then there's fish.
• to actually test shells you want to have shells installed, of supported version range and expected configuration. To run in bpaf CI I was thinking about using something like nix. While it might be outside of the completest scope anyone using it will have to do it in some way or suffer the fun. I'm still exploring the options here and currently I'm working on documentation rewrite for bpaf.
• speaking of - documentation, how this project should be used. In my current workflow comptester configures the shells to run completion with all the examples I'm testing, changes the configuration when needed, etc, runs/complains about mismatches, but also generates files for documentation.

How do you see the scope for completest?

[epage]

some controlled amount of concurrency

What are you thinking of? I'm not too sure where concurrency would cause problems.

reuse of the shell if one configuration is tested with multiple inputs

Would this offer much in the way of gains?

ANSI sequences - currently they are stripped but it should be possible to handle them while rendering HTML. Colorizing completions is PITA but possible. Then there's fish.

I definitely see value in this. I want to pull the escape code -> svg logic out of https://docs.rs/term-transcript/latest/term_transcript/ for some of my other CLi testoing tools

to actually test shells you want to have shells installed, of supported version range and expected configuration. To run in bpaf CI I was thinking about using something like nix. While it might be outside of the completest scope anyone using it will have to do it in some way or suffer the fun. I'm still exploring the options here and currently I'm working on documentation rewrite for bpaf.

Documenting a happy path would be helpful

speaking of - documentation, how this project should be used. In my current workflow comptester configures the shells to run completion with all the examples I'm testing, changes the configuration when needed, etc, runs/complains about mismatches, but also generates files for documentation.

I see it mostly be agnostic of this. Right now, to create a runtime will re-initialize it (the dot files). I can see adding a constructor that skips this.

How do you see the scope for completest?

For now, roughly what is in the code base though I can see that evolving with time.

One example is I'm considering pulling over my nushell completion code. The difference with this is that it runs nushell in-process calling its APIs. So its not a full end-to-end test but its pretty close still. Granted, if we want all of the escape codes being processed, that would probably want to move us to end-to-end.

[pacak]

What are you thinking of? I'm not too sure where concurrency would cause problems.
Would this offer much in the way of gains?

Shell takes about half a second to start and start replying but once it starts replies are pretty fast, from my devlops past I remember that having too much fancy shell activity so limiting concurrency is probably a good idea, especially on weak CI machines. Reusing should lower the CI time from half a second per test to half a second per test plus some milliseconds per test case.

I see it mostly be agnostic of this. Right now, to create a runtime will re-initialize it (the dot files). I can see adding a constructor that skips this.

I see. With dynamic completion it's a bit more tricky, binary that generates completion snippet needs to be up to date (cargo run works here but it's inefficient when it is called concurrently and needs to compile multiple binaries) and in $PATH so snippet an call it cauwugo run can propagate completion requests for bpaf apps, but cargo run cannot.

For now, roughly what is in the code base though I can see that evolving with time.

Sounds good.

[epage]

I see. With dynamic completion it's a bit more tricky, binary that generates completion snippet needs to be up to date (cargo run works here but it's inefficient when it is called concurrently and needs to compile multiple binaries) and in $PATH so snippet an call it cauwugo run can propagate completion requests for bpaf apps, but cargo run cannot.

In my tests, I'm looking at generating the completions as part of the test. You could alternatively keep using a build.rs or you could use snapshot testing to check-in the content.

So the hope is you can build the solution that works for you.

[pacak]

Yea, should be possible. Will see once I get back to it. Do you already have any code that uses it?

[epage]

I have a draft PR up that still needs a lot of iteration (code reuse, apply to all shells, etc)

clap-rs/clap#5026

[pacak]

Hmm...

Dependencies are getting crazy assert-rs/completest#3.

Having a concrete shell version is nice, other shells can't be installed the same way - this means you still have to manage external non rust dependencies, but also it takes 90 seconds to compile everything.

And this I think is a step in the wrong direction: assert-rs/completest#4. Instead of scrubbing existing environment from anything that can cause variations - it is better to make one that is known to be clean.

precmd_functions=\"\"  # avoid the prompt being overwritten

[epage]

Dependencies are getting crazy assert-rs/completest#3.

Those are behind the nu feature flag

[epage]

And this I think is a step in the wrong direction: assert-rs/completest#4. Instead of scrubbing existing environment from anything that can cause variations - it is better to make one that is known to be clean.

If you know of a better way, you are welcome to make a PR. I am not an expert on these shells and so long as its an improvement, even if it isn't perfect, I'm ok with it.

[pacak]

I'm planning to release 0.9.3 with current set of documentation updates soon-ish and will be looking to update documentation and completion stuff to run in CI. Will ping you once I have something to show then we can decide on PR.