testing: use Hurl in CI to test Caddy against spec #6255
+222
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since #5704 was posted, we've been on-and-off brainstorming how to approach testing of a web server. We sorta agreed that declarative approach is desired but weren't aware of any tools that'd facilitate the declarative approach nor had a concrete plan. We just knew we need solid tests.
I have recently come across Hurl (https://github.com/Orange-OpenSource/hurl) and was curious if it meets our needs. It is declarative. It makes HTTP calls. It stands on shoulders of The Giant®, namely curl. The PoC presented in this branch seems to work. In fact, the PR #6249 is a fix for a bug found while building this PoC.
This PR is to discuss the approach and to collaboratively add the tests. The core idea is simple:
TODO:
For TODO number 2, code coverage is a helpful tool. There's a way to extract execution coverage of the hurl tests†, but I haven't found a neat way to present it on GitHub PRs/Actions.
Based on the work done to resolve #5849 and the existence of the project REDbot, we can translate those expectations and rules into Hurl files.
† Using this article as guide: Build Caddy with coverage instrumentation using
go build -cover
. Run Caddy using the commandGOCOVERDIR=./coverdir caddy run
, then run the Hurl tests. Stop Caddy with eithercaddy stop
orCtrl-C
. Rungo tool covdata textfmt -i=coverdir -o profile.txt
. Rungo tool cover -html profile.txt
. An HTML page is opened in the browser with each file annotated by color for whether it was executed or not.