promql+alert_generator: Dockerized the toolset, updated docs. (#46)

* promql+alert_generator: Dockerized the toolset, updated docs.

Motivation for this is that we can now set automated tests where in future we might want to provide
docker images with exact set of test cases in versioned image.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Julius Volz <julius.volz@gmail.com>

* Apply suggestions from code review

Co-authored-by: Julius Volz <julius.volz@gmail.com>

Co-authored-by: Julius Volz <julius.volz@gmail.com>

alert_generator/Dockerfile

@@ -0,0 +1,14 @@
+FROM golang:1.18 as build-env
+
+WORKDIR /go/src/alert_generator
+COPY . /go/src/alert_generator
+
+ENV CGO_ENABLED 0
+
+RUN go build ./cmd/alert_generator_compliance_tester
+
+FROM quay.io/prometheus/busybox
+COPY --from=build-env /go/src/alert_generator/alert_generator_compliance_tester /
+COPY --from=build-env /go/src/alert_generator/rules.yaml /
+
+ENTRYPOINT ["/alert_generator_compliance_tester"]

alert_generator/Makefile

@@ -1,6 +1,10 @@
 build:
 	go build ./cmd/alert_generator_compliance_tester/
 
+.PHONY: docker
+docker:
+	docker build . -t alert_generator_compliance_tester:latest
+
 run: build
 	./alert_generator_compliance_tester -config-file=$(CONFIG)
 

alert_generator/README.md

@@ -43,7 +43,15 @@ Feel free to modify the example config itself. Config for some softwares are pro
 
 If you are running your software in a local environment, you can set the alertmanager URL to `http://<host>:<port>` with port being the one set in the test suite config. For example `http://localhost:8080`.
 
-If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, here is an alternative:
+If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, there are two alternatives:
+
+##### Step 4a
+
+You can run `./cmd/alert_generator_compliance_tester` in docker and run it inside your infrastructure as a batch job. To do so, build the image with `make docker` and instead of step 6, simply run `alert_generator_compliance_tester:latest` with `-config-file=<config file you have done in step 3>`.
+
+See example [here](https://github.com/thanos-io/thanos/blob/89077c9df109dac8dbe6c979ebc181071c5e0db8/test/e2e/compatibility_test.go#L132) 
+
+##### Step 4b
 
 1. Open https://webhook.site/. It will give a unique link to use as a webhook. Let's take https://webhook.site/12345 as an example.
 2. Set https://webhook.site/12345 to be the alertmanager URL in your software - hence all alerts will be set to this webhook.

promql/Dockerfile

@@ -0,0 +1,14 @@
+FROM golang:1.18 as build-env
+
+WORKDIR /go/src/promql
+COPY . /go/src/promql
+
+ENV CGO_ENABLED 0
+
+RUN go build ./cmd/promql-compliance-tester
+
+FROM quay.io/prometheus/busybox
+COPY --from=build-env /go/src/promql/promql-compliance-tester /
+COPY --from=build-env /go/src/promql/promql-test-queries.yml /
+
+ENTRYPOINT ["/promql-compliance-tester"]

promql/Makefile

@@ -0,0 +1,3 @@
+.PHONY: docker
+docker:
+	docker build . -t promql-compliance-tester:latest

promql/README.md

@@ -4,21 +4,31 @@ The PromQL Compliance Tester is a tool for running comparison tests between nati
 
 The tool was [first published and described](https://promlabs.com/blog/2020/08/06/comparing-promql-correctness-across-vendors) in August 2020. [Test results have been published](https://promlabs.com/promql-compliance-tests) on 2020-08-06 and 2020-12-01.
 
-## Build Requirements
+## Building via Docker
+
+If you have docker installed, you can build the tool using docker.
+
+```bash
+make docker
+```
+
+## Building from source
+
+### Requirements
 
 This tool is written in Go and requires a working Go setup to build. Library dependencies are handled via [Go Modules](https://blog.golang.org/using-go-modules).
 
-## Building
+### Building
 
 To build the tool:
 
 ```bash
 go build ./cmd/promql-compliance-tester
 ```
 
-## Available flags
+## Executing
 
-To list available flags:
+The tool allows setting the following flags:
 
 ```
 $ ./promql-compliance-tester -h
@@ -35,6 +45,18 @@ Usage of ./promql-compliance-tester:
     	Maximum number of comparison queries to run in parallel. (default 20)
 ```
 
+Running the tool will execute all test cases in `-config-file` and compare results between reference and target provided in the same file.
+
+At the end of the run, the output is provided in the form of the number of executed tests and errors if any. Example output can be seen here:
+
+```bash
+./promql-compliance-tester -config-file config.yaml -config-file ./promql-test-queries.yml
+529 / 529 [-----------------------------------------------------------------------------------------------------------] 100.00% 278 p/s
+Total: 529 / 529 (100.00%) passed, 0 unsupported
+```
+
+If all tests were executed correctly and passing the tool returns a 0 exit code, otherwise it returns 1.
+
 ## Configuration
 
 A standard suite of test cases is defined in the [`promql-test-queries.yml`](./promql-test-queries.yml) file, while separate `test-<vendor>.yml` config files specify test target configurations and query tweaks for a number of individual projects and vendors. To run the tester tool, you need to specify both the test suite config file as well as a config file for a single vendor.
@@ -64,7 +86,7 @@ If the tool reports a test score of 100% without any cross-cutting query tweaks,
 
 ## Contributing
 
-It's still early days for the PromQL Compliance Tester. In particular, we would love to add and improve the following points:
+Help is wanted to improve the PromQL Compliance Tester. In particular, we would love to add and improve the following points:
 
 * Test instant queries in addition to range queries.
 * Add more variation and configurability to input timestamps.