promql+alert_generator: Dockerized the toolset, updated docs.

alert_generator/Dockerfile

@@ -0,0 +1,14 @@
+FROM golang:1.18 as build-env
+
+WORKDIR /go/src/alert_generator
+COPY . /go/src/alert_generator
+
+ENV CGO_ENABLED 0
+
+RUN go build ./cmd/alert_generator_compliance_tester
+
+FROM quay.io/prometheus/busybox
+COPY --from=build-env /go/src/alert_generator/alert_generator_compliance_tester /
+COPY --from=build-env /go/src/alert_generator/rules.yaml /
+
+ENTRYPOINT ["/alert_generator_compliance_tester"]

alert_generator/Makefile

@@ -1,6 +1,10 @@
 build:
 	go build ./cmd/alert_generator_compliance_tester/
 
+.PHONY: docker
+docker:
+	docker build . -t alert_generator_compliance_tester:latest
+
 run: build
 	./alert_generator_compliance_tester -config-file=$(CONFIG)
 

alert_generator/README.md

@@ -43,7 +43,15 @@ Feel free to modify the example config itself. Config for some softwares are pro
 
 If you are running your software in a local environment, you can set the alertmanager URL to `http://<host>:<port>` with port being the one set in the test suite config. For example `http://localhost:8080`.
 
-If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, here is an alternative:
+If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, there are two alternatives:
+
+##### Step 4a
+
+You can run `./cmd/alert_generator_compliance_tester` in docker and run it inside your infrastructure as a batch job. To do so, build image with `make docker` and instead of step 6, simply run `alert_generator_compliance_tester:latest` with `-config-file=<config file you have done in step 3>`.
+
+See example [here](https://github.com/thanos-io/thanos/blob/89077c9df109dac8dbe6c979ebc181071c5e0db8/test/e2e/compatibility_test.go#L132) 
+
+##### Step 4b
 
 1. Open https://webhook.site/. It will give a unique link to use as a webhook. Let's take https://webhook.site/12345 as an example.
 2. Set https://webhook.site/12345 to be the alertmanager URL in your software - hence all alerts will be set to this webhook.

promql/Dockerfile

@@ -0,0 +1,14 @@
+FROM golang:1.18 as build-env
+
+WORKDIR /go/src/promql
+COPY . /go/src/promql
+
+ENV CGO_ENABLED 0
+
+RUN go build ./cmd/promql-compliance-tester
+
+FROM quay.io/prometheus/busybox
+COPY --from=build-env /go/src/promql/promql-compliance-tester /
+COPY --from=build-env /go/src/promql/promql-test-queries.yml /
+
+ENTRYPOINT ["/promql-compliance-tester"]

promql/Makefile

@@ -0,0 +1,3 @@
+.PHONY: docker
+docker:
+	docker build . -t promql-compliance-tester:latest

promql/README.md

@@ -4,21 +4,31 @@ The PromQL Compliance Tester is a tool for running comparison tests between nati
 
 The tool was [first published and described](https://promlabs.com/blog/2020/08/06/comparing-promql-correctness-across-vendors) in August 2020. [Test results have been published](https://promlabs.com/promql-compliance-tests) on 2020-08-06 and 2020-12-01.
 
-## Build Requirements
+## Building via Docker
+
+If you have docker installed, you can build the tool using docker.
+
+```bash
+make docker
+```
+
+## Build from source
+
+### Requirements
 
 This tool is written in Go and requires a working Go setup to build. Library dependencies are handled via [Go Modules](https://blog.golang.org/using-go-modules).
 
-## Building
+### Building
 
 To build the tool:
 
 ```bash
 go build ./cmd/promql-compliance-tester
 ```
 
-## Available flags
+## Executing
 
-To list available flags:
+Tool allows setting following flags:
 
 ```
 $ ./promql-compliance-tester -h
@@ -35,6 +45,18 @@ Usage of ./promql-compliance-tester:
     	Maximum number of comparison queries to run in parallel. (default 20)
 ```
 
+Running the tool will execute all test cases in `-config-file` and compare results between reference and target provided in the same file.
+
+At the end of run, the output is provided in form or number of executed tests and errors if any. Example output can be seen here:
+
+```bash
+./promql-compliance-tester -config-file config.yaml -config-file ./promql-test-queries.yml
+529 / 529 [-----------------------------------------------------------------------------------------------------------] 100.00% 278 p/s
+Total: 529 / 529 (100.00%) passed, 0 unsupported
+```
+
+If all tests were executed correctly and passing the tool returns 0 code, otherwise it returns 1.
+
 ## Configuration
 
 A standard suite of test cases is defined in the [`promql-test-queries.yml`](./promql-test-queries.yml) file, while separate `test-<vendor>.yml` config files specify test target configurations and query tweaks for a number of individual projects and vendors. To run the tester tool, you need to specify both the test suite config file as well as a config file for a single vendor.
@@ -64,7 +86,7 @@ If the tool reports a test score of 100% without any cross-cutting query tweaks,
 
 ## Contributing
 
-It's still early days for the PromQL Compliance Tester. In particular, we would love to add and improve the following points:
+Help wanted to improve the PromQL Compliance Tester. In particular, we would love to add and improve the following points:
 
 * Test instant queries in addition to range queries.
 * Add more variation and configurability to input timestamps.