Skip to content

Commit

Permalink
OCPBUGS-2873: fix certificate reloads after rotation
Browse files Browse the repository at this point in the history
When the TLS certificate (used by Prometheus to authenticate to the
scraped targets) gets rotated, Prometheus doesn't pick up the new
certificate until the connection to the target is re-established.
Because Prometheus uses keep-alive HTTP connections, the consequence is
that the scrapes start failing after about 1 day and the TargetDown
alert fires.

There's an upstream pull request [1] to address the issue but it isn't
merged yet. This commit pulls the changes from [1] into our downstream
fork by adding a replace directive to go.mod for the
github.com/prometheus/common. The replacement code is under
patches/github.com/prometheus/common which is the same version as
upstream (v0.37.0) + the upstream PR applied on top of it.

As soon as upstream Prometheus depends on a version of
github.com/prometheus/common that fixes the issue, the replace directive
in go.mod and the code under the patches/ directory can be removed.

[1] prometheus/common#345

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
  • Loading branch information
simonpasquier committed Oct 26, 2022
1 parent c3cbff9 commit d956fbc
Show file tree
Hide file tree
Showing 159 changed files with 17,263 additions and 61 deletions.
18 changes: 15 additions & 3 deletions go.mod
Expand Up @@ -23,6 +23,7 @@ require (
github.com/go-openapi/strfmt v0.21.3
github.com/go-zookeeper/zk v1.0.3
github.com/gogo/protobuf v1.3.2
github.com/golang/protobuf v1.5.2
github.com/golang/snappy v0.0.4
github.com/google/pprof v0.0.0-20220829040838-70bd9ae97f40
github.com/gophercloud/gophercloud v1.0.0
Expand All @@ -33,8 +34,10 @@ require (
github.com/hetznercloud/hcloud-go v1.35.3
github.com/ionos-cloud/sdk-go/v6 v6.1.3
github.com/json-iterator/go v1.1.12
github.com/julienschmidt/httprouter v1.3.0
github.com/kolo/xmlrpc v0.0.0-20220919000247-3377102c83bd
github.com/linode/linodego v1.9.1
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
github.com/miekg/dns v1.1.50
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/oklog/run v1.1.0
Expand Down Expand Up @@ -121,7 +124,6 @@ require (
github.com/golang-jwt/jwt/v4 v4.2.0 // indirect
github.com/golang/glog v1.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/gnostic v0.5.7-v3refs // indirect
github.com/google/go-cmp v0.5.8 // indirect
github.com/google/go-querystring v1.1.0 // indirect
Expand All @@ -143,11 +145,9 @@ require (
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
github.com/julienschmidt/httprouter v1.3.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect
Expand Down Expand Up @@ -187,6 +187,18 @@ replace (
k8s.io/klog/v2 => github.com/simonpasquier/klog-gokit/v3 v3.0.0
)

// Use a patched version of github.com/prometheus/common v0.37.0 to have proper
// reloading of the TLS certificates on disk.
//
// Bug link: https://issues.redhat.com/browse/OCPBUGS-2873
//
// Upstream pull request: https://github.com/prometheus/common/pull/345
//
// This directive (as well as the patches/ directory) can be removed once
// upstream Prometheus depends on a version of github.com/prometheus/common
// including the aforementioned fix.
replace github.com/prometheus/common => ./patches/github.com/prometheus/common

// Exclude linodego v1.0.0 as it is no longer published on github.
exclude github.com/linode/linodego v1.0.0

Expand Down
37 changes: 0 additions & 37 deletions go.sum

Large diffs are not rendered by default.

105 changes: 105 additions & 0 deletions patches/github.com/prometheus/common/.circleci/config.yml
@@ -0,0 +1,105 @@
---
version: 2.1

orbs:
go: circleci/go@0.2.0

jobs:
test:
parameters:
go_version:
type: string
use_gomod_cache:
type: boolean
default: true
docker:
- image: cimg/go:<< parameters.go_version >>
steps:
- checkout
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/load-cache:
key: v1-go<< parameters.go_version >>
- run: make test
- run: make -C sigv4 test
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/save-cache:
key: v1-go<< parameters.go_version >>
- store_test_results:
path: test-results
test-assets:
parameters:
go_version:
type: string
use_gomod_cache:
type: boolean
default: true
docker:
- image: cimg/go:<< parameters.go_version >>
steps:
- checkout
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/load-cache:
key: v1-go<< parameters.go_version >>
- run: make -C assets test
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/save-cache:
key: v1-go<< parameters.go_version >>
- store_test_results:
path: test-results
style:
parameters:
go_version:
type: string
use_gomod_cache:
type: boolean
default: true
docker:
- image: cimg/go:<< parameters.go_version >>
steps:
- checkout
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/load-cache:
key: v1-go<< parameters.go_version >>
- run: make style
- run: make -C sigv4 style
- run: make -C assets style
- when:
condition: << parameters.use_gomod_cache >>
steps:
- go/save-cache:
key: v1-go<< parameters.go_version >>
- store_test_results:
path: test-results

workflows:
version: 2
tests:
jobs:
# Supported Go versions are synced with github.com/prometheus/client_golang.
- test:
name: go-<< matrix.go_version >>
matrix:
parameters:
go_version:
- "1.16"
- "1.17"
- "1.18"
- test-assets:
name: assets-go-<< matrix.go_version >>
matrix:
parameters:
go_version:
- "1.18"
- style:
name: style
go_version: "1.18"
@@ -0,0 +1,27 @@
name: golangci-lint
on:
push:
paths:
- "go.sum"
- "go.mod"
- "**.go"
- "scripts/errcheck_excludes.txt"
- ".github/workflows/golangci-lint.yml"
- ".golangci.yml"
pull_request:

jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: install Go
uses: actions/setup-go@v2
with:
go-version: 1.17.x
- name: Lint
uses: golangci/golangci-lint-action@v3.1.0
with:
version: v1.44.2
1 change: 1 addition & 0 deletions patches/github.com/prometheus/common/.gitignore
@@ -0,0 +1 @@
vendor/
27 changes: 27 additions & 0 deletions patches/github.com/prometheus/common/.yamllint
@@ -0,0 +1,27 @@
---
extends: default

rules:
braces:
max-spaces-inside: 1
level: error
brackets:
max-spaces-inside: 1
level: error
commas: disable
comments: disable
comments-indentation: disable
document-start: disable
indentation:
spaces: consistent
key-duplicates:
ignore: |
config/testdata/section_key_dup.bad.yml
line-length: disable
truthy:
ignore: |
.github/workflows/codeql-analysis.yml
.github/workflows/funcbench.yml
.github/workflows/fuzzing.yml
.github/workflows/prombench.yml
.github/workflows/golangci-lint.yml
3 changes: 3 additions & 0 deletions patches/github.com/prometheus/common/CODE_OF_CONDUCT.md
@@ -0,0 +1,3 @@
## Prometheus Community Code of Conduct

Prometheus follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
18 changes: 18 additions & 0 deletions patches/github.com/prometheus/common/CONTRIBUTING.md
@@ -0,0 +1,18 @@
# Contributing

Prometheus uses GitHub to manage reviews of pull requests.

* If you have a trivial fix or improvement, go ahead and create a pull request,
addressing (with `@...`) the maintainer of this repository (see
[MAINTAINERS.md](MAINTAINERS.md)) in the description of the pull request.

* If you plan to do something more involved, first discuss your ideas
on our [mailing list](https://groups.google.com/forum/?fromgroups#!forum/prometheus-developers).
This will avoid unnecessary work and surely give you and us a good deal
of inspiration.

* Relevant coding style guidelines are the [Go Code Review
Comments](https://code.google.com/p/go-wiki/wiki/CodeReviewComments)
and the _Formatting and style_ section of Peter Bourgon's [Go: Best
Practices for Production
Environments](http://peter.bourgon.org/go-in-production/#formatting-and-style).

0 comments on commit d956fbc

Please sign in to comment.