Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend prometheus.Registry to implement Collector #1103

Merged
merged 3 commits into from Aug 23, 2022

Conversation

rfratto
Copy link
Member

@rfratto rfratto commented Aug 3, 2022

This change allows Registries to be used as Collectors.

This enables new instances of Registry to be passed to ephemeral subroutines for collecting metrics from subroutines which are still running:

package main

import (
  "fmt"

  "github.com/prometheus/client_golang/prometheus"
)

func main() {
  globalReg := prometheus.NewRegistry()

  for i := 0; i < 100; i++ {
    workerReg := prometheus.WrapRegistererWith(prometheus.Labels{
      // Add an ID label so registered metrics from workers don't
      // collide.
      "worker_id": fmt.Sprintf("%d", i),
    }, prometheus.NewRegistry())

    globalReg.MustRegister(workerReg)

    go func(i int) {
      runWorker(workerReg)

      // Unregister any metrics the worker may have created.
      globalReg.Unregister(workerReg)
    }(i)
  }
}

// runWorker runs a worker, registering worker-specific metrics.
func runWorker(reg *prometheus.Registry) {
  // ... register metrics ...
  // ... do work ...
}

This change makes it easier to avoid leaking metrics from subroutines which do not consistently properly unregister metrics.

This would primarily make things easier for Grafana Agent, where it can spawn different subroutines throughout the process lifetime which have different sets of metrics (i.e., remote_write). Ensuring that metrics from these subroutines consistently get unregistered can be a source of bugs, and something like this would make it easier for us to avoid leaking no-longer-needed metrics.

I'm not completely sure this change works as-is (it seems reasonable), but I wanted to open it as a PR to spawn discussion about the concept.

This change allows Registries to be used as Collectors.

This enables new instances of Registry to be passed to ephemeral
subroutines for collecting metrics from subroutines which are still
running:

```go
package main

import (
  "fmt"

  "github.com/prometheus/client_golang/prometheus"
)

func main() {
  globalReg := prometheus.NewRegistry()

  for i := 0; i < 100; i++ {
    workerReg := prometheus.WrapRegistererWith(prometheus.Labels{
      // Add an ID label so registered metrics from workers don't
      // collide.
      "worker_id": fmt.Sprintf("%d", i),
    }, prometheus.NewRegistry()

    globalReg.MustRegister(workerReg)

    go func(i int) {
      runWorker(workerReg)

      // Unregister any metrics the worker may have created.
      globalReg.Unregister(workerReg)
    }(i)
  }
}

// runWorker runs a worker, registering worker-specific metrics.
func runWorker(reg *prometheus.Registry) {
  // ... register metrics ...
  // ... do work ...
}
```

This change makes it easier to avoid leaking metrics from subroutines
which do not consistently properly unregister metrics.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
@rfratto rfratto force-pushed the registry-implement-collector branch from c76e64d to 688d41e Compare August 3, 2022 15:17
@rfratto
Copy link
Member Author

rfratto commented Aug 3, 2022

This would also make #945 no longer needed, since you could instead pass new Registries to metric containers and easily unregister the container as a whole.

Copy link
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the pattern. I can see the places where I'd use this pattern.
I want to think about it for a second and check if there's any downside (possible abuse?) to complying with Collector interface 🤔

@bwplotka
Copy link
Member

bwplotka commented Aug 4, 2022

Hm, so essentially you want to be able to group collectors better and collect, register and unregister them in one go. I think it's pretty nice characteristics, although at first it the Registry might feel like a separate functionality. The only downside I can see is that we put too many functionalities into one struct which can cause a bit of confusion and many ways of doing same thing.

The alternative is to create dedicated structure MultiRegistry (or multiCollector) attempt I did in https://github.com/prometheus/client_golang/pull/929/files#diff-6f0796727ecbd7f1d226589f6cf66cbb90a2c30d1823ce35d7a480e2561358d6R966 or something like that... But given this code is so little and clean I would be happy with your proposal @rfratto 💪🏽

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGMT, I like this logic 👍🏽

A bit more docs would be amazing if you have time (:

// NewPedanticRegistry.
// them into MetricFamilies for exposition. It implements Registerer, Gatherer,
// and Collector. The zero value is not usable. Create instances with
// NewRegistry or NewPedanticRegistry.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind adding a paraph about implementing Collector, it's behavior and what it enables?

Ideally we could add short ExampleRegistryGrouping example in test file too! 🤗

@rfratto
Copy link
Member Author

rfratto commented Aug 4, 2022

It's a good thing @bwplotka suggested writing an Example function, because I definitely missed something on my first pass at this :)

The snippet I gave in the PR description doesn't work; prometheus.WrapRegistererWith returns an instance of the Registerer interface, which doesn't implement Collector.

I'm not sure what we should do about this change in that case; it's a lot more limiting than I originally thought since most of the API erases the Collector implementation from the Registry.

There could be a new interface:

type RegistererCollector interface {
  Registerer 
  Collector 
}

which methods like WrapRegistererWith can be updated to return, but I'm not sure if this would bloat the proposal too much.

@kakkoyun kakkoyun modified the milestones: v1.13.0, v1.14.0 Aug 5, 2022
@bwplotka
Copy link
Member

bwplotka commented Aug 5, 2022

Hmmm what about casting to Collector interface in your code for now?

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
@rfratto
Copy link
Member Author

rfratto commented Aug 8, 2022

Added more docs with an example, PTAL :)

Also, prometheus.WrapRegistererWith would work, it's just not very obvious:

package main

import (
  "fmt"

  "github.com/prometheus/client_golang/prometheus"
)

func main() {
  globalReg := prometheus.NewRegistry()

  for i := 0; i < 100; i++ {
    workerMetrics := prometheus.NewRegistry() 
    globalReg.MustRegister(workerMetrics)

    go func(i int) {
      // Wrap worker metrics with a unique label so workers don't 
      // collide. 
      runWorker(prometheus.WrapRegistererWith(prometheus.Labels{
        "worker_id": fmt.Sprintf("%d", i),
      }, workerMetrics)

      // Unregister any metrics the worker may have created.
      globalReg.Unregister(workerMetrics)
    }(i)
  }
}

// runWorker runs a worker, registering worker-specific metrics.
func runWorker(reg prometheus.Registerer) {
  // ... register metrics ...
  // ... do work ...
}

@bwplotka
Copy link
Member

I like it 👍🏽 Again, casting to different interface would also work just fine. Thanks!

@bwplotka bwplotka merged commit 83d56b1 into prometheus:main Aug 23, 2022
patrickdevivo added a commit to mergestat/mergestat that referenced this pull request Nov 28, 2022
….0 (#524)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
|
[github.com/prometheus/client_golang](https://togithub.com/prometheus/client_golang)
| require | minor | `v1.13.0` -> `v1.14.0` |

---

### Release Notes

<details>
<summary>prometheus/client_golang</summary>

###
[`v1.14.0`](https://togithub.com/prometheus/client_golang/releases/tag/v1.14.0):
1.14.0 / 2022-11-08

[Compare
Source](https://togithub.com/prometheus/client_golang/compare/v1.13.1...v1.14.0)

It might look like a small release, but it's quite opposite 😱 There were
many non user facing changes and fixes and enormous work from engineers
from Grafana to add native histograms in 💪🏾 Enjoy! 😍

#### What's Changed

- \[FEATURE] Add Support for Native Histograms.
[#&#8203;1150](https://togithub.com/prometheus/client_golang/issues/1150)
- \[CHANGE] Extend `prometheus.Registry` to implement
`prometheus.Collector` interface.
[#&#8203;1103](https://togithub.com/prometheus/client_golang/issues/1103)

#### New Contributors

- [@&#8203;hairyhenderson](https://togithub.com/hairyhenderson) made
their first contribution in
[prometheus/client_golang#1118
- [@&#8203;rfratto](https://togithub.com/rfratto) made their first
contribution in
[prometheus/client_golang#1103
- [@&#8203;donotnoot](https://togithub.com/donotnoot) made their first
contribution in
[prometheus/client_golang#1125
- [@&#8203;rogerogers](https://togithub.com/rogerogers) made their first
contribution in
[prometheus/client_golang#1130
- [@&#8203;balintzs](https://togithub.com/balintzs) made their first
contribution in
[prometheus/client_golang#1148
- [@&#8203;fstab](https://togithub.com/fstab) made their first
contribution in
[prometheus/client_golang#1146
- [@&#8203;jessicalins](https://togithub.com/jessicalins) made their
first contribution in
[prometheus/client_golang#1152

**Full Changelog**:
prometheus/client_golang@v1.13.1...v1.14.0

###
[`v1.13.1`](https://togithub.com/prometheus/client_golang/releases/tag/v1.13.1):
1.13.1 / 2022-11-02

[Compare
Source](https://togithub.com/prometheus/client_golang/compare/v1.13.0...v1.13.1)

- \[BUGFIX] Fix race condition with Exemplar in Counter.
[#&#8203;1146](https://togithub.com/prometheus/client_golang/issues/1146)
- \[BUGFIX] Fix `CumulativeCount` value of `+Inf` bucket created from
exemplar.
[#&#8203;1148](https://togithub.com/prometheus/client_golang/issues/1148)
- \[BUGFIX] Fix double-counting bug in
`promhttp.InstrumentRoundTripperCounter`.
[#&#8203;1118](https://togithub.com/prometheus/client_golang/issues/1118)

**Full Changelog**:
prometheus/client_golang@v1.13.0...v1.13.1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://app.renovatebot.com/dashboard#github/mergestat/mergestat).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMjYuNCJ9-->
estahn pushed a commit to estahn/k8s-image-swapper that referenced this pull request Dec 1, 2022
## [1.3.3](v1.3.2...v1.3.3) (2022-12-01)

### ⬆️ Dependencies

* **deps:** Bump alpine from 3.16.2 to 3.16.3 ([#388](#388)) ([ffae497](ffae497))
* **deps:** Bump alpine from 3.16.3 to 3.17.0 ([#395](#395)) ([d32255d](d32255d))
* **deps:** Bump github.com/aws/aws-sdk-go from 1.44.126 to 1.44.136 ([#391](#391)) ([61a6ae2](61a6ae2)), closes [#4620](https://github.com/estahn/k8s-image-swapper/issues/4620) [#4619](https://github.com/estahn/k8s-image-swapper/issues/4619) [#4617](https://github.com/estahn/k8s-image-swapper/issues/4617) [#4616](https://github.com/estahn/k8s-image-swapper/issues/4616) [#4615](https://github.com/estahn/k8s-image-swapper/issues/4615) [#4614](https://github.com/estahn/k8s-image-swapper/issues/4614) [#4613](https://github.com/estahn/k8s-image-swapper/issues/4613) [#4611](https://github.com/estahn/k8s-image-swapper/issues/4611) [#4608](https://github.com/estahn/k8s-image-swapper/issues/4608) [#4609](https://github.com/estahn/k8s-image-swapper/issues/4609)
* **deps:** Bump github.com/aws/aws-sdk-go from 1.44.136 to 1.44.146 ([#397](#397)) ([d4a6136](d4a6136)), closes [#4638](https://github.com/estahn/k8s-image-swapper/issues/4638) [#4636](https://github.com/estahn/k8s-image-swapper/issues/4636) [#4633](https://github.com/estahn/k8s-image-swapper/issues/4633) [#4632](https://github.com/estahn/k8s-image-swapper/issues/4632) [#4630](https://github.com/estahn/k8s-image-swapper/issues/4630) [#4628](https://github.com/estahn/k8s-image-swapper/issues/4628) [#4627](https://github.com/estahn/k8s-image-swapper/issues/4627) [#4626](https://github.com/estahn/k8s-image-swapper/issues/4626) [#4625](https://github.com/estahn/k8s-image-swapper/issues/4625) [#4624](https://github.com/estahn/k8s-image-swapper/issues/4624)
* **deps:** Bump github.com/containers/image/v5 from 5.23.0 to 5.23.1 ([#393](#393)) ([84f4d18](84f4d18))
* **deps:** Bump github.com/go-co-op/gocron from 1.17.1 to 1.18.0 ([#390](#390)) ([1750ee9](1750ee9)), closes [go-co-op/gocron#388](go-co-op/gocron#388) [go-co-op/gocron#389](go-co-op/gocron#389) [go-co-op/gocron#392](go-co-op/gocron#392) [go-co-op/gocron#394](go-co-op/gocron#394) [go-co-op/gocron#393](go-co-op/gocron#393) [go-co-op/gocron#392](go-co-op/gocron#392) [go-co-op/gocron#394](go-co-op/gocron#394) [#393](#393) [#394](#394) [#392](#392) [#389](#389)
* **deps:** Bump github.com/gruntwork-io/terratest from 0.40.24 to 0.41.3 ([#398](#398)) ([ab35b1a](ab35b1a)), closes [gruntwork-io/terratest#1203](gruntwork-io/terratest#1203) [gruntwork-io/terratest#1202](gruntwork-io/terratest#1202) [gruntwork-io/terratest#1201](gruntwork-io/terratest#1201) [gruntwork-io/terratest#1199](gruntwork-io/terratest#1199) [gruntwork-io/terratest#1196](gruntwork-io/terratest#1196) [#1202](https://github.com/estahn/k8s-image-swapper/issues/1202) [#1203](https://github.com/estahn/k8s-image-swapper/issues/1203) [#1201](https://github.com/estahn/k8s-image-swapper/issues/1201) [#1199](https://github.com/estahn/k8s-image-swapper/issues/1199) [#1196](https://github.com/estahn/k8s-image-swapper/issues/1196)
* **deps:** Bump github.com/prometheus/client_golang from 1.13.0 to 1.13.1 ([#387](#387)) ([b155a16](b155a16)), closes [#1146](https://github.com/estahn/k8s-image-swapper/issues/1146) [#1148](https://github.com/estahn/k8s-image-swapper/issues/1148) [#1118](https://github.com/estahn/k8s-image-swapper/issues/1118) [#1146](https://github.com/estahn/k8s-image-swapper/issues/1146) [#1148](https://github.com/estahn/k8s-image-swapper/issues/1148) [#1118](https://github.com/estahn/k8s-image-swapper/issues/1118) [#1157](https://github.com/estahn/k8s-image-swapper/issues/1157) [#1146](https://github.com/estahn/k8s-image-swapper/issues/1146) [#1148](https://github.com/estahn/k8s-image-swapper/issues/1148) [#1118](https://github.com/estahn/k8s-image-swapper/issues/1118)
* **deps:** Bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 ([#392](#392)) ([af00594](af00594)), closes [#1150](https://github.com/estahn/k8s-image-swapper/issues/1150) [#1103](https://github.com/estahn/k8s-image-swapper/issues/1103) [prometheus/client_golang#1118](prometheus/client_golang#1118) [prometheus/client_golang#1103](prometheus/client_golang#1103) [prometheus/client_golang#1125](prometheus/client_golang#1125) [prometheus/client_golang#1130](prometheus/client_golang#1130) [prometheus/client_golang#1148](prometheus/client_golang#1148) [prometheus/client_golang#1146](prometheus/client_golang#1146) [prometheus/client_golang#1152](prometheus/client_golang#1152) [#1150](https://github.com/estahn/k8s-image-swapper/issues/1150) [#1103](https://github.com/estahn/k8s-image-swapper/issues/1103) [#1162](https://github.com/estahn/k8s-image-swapper/issues/1162) [#1161](https://github.com/estahn/k8s-image-swapper/issues/1161) [#1160](https://github.com/estahn/k8s-image-swapper/issues/1160) [#1136](https://github.com/estahn/k8s-image-swapper/issues/1136) [#1133](https://github.com/estahn/k8s-image-swapper/issues/1133) [#1150](https://github.com/estahn/k8s-image-swapper/issues/1150) [#1152](https://github.com/estahn/k8s-image-swapper/issues/1152)
* **deps:** Bump github.com/spf13/viper from 1.13.0 to 1.14.0 ([#385](#385)) ([6f79498](6f79498)), closes [spf13/viper#1457](spf13/viper#1457) [spf13/viper#1458](spf13/viper#1458) [spf13/viper#1460](spf13/viper#1460) [spf13/viper#1428](spf13/viper#1428) [spf13/viper#1406](spf13/viper#1406) [spf13/viper#1437](spf13/viper#1437) [spf13/viper#1453](spf13/viper#1453) [spf13/viper#1449](spf13/viper#1449) [spf13/viper#1461](spf13/viper#1461)
* **deps:** Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 ([#389](#389)) ([0b50f7b](0b50f7b)), closes [golangci/golangci-lint-action#590](golangci/golangci-lint-action#590) [golangci/golangci-lint-action#591](golangci/golangci-lint-action#591) [golangci/golangci-lint-action#592](golangci/golangci-lint-action#592) [golangci/golangci-lint-action#593](golangci/golangci-lint-action#593) [golangci/golangci-lint-action#594](golangci/golangci-lint-action#594) [golangci/golangci-lint-action#595](golangci/golangci-lint-action#595) [golangci/golangci-lint-action#596](golangci/golangci-lint-action#596) [golangci/golangci-lint-action#597](golangci/golangci-lint-action#597) [golangci/golangci-lint-action#598](golangci/golangci-lint-action#598) [golangci/golangci-lint-action#599](golangci/golangci-lint-action#599) [#599](#599) [#598](#598) [#596](#596) [#595](#595) [#593](#593) [#591](#591) [#590](#590)
* **deps:** Bump hmarr/auto-approve-action from 2 to 3 ([#396](#396)) ([0b982a2](0b982a2)), closes [hmarr/auto-approve-action#205](hmarr/auto-approve-action#205) [hmarr/auto-approve-action#202](hmarr/auto-approve-action#202) [hmarr/auto-approve-action#202](hmarr/auto-approve-action#202) [hmarr/auto-approve-action#200](hmarr/auto-approve-action#200) [hmarr/auto-approve-action#200](hmarr/auto-approve-action#200) [hmarr/auto-approve-action#186](hmarr/auto-approve-action#186) [hmarr/auto-approve-action#191](hmarr/auto-approve-action#191) [#210](#210) [#205](#205)
* **deps:** Bump k8s.io/api from 0.25.3 to 0.25.4 ([#401](#401)) ([0f80b5d](0f80b5d))
* **deps:** Bump k8s.io/apimachinery from 0.25.3 to 0.25.4 ([#399](#399)) ([1f0944f](1f0944f)), closes [#112218](https://github.com/estahn/k8s-image-swapper/issues/112218) [haoruan/automated-cherry-pick-of-#111936](https://github.com/haoruan/automated-cherry-pick-of-/issues/111936)
* **deps:** Bump k8s.io/client-go from 0.25.3 to 0.25.4 ([#400](#400)) ([ad036e0](ad036e0))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants