Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MetricsExporter does not configure a record filter #9240

Closed
korthout opened this issue Apr 27, 2022 · 0 comments · Fixed by #9371
Closed

MetricsExporter does not configure a record filter #9240

korthout opened this issue Apr 27, 2022 · 0 comments · Fixed by #9371
Assignees
Labels
area/performance Marks an issue as performance related area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) kind/bug Categorizes an issue or PR as a bug scope/broker Marks an issue or PR to appear in the broker section of the changelog version:1.3.9 version:8.1.0-alpha2 version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0

Comments

@korthout
Copy link
Member

korthout commented Apr 27, 2022

Describe the bug

The MetricsExporter does not override the configure method. This method allows to set a filter for which records are available to export. Since the MetricsExporter only exports records of a few value types (i.e. JOB, JOB_BATCH, PROCESS_INSTANCE), it unnecessarily reads the record values of all other records.

This can completely halt exporting on a partition when combined with #6442.

To Reproduce

  • configure the MetricsExporter for the broker
  • run the broker
  • deploy a very large process with mistakes that lead to zeebe rejecting the deployment with validation errors
  • note that the exporter stops making progress
  • note an increase in CPU because it keeps trying to re-read the record value indefinitely

Expected behavior

The metrics exporter should be configured in such a way that only those records that it wants to export are actually read from the log by the exporter director.

Log/Stacktrace

Full Stacktrace

java.lang.IllegalArgumentException: invalid offset: -31808
	at org.agrona.concurrent.UnsafeBuffer.boundsCheckWrap(UnsafeBuffer.java:2425) ~[agrona-1.15.0.jar:1.15.0]
	at org.agrona.concurrent.UnsafeBuffer.wrap(UnsafeBuffer.java:277) ~[agrona-1.15.0.jar:1.15.0]
	at io.camunda.zeebe.msgpack.spec.MsgPackReader.wrap(MsgPackReader.java:49) ~[zeebe-msgpack-core-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.msgpack.UnpackedObject.wrap(UnpackedObject.java:30) ~[zeebe-msgpack-value-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.logstreams.impl.log.LoggedEventImpl.readValue(LoggedEventImpl.java:135) ~[zeebe-logstreams-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.RecordValues.readRecordValue(RecordValues.java:35) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.wrap(ExporterDirector.java:489) ~[zeebe-broker-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.broker.exporter.stream.ExporterDirector.lambda$exportEvent$9(ExporterDirector.java:391) ~[zeebe-broker-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.retry.ActorRetryMechanism.run(ActorRetryMechanism.java:36) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.retry.EndlessRetryStrategy.run(EndlessRetryStrategy.java:50) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) ~[zeebe-util-8.0.0.jar:8.0.0]

Environment:

  • Zeebe Version: 8.0
  • Configuration: MetricsExporter
@korthout korthout added kind/bug Categorizes an issue or PR as a bug scope/broker Marks an issue or PR to appear in the broker section of the changelog area/performance Marks an issue as performance related area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) team/process-automation labels Apr 27, 2022
zeebe-bors-camunda bot added a commit that referenced this issue May 13, 2022
9371: Configure record filter for metrics exporter r=korthout a=korthout

## Description

<!-- Please explain the changes you made here. -->

Configures a record filter for the Metrics exporter.

## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #9240 
relates to #6442 



9373: deps(go): bump github.com/docker/docker from 20.10.15+incompatible to 20.10.16+incompatible in /clients/go r=Zelldon a=dependabot[bot]

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.15+incompatible to 20.10.16+incompatible.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/docker/docker/releases">github.com/docker/docker's releases</a>.</em></p>
<blockquote>
<h2>v20.10.16</h2>
<p>This release of Docker Engine fixes a regression in the Docker CLI builds for
macOS, fixes an issue with <code>docker stats</code> when using containerd 1.5 and up,
and updates the Go runtime to include a fix for <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-29526">CVE-2022-29526</a>.</p>
<h3>Client</h3>
<ul>
<li>Fix a regression in binaries for macOS introduced in <a href="%5B#201015%5D(https://github-redirect.dependabot.com/docker/docker/issues/201015)">20.10.15</a>, which
resulted in a panic <a href="https://github-redirect.dependabot.com/docker/cli/pull/3592">docker/cli#43426</a>.</li>
<li>Update golang.org/x/sys dependency which contains a fix for
<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-29526">CVE-2022-29526</a>.</li>
</ul>
<h3>Daemon</h3>
<ul>
<li>Fix an issue where <code>docker stats</code> was showing empty stats when running with
containerd 1.5.0 or up <a href="https://github-redirect.dependabot.com/moby/moby/pull/43567">moby/moby#43567</a>.</li>
<li>Update the <code>golang.org/x/sys</code> build-time dependency which contains a fix for <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-29526">CVE-2022-29526</a>.</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Update Go runtime to <a href="https://go.dev/doc/devel/release#go1.17.minor">1.17.10</a>,
which contains a fix for <a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-29526">CVE-2022-29526</a>.</li>
<li>Use &quot;weak&quot; dependencies for the <code>docker scan</code> CLI plugin, to prevent a
&quot;conflicting requests&quot; error when users performed an off-line installation from
downloaded RPM packages <a href="https://github-redirect.dependabot.com/docker/docker-ce-packaging/pull/659">docker/docker-ce-packaging#659</a>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/moby/moby/commit/f756502055d2e36a84f2068e6620bea5ecf09058"><code>f756502</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/docker/docker/issues/43582">#43582</a> from thaJeztah/20.10_bump_golang_1.17.10</li>
<li><a href="https://github.com/moby/moby/commit/a15acb4bd6b9b65f93de5f52de56a29ae4d63a4c"><code>a15acb4</code></a> [20.10] vendor: golang.org/x/sys v0.0.0-20220412211240-33da011f77ad</li>
<li><a href="https://github.com/moby/moby/commit/5f2e0b79adc4d1786043176ee599e73f33430989"><code>5f2e0b7</code></a> [20.10] update golang to 1.17.10</li>
<li><a href="https://github.com/moby/moby/commit/462cd7de50edbba462ca2de82ceb10d26ac1e6bb"><code>462cd7d</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/docker/docker/issues/43567">#43567</a> from 42wim/fixstats</li>
<li><a href="https://github.com/moby/moby/commit/be7855fdbe3136f144feb3b9340e7d21c5c45e1e"><code>be7855f</code></a> vendor: update github.com/containerd/cgroups and github.com/cilium/ebpf</li>
<li>See full diff in <a href="https://github.com/docker/docker/compare/v20.10.15...v20.10.16">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/docker/docker&package-manager=go_modules&previous-version=20.10.15+incompatible&new-version=20.10.16+incompatible)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
zeebe-bors-camunda bot added a commit that referenced this issue May 25, 2022
9425: [Backport stable/1.3] Configure record filter for metrics exporter r=npepinpe a=korthout

# Description
Backport of #9371 to `stable/1.3`.

relates to #9240 #6442

Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>
zeebe-bors-camunda bot added a commit that referenced this issue May 25, 2022
9378: [Backport stable/8.0] Configure record filter for metrics exporter r=npepinpe a=github-actions[bot]

# Description
Backport of #9371 to `stable/8.0`.

relates to #9240 #6442

Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>
zeebe-bors-camunda bot added a commit that referenced this issue May 25, 2022
9425: [Backport stable/1.3] Configure record filter for metrics exporter r=npepinpe a=korthout

# Description
Backport of #9371 to `stable/1.3`.

relates to #9240 #6442

Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>
@Zelldon Zelldon added the version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0 label Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/performance Marks an issue as performance related area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) kind/bug Categorizes an issue or PR as a bug scope/broker Marks an issue or PR to appear in the broker section of the changelog version:1.3.9 version:8.1.0-alpha2 version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants