Add Subscribe::event_enabled to conditionally dis/enable events based on fields #2008

CAD97 · 2022-03-21T00:48:18Z

Motivation

Allow filter layers to filter on the contents of events (closes #2007).

Solution

Add Subscribe::event_enabled(&self, event: &Event<'_>, ctx: Context<'_, C>) -> bool;. This is called before Subscribe::on_event, and if event_enabled returns false, on_event is not called (nor is Collect::event).

Previously, this PR did

Subscribe::on_event returns ControlFlow, and the Layered subscriber only continues to call on_event on ControlFlow::Continue.

This PR exists mainly as a place for discussion of this potential solution. Looking at this diff, and given that most subscriber layers will want to continue and not break the layering, it might make sense to make this a new method which delegates to on_event by default. Since Layered is the primary way of layering two subscribers together, perhaps the footgun of calling on_event rather than "filter_on_event" would not be a big deal in practice?

hawkw · 2022-03-22T18:24:12Z

I'm on board with adding this. I think it makes sense to add the ability to filter out an event after its fields are recorded, people have definitely asked for this in the past. In addition to the use-case of filtering based on fields, it would also help with something else I've heard some demand for --- rate limiting event instances based on their fields, so that an event is rate-limited only if it's recorded multiple times with the same field values.

The one thing I would note that's important to document is that this method is going to behave differently from enabled filtering: in enabled, any layer that returns false will disable the event, while with a filtered on_event method, the event is only disabled for lower layers --- any layers above the filter will still see the event. I think it's important to call this out clearly in the documentation, both for the method itself and for filter implementations. Currently, I've seen code where a filter such as e.g. an EnvFilter is not placed at the top of the stack, and if we add event field value filtering to EnvFilter based on this change, it would mean that this filtering would not work unless the filter is added last. So, it's important that we make it as clear as possible to users that this would behave differently from other forms of filtering.

Looking at this diff, and given that most subscriber layers will want to continue and not break the layering, it might make sense to make this a new method which delegates to on_event by default.

I think this makes sense, as it would also allow us to add this API without a breaking change. Since we don't really expect user code to call Layer methods manually (they will basically only ever be called by Layered and friends), I think the footgun of accidentally calling on_event rather than calling it through filter_on_event is not all that big a problem, as long as Layered does the Right Thing.

tracing-journald/src/lib.rs

CAD97 · 2022-03-23T22:37:46Z

This now uses a design where events are separately checked for enabled. This now makes it behave similarly to enabled; the on_event notification is disabled for the entire subscriber stack, rather than just lower layers of the stack. This also makes it a simple API addition, and unlike -> ControlFlow, a) doesn't raise MSRV, and b) doesn't create a pitfall of two methods to do the same thing.

The one downside I see to this design is that there might be a layer which could fuse some work between event_enabled and on_event if they weren't separate notifications. Thinking about that some more, though, this seems unlikely; layers should typically be doing one or the other, not both.

This PR is now ready for review to merge.

CAD97 · 2022-03-26T04:00:16Z

Interesting alternative solution: just add event_enabled to Filter, then add e.g. a GlobalFilter subscriber which takes a Filter and applies it to the entire subscriber stack a la EnvFilter as Subscribe.

EDIT: wait, no, Subscribe still needs to support event_enabled, so that's not really an alternative

CAD97 · 2022-03-31T10:11:27Z

Rebased. I'd appreciate if this could move forward, as it has implications for my tracing-filter work.

hawkw

I'd definitely like to move forward with this change, as well!

The current API proposal seems good to me --- @davidbarsky, want to weigh in?

I think the next step will be actually implementing the intended behavior (i.e. actually calling event_enabled and skipping events based on it). IMO, the current API design seems good and we can go ahead and start on actually implementing it.

We may also want to consider adding similar functionality to Filter, to allow a per-subscriber filter to disable recording an event for its attached subscriber based on field values, but that can be done in a follow-up change.

tracing-subscriber/src/subscribe/mod.rs

CAD97 · 2022-03-31T22:05:57Z

I think the next step will be actually implementing the intended behavior

Whoops, that was there in ad609d5 but got lost in the rebase 😅 It should be back now!

I also added some trait-level docs that should help clarify the relationship between the three filtering methods.

hawkw

One thing that I think we need to figure out is that, unlike enabled, the event_enabled method is order-dependent. If a subscriber returns false in event_enabled, any subscribers above it in the stack will still have recorded the event.

Ideally, I think we would want to change this to behave like enabled does, which it looks like we've made a partial attempt to do: the implementation of event_enabled for Layered is non-order-dependent...when a Layered consists of two Subscribers. However, in most cases, a Layered consists of a Subscriber and a Collector, which may itself be a Layered.

Since we don't have an event_enabled method for Collect, Layered's Collect implementation doesn't continue asking the rest of the stack before returning true or false, so the non-order-dependent behavior only exists when subscribers are composed individually before being layered onto a Collector.

This seems unfortunate. We should try to have consistent behavior here.

I think that if we want to make event_enabled non-order-dependent, we would need to also add an event_enabled method to the Collect trait. This could have a default impl that returns true, so it wouldn't be a breaking change to add this in v0.1.x. This would allow the Collect impl for Layered to behave the same as the Subscribe impl for Layered.

Alternatively, we could just decide that we are okay with order-dependent behavior for event_enabled, and change the Subscribe impl for Layered to match. However, this seems not great, because it's inconsistent with all the rest of the filtering methods...

CAD97 · 2022-04-11T22:58:28Z

Honestly, I just hadn't realized that it was still order-dependent; the point of event_enabled -> bool instead of event -> ControlFlow was to make it order-independent, not just to make the change non-breaking / msrv compatible. So I just went ahead and added event_enabled to Collect.

I'm not super happy with the docs -- it's kind of weird to talk about event_enabled on Collect, when a collector could "just" filter in event -- but I think this gets the point across.

The other thing that should be added is a test that this commit fixes, but I'm completely unsure how to go about structuring such a test.

tracing-core/src/collect.rs

hawkw

overall, the code change change looks good to me, thanks for working on this! i had some minor suggestions for the documentation, but once those are addressed, i'll be very happy to merge this!

tracing-core/src/dispatch.rs

tracing-core/src/collect.rs

tracing-subscriber/src/subscribe/mod.rs

hawkw · 2022-06-14T21:00:56Z

tracing-subscriber/src/subscribe/mod.rs

@@ -686,7 +686,31 @@ pub(crate) mod tests;
 /// be composed together with other subscribers to build a [collector]. See the
 /// [module-level documentation](crate::subscribe) for details.
 ///
+/// # Enabling interest
+///
+/// Whenever an tracing event (or span) is emitted, it goes through a number of


might be nice if "event" and "span" linked to the tracing docs, but it's not a big deal --- people reading this documentation probably know what those are by now.

tracing-subscriber/src/subscribe/mod.rs

hawkw · 2022-06-14T21:06:21Z

also, @davidbarsky, do you also want to take a look at the docs change in this PR? no pressure either way!

davidbarsky · 2022-06-14T21:31:00Z

i think you got most of concerns :)

Co-authored-by: Eliza Weisman <eliza@buoyant.io>

hawkw

this looks good to me overall. i left a couple last suggestions, but i'm happy to merge the PR regardless.

tracing-subscriber/tests/event_enabling.rs

tracing-subscriber/src/subscribe/mod.rs

Co-authored-by: Eliza Weisman <eliza@buoyant.io>

hawkw

thanks for the additional tests, this looks good to me!

hawkw · 2022-06-21T17:58:22Z

Ugh, the CI failure is unrelated and is due to another crate's dependency changing its MSRV: https://github.com/tokio-rs/tracing/runs/6990243761?check_suite_focus=true

I'll fix that separately.

## Motivation Allow filter layers to filter on the contents of events (see #2007). ## Solution This branch adds a new `Subscriber::event_enabled` method, taking an `Event` and returning `bool`. This is called before the `Subscriber::event` method, and if it returns `false`, `Subscriber::event` is not called. For simple subscriber (e.g. not using `Layer`s), the `event_enabled` method is not particulary necessary, as the subscriber can just skip the `event` call. However, this branch also adds a new `Layer::event_enabled` method, with the signature: ```rust fn event_enabled(&self, event: &Event<'_>, ctx: Context<'_, S>) -> bool; ``` This is called before `Layer::on_event`, and if `event_enabled` returns `false`, `on_event` is not called (nor is `Subscriber::event`). This allows filter `Layer`s to filter out an `Event` based on its fields. Closes #2007

@jswrenn

# 0.1.28 (June 23, 2022) This release of `tracing-core` adds new `Value` implementations, including one for `String`, to allow recording `&String` as a value without having to call `as_str()` or similar, and for 128-bit integers (`i128` and `u128`). In addition, it adds new methods and trait implementations for `Subscriber`s. ### Added - `Value` implementation for `String` ([#2164]) - `Value` implementation for `u128` and `i28` ([#2166]) - `downcast_ref` and `is` methods for `dyn Subscriber + Sync`, `dyn Subscriber + Send`, and `dyn Subscriber + Send + Sync` ([#2160]) - `Subscriber::event_enabled` method to enable filtering based on `Event` field values ([#2008]) - `Subscriber` implementation for `Box<S: Subscriber + ?Sized>` and `Arc<S: Subscriber + ?Sized>` ([#2161]) Thanks to @jswrenn and @CAD97 for contributing to this release! [#2164]: #2164 [#2166]: #2166 [#2160]: #2160 [#2008]: #2008 [#2161]: #2161

@jswrenn

# 0.1.28 (June 23, 2022) This release of `tracing-core` adds new `Value` implementations, including one for `String`, to allow recording `&String` as a value without having to call `as_str()` or similar, and for 128-bit integers (`i128` and `u128`). In addition, it adds new methods and trait implementations for `Subscriber`s. ### Added - `Value` implementation for `String` ([#2164]) - `Value` implementation for `u128` and `i28` ([#2166]) - `downcast_ref` and `is` methods for `dyn Subscriber + Sync`, `dyn Subscriber + Send`, and `dyn Subscriber + Send + Sync` ([#2160]) - `Subscriber::event_enabled` method to enable filtering based on `Event` field values ([#2008]) - `Subscriber` implementation for `Box<S: Subscriber + ?Sized>` and `Arc<S: Subscriber + ?Sized>` ([#2161]) Thanks to @jswrenn and @CAD97 for contributing to this release! [#2164]: #2164 [#2166]: #2166 [#2160]: #2160 [#2008]: #2008 [#2161]: #2161

@ben0x539

# 0.3.12 (Jun 29, 2022) This release of `tracing-subscriber` adds a new `Layer::event_enabled` method, which allows `Layer`s to filter events *after* their field values are recorded; a `Filter` implementation for `reload::Layer`, to make using `reload` with per-layer filtering more ergonomic, and additional inherent method downcasting APIs for the `Layered` type. In addition, it includes dependency updates, and minor fixes for documentation and feature flagging. ### Added - **layer**: `Layer::event_enabled` method, which can be implemented to filter events based on their field values (#2008) - **reload**: `Filter` implementation for `reload::Layer` (#2159) - **layer**: `Layered::downcast_ref` and `Layered::is` inherent methods (#2160) ### Changed - **parking_lot**: Updated dependency on `parking_lot` to 0.13.0 (#2143) - Replaced `lazy_static` dependency with `once_cell` (#2147) ### Fixed - Don't enable `tracing-core` features by default (#2107) - Several documentation link and typo fixes (#2064, #2068, #2077, #2161, #1088) Thanks to @ben0x539, @jamesmunns, @georgemp, @james7132, @jswrenn, @CAD97, and @guswynn for contributing to this release!

@ben0x539

# 0.3.12 (Jun 29, 2022) This release of `tracing-subscriber` adds a new `Layer::event_enabled` method, which allows `Layer`s to filter events *after* their field values are recorded; a `Filter` implementation for `reload::Layer`, to make using `reload` with per-layer filtering more ergonomic, and additional inherent method downcasting APIs for the `Layered` type. In addition, it includes dependency updates, and minor fixes for documentation and feature flagging. ### Added - **layer**: `Layer::event_enabled` method, which can be implemented to filter events based on their field values (#2008) - **reload**: `Filter` implementation for `reload::Layer` (#2159) - **layer**: `Layered::downcast_ref` and `Layered::is` inherent methods (#2160) ### Changed - **parking_lot**: Updated dependency on `parking_lot` to 0.13.0 (#2143) - Replaced `lazy_static` dependency with `once_cell` (#2147) ### Fixed - Don't enable `tracing-core` features by default (#2107) - Several documentation link and typo fixes (#2064, #2068, #2077, #2161, #1088) Thanks to @ben0x539, @jamesmunns, @georgemp, @james7132, @jswrenn, @CAD97, and @guswynn for contributing to this release!

PR #2008 added a new method to the `tracing_core::Subscriber` trait, and modified `tracing-subscriber` to and implement that method. However, that PR did *not* increase the minimum `tracing-core` dependency for the `tracing-subscriber` crate. This means that if a dependent of `tracing-subscriber` updates *only* `tracing-subscriber` dependency version (e.g. by running `cargo update -p tracing-subscriber`), it will not also update its `tracing-core` version to one that contains the new method, and `tracing-subscriber` will fail to compile. This is a common occurence with projects using Dependabot, for example. This commit updates `tracing-subscriber`'s minimum `tracing-core` dep to 0.1.28. Once this merges, I'll release 0.3.13 of `tracing-subscriber` and yank 0.3.12.

## Motivation PR #2008 added a new method to the `tracing_core::Subscriber` trait, and modified `tracing-subscriber` to and implement that method. However, that PR did *not* increase the minimum `tracing-core` dependency for the `tracing-subscriber` crate. This means that if a dependent of `tracing-subscriber` updates *only* `tracing-subscriber` dependency version (e.g. by running `cargo update -p tracing-subscriber`), it will not also update its `tracing-core` version to one that contains the new method, and `tracing-subscriber` will fail to compile. This is a common occurrence with projects using Dependabot, for example. ## Solution This commit updates `tracing-subscriber`'s minimum `tracing-core` dep to 0.1.28. Once this merges, I'll release 0.3.13 of `tracing-subscriber` and yank 0.3.12.

hawkw reviewed Mar 22, 2022

View reviewed changes

tracing-journald/src/lib.rs Outdated Show resolved Hide resolved

CAD97 force-pushed the on_event_control_flow branch from 22dc571 to ad609d5 Compare March 23, 2022 22:32

CAD97 marked this pull request as ready for review March 23, 2022 22:38

CAD97 requested a review from a team as a code owner March 23, 2022 22:38

CAD97 force-pushed the on_event_control_flow branch from ad609d5 to fde00c9 Compare March 31, 2022 10:11

hawkw reviewed Mar 31, 2022

View reviewed changes

tracing-subscriber/src/subscribe/mod.rs Outdated Show resolved Hide resolved

tracing-subscriber/src/subscribe/mod.rs Show resolved Hide resolved

CAD97 force-pushed the on_event_control_flow branch from 6968e3c to 40db607 Compare March 31, 2022 22:10

hawkw requested changes Apr 11, 2022

View reviewed changes

CAD97 requested a review from carllerche as a code owner April 11, 2022 22:55

hawkw reviewed Apr 11, 2022

View reviewed changes

tracing-core/src/collect.rs Show resolved Hide resolved

CAD97 changed the title ~~Let Subscribe::on_event hide event from lower layers~~ Add Subscribe::event_enabled to conditionally dis/enable events based on fields Jun 14, 2022

CAD97 added 5 commits June 14, 2022 14:51

Let Subscribe disable specific events

029be0e

Explain how various interest enablers work together

2f69943

Rescue event_enabled impls from the rebase wastes

30f0644

Improve event_enabled order independence

ce016e1

Properly call Collect::event_enabled in more places

53ed1dc

CAD97 force-pushed the on_event_control_flow branch from d9591bd to 53ed1dc Compare June 14, 2022 18:54

CAD97 requested a review from hawkw June 14, 2022 18:54

hawkw reviewed Jun 14, 2022

View reviewed changes

Apply code review documentation improvements

f5f1758

Co-authored-by: Eliza Weisman <eliza@buoyant.io>

CAD97 force-pushed the on_event_control_flow branch from e065edb to f5f1758 Compare June 15, 2022 00:34

Ensure event_enabled is not called extra times

952538f

CAD97 requested a review from hawkw June 15, 2022 00:46

CAD97 and others added 2 commits June 14, 2022 20:53

Gate event_enabling test for registry presence

d400cf6

Merge branch 'master' into on_event_control_flow

0eda53b

hawkw approved these changes Jun 16, 2022

View reviewed changes

tracing-subscriber/tests/event_enabling.rs Show resolved Hide resolved

tracing-subscriber/src/subscribe/mod.rs Outdated Show resolved Hide resolved

CAD97 and others added 2 commits June 17, 2022 16:27

Update tracing-subscriber/src/subscribe/mod.rs

57e2a47

Co-authored-by: Eliza Weisman <eliza@buoyant.io>

More event_enabled tests

536ed44

hawkw approved these changes Jun 21, 2022

View reviewed changes

Merge branch 'master' into on_event_control_flow

4620bd2

hawkw enabled auto-merge (squash) June 21, 2022 17:57

hawkw merged commit acb078f into tokio-rs:master Jun 21, 2022

CAD97 deleted the on_event_control_flow branch June 21, 2022 19:14

hawkw mentioned this pull request Jun 22, 2022

backport 9 commits from master #2174

Merged

hawkw mentioned this pull request Jun 23, 2022

chore: prepare to release tracing-core v0.1.28 #2177

Merged

hawkw mentioned this pull request Jun 29, 2022

subscriber: prepare to release v0.3.12 #2187

Merged

msfjarvis mentioned this pull request Jun 30, 2022

tracing-subscriber 0.3.12 fails to compile with env-filter feature #2189

Closed

hawkw mentioned this pull request Jun 30, 2022

subscriber: update minimum tracing-core dep to 0.1.28 #2190

Merged

davidbarsky mentioned this pull request Sep 26, 2023

chore: backport roughly a year's worth of changes #2728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Subscribe::event_enabled to conditionally dis/enable events based on fields #2008

Add Subscribe::event_enabled to conditionally dis/enable events based on fields #2008

CAD97 commented Mar 21, 2022 •

edited

hawkw commented Mar 22, 2022

CAD97 commented Mar 23, 2022

CAD97 commented Mar 26, 2022 •

edited

CAD97 commented Mar 31, 2022

hawkw left a comment

CAD97 commented Mar 31, 2022

hawkw left a comment

CAD97 commented Apr 11, 2022

hawkw left a comment

hawkw Jun 14, 2022

hawkw commented Jun 14, 2022

davidbarsky commented Jun 14, 2022

hawkw left a comment

hawkw left a comment

hawkw commented Jun 21, 2022

Add Subscribe::event_enabled to conditionally dis/enable events based on fields #2008

Add Subscribe::event_enabled to conditionally dis/enable events based on fields #2008

Conversation

CAD97 commented Mar 21, 2022 • edited

Motivation

Solution

Previously, this PR did

hawkw commented Mar 22, 2022

CAD97 commented Mar 23, 2022

CAD97 commented Mar 26, 2022 • edited

CAD97 commented Mar 31, 2022

hawkw left a comment

Choose a reason for hiding this comment

CAD97 commented Mar 31, 2022

hawkw left a comment

Choose a reason for hiding this comment

CAD97 commented Apr 11, 2022

hawkw left a comment

Choose a reason for hiding this comment

hawkw Jun 14, 2022

Choose a reason for hiding this comment

hawkw commented Jun 14, 2022

davidbarsky commented Jun 14, 2022

hawkw left a comment

Choose a reason for hiding this comment

hawkw left a comment

Choose a reason for hiding this comment

hawkw commented Jun 21, 2022

CAD97 commented Mar 21, 2022 •

edited

CAD97 commented Mar 26, 2022 •

edited