Finding what triggered the reconciliation

[kazk]

(I realized I shouldn't be thinking this way, but decided to post anyway because it might be helpful for others to realize the flaw and also interested in any thoughts)

Question For a controller with core object T that owns A and B, is there a way to find what triggered its reconciler? (T changed or A changed or B changed or time)

This question came up when I was implementing a controller with T that has status conditions based on each child. I wanted to know what triggered it, so reconciler can focus on that case. But this is not a good way to think about reconciliation because that specific event may be lost. By not focusing on the trigger and using requeue_after: Some(duration), the outdated status condition depending on A can be recovered on reconciliation triggered by B or time.

I understood this as a concept, but didn't realize at first when writing code, probably because of the mindset. I'm used to writing code reacting to specific events.

[nightkr]

1. Yeah.. might be worth mentioning this in the Controller docs
2. It might actually be worth adding some kind of logging for this, to help people understand where the updates are coming from? In general the philosophy of kube-runtime has been to stay logger-agnostic, but I'm really not keen on making this a part of the public interface. Maybe we could make an opaque Display-only object (like std::fmt::Arguments) describing the cause?

[kazk]

I found that tracing has a feature log that enables emitting log Record instead when no Subscriber is active. It can be a log unless a user opts-in with Subscriber.

• log (+ lowest common denominator) (- no structure)
• slog (+ ?) (- new dependency, not logger agnostic)
• tracing (+ used by Tokio ecosystem, can be logger agnostic, opt-in structure) (- ?)

Did I miss anything? I don't see any downsides with tracing now assuming log records won't annoy slog users.

So for kube, it can be a drop-in replacement. Adding to kube-runtime will be nice too.

[clux]

I think this sounds like a good idea. I found this sloggish-collector which may smoothen the path even further.

At any rate, I'm going to be exploring tracing more myself as $DAYJOB is going all in on it. We can't really justify passing loggers around in kube, but the way tracing does it with subscribers is very attractive. In big kubernetes environments there's a lot of emphasis on big o11y stacks with metrics/traces anyway so given where we are, we should try this out more I think.

[davidbarsky]

(Hiya, I'm a tracing contributor. I found this via https://clux.github.io/probes/post/2021-02-28-kube-evolution/.)

You don't need to take a dependency on tracing-futures unless you're interested in instrumenting futures 0.3-specific types such as futures::stream::Stream or futures::sink::Sink. The Instrument trait is implemented on std::future::Future, and once streams land on stable, std::stream::Stream.

We're starting to look into migrating towards tracing (because that's where the momentum seems to be), but custom tracing Subscribers seem to be a lot more complicated than slog Drains are (and we already have some slog Drains that we'd need to migrate before actually using tracing is on the table at all).

You're not wrong about that and it's something that I'd like to improve. A decent portion of the complexity comes from tracing needing to buffer spans prior to them being emitted, but high-level idea is that tracing_subscriber::Registry can be used with a Layer, where a registry provides storage and a Layer provides a Subscriber-style interface. I should probably write a simple example for the examples...

(Note that in tracing 0.2, we'll rename Layer to Subscribe and tracing::Subscriber to tracing::Collect.)

[clux]

Having tried out tracing instrumentation as part of a grafana cloud and tempo deep dive this week, I have found the results to be very helpful in terms of debuggability, and lets us give users some very select slices into the underlying behaviour by inserting a few key #[instrument] macros within our crates. See #455 for one dealing with tracing Api calls.

Compatibility issues with people who just want stdout seem relatively easy to deal with, as the customizability of tracing::subscriber::fmt feels sufficient for most defaults. It even provides env_logger style for 2/3 lines in your main so people using RUST_LOG evars can carry on. AFAIKT from a few slog vs tracing discussions, people aren't left without options for more complicated ones either.

Personally, I think the real complexity of the tracing ecosystem comes when you need an actual tracing collector and need to start to hook that up with a library that embeds an http/grpc client. Then you'll have new fun build issues, potential quirks in the various implementations, and potentially 5 crates to battle with to get span ids and state through.

However, a modern setup with opentelemetry not too bad to setup (and once we have examples it'll only be easier for the rest). It worked very nicely with a modern setup like tempo, which I will encourage. At any rate, wanted to share a bit. I will suggest some approaches for kube and kube-runtime in a separate issue.

[nightkr]

FWIW, there is now a PoC for this up at #571.

[nightkr]

0.58's Controller (and applier) now attaches a tracing span with the (approximate) reason for the reconciliation.