by default, don't panic unless there is no alternative

[bhs]

Monitoring should always be "first, do no harm".

Though it's formally undefined behavior in OpenTracing, I propose that the basictracer (and thus basictracer-go) default should be to never-panic unless there is truly no alternative; even if that means dropping data.

(Loud logging messages are fine, though)

cc: @bg451 @yurishkuro @jmacd @tschottdorf

[tbg]

Fine by me, though I like the option to turn assertions on to catch "undefined" behaviour.

[bhs]

@tschottdorf yes definitely... "default" is the key word here :)

[bhs]

btw, I was imagining something like this:

• add an (optional) Logger interface to the basictracer.Options struct
  • default to the Go global Logger if it’s not provided
• only panic when DebugAssertUseAfterFinish is true
• drop data if need be when we would have otherwise panicked, etc (if there’s no tracer, what better can we do?)
• and always (regardless of the debug bit) log something scary to the logger where we presently panic… bonus to log stuff about the span in question if it won’t crash the process

I could also imagine s/DebugAssertUseAfterFinish/DebugUsageErrors/, but that's a nit.

[jmacd]

I can't think of many situations where there is truly no alternative to an intentional crash.

[tbg]

When there's a data race, there aren't many other sane options as sweeping under the rug is only going to make the bugs more subtle. But with the span pool disabled, (internal) data races aren't an issue any more. However, with the span pool on, I think it should panic when it seens an obviously reused span or the panic will eventually occur in a completely random place.