Log record omissions?

[io7m]

Hello!

Following on from my original question about the events API, what I've done as a stopgap measure is to centralize the handling of "things that would be events" in the application(s) I'm working on to make it easier to move to Events when it's available. For everything I would log as an event if the Event API were stable, I'm instead manually constructing an "event" using SLF4J:

  @Override
  public void emit(
    final CSEventType event)
  {
    LoggingEventBuilder builder;
    if (event.isFailure()) {
      builder = LOG.atError();
    } else {
      builder = LOG.atInfo();
    }

    builder = builder.setMessage(event.message());
    builder = builder.addKeyValue("certusine.type", event.getClass().getSimpleName());
    for (final var entry : event.attributes().entrySet()) {
      builder = builder.addKeyValue(entry.getKey(), entry.getValue());
    }
    builder.log();

These logged events show up in traces, which is nice, but I'd also like to be able to have them sent to my logging backend (Grafana Loki) for alerting purposes. The upcoming Log API (in 1.27.0) can obviously do this, but I can't tell from looking at the API if it's possible to essentially eliminate all log events except for those specific events published above. It seems like it's possible to remove attributes from log records in a LogRecordProcessor, but not actually discard entire records.

Essentially, I'd like to be able to say "Don't send any log events over OTLP except for those that are specifically coming from CSEventService.class". I don't want every little thing the application says being sent to the log server.

One way to do this would be to install an extra logging appender in Logback and do the filtering there, but I'd prefer not to do that if I can remove the events in the OpenTelemetry LogRecordProcessor and not have to deal with Logback specifics.

[jack-berg]

It seems like it's possible to remove attributes from log records in a LogRecordProcessor, but not actually discard entire records.

Correct. You can overwrite attribute values in a LogRecordProcessor, though that's not the same as deleting them since an empty value is semantically meaningful. If you want to strip the attributes altogether, or filter a log record altogether, you'll have to do that in a custom LogRecordExporter, which can mutate the Collection<LogRecordData> batch before delegating to to some other exporter to send over the wire (e.g. OtlpGrpcLogRecordExporter).

One way to do this would be to install an extra logging appender in Logback and do the filtering there, but I'd prefer not to do that if I can remove the events in the OpenTelemetry LogRecordProcessor and not have to deal with Logback specifics.

That's something you should consider - it may be easier / more configurable than writing a custom exporter.

[io7m]

OK, thanks!

Looking at the implementations of LogRecordExporter, it doesn't seem so bad...

[io7m]

The implementation of this was straightforward with one extremely minor exception: The collection passed in to the exporter is immutable and so can't be modified directly. It was obviously trivial to filter the items with a stream, so no problems there.