Can we export application logs to s3?

[anjulaendowus]

We have a requirement where we need to export application logs to s3. One of the possible option would be to use fluentd and we are looking at the OTel collector as well. Please provide your suggestions on this.

[jack-berg]

OpenTelemetry logs supports log forwarding, where application logs are forwarded from the console using something like fluentd, or exporting logs directly to a collector, where application logs are exported to a collector or other storage directly from the application using a network protocol like OTLP. These solutions are described in more detail here and here.

Note that for the direct to collector approach, opentelemetry-java is bundled with an OTLP exporter (OtlpHttpLogRecordExporter / OtlpGrpcLogRecordExporter), but its also possible to implement your own LogRecordExporter.

[anjulaendowus]

So if we use the direct to collector approach, are we able to store our logs in S3 in parquet format?

[jack-berg]

You'll have to explore the options available in the collector. The collector has an extensible architecture which allows you to write your own receivers, processors, exporters, etc, but has a number bundled in (the set of components bundled will vary based on the distribution). Not sure if any of the community-provided exporters support s3 in parquest format: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter

[trask]

this would be a better question for https://github.com/open-telemetry/opentelemetry-collector-contrib or #otel-collector slack channel (see @jack-berg's response above)