Logging on standard err

[ghost]

It seems like the logging library used (github.com/Sirupsen/logrus) logs everything to stderr.

@freddd says that it’s a known logrus problem and would suggest using https://github.com/uber-go/zap instead).

[freddd]

For reference:

• How to separate Stdout and Stderr output? sirupsen/logrus#403
• logrus outputs all the logs to stderr by default Azure/open-service-broker-azure#25

Are you interested in assisting in maintaining Logrus? Currently I have a lot of obligations, and I am unable to provide Logrus with the maintainership it needs. If you'd like to help, please reach out to me at simon at author's username dot com. - https://github.com/sirupsen/logrus/blob/master/README.md

[tredman]

Hi, can you help me understand what problem you're trying to solve by having the agent write to stdout? My understanding is that the usual pattern is: program output -> stdout, program logs -> stderr. Given that the "output" of the agent is Honeycomb events and the logs emitted by the agent are mostly for debugging and informational purposes, this seems appropriate, but maybe I am missing something.

[freddd]

Hi @tredman, Sure. In Google Stackdriver if you pipe to stderr it will automatically assume it's an error no matter what level or severity is set to. Because of this all debugging/informational logs going to stderr will basically drown the real errors causing lots of false positives and more or less making the alerts you might have on error logs useless.

Thanks for responding so quickly!

[tredman]

Hi @freddd and @mariana-bocoi - we've patched our agent to allow optionally splitting stdout and stderr by severity. (trace, debug, info, warn to stdout, everything higher to stderr) It's sort of a workaround as we don't have the cycles to swap out our log lib at the moment. If you redeploy the agent with the latest image and specify splitLogging: true in the ConfigMap, you should get the behavior you need. Here's a link to a sample config: https://github.com/honeycombio/kubernetes-manifests/blob/master/logs/quickstart.yaml#L69-L71

[ghost]

Thanks!