Add severity and message to logging payload to improve stackdriver log viewing

[diox]

Stackdriver currently displays the whole json payload for each line of log because we don't set a message property. Similarly, because we don't set severity all messages are considered to be of the same severity, preventing us to filter by log level.

If we were to add those fields it would increase payload size (since they'd be duplicated) but we'd get better logging viewing experience.

[diox]

Note: ideally, all this would be part of some new spec, maybe mozLog 3.0, and handled in https://github.com/mozilla-services/python-dockerflow/blob/master/src/dockerflow/logging.py. In the meantime though I'd like to get nice logs in stackdriver so we'll probably have to implement it ourselves.

[bqbn]

I think the downside of doing this is probably increased payload, networking traffic and cost. Cost wise, we currently emit about 5 GB application logs per day. Doubling that should not be a huge deal.

Also maybe this can be made so that it can be switched on/off by an environment variable, so that if/when we stop using stackdriver, we can turned it off. The scenario is less likely to happen near-term, but given that we have changed our logging platform a few times, it's not entirely impossible. :P

[diox]

Once a new release of python-dockerflow includes the changes I made in mozilla-services/python-dockerflow#37 we'll be able to customize the payload more easily.

[diox]

we merged mozilla/addons-server#12439 which means we're now using a python-dockerflow version including those changes.

[diox]

@bqbn I have a patch locally that works. Do you have an opinion regarding duplicating the message vs removing Fields['msg'] from the payload entirely and instead emit it as message ? I can do either, behind an env variable as you suggested. The latter would kinda be breaking mozLog 2.0 spec though.

[willdurand]

FWIW, I personally use custom fields in stackdriver: https://cloud.google.com/logging/docs/view/overview#custom-fields

I have this one configured all the time:

jsonPayload.Fields.msg

[diox]

Would it be a problem to switch to message instead ? I guess we'd try it on dev in any case.

[bqbn]

The latter would kinda be breaking mozLog 2.0 spec though.

If this is true, then I prefer the former for the reason that it doesn't break the spec.

The logging platform and tools built around it are shared across all services cloudops has to manage, and it's assumed that the spec is adhered to. If we have something that breaks the spec, I fear that it'll cause problems in the future, say, when new feature or tools are introduced to the logging platform.

[diox]

FWIW, I personally use custom fields in stackdriver: https://cloud.google.com/logging/docs/view/overview#custom-fields

I do too but the problem with custom fields is that you can't remove the big json blob column when you use them, so the view is still a little crowded, with lots of extra garbage being displayed. Worse, if the message is a bit too long, it gets truncated in favor of the json blob column.

[diox]

@bqbn I might just set severity for now and ignore message, cause the duplication of the text bothers me. Do you know if anything in cloudops logging pipeline/services would have issues if I add a severity set in addition to the Severity field that is already set? JSON is case-sensitive, and Stackdriver should be as well.

A typical record would then contain:

{
    //...
    "Severity": 7,
    "severity": 100
}

(Note the different values, since MozLog's Severity (uppercase s) are syslog levels, and Stackdriver's severity (lowercase s) are different.

[bqbn]

Quote from Slack,

@diox i don't remember the context of the question, but right off the bat, cloudops doesn't manipulate either field, Severity or severity, if there's going to be an issue, it's going to be either stackdriver, the flentd agent or the mozLog client lib.

[diox]

Verified myself, it's working, we now have severity display/filtering in stackdriver for addons-server dev: