Enable debug logging for ShardCoordinator/ShardRegion/Shard actors

[LeonHartley]

Is your feature request related to a problem? Please describe.
We are seeing potential issues with the shard coordinator when recovering millions of entities, where shard regions do not receive acknowledgement from the coordinator. We'd like to allow debug logs to be switched on for sharding system-level actors without having to enable debug logging for the whole system

Describe the solution you'd like
A configuration switch to perhaps change the log level of all logs from the shard coordinator and shard region actors, so we can view more information, even when the rest of the system is set to Information or Warning level.

Describe alternatives you've considered
We could switch debug logging on for the whole system but this would cause unnecessary stress on our logging infrastructure.

[Aaronontheweb]

There is a setting that will do this for Akka.Cluster.Sharding already, have you tried setting this to on ?

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/reference.conf

Line 239 in f7ed2ac

verbose-debug-logging = off

[LeonHartley]

From the looks of it, that would still require us to enable debug logging for the whole system, this feature request is more around allowing debug logs without turning full system level log level to debug.

[Aaronontheweb]

I guess we could change that setting to accept a LOGLEVEL argument - that should do it

[LeonHartley]

Yeah that would work.

Cheers

[Aaronontheweb]

@LeonHartley so I implemented a fix for this in #7118 but decided not to merge it for 1.5.18 because it's kind of weird / sloppy.

What I think we're going to do instead is introduce a new "log filtering" feature on the backend that allows you to scrub out logs you don't care about at various log levels. I'll create a separate GH issue to spec it out, but I think that will be a much cleaner solution to your problems - it'll let you enable LogLevel.Debug and just say "get rid of everything that isn't from LogSources that start with /system/sharding/. Would that work for you?

[LeonHartley]

Yeah that sounds perfect, nice one @Aaronontheweb.

[Aaronontheweb]

Working on a preview of this feature here #7179