Actor mailbox delays when service is hosted as an Azure App Service

[CalumPeebles]

I've been having an issue for a while now (around 2 months) where I'm experiencing delays only when the service is ran as an Azure App Service. There are often delays of up to 2 seconds between the IActorRef.Tell() call and the message being picked up by the recipient actor, even when there are no messages being waited for.

I've only seen this happen when running the service as an Azure App Service, it works fine on my laptop and on an Azure Windows 11 VM with much lower RAM and CPU capacity than the App Service plan. The service is lightweight and I've tried increasing the App Service plan to an insanely high spec one and the issue still occurs, so that rules out computing power as the cause.

The technical support team at Azure have been looking into it for nearly 2 months but are stumped as they have little knowledge of akka.net and haven't been able to debug it properly.

Has anyone else experienced this? Or know what might be causing it?

Thanks

[Aaronontheweb]

@CalumPeebles so a 2-second delay between IActorRef.Tell and a message being received by the actor is pretty unusual - the actor just gets scheduled to execute on the normal ThreadPool in .NET typically is what happens.

By any chance, does this only happen the first time the application has been used in a while? i.e. is this an infrequently used service? Or is this a regular issue that occurs with a system that has a consistent amount of traffic?

[Aaronontheweb]

And this problem does not occur when running locally / not on Azure App Service? Could you share the following with us:

1. OS & .NET Version
2. Akka.NET Version
3. HOCON configuration for the affected actors and
4. Ideally, it'd be great to see the actor source itself - it'd be useful to see if there's anything inside the actor that can be "long-running" as that is an easily explainable cause for this issue too.

[CalumPeebles]

The "unsolicited" waiting happens very rarely when running elsewhere (both on my laptop and running the DLL on an Azure Windows 11 VM), and the wait times are very short when it does happen - if I remember correctly I believe around sub 200ms. I assume this is due to garbage collection.

1. The OS version is: Microsoft Windows [Version 10.0.14393], and .NET version is .NET 6.0
2. Akka.NET version is 1.4.40
3. How can I output the HOCON configuration for the actors?
4. Sorry I can't distribute the full solution, but I can show specific sections of how the actors are set up if that'd help? Also I don't see how long running threads could be the cause as the waiting is happening when the actor isn't currently processing any thread

Do you know of any affordable tools I can use to get better logging/metrics for the actor states/mailboxes etc? (so far the only ones I've come across seem to be a bit over my budget as it's just me working on this). Or even one that provides a trial so I can just dig into this specific issue?

Thanks

[Arkatufus]

Are you using Akka.Hosting or ActorSystemSetup?
You can add akka.log-config-on-start = on in your HOCON, this will dump the HOCON config during ActorSystem startup, though settings being passed as Setup programmatically might not show up in the final configuration.

[CalumPeebles]

Sorry have been away!

I'm using ActorSystemSetup.

I've just added an app.conf file containing the following:

akka{
	akka.log-config-on-start = on
}

However nothing appears to be output - perhaps I'm not setting it up correctly?

Here is the section of code where I'm creating the system and the orchestration actor. I should point out that there are other actors that are created elsewhere, however the orchestration actor sees by far the majority of the traffic and is where I'm seeing the delays.

[Aaronontheweb]

@CalumPeebles we'll take a look - I've been overseas presenting at some tech conferences but I'm now back in my office.