Adjusted frame length exceeds 128000: 542393675

[mnoroozi1990]

Version Information
Version of Akka.NET? 1.4.42
Which Akka.NET Modules? Akka.Remote

Describe the bug
I have an actorsystem that I hosted it as a workerService. It's a node of a sharding cluster and also seed node of that cluster. It's completely independent and I want to test and launch it separately.

I get 4 errors almost everyday at a specific time as I explain below.

Errors occurred in less than a second with the same message of :
"Error caught channel "[::ffff:x.x.x.x]:2006"->"[::ffff:x.x.x.x]:59312""

and different exception messages of :
"Adjusted frame length exceeds 128000: 542393675 - discarded"
"Adjusted frame length exceeds 128000: 542393675 - discarded"
"Adjusted frame length exceeds 128000: 197402 - discarded"
"Adjusted frame length exceeds 128000: 131866 - discarded"
the order of occurrence is always like this.

I don't understand the port specified in error message. I mean "59312". because I host the actorsystem on port "2006" and seed node of cluster as I said, is itself.
I think the system is internally hosting something on port "59312". A more information I should tell you is that another log with debug level I get right after explained error as below:
Association between local ["tcp://XXX@x.x.x.x.:2006"] and remote ["tcp://XXX@[::ffff:x.x.x.x]:59312"] was disassociated because "the ProtocolStateActor failed: Unknown".

Finally...
*******The most important thing in my scenario is this, these errors occurred while system(workerService) is idle and there is no external request or internal scheduling jobs out there. If I say simply, system is just Up and is doing nothing.

I have this problem in both Alpha and Staging environment.

Environment
I'm running worker sevice on Windows and .NET core 3.1?

[Aaronontheweb]

Have you consider increasing your max frame size in Akka.Remote? It defaults to 128kb:

akka.remote.dot-netty.tcp.maximum-frame-size = 128000b

This error appears to be that one of the nodes is accepting messages that are a significantly larger size than this

[mnoroozi1990]

Thank you Aaron for your attention.
I've read and checked this config before and I know the effect.

But as I mentioned in issue, system is totally idle when errors happen.

I must emphasis the Size of frame in error. it is about 500mb. All of messages that are cycling in system are command objects with less than 10 properties.most of them are primitive data types. and I never produce such an object with that size.

system doesn't receive anything and I set these Akka.Remote.log-received-messages and Akka.Remote.log-sent-messages configs to ON in order to log inbound and outbound logs. but No logs there.

when system works, there isn't such frame size error and everything works fine as expected. as I mentioned I'm testing system and I checked every possible scenarios of system business. everything is great.

with these facts, is there any thing that I missed in configuration?
on more time, I should say:

I don't understand the port specified in error message. I mean "59312". because I host the actorsystem on port "2006" and seed node of cluster is itself.
I think the system is internally hosting something on port "59312". A more information I should tell you is that another log with debug level I get right after explained error as below:
Association between local ["tcp://XXX@x.x.x.x.:2006"] and remote ["tcp://XXX@[::ffff:x.x.x.x]:59312"] was disassociated because "the ProtocolStateActor failed: Unknown".

Isn't it weird? or there is some thing in framework that I don't know.

thanks in advanced for helping.

[to11mtm]

If I had to guess, the "59312" port is the 'outbound' port for the node in question.

I could be wrong (it's been a while since I've debugged remoting) but if I am remembering properly... there are TWO connections between nodes. Basically, there are separate ports for Inbound and outbound messages. The outbound message port is typically semi-random, because TCP (and, AFAIK you can't really re-use ports for multiple outbounds, but I could be totally wrong on that).

So, if you have Node 'A' listening on port 2006 and Node 'B' listening on port 2007, it's possible that the inbound messages from B to A will show a different port than 2007, at least at the transport layer. I think normally in most of akka remote and the rest of akka, this port number is instead corrected to be the system's inbound port, but some internals of remote the outbound port of the sender is shown instead (@Aaronontheweb please please please correct me if I'm wrong here 😅.)

[RomanChernyshev]

Hi. I have the same problem. And I don't send any messages at all. Receive exactly this message in couple of minutes after the start. I use <PackageReference Include="Akka.Cluster.Tools" Version="1.4.0" />, Env - k8s + istio. Is there any idea?

[ERROR][02/22/2024 13:52:28][Thread 0019][TcpServerHandler (akka://cluster)] Error caught channel [[::ffff:10.109.115.67]:25520->[::ffff:10.109.19.67]:3 │
│ Cause: DotNetty.Codecs.TooLongFrameException: Adjusted frame length exceeds 128000: 542393675 - discarded                                                │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Fail(Int64 frameLength)                                                                               │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input)                                             │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input, List`1 output)                              │
│    at DotNetty.Codecs.ByteToMessageDecoder.CallDecode(IChannelHandlerContext context, IByteBuffer input, List`1 output)                                  │
│    at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext context, Object message)                                                   │
│    at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelRead(Object msg)

[Aaronontheweb]

Hi. I have the same problem. And I don't send any messages at all. Receive exactly this message in couple of minutes after the start. I use <PackageReference Include="Akka.Cluster.Tools" Version="1.4.0" />, Env - k8s + istio. Is there any idea?

[ERROR][02/22/2024 13:52:28][Thread 0019][TcpServerHandler (akka://cluster)] Error caught channel [[::ffff:10.109.115.67]:25520->[::ffff:10.109.19.67]:3 │
│ Cause: DotNetty.Codecs.TooLongFrameException: Adjusted frame length exceeds 128000: 542393675 - discarded                                                │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Fail(Int64 frameLength)                                                                               │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input)                                             │
│    at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input, List`1 output)                              │
│    at DotNetty.Codecs.ByteToMessageDecoder.CallDecode(IChannelHandlerContext context, IByteBuffer input, List`1 output)                                  │
│    at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext context, Object message)                                                   │
│    at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelRead(Object msg)

Do you have K8s healthchecks hitting the Akka.Remote inbound endpoint? If so, that would do this.

[Aaronontheweb]

Reason being is that Akka.Remote expects packets with a specific encoding present - K8s socket healthchecks basically send telnet messages, which don't support the frame-length encoding that Akka.Remote expects.

[RomanChernyshev]

I checked - nothing. I don't see any explicit definitions of health checks for the akka port. But I tried to send an http request to the akka port locally and it results in this error.

Now I think about akka-heartbeats. Maybe my k8s network somehow is altering this traffic.