Remember entities issues in AKKA.net 1.5.2 #6843
Replies: 7 comments 6 replies
-
|
This is in Akka.NET v1.4.39? I'm skeptical because the RememberEntitiesStore was introduced in 1.5.0, IIRC |
Beta Was this translation helpful? Give feedback.
-
|
But the reason for this, even in 1.5.0, would more than likely be a bad remember-entities configuration. And if it's not that, then busy-ness - I'd need more logs from the failed RE store to know for certain. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry yeah 1.5.2, pretty sure we have no configuration except for turning
it on, how would I get logs from RE store
…On Fri, 21 Jul 2023 at 15:55, Aaron Stannard ***@***.***> wrote:
But the reason for this, even in 1.5.0, would more than likely be a bad
remember-entities configuration. And if it's not that, then busy-ness - I'd
need more logs from the failed RE store to know for certain.
—
Reply to this email directly, view it on GitHub
<#6843 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSAOP7PU5XJ5I2KWAP4HD3XRKJ7TANCNFSM6AAAAAA2S57M2M>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Can you share your configuration? Edit: that'll help me answer how to get logs |
Beta Was this translation helpful? Give feedback.
-
|
Yeah thanks, her you go
…On Fri, 21 Jul 2023 at 16:18, Aaron Stannard ***@***.***> wrote:
Can you share your configuration?
—
Reply to this email directly, view it on GitHub
<#6843 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSAOP5G7YNB6QGC2RLCXS3XRKMV7ANCNFSM6AAAAAA2S57M2M>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
petabridge.cmd {
host = "0.0.0.0"
port = 9110
}
akka {
coordinated-shutdown {
exit-clr = on
}
extensions =
["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider,
Akka.Cluster.Tools"]
loglevel = DEBUG
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
actor {
provider = cluster
default-dispatcher = {
executor = channel-executor
fork-join-executor { #channelexecutor will re-use these settings
parallelism-min = 2
parallelism-factor = 1
parallelism-max = 64
}
}
serializers {
hyperion = "Akka.Serialization.HyperionSerializer,
Akka.Serialization.Hyperion"
akka-cluster-client =
"Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer,
Akka.Cluster.Tools"
}
serialization-bindings {
"Akka.Cluster.Tools.Client.IClusterClientMessage,
Akka.Cluster.Tools" = akka-cluster-client
"System.Object" = hyperion
}
serialization-identifiers {
"Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer,
Akka.Cluster.Tools" = 15
}
}
test {
timefactor = 10
}
cluster {
downing-provider-class =
"Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"
split-brain-resolver {
active-strategy = keep-majority
}
client {
receptionist {
role = "bet-processor"
}
}
sharding {
least-shard-allocation-strategy.rebalance-threshold = 3
rebalance-interval = 60s
remember-entities = on
remember-entities-store = "eventsourced"
snapshot-after = 20
role = "bet-processor"
state-store-mode = persistence
# fail if upgrade doesn't succeed
fail-on-invalid-entity-state-transition = on
}
failure-detector {
# FQCN of the failure detector implementation.
# It must implement akka.remote.FailureDetector and have
# a public constructor with a com.typesafe.config.Config and
# akka.actor.EventStream parameter.
implementation-class = "Akka.Remote.PhiAccrualFailureDetector,
Akka.Remote"
heartbeat-interval = 5s
acceptable-heartbeat-pause = 13s
expected-response-after = 5s
monitored-by-nr-of-members = 7
}
seed-nodes = []
roles = ["bet-processor", "pub-sub-user"]
role.bet-processor.min-nr-of-members = 3
# Enable/disable info level logging of cluster events
log-info = on
# Enable/disable verbose info-level logging of cluster events
# for temporary troubleshooting. Defaults to 'off'.
log-info-verbose = off
}
remote {
maximum-payload-bytes = 2000000 bytes
log-sent-messages = on
dot-netty.tcp {
port = 4256
message-frame-size = 300000b
send-buffer-size = 300000b
receive-buffer-size = 300000b
maximum-frame-size = 2000000b
}
}
healthcheck {
log-config-on-start = off
log-info = off
liveness {
providers {
default =
"Akka.HealthCheck.Liveness.DefaultLivenessProvider, Akka.HealthCheck"
cluster =
"Akka.HealthCheck.Cluster.ClusterLivenessProbeProvider,
Akka.HealthCheck.Cluster"
}
}
readiness {
providers {
default =
"Akka.HealthCheck.Readiness.DefaultReadinessProvider, Akka.HealthCheck"
cluster =
"Akka.HealthCheck.Cluster.ClusterReadinessProbeProvider,
Akka.HealthCheck.Cluster"
}
}
}
}
…On Fri, 21 Jul 2023 at 16:23, Derek Goodwin ***@***.***> wrote:
Yeah thanks, her you go
On Fri, 21 Jul 2023 at 16:18, Aaron Stannard ***@***.***>
wrote:
> Can you share your configuration?
>
> —
> Reply to this email directly, view it on GitHub
> <#6843 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACSAOP5G7YNB6QGC2RLCXS3XRKMV7ANCNFSM6AAAAAA2S57M2M>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
|
Ok, this was very helpful: sharding {
least-shard-allocation-strategy.rebalance-threshold = 3
rebalance-interval = 60s
remember-entities = on
remember-entities-store = "eventsourced"
snapshot-after = 20
role = "bet-processor"
state-store-mode = persistence
# fail if upgrade doesn't succeed
fail-on-invalid-entity-state-transition = on
}
Ok, with these settings in mind you're using Akka.Persistence for everything. That explains: So if you see this error occurring it usually means that Akka.Persistence is having trouble connecting to or deserializing messages from the underlying database. Aside from database timeouts perhaps, those will appear as giant screaming errors in the Akka.NET error log and will automatically fail the Akka.Persistence liveness check in Akka.HealthChecks. I don't see any Akka.Persistence configuration in your HOCON - are you using it? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah here you go, thanks for trying to help with this
akka {
persistence {
max-concurrent-recoveries = 5
journal {
plugin = "akka.persistence.journal.redis"
redis {
class = "Akka.Persistence.Redis.Journal.RedisJournal,
Akka.Persistence.Redis"
plugin-dispatcher =
"akka.persistence.dispatchers.default-plugin-dispatcher"
configuration-string = ""
database = 0
replay-filter {
mode = repair-by-discard-old
}
key-prefix = "Bet"
recovery-event-timeout = 60s
circuit-breaker {
max-failures = 10
call-timeout = 60s
reset-timeout = 30s
}
}
event-adapters {
coordinator-migration =
"Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter,
Akka.Cluster.Sharding"
}
event-adapter-bindings {
"Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent,
Akka.Cluster.Sharding" = coordinator-migration
}
}
snapshot-store {
plugin = "akka.persistence.snapshot-store.redis"
redis {
class =
"Akka.Persistence.Redis.Snapshot.RedisSnapshotStore, Akka.Persistence.Redis"
plugin-dispatcher =
"akka.persistence.dispatchers.default-plugin-dispatcher"
configuration-string = ""
database = 0
replay-filter {
mode = repair-by-discard-old
}
key-prefix = "Bet"
}
}
}
}
…On Fri, 21 Jul 2023 at 17:06, Aaron Stannard ***@***.***> wrote:
Ok, this was very helpful:
sharding {least-shard-allocation-strategy.rebalance-threshold = 3rebalance-interval = 60sremember-entities = onremember-entities-store = "eventsourced"snapshot-after = 20role = "bet-processor"state-store-mode = persistence
# fail if upgrade doesn't succeedfail-on-invalid-entity-state-transition = on
}
Ok, with these settings in mind you're using Akka.Persistence for
everything. That explains:
[RecoveryTick] from [akka://xxx/system/sharding/xxx/17/RememberEntitiesStore#346032082] to [akka://xxx/system/sharding/xxx/17/RememberEntitiesStore#346032082] was not delivered. [4] dead letters encountered. If this is not an expected behavior then [akka://xxx/system/sharding/Bet/17/RememberEntitiesStore#346032082] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Akka.Persistence.RecoveryTick
So if you see this error occurring it usually means that Akka.Persistence
is having trouble connecting to or deserializing messages from the
underlying database. Aside from database timeouts perhaps, those will
appear as giant screaming errors in the Akka.NET error log and will
automatically fail the Akka.Persistence liveness check in Akka.HealthChecks.
I don't see any Akka.Persistence configuration in your HOCON - are you
using it?
—
Reply to this email directly, view it on GitHub
<#6843 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSAOP3HLQV2UYSAR6FR3I3XRKSGXANCNFSM6AAAAAA2S57M2M>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Ok, that looks good to me - how often are you running into these errors from the sharding system and are there underlying logs from Akka.Persistence.Redis itself? Timeouts? Serialization errors? Anything notable? |
Beta Was this translation helpful? Give feedback.
-
|
Would logging for Akka.Persistence.Redis go into a different file? Maybe we
just haven't got it turned on, how do I do that?
…On Fri, 21 Jul 2023 at 17:10, Aaron Stannard ***@***.***> wrote:
Ok, that looks good to me - how often are you running into these errors
from the sharding system and are there underlying logs from
Akka.Persistence.Redis itself? Timeouts? Serialization errors? Anything
notable?
—
Reply to this email directly, view it on GitHub
<#6843 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSAOP7WQUNEKZ6OZL52K3LXRKSYXANCNFSM6AAAAAA2S57M2M>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
It'd show up in the default log as an ERROR, which will get captured automatically at default logging levels. |
Beta Was this translation helpful? Give feedback.
-
|
Hi seeing these as well, is this an issue with Redis not coping with the
load
Exception: System.InvalidOperationException: Async write timed out after
00:00:05 at
Akka.Cluster.Sharding.Shard.<>c__DisplayClass73_0.<WaitingForRememberEntitiesStore>g__WaitingForRememberEntitiesStore|0(Object
message) at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object
message) at Akka.Actor.ActorCell.ReceiveMessage(Object message) at
Akka.Actor.ActorCell.Invoke(Envelope envelope)
"Bet": Remember entity store did not respond, restarting shard
}
…On Fri, 21 Jul 2023 at 17:35, Aaron Stannard ***@***.***> wrote:
It'd show up in the default log as an ERROR, which will get captured
automatically at default logging levels.
—
Reply to this email directly, view it on GitHub
<#6843 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSAOP25NHROLA526HF6A73XRKVWRANCNFSM6AAAAAA2S57M2M>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
@derekgoodwin ok, so that would do it - Redis couldn't complete the write within 5s. Is Redis running under a high load at the time this error occurs? |
Beta Was this translation helpful? Give feedback.
-
Getting these errors from AKKA on our application, anyone got any ideas
"XXX": Failed to load initial entity ids from remember entities store within [00:00:05], stopping shard for backoff and restart[RecoveryTick] from [akka://xxx/system/sharding/xxx/17/RememberEntitiesStore#346032082] to [akka://xxx/system/sharding/xxx/17/RememberEntitiesStore#346032082] was not delivered. [4] dead letters encountered. If this is not an expected behavior then [akka://xxx/system/sharding/Bet/17/RememberEntitiesStore#346032082] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Akka.Persistence.RecoveryTickBeta Was this translation helpful? Give feedback.
All reactions