Error in SplitBrainResolver.PreStart when using ChannelTaskScheduler for internal-dispatcher #5962

SchiessMax · 2022-05-25T06:33:30Z

Version Information
Version of Akka.NET? 1.4.38
Which Akka.NET Modules? Akka.Cluster

Describe the bug
When using the Akka.Cluster.SBR.SplitBrainResolver while having configured the ChannelTaskScheduler as internal dispatcher an error is thrown in the PreStart method. The error is thrown when Cluster.Get(Context.System) is called.
I think this might be related to #5861

The error that is logged:

ValueFactory attempted to access the Value property of this instance.
      [akka://CasActorSystem/system/cluster/core/daemon/downingProvider#1885816536]: Akka.Actor.ActorInitializationException: Exception during creation
       ---> System.InvalidOperationException: ValueFactory attempted to access the Value property of this instance.
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
         at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension[T](T& extension)
         at Akka.Actor.Internal.ActorSystemImpl.GetExtension[T]()
         at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
         at Akka.Cluster.Cluster.Get(ActorSystem system)
         at Akka.Cluster.SBR.SplitBrainResolver.PreStart()
         at Akka.Actor.ActorBase.AroundPreStart()
         at Akka.Actor.ActorCell.<>c__DisplayClass176_0.<Create>b__0()
         at Akka.Actor.ActorCell.UseThreadContext(Action action)
         at Akka.Actor.ActorCell.Create(Exception failure)
         --- End of inner exception stack trace ---
         at Akka.Actor.ActorCell.Create(Exception failure)
         at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

To Reproduce
Steps to reproduce the behavior:

Create an ActorSystem with the following configuration:

akka {
    actor {
        provider = cluster
        default-dispatcher = {
            executor = "channel-executor"
            channel-executor.priority = "normal"
        }
       # Adding this part in combination with the SplitBrainResolverProvider causes the error
        internal-dispatcher = {
            executor = "channel-executor"
            channel-executor.priority = "high"
        }
    }
    remote {
        dot-netty.tcp {
            port = 5508
            hostname = 127.0.0.1
        }
        default-remote-dispatcher {
            executor = "channel-executor"
            channel-executor.priority = "high"
        }
        backoff-remote-dispatcher {
            executor = "channel-executor"
            channel-executor.priority = "low"
        }
    }
    cluster {
        seed-nodes = ["akka.tcp://ActorSystem@127.0.0.1:5508"]
        downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"
    }
}

Expected behavior
The node should startup without errors

Actual behavior
An error is thrown in the SplitBrainResolver class

Environment
Are you running on Linux? Windows? Docker? Which version of .NET?
=> Error occurs in Linux, Windows and Docker. .NET Version: net6.0

The text was updated successfully, but these errors were encountered:

abeltsios · 2022-05-25T06:58:35Z

Hello we have reached the same conclusion. (I was about to post the same bug here)

Some more context:

Split brain resolver calls _cluster = Cluster.Get(Context.System); in order to be able to subscribe to cluster events on PreStart phase.

That leads to exceptions being thrown during ActorSystems Extension's initialization code (ie, Lazy.Value fails) when called from SplitBrainResolver.
As such SBR actor never initializes (as it is unsupervised).

The exception thrown is

Cause: [akka://stx-pandora/system/cluster/core/daemon/downingProvider#2113384032]: Akka.Actor.ActorInitializationException: Exception during creation
 ---> System.InvalidOperationException: ValueFactory attempted to access the Value property of this instance.
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension[T](T& extension)
   at Akka.Actor.Internal.ActorSystemImpl.GetExtension[T]()
   at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
   at Akka.Cluster.Cluster.Get(ActorSystem system)
   at Akka.Cluster.SBR.SplitBrainResolver.PreStart()
   at Akka.Actor.ActorBase.AroundPreStart()
   at Akka.Actor.ActorCell.<>c__DisplayClass176_0.<Create>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

After a quick debug a simple change like the one bellow seems to resolve the issue (sorry for the git diff dump)

diff --git a/src/core/Akka.Cluster/SBR/SplitBrainResolver.cs b/src/core/Akka.Cluster/SBR/SplitBrainResolver.cs
index 12c203e2c..1a528e0f7 100644
--- a/src/core/Akka.Cluster/SBR/SplitBrainResolver.cs
+++ b/src/core/Akka.Cluster/SBR/SplitBrainResolver.cs
@@ -39,13 +39,21 @@ public static Props Props2(TimeSpan stableAfter, DowningStrategy strategy)
             return Props.Create(() => new SplitBrainResolver(stableAfter, strategy));
         }

+        protected override bool Receive(object message)
+        {
+            if (_cluster == null)
+            {
+                _cluster = Cluster.Get(Context.System);
+                _cluster.Subscribe(Self, InitialStateAsEvents, typeof(IClusterDomainEvent));
+            }
+            return base.Receive(message);
+        }
+
         // re-subscribe when restart
         protected override void PreStart()
         {
-            _cluster = Cluster.Get(Context.System);
-            _cluster.Subscribe(Self, InitialStateAsEvents, typeof(IClusterDomainEvent));
-
-            base.PreStart();
+           Self.Tell(new object());
+           base.PreStart();
         }

Aaronontheweb · 2022-05-25T10:32:44Z

So #5861 (comment) was my attempted fix for this issue as Akka.NET v1.4.37 also has this bug, but it would occur in the constructor and blow up the cluster supervisor.

As such SBR actor never initializes (as it is unsupervised).

All actors are supervised - and in this case the SBR is supervised by the default supervision strategy (restart.)

The worst case scenario right now is the SplitBrainResolver should just restart, whereas before cluster initialization itself could potentially fail (because throwing in the constructor blows up the parent instantiating the actor too, which in this case was one of the core clustering actors.) While this issue is disconcerting I don't think it's going to have much production impact.

SchiessMax · 2022-05-25T10:40:45Z

Ok, so the SplitBrainResolver should still work. I will try it out, and if it does work I will just have to live with the error log for now.

Aaronontheweb · 2022-05-25T10:42:26Z

It should definitely still work - but we'll fix this, either by breaking the SBR public API and passing in Cluster directly or by implementing some of @Zetanova 's suggestions here: #5447

Zetanova · 2022-05-25T13:56:08Z

I have multiple PR's out for Cluster, ClusterDaemon out.

I am still at 1.4.29-beta637735681605651069 in this version one PR got merged and after because of some minor and already foreseen issue with legacy DI pulled back.

The main issue simply the current extensions management over Lazy<>.
see: #5447

I try to implement an ExtensionManager soon.

abeltsios · 2022-05-25T19:51:14Z

As such SBR actor never initializes (as it is unsupervised).

All actors are supervised - and in this case the SBR is supervised by the default supervision strategy (restart)

Pardon my phrasing!
No-one triggers any restart and as such if someone tries to reproduce it, given that he adds or removes nodes in a cluster, several deadlettered messages can be observed

My understanding from the logs bellow is that SBR doesn't restart.

remoting (akka://akkasys) - Starting remoting
remoting (akka://akkasys) - Remoting started; listening on addresses : [akka.tcp://akkasys@hostname:4061]
remoting (akka://akkasys) - Remoting now listens on addresses: [akka.tcp://akkasys@hostname:4061]
Cluster (akka://akkasys) - Cluster Node [akka.tcp://akkasys@hostname:4061] - Starting up...
akka://akkasys/system/cluster/core/daemon/downingProvider - ValueFactory attempted to access the Value property of this instance.
[akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128]: Akka.Actor.ActorInitializationException: Exception during creation
---> System.InvalidOperationException: ValueFactory attempted to access the Value property of this instance.
	at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
	at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
	at System.Lazy`1.CreateValue()
	at System.Lazy`1.get_Value()
	at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension[T](T& extension)
	at Akka.Actor.Internal.ActorSystemImpl.GetExtension[T]()
	at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
	at Akka.Cluster.Cluster.Get(ActorSystem system)
	at Akka.Cluster.SBR.SplitBrainResolver.PreStart()
	at Akka.Actor.ActorBase.AroundPreStart()
	at Akka.Actor.ActorCell.<>c__DisplayClass176_0.<Create>b__0()
	at Akka.Actor.ActorCell.UseThreadContext(Action action)
	at Akka.Actor.ActorCell.Create(Exception failure)
	--- End of inner exception stack trace ---
	at Akka.Actor.ActorCell.Create(Exception failure)
	at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)
Cluster (akka://akkasys) - Cluster Node [akka.tcp://akkasys@hostname:4061] - Started up successfully
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [1] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [2] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [3] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [4] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [5] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [6] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [7] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [8] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [9] dead letters encountered. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka://akkasys/system/cluster/core/daemon/downingProvider - Message [TimerMsg] to [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [00:05:00]. If this is not an expected behavior then [akka://akkasys/system/cluster/core/daemon/downingProvider#2124238128] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Should the above happen if the SBR had restarted?

Aaronontheweb · 2022-05-26T13:29:22Z

Well I learned something new today

Throwing an exception in an actor's PreStart block will not allow the actor to be restarted: https://replit.com/@Aaronontheweb/ActorsPreStart#main.cs

I'm trying to imagine if that's by design or not (it might be since the child actor can't really complete its startup sequence if it fails there.)

Either way, that means this bug is more severe than I thought. We'll get this patched right away.

close akkadotnet#5962

…t restarted after a PreStart failure.

Aaronontheweb · 2022-05-26T15:47:04Z

Have a fix for this in #5965

Aaronontheweb · 2022-05-26T15:49:59Z

Verifiable reproduction by @Arkatufus here #5966

* pass `Akka.Cluster.Cluster` into `IDowningProvider` directly close #5962 * enhanced `StartupWithOneThreadSpec` to include SBR * C#-ified HOCON * added #5962 repro * fixed `AutoDown` * fixed other `IDowningProvider` instances * updated Akka.Cluster API approvals

Aaronontheweb · 2022-05-26T19:21:24Z

Resolved via #5965

Aaronontheweb mentioned this issue May 25, 2022

Exception during creation --- ValueFactory attempted to access the Value property of this instance. petabridge/lighthouse#259

Closed

Aaronontheweb added the akka-cluster label May 25, 2022

Aaronontheweb added this to the 1.4.39 milestone May 25, 2022

Aaronontheweb added the confirmed bug label May 25, 2022

Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this issue May 26, 2022

pass Akka.Cluster.Cluster into IDowningProvider directly

ddfe27f

close akkadotnet#5962

Aaronontheweb mentioned this issue May 26, 2022

pass Akka.Cluster.Cluster into IDowningProvider directly #5965

Merged

6 tasks

Arkatufus added a commit to Arkatufus/akka.net that referenced this issue May 26, 2022

Spec for akkadotnet#5962, SplitBrainResolver DowningProvider never go…

4bd95ca

…t restarted after a PreStart failure.

Arkatufus mentioned this issue May 26, 2022

Spec for #5962, SplitBrainResolver DowningProvider never got restarted after a PreStart failure. #5966

Closed

Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this issue May 26, 2022

added akkadotnet#5962 repro

a636343

Aaronontheweb closed this as completed May 26, 2022

This was referenced May 26, 2022

Need to document ActorInitializationException and how it occurs in actor lifecycle #5969

Open

added v1.4.39 release notes #5977

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in SplitBrainResolver.PreStart when using ChannelTaskScheduler for internal-dispatcher #5962

Error in SplitBrainResolver.PreStart when using ChannelTaskScheduler for internal-dispatcher #5962

SchiessMax commented May 25, 2022

abeltsios commented May 25, 2022

Aaronontheweb commented May 25, 2022

SchiessMax commented May 25, 2022 •

edited

Aaronontheweb commented May 25, 2022

Zetanova commented May 25, 2022

abeltsios commented May 25, 2022 •

edited

Aaronontheweb commented May 26, 2022 •

edited

Aaronontheweb commented May 26, 2022

Aaronontheweb commented May 26, 2022

Aaronontheweb commented May 26, 2022 •

edited

Error in SplitBrainResolver.PreStart when using ChannelTaskScheduler for internal-dispatcher #5962

Error in SplitBrainResolver.PreStart when using ChannelTaskScheduler for internal-dispatcher #5962

Comments

SchiessMax commented May 25, 2022

abeltsios commented May 25, 2022

Aaronontheweb commented May 25, 2022

SchiessMax commented May 25, 2022 • edited

Aaronontheweb commented May 25, 2022

Zetanova commented May 25, 2022

abeltsios commented May 25, 2022 • edited

Aaronontheweb commented May 26, 2022 • edited

Aaronontheweb commented May 26, 2022

Aaronontheweb commented May 26, 2022

Aaronontheweb commented May 26, 2022 • edited

SchiessMax commented May 25, 2022 •

edited

abeltsios commented May 25, 2022 •

edited

Aaronontheweb commented May 26, 2022 •

edited

Aaronontheweb commented May 26, 2022 •

edited