looking for a way to reliably send a message to self

[benbenwilde]

i'm looking for a way for an actor to reliably deliver messages to itself, i.e. a Self.Tell() but i want to do it such that if there is a failure at any point that the actor restarts as in an error scenario.

starting from the end and going backwards, my understanding is:

1. once the actor starts handling a message, if it fails the actor is restarted.
2. once the message is in an actors mailbox, it will either handle it, or it will fail at some point and restart.
3. the dispatch process for a local actor is just putting the message in its mailbox and then registering it for execution, of course this has the theoretical possiblity of failing

the issue i have with Self.Tell() is that for number 3, any error in the dispatch process ends up getting swallowed (purposefully), since .Tell() in reality is just "please try to send this, if you can". but in theory when sending a message to yourself you should be able to close the loop instead of having the possiblity of a message being lost to the ether, however unlikely that is.

so i'm thinking about doing my own direct dispatch, something like this

	var cell = (Self as ActorRefWithCell).Underlying as ActorCell;
	cell.Dispatcher.Dispatch(cell, new Envelope(message, ActorRefs.NoSender, Context.System));

is this totally crazy? or is there a better way?

[Danthar]

I think your operating on a misconception here. Ill go over your remarks point by point and then end with some advice.

1: It depends on the supervision strategy being applied, but the default supervision strategy is to restart. See here for some specifics.
2: Correct. With the caveat of point 1 and also the message that causes the failure is not automatically retried.
3: Im not sure what you mean exactly by 'registering it for execution'. Conceptually the mailbox is a queue and the actor takes a message from this queue and processes it.

Your talking about errors in the dispatch process. If by that you mean the part of a message being in the mailbox, and the actor taking it from the mailbox, there is no way this can fail. Not unless we have a serious bug in the framework that is. So that is not something you need to concern yourself with.
Furthermore because we are talking about a Self.Tell no matter how your actor system is deployed, whether the actor system is in a clustered environment, or if your using Akka.Remote directly, your actually communicating with a local actor. Thus there is no networking involved and the AtMostOnce delivery guarantee applies here.

So in short the snippet in your question is not needed, and does not provide any added benefits as opposed to simply doing Self.Tell. The only thing you need to think about in this case is errors in your actor implementation itself.

My advice would be that if the message your sending has an considerate chance of failing (because your doing some kind of dangerous operation) and you want to be able to retry it without resetting the state of your actor, is to instead delegate this 'dangerous' work to a child actor and use the Watch feature to decide what to do if that child actor fails, and/or listen to an functional Ack message that the child actor sends back when its done doing its work.

[benbenwilde]

Thanks, yes this is very helpful.

So basically I have a persistent actor that does a check after each message to see if i has some work to do, which will in turn further update the state. So it would see it needs to do work "123" for example. it would first set a local in memory flag on the actor (not persisted) so that the work is not repeated. then it would Self.Tell() the update produced by the work to itself. In akka persistence fashion, when the actor gets to the message it will be persisted and the overall state will also be updated.

So the TLDR is actor sees that it's in a state that it should do work and then update its state with the result of that work.

So the key goals here are:

1. If possible, don't repeat the work
2. don't lose the update. (meaning, either the update is persisted, or there is a failure in which case the actor can restart and do the work again)

If there is no way for Self.Tell to fail without the actor seeing it and erroring out, then yes I am good to go! However, when i look at the code for sending a message I see this:

            try
            {
                var messageToDispatch = _systemImpl.Settings.SerializeAllMessages
                    ? SerializeAndDeserialize(message)
                    : message;

                Dispatcher.Dispatch(this, messageToDispatch);
            }
            catch (Exception e)
            {
                _systemImpl.EventStream.Publish(new Error(e, _self.Parent.ToString(), ActorType, "Swallowing exception during message send"));
            }

So as you can see an error in this block will be swallowed, which is definitely not what i want in this case. I understand it since .Tell() is a fire and forget / at most once strategy, but in this case with Self.Tell() I don't want to be swallowing any exceptions, such that a message simply disappears. Of course i don't have SerializeAllMessages on so that leaves the Dispatch method which does this:

            var mbox = cell.Mailbox;
            mbox.Enqueue(cell.Self, envelope);
            RegisterForExecution(mbox, true, false);

(this is what i was referring to by the way with "register for execution", it registers the mailbox to be processed if it is not already, so the messages aren't simply left in there)

Anyways, maybe all the code in here is "safe" and should never throw in which case i could just use Self.Tell directly, however, as you can imagine the try catch block of course would give me some pause. At the end of the day, if I can avoid a situation where an error happens here and a message goes poof as a result. that seems like a good thing.

[Danthar]

Ok, im seeing some issues in the design your describing. But first lets address the code snippets you lifted from the codebase:

Its been a while since i have looked in this part of the framework, but i have an educated guess.
So yes, we have a try catch there. But no its not something you would have to worry about.
The MessageDispatcher is where 2 things come together. Its where the message is taken from the mailbox, and where its determined how the message is executed on the actor.
By default this translates to something like this:

• The default mailbox implementation in akka.net hands over the message
• The message is scheduled for execution using the default threadpool

Now wording is important here, i emphasized the word default because those 2 elements that we are talking about here can be configured using HOCON. You can configure your actor to use your own Mailbox implementation. And you can configure your actor to run using a different type of threading model, which in the config we call a different type of dispatcher. One such example is the ForkJoinDispatcher. This Dispatcher configuration options allows people to force an actor to, for example, always run on 1 particular thread.

Our defaults, and anything thats baked into the framework, are extensively tested using the expansive testsuite of the akka.net framework. They work, and they wont fail in ways the framework does not expect. (barring extra-ordinary circumstances)

But we cannot make any such guarantees for custom code. However we still want to system to at least fail gracefully and not explode in spectacular fashion. Hence the try catch.

Now on to description of your persistent actor.
It sounds like its doing to much. And im also wondering if you thought about what happens if the state is replayed and potential side effects are executed again. Im assuming your using an akka.net PersistentActor here.

How i would model this is as follows:

• WorkMessage comes in at PersistentActor. its job is to keep track of outstanding work and make sure stuff is not done twice
• Register the message in an internal list or something that represents outstanding work (this list is part of the state of the actor)
• If work needs to be done, tell another actor to go do the work
• PersistentActor listens to Ack or Failure message for its outstanding work, stores any such message and updates its internal state accordingly.
• Upon replay/rebuilding of the state of the PersistentActor we do not send out any work, we only rebuild our internal state
• We listen to the RecoveryCompleted message and once received we query our internal state, to check if there is any outstanding work which wasn't completed and needs to be restarted.

Obviously your use case might be more complex then what your describing here. But the key take-away should be (i think) that to me it sounds like your actorsystem design is wrong, and it would be much better to separate the tracking of work, and execution of work between multiple actors. You would then also not need this self.tell in your persistent actor anymore.

[benbenwilde]

Thanks for the reply, all that is very helpful. Also the reasoning for the try catch being that to handle custom mailbox/dispatcher implementations gracefully makes alot of sense. In the default implementations of those (which i'm using) I didn't really see anything that I would expect to throw, but i also didn't fully grok all the scheduling magic where it's making sure the mailbox will run, but anyways, the key takeaway I think is that .Tell() to a local actor is reliable, meaning we can say with confidence that the message will be successfully dispatched assuming the target actor exists. Thanks for taking the time to explain that for me :)

Regarding actor design, you could definitely say that the one actor is doing alot. However it is not actually doing the work (which would of course be bad since it would tie up the actor while it should be handling other messages). But as far as work like this goes, I actually see two ways of making it work. It seems the methodology that this framework has embraced is to basically have an actor for everything. Which is a very neat design. So I see that i could do "extra work" reliably with child actors and if it fails then the parent actor would be notified and can handle accordingly. But simplifying the problem, you could say that I merely need to run this work on a separate thread from the current actor, and the result of that work needs to be sent back to the current actor. So yes a child actor solves that but I can also just spin up a new task/thread for the work. This seems more convenient when no other interaction is needed except for the work to run and then report back the result. Maybe with time I will learn to embrace child actors more!

For my persistent actors I've basically placed a more opinionated layer on top where the methodology for handling incoming messages is basically this:

1. Validate the message
2. Persist the message
3. After persisted, update actor state (this step is automatically rerun during recovery)
4. If N messages have been persisted, then persist a snapshot (and clean up messages after successful)
5. One time message handling, like replying to the sender

Most of this is baked in so i don't have to reimplement all of it with every actor.

Then I have a callback method that runs after every message and after recovery completes where I can simply evaluate the current state and determine if any work should be done.

Anyways that's a bit of an oversimplification but I thought you might be interested!

Thanks again for the answers it is very appreciated!

[Danthar]

But simplifying the problem, you could say that I merely need to run this work on a separate thread from the current actor, and the result of that work needs to be sent back to the current actor.

If your starting an async operation that returns a task you can also use the PipeTo extension method. To start the operation, continue processing, and 'pipe' the response back to the current actor as a message. Effectively you'd be doing the same as delegating it to a child actor, only without actually using a childactor.
Doing things this way is perfectly fine and a perfect fit for some cases.

[benbenwilde]

ah, very convenient, thank you!