Transient failures in instance state monitor cause instances to be lost at sea

[gjcolombo]

Sled agent spawns a state monitor task per instance that calls Propolis's instance_state_monitor endpoint, watches for VM state transitions, processes them, and relays any resulting instance state changes to Nexus. Any error in Instance::monitor_state_task or any of its callees bubbles up through the task and causes it to exit. After this, nothing monitors Propolis for subsequent state changes or notifies Nexus if they occur.

The DNS resolution error in #2726 brought this to my immediate attention, but any failure in the monitoring task will do the trick; another not-too-esoteric one is a Propolis panic that causes the call to instance_state_monitor to fail.

[smklein]

Good catch, I'm looking at this bug in the context of #2765 .

A quick fix here would be to just add a retry loop within these external calls of state monitoring (e.g., if we cannot contact nexus, retry with exponential backoff).

However, this raises the question: if we fail to alert nexus for long enough, or if we fail to monitor the instance, is that a valid condition for "setting the instance to failed"?

[askfongjojo]

However, this raises the question: if we fail to alert nexus for long enough, or if we fail to monitor the instance, is that a valid condition for "setting the instance to failed"?

It makes sense to give up after reaching a certain threshold. I mentioned in an earlier discussion today (in the context of sled fault tolerance) that we probably shouldn't automatically reap the failed instances because they could still be working perfectly from the end-user's POV. What may be a good compromise is some amount of human intervention, e.g. the failed instance status handling described here.