Inventory collection

[jcstanaway]

When run with --inventory-kubernetes does seal collect the set of nodes only at startup or at the start of each "run" (when in autonomous mode)? I'm really hoping it refreshes the node list at the start of each run to account for dynamically scaling clusters.

[Niksko]

@jcstanaway I think it's just at startup.

The result is that if you leave a strategy running that stops nodes without restarting them (which is the sensible behaviour if you run your nodes in an ASG), it gradually exhausts its list of nodes, and eventually ends up in the state where the only node left in the inventory is the one that the PowerfulSeal pod is on, at which point the stop pods strategy does nothing.

[seeker89]

That's right: currently it reads the list of nodes at startup only: https://github.com/powerfulseal/powerfulseal/blob/master/powerfulseal/cli/__main__.py#L525

@jcstanaway do you need support for refreshing the inventory mid-flight?

[jcstanaway]

I would suggest refreshing the inventory each time seal wakes up before each run. Our clusters utilize dynamic scaling (using k8s autoscaler) with multiple ASGs and so we have nodes coming and going quite frequently depending on load. Since seal only reads the list of nodes at startup, this effectively makes it so that we can't run most node chaos scenarios.