Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial sketch of --exec #104

Merged
merged 5 commits into from
Jan 20, 2019
Merged

Initial sketch of --exec #104

merged 5 commits into from
Jan 20, 2019

Conversation

jakewins
Copy link
Contributor

@jakewins jakewins commented Sep 5, 2018

This is a proposal to introduce alternative means of killing
pods, outside of simply asking kubernetes to stop them.

Specifically, it allows executing a shell command on a victim
container; that shell command can then be a simple kill -9,
a fork bomb, or any other means of destruction.

There are, at least, two reasons why this is useful. First,
it gives an option of more realistic failures. Simply deleting
a pod is a rather kind way to stop a pod - in reality a pod is likely
to go down in flames from lack of memory, segfaulting or instant
hardware failure. This brings the chaos closer to reality.

The second reason is that this helps test operators that directly
manipulate pods, rather than rely on higher-level concepts like
StatefulSets, Deployments and so on. In this case, the thing we want
to test is the operators ability to realize a pod is severely broken,
and to take appropriate action, depending on what software the operator
is running in the failed pod.

Note: These code changes contain a TODO and no tests; I stopped when I'd verified it worked, so I wouldn't spend more effort than necessary in case this is rejected. If y'all are keen to accept this in some form, I'm super happy to polish it up and add appropriate tests.

Let me know what you think!

Closes #103

This is a proposal to introduce alternative means of killing
pods, outside of simply asking kubernetes to stop them.

Specifically, it allows executing a shell command on a victim
container; that shell command can then be a simple kill -9,
a fork bomb, or any other means of destruction.

There are, at least, two reasons why this is useful. First,
it gives an option of more realistic failures. Simply deleting
a pod is a rather kind way to stop a pod - in reality a pod is likely
to go down in flames from lack of memory, segfaulting or instant
hardware failure. This brings the chaos closer to reality.

The second reason is that this helps test operators that directly
manipulate pods, rather than rely on higher-level concepts like
StatefulSets, Deployments and so on. In this case, the thing we want
to test is the operators ability to realize a pod is severely broken,
and to take appropriate action, depending on what software the operator
is running in the failed pod.
@linki
Copy link
Owner

linki commented Sep 17, 2018

@jakewins I'll have a look this week.

@linki
Copy link
Owner

linki commented Sep 20, 2018

@jakewins sounds like a good idea. How do you want to deal with containers that don't support the command you specify in --exec? (e.g. if kill isn't available or allowed to exec in the container?)

@palmerabollo
Copy link
Contributor

I think that abstracting the chaos action it's a good idea. However, different containers might need different --exec commands. Annotating each pod with a chaoschube/exec-action would be an option in case you need it (e.g. kill not being available in some containers).

@linki
Copy link
Owner

linki commented Oct 22, 2018

I merged this PR with master and did my best to make everything compile correctly.

Conflicts mostly included

  • the addition of gracePeriod which I moved from chaoskube's New to the deletePod action and
  • creating events which I left in the main chaoskube object

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Does chaoskube really kill the pods?
3 participants