Crash or Stop

[andrewgrabbs]

Is it possible to stop or crash a thread on purpose? For instance, we have 5 devices on the start of a test run, one thread per device. If a device crashes we have a scenario where we have 5 threads and 4 devices. The thread attempts to get another device but they are all already tied to other threads. We are sleeping the thread, but that impacts all of the other threads.

I noticed there is an option for --max-worker-restart 0 we just need to know how to force a crash at that point, but ideally a way to elegantly stop a thread would be much nicer.

[RonnyPfannschmidt]

Do you parameterize per device or do you run all tests on all workers, where each worker is configured different

Also please note each worker is a own process

[andrewgrabbs]

@RonnyPfannschmidt at the moment we spawn as many workers as devices available. Then the workers reach out to a "device controller" and ask for a idle device on each test run. Ideally the worker would be hard coded with the device id, but we can't go back at this point.

[RonnyPfannschmidt]

That leaves more questions,

With what i read so far i suspect there is nothing to fix your situation

[andrewgrabbs]

I can identify the worker/process that has the initial device lost. For example, say we go from 5 threads 5 devices to 5 threads and 4 devices. I just need to kill 1 thread so it doesn't continually fight the other 4 threads for a device. We know when the process loses a watch. Is it possible to send the controller process a kill on the worker... or even send a kill to the OS on the process?

[RonnyPfannschmidt]

My recommendation would be to send the device I'd in for the worker, so it can fail when it's device goes out of commission instead of having tests re-request devices at each test thus running into races

[andrewgrabbs]

If we moved that direction, how would we signal the thread to go down once the device goes down?

[andrewgrabbs]

@RonnyPfannschmidt I moved the repo over to assigning a thread a device via a global variable map; however, if a test fails. The thread continues. I have specified --max-worker-restart=0 in the test execution. I have tried several ways to initiate the "crash" of the thread.
@pytest.mark.filterwarnings, pytest.fail(), pytest.exit(), sys.exit(), the thread just continues going. Any advice?

[RonnyPfannschmidt]

Right now I don't have any advice,
A lot of context is missing to come up with reasonable advice

[andrewgrabbs]

So I figured out how to crash the thread intentionally; however, if a thread dies, the entire process dies. I have max-worker-resart set to 0 (because I don't want the thread to restart if a device is gone), however, I want the remaining threads to continue on if a thread dies.

Is this a possibility?

worker gw5 crashed and worker restarting disabled