submariner test stuck for hours

[nirs]

Seen once, submariner test stuck on cleanup after successful run.

kubectl delete -k never completed - needs a timeout.

Why it was stuck? maybe we need to wait when unexporting a service?
2024-03-26 03:21:28,485 ERROR   Command failed
Traceback (most recent call last):
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 72, in main
    func(env, args)
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 167, in cmd_start
    execute(
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 252, in execute
    f.result()
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 282, in start_cluster
    execute(
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 252, in execute
    f.result()
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 428, in run_worker
    run_addon(addon, worker["name"], hooks=hooks, allow_failure=allow_failure)
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 459, in run_addon
    run_hook(hook, addon["args"], name, allow_failure=allow_failure)
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 470, in run_hook
    run(hook, *args, name=name)
  File "/home/nsoffer/ramen/test/drenv/__main__.py", line 485, in run
    for line in commands.watch(*cmd):
  File "/home/nsoffer/ramen/test/drenv/commands.py", line 155, in watch
    raise Error(args, error, exitcode=p.returncode)
drenv.commands.Error: Command failed:
   command: ('addons/submariner/test', 'hub', 'dr1', 'dr2')
   exitcode: 1
   error:
      Traceback (most recent call last):
        File "/home/nsoffer/ramen/test/addons/submariner/test", line 268, in <module>
          wait_for_delete(NS2, clusters[1], clusters[0])
        File "/home/nsoffer/ramen/test/addons/submariner/test", line 79, in wait_for_delete
          kubectl.delete(
        File "/home/nsoffer/ramen/test/drenv/kubectl.py", line 127, in delete
          _watch("delete", *args, input=input, context=context, log=log)
        File "/home/nsoffer/ramen/test/drenv/kubectl.py", line 157, in _watch
          for line in commands.watch(*cmd, input=input):
        File "/home/nsoffer/ramen/test/drenv/commands.py", line 155, in watch
          raise Error(args, error, exitcode=p.returncode)
      drenv.commands.Error: Command failed:
         command: ('kubectl', 'delete', '--context', 'dr2', '--kustomize=ns2/dst', '--wait=true', '--ignore-not-found')
         exitcode: 1
         error:

Tasks

• Add timeout to all kubectl delete calls in all addons, ramenctl, basic-test
• Find why the delete fails in submariner test - maybe we need to wait after calling subctl unexport?