Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing retry/exception handling for Nova overcloud API call #575

Open
fnordahl opened this issue May 22, 2021 · 2 comments
Open

Missing retry/exception handling for Nova overcloud API call #575

fnordahl opened this issue May 22, 2021 · 2 comments

Comments

@fnordahl
Copy link
Contributor 

2021-05-22 06:25:07.362241 | bionic-medium | 2021-05-22 06:25:07 [ERROR] {'default_alias': 'zaza-21f88ed5ac6e'}
2021-05-22 06:25:07.362486 | bionic-medium | 2021-05-22 06:25:07 [ERROR] Model default_alias (zaza-21f88ed5ac6e)
2021-05-22 06:25:07.987688 | bionic-medium | Traceback (most recent call last):
2021-05-22 06:25:07.987777 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/bin/functest-run-suite", line 8, in <module>
2021-05-22 06:25:07.987821 | bionic-medium |     sys.exit(main())
2021-05-22 06:25:07.987854 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 314, in main
2021-05-22 06:25:07.987901 | bionic-medium |     test_directory=args.test_directory)
2021-05-22 06:25:07.987944 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 243, in func_test_runner
2021-05-22 06:25:07.987968 | bionic-medium |     force=force, test_directory=test_directory)
2021-05-22 06:25:07.987985 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 144, in run_env_deployment
2021-05-22 06:25:07.987998 | bionic-medium |     test_directory=test_directory)
2021-05-22 06:25:07.988011 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
2021-05-22 06:25:07.988028 | bionic-medium |     run_configure_list(functions)
2021-05-22 06:25:07.988053 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
2021-05-22 06:25:07.988069 | bionic-medium |     utils.get_class(func)()
2021-05-22 06:25:07.988082 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 99, in basic_overcloud_network
2021-05-22 06:25:07.988106 | bionic-medium |     use_juju_wait=use_juju_wait)
2021-05-22 06:25:07.988120 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/configure/network.py", line 237, in setup_gateway_ext_port
2021-05-22 06:25:07.988133 | bionic-medium |     use_juju_wait=use_juju_wait)
2021-05-22 06:25:07.988146 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1043, in configure_gateway_ext_port
2021-05-22 06:25:07.988159 | bionic-medium |     add_dataport_to_netplan)
2021-05-22 06:25:07.988182 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 943, in create_additional_port_for_machines
2021-05-22 06:25:07.988202 | bionic-medium |     net_id=None, fixed_ip=None)
2021-05-22 06:25:07.988217 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution
2021-05-22 06:25:07.988249 | bionic-medium |     return methods[-1].func(obj, *args, **kwargs)
2021-05-22 06:25:07.988278 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/v2/servers.py", line 613, in interface_attach
2021-05-22 06:25:07.988300 | bionic-medium |     return self.manager.interface_attach(self, port_id, net_id, fixed_ip)
2021-05-22 06:25:07.988314 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution
2021-05-22 06:25:07.988345 | bionic-medium |     return methods[-1].func(obj, *args, **kwargs)
2021-05-22 06:25:07.988360 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2067, in interface_attach
2021-05-22 06:25:07.988380 | bionic-medium |     obj_class=NetworkInterface)
2021-05-22 06:25:07.988401 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/base.py", line 364, in _create
2021-05-22 06:25:07.988414 | bionic-medium |     resp, body = self.api.client.post(url, body=body)
2021-05-22 06:25:07.988427 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2021-05-22 06:25:07.988440 | bionic-medium |     return self.request(url, 'POST', **kwargs)
2021-05-22 06:25:07.988452 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/client.py", line 78, in request
2021-05-22 06:25:07.988465 | bionic-medium |     raise exceptions.from_response(resp, body, url, method)
2021-05-22 06:25:07.988494 | bionic-medium | novaclient.exceptions.ClientException: Unknown Error (HTTP 500)
@ajkavanagh
Copy link
Contributor

This would be an excellent test/use of the ObjectRetrier class that, er, does retries for an object automatically. Built exactly for this scenario.

In

zaza-openstack-tests/zaza/openstack/configure/network.py

Line 186 in eae35cc

def setup_gateway_ext_port(network_config, keystone_session=None, 

    # Get authenticated clients
    nova_client = ObjectRetrierWraps(
        openstack_utils.get_nova_session_client(keystone_session))
    neutron_client = ObjectRetrierWraps(
        openstack_utils.get_neutron_session_client(keystone_session))

Essentially, any exception generated by an method on nova_client or neutron_client will be retried. If that's too broad we could restrict it to relevant client and connection errors?

What do you think?

@fnordahl
Copy link
Contributor Author

@ajkavanagh that is an excellent idea, and thank you for the pointer. I'll have a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ajkavanagh @fnordahl