Block tests from opening sockets

[emontnemery]

Proposed change

Block tests from opening sockets

The blocking is implemented by using the pytest_socket package which is enabled in a pytest_runtest_setup in the top-level conftest.py.
Workarounds for two pytest_socket PRs, miketheman/pytest-socket#75, miketheman/pytest-socket#76 are added in the top-level conftest.py.

Related PRs

• Use respx.mock in generic camera tests #55521
• Fix zeroconf mock and use it in CI group 1's tests #55526
• Add test fixture for unauthenticated HTTP client #55561
• Use hass_client_no_auth test fixture in integrations a-g #55581
• Use hass_client_no_auth test fixture in integrations h-p #55583
• Use hass_client_no_auth test fixture in integrations s-x #55585
• Mock out network.util.async_get_source_ip in tests #55592
• Prevent 3rd party lib from opening sockets in broadlink tests #55593
• Mock sockets in the network integration tests #55594
• Prevent 3rd party lib from opening sockets in epson tests #55595
• Prevent 3rd party lib from opening sockets in freedompro tests #55596
• Prevent 3rd party lib from opening sockets in broadlink tests #55636
• Make scapy imports in DHCP local #55647
• Avoid creating sockets in homekit port available tests #55668
• Mock out zeroconf in homekit_controller tests #56307
• Prevent 3rd party lib from opening sockets in wallbox tests #56308
• Prevent 3rd party lib from opening sockets in wilight tests #56310
• Prevent 3rd party lib from opening sockets in zeroconf tests #56324
• Prevent 3rd party lib from opening sockets in sia tests #56325
• Use hass_client_no_auth test fixture in nest tests #56326
• Prevent 3rd party lib from opening sockets in cloud tests #56328
• Prevent 3rd party lib from opening sockets in ping tests #56329
• Prevent 3rd party lib from opening sockets in ps4 tests #56330
• Prevent 3rd party lib from opening sockets in rfxtrx tests #56331
• Prevent 3rd party lib from opening sockets in samsungtv tests #56334
• Use hass_client_no_auth test fixture in withings tests #56337
• Mock out network.util.async_get_source_ip in tests #56339
• Avoid creating sockets in homekit port available tests #56342
• Prevent 3rd party lib from opening sockets in glances tests #56345
• Use hass_client_no_auth test fixture in additional tests #57037
• Prevent opening of sockets in kira tests #57038
• Mock out network.util.async_get_source_ip in tests #57039
• Prevent opening of sockets in watttime tests #57040
• Prevent tplink from opening sockets in tests #57058
• Prevent dlna_dmr from opening sockets in tests #57059
• Prevent opening of sockets in mqtt tests #57101

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[emontnemery]

This is not so nice.
There are other integration alternative: https://github.com/miketheman/pytest-socket#usage, the best option might be to do this in the top level conftest.py:

import pytest_socket

def pytest_runtest_setup():
    pytest_socket.disable_socket(allow_unix_socket=True)

[janiversen]

Adding the limitation to the top-level conftest.py is clearly better.

But why do we allow unix_sockets, I thought the tests should operate with mock and sort of "offline".

[emontnemery]

unix sockets are used by asyncio, so we have to allow it.

[emontnemery]

The limitation is now added to the top-level conftest.py instead

[emontnemery]

A summary of tests failing in group 1:
almond: hits external API
api: connects to HA API@127.0.0.1 - allow, or update test to use the common API mock
august: connects to HA API@127.0.0.1 - allow, or update test to use the common API mock
auth: connects to HA API@127.0.0.1 - allow, or update test to use the common API mock
bosch_sch: sets up live zeroconf
broadlink: opens a socket, should probably be mocked out in test
devolo_home_control: sets up live zeroconf
dialogflow: connects to HA API@127.0.0.1 - allow, or update test to use the common API mock
emulated_hue: connects to HA API@127.0.0.1 - allow, or update test to use the common API mock
emulated_roku: uses homeassistant.components.network.util.async_get_source_ip, which opens a socket to find the local IP
epson: opens a socket, should probably be mocked out in test
esphome: sets up live zeroconf
freedompro: opens a socket, should probably be mocked out in test
fritz: uses homeassistant.components.network.util.async_get_source_ip, which opens a socket to find the local IP
frontend: connects to HA API@127.0.0.1 - allow, the tests can't use the common API mock
generic: broken tests, fixed in #55521

Common issues are:

• Components relying on zeroconf, we should probably use a mocked out zeroconf in most cases
• Components using homeassistant.components.network.util.async_get_source_ip, we may want to mock it out and just return a predetermined IP.
• Tests connecting to the HA API, but not using the hass_client or hass_ws_client fixtures

[emontnemery]

This fixes the zeroconf mock and enables it for group 1: #55526

[frenck]

IMHO: Awesomeness!

[bdraco]

Just awesome 👍

[bdraco]

Let's merge this so we don't have to keep playing whack-a-mole.