Replace lambdas with function calls

[Solomon1732]

Replace lambdas with function calls. This was done either by extracting a lambda into a function or method, or by replacing it with a library function call. Also some minor code quality changes.

This PR fixes #7432

[Solomon1732]

This pull request is instead of #7441 which I messed up.

[Solomon1732]

I didn't see the test result for "codecov/patch", so I thought it was fine to convert to "ready for review." Now I see I still need to fix it, so I converted it back to "draft".

Edit: My mistake.

[drew2a]

Thank you for the tremendous effort you've put into this work. The quality of the Tribler codebase has improved with these changes. However, I have some comments which I've added for your consideration.

[Solomon1732]

When this PR will be ready to be accepted, I will of course rewrite commit messages and squash accordingly. Right now it's a work in progress so it's a mess.

[Solomon1732]

All those *Mock instantiations I did are going wrong somewhere and I'm not sure where 😓. I'd test it locally, except #7252 prevents me.

Here's how I know:
https://github.com/Tribler/tribler/actions/runs/5448403366/jobs/9911544924?pr=7466

Edit: Never mind, it's fine now. The latest rebasing from main may have fixed the issues.

[Solomon1732]

Does this coverage error relate to #7382? I'm asking since the latest force push from my fork only rebased from upsteam. I did no additional changes since the last one. Yet the test failed. Do my changes affect the failed file?
> /home/runner/work/_temp/7520c105-57cf-491b-9035-f33c995d6039.sh: line 1: 1875 Segmentation fault (core dumped) coverage run --source=./src/tribler/core -p -m pytest ${PYTEST_CORE_ARGUMENTS_LINUX}
src/tribler/core/upgrade/tags_to_knowledge/tests/test_migration.py::test_migration
Error: Process completed with exit code 139.

Edit: Since I don't know whether I caused this or not, I mention it here and not in the issue. At worst I can always comment there about it.

Edit 2: I forgot I also changed if-return, if-return I wrote into if-elif. I changed it back. Maybe that's what caused it, although I doubt it. I changed it back. Nvm, I didn't do it. I already wrote it as if-elif.

Edit 3: I ran it again by changing them to if-return, if-return and the issue is gone. Must have been a spurious error 🤔.

[Solomon1732]

This came from a recent pull request:

def test_start_download_existing_download(fake_dlmgr):
    """
    Testing the addition of a torrent to the libtorrent manager, if there is a pre-existing download.
    """
    infohash = b'a' * 20

    mock_download = MagicMock(
        get_def=MagicMock(return_value=MagicMock(get_trackers=set))
    )
    mock_ltsession = MagicMock()

    fake_dlmgr.downloads[infohash] = mock_download
    fake_dlmgr.get_session = lambda *_: mock_ltsession

    download = await fake_dlmgr.start_download(tdef=TorrentDefNoMetainfo(infohash, 'name'), checkpoint_disabled=True)
    assert download == mock_download
    fake_dlmgr.downloads.clear()

Notice there's an await outside of an async context. It causes the test to fail.

It's in tribler\src\tribler\core\components\libtorrent\tests\test_download_manager.py.

[drew2a]

The Application Tester is failing for some reason: https://github.com/Tribler/tribler/actions/runs/7049092932/job/19186780929?pr=7466

See: #7739

I'll fix it soon.

[drew2a]

@Solomon1732, the Application Tester has been fixed. Please, rebase your PR into the main branch.
When the PR is ready, please mark it as ready for review 🙏.

[qstokkink]

@Solomon1732 I see that this PR has been open for quite a while now. Do you need any help in getting this PR ready?

[Solomon1732]

@Solomon1732 I see that this PR has been open for quite a while now. Do you need any help in getting this PR ready?

@qstokkink I admit I do. I don't know how to cover the following warnings in codecove/patch with tests:

• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491167
• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491170
• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491172
• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491174
• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491177
• https://github.com/Tribler/tribler/pull/7466/files#annotation_16642491181

You can see they're all code changes which are internal, and don't necessarily affect a function's output. I don't know how to approach them. If I can get ideas or pointers for how to write tests for these failures, I would be happy.

[qstokkink]

Ah yes, those are pretty nasty. I could cover most of them (except the dummy_mode block in download_manager.py) as follows:

Click me for changes!

diff --git a/src/tribler/core/components/libtorrent/tests/test_download_manager.py b/src/tribler/core/components/libtorrent/tests/test_download_manager.py
index 493ee66eb..e3b1a598b 100644
--- a/src/tribler/core/components/libtorrent/tests/test_download_manager.py
+++ b/src/tribler/core/components/libtorrent/tests/test_download_manager.py
@@ -1,6 +1,7 @@
 import asyncio
 from asyncio import Future, gather, get_event_loop, sleep
-from unittest.mock import MagicMock
+from functools import partial
+from unittest.mock import MagicMock, Mock
 from unittest.mock import AsyncMock
 
 import pytest
@@ -491,3 +492,71 @@ async def test_check_for_dht_ready(fake_dlmgr):
     fake_dlmgr.get_session().status().dht_nodes = 1000
     # If the session has enough peers, it should finish instantly
     await fake_dlmgr._check_dht_ready()
+
+
+def test_update_trackers(fake_dlmgr: DownloadManager) -> None:
+    fake_download, dl_state = create_fake_download_and_state()
+    fake_dlmgr.downloads[fake_download.infohash] = fake_download
+    fake_metainfo = {b'info': {b'name': b"test_download"}}
+    fake_download.get_def().metainfo = fake_metainfo
+
+    fake_tracker1 = "127.0.0.1/test-announce1"
+
+    fake_dlmgr.update_trackers(fake_download.infohash, [fake_tracker1])
+
+    assert fake_metainfo["announce"] == fake_tracker1
+    assert "announce-list" not in fake_metainfo
+
+
+def test_update_trackers_list(fake_dlmgr: DownloadManager) -> None:
+    fake_download, dl_state = create_fake_download_and_state()
+    fake_dlmgr.downloads[fake_download.infohash] = fake_download
+    fake_metainfo = {b'info': {b'name': b"test_download"}}
+    fake_download.get_def().metainfo = fake_metainfo
+
+    fake_tracker1 = "127.0.0.1/test-announce1"
+    fake_tracker2 = "127.0.0.1/test-announce2"
+
+    fake_dlmgr.update_trackers(fake_download.infohash, [fake_tracker1, fake_tracker2])
+
+    assert "announce" not in fake_metainfo
+    assert fake_metainfo["announce-list"] == [[fake_tracker1, fake_tracker2]]
+
+
+def test_update_trackers_list_append(fake_dlmgr: DownloadManager) -> None:
+    fake_download, dl_state = create_fake_download_and_state()
+    fake_dlmgr.downloads[fake_download.infohash] = fake_download
+    fake_metainfo = {b'info': {b'name': b"test_download"}}
+    fake_download.get_def().metainfo = fake_metainfo
+
+    fake_tracker1 = "127.0.0.1/test-announce1"
+    fake_tracker2 = "127.0.0.1/test-announce2"
+
+    fake_dlmgr.update_trackers(fake_download.infohash, [fake_tracker1])
+    fake_download.get_def().get_tracker = Mock(return_value=fake_tracker1)
+    fake_dlmgr.update_trackers(fake_download.infohash, [fake_tracker2])
+
+    assert fake_metainfo["announce"] == fake_tracker1
+    assert fake_metainfo["announce-list"] == [[fake_tracker1, fake_tracker2]]
+
+
+def test_get_download_rate_limit(fake_dlmgr: DownloadManager) -> None:
+    fake_value = 42
+    settings = {}
+    fake_dlmgr.get_session = Mock(return_value=Mock(get_settings=Mock(return_value=settings)))
+    fake_dlmgr.get_session().download_rate_limit = partial(settings.get, "download_rate_limit")
+    fake_dlmgr.ltsessions = {0: fake_dlmgr.get_session()}
+    fake_dlmgr.set_download_rate_limit(fake_value, 0)
+
+    assert fake_value == fake_dlmgr.get_download_rate_limit()
+
+
+def test_get_upload_rate_limit(fake_dlmgr: DownloadManager) -> None:
+    fake_value = 42
+    settings = {}
+    fake_dlmgr.get_session = Mock(return_value=Mock(get_settings=Mock(return_value=settings)))
+    fake_dlmgr.get_session().upload_rate_limit = partial(settings.get, "upload_rate_limit")
+    fake_dlmgr.ltsessions = {0: fake_dlmgr.get_session()}
+    fake_dlmgr.set_upload_rate_limit(fake_value, 0)
+
+    assert fake_value == fake_dlmgr.get_upload_rate_limit()
diff --git a/src/tribler/core/components/tunnel/tests/test_dispatcher.py b/src/tribler/core/components/tunnel/tests/test_dispatcher.py
index c31c5cea7..e068570a2 100644
--- a/src/tribler/core/components/tunnel/tests/test_dispatcher.py
+++ b/src/tribler/core/components/tunnel/tests/test_dispatcher.py
@@ -1,7 +1,8 @@
+import asyncio
 from unittest.mock import Mock
 
 import pytest
-from ipv8.messaging.anonymization.tunnel import CIRCUIT_STATE_EXTENDING, CIRCUIT_STATE_READY, CIRCUIT_TYPE_DATA
+from ipv8.messaging.anonymization.tunnel import CIRCUIT_STATE_EXTENDING, CIRCUIT_STATE_READY, CIRCUIT_TYPE_DATA, Circuit
 from ipv8.util import succeed
 
 from tribler.core.components.tunnel.community.dispatcher import TunnelDispatcher
@@ -127,3 +128,35 @@ def test_check_connections(dispatcher, mock_circuit):
     assert connection not in dispatcher.con_to_cir
     assert mock_circuit.circuit_id not in dispatcher.cid_to_con
     assert 2 in dispatcher.cid_to_con
+
+
+async def test_on_data_after_select(dispatcher: TunnelDispatcher, mock_circuit: Circuit) -> None:
+    mock_connection = Mock(udp_connection=None)
+    mock_request = Mock(destination=("0.0.0.0", 1024), data=b'a')
+    dispatcher.set_socks_servers([mock_connection.socksserver])
+    dispatcher.tunnels.circuits = {}
+    dispatcher.tunnels.create_circuit = Mock(return_value=mock_circuit)
+    dispatcher.on_socks5_udp_data = Mock(return_value=None)
+    mock_circuit.ready = asyncio.Future()
+
+    mock_circuit.ready.set_result(True)
+    dispatcher.select_circuit(mock_connection, mock_request)
+    await asyncio.sleep(0)
+
+    assert dispatcher.on_socks5_udp_data.called
+
+
+async def test_on_data_after_select_no_result(dispatcher: TunnelDispatcher, mock_circuit: Circuit) -> None:
+    mock_connection = Mock(udp_connection=None)
+    mock_request = Mock(destination=("0.0.0.0", 1024), data=b'a')
+    dispatcher.set_socks_servers([mock_connection.socksserver])
+    dispatcher.tunnels.circuits = {}
+    dispatcher.tunnels.create_circuit = Mock(return_value=mock_circuit)
+    dispatcher.on_socks5_udp_data = Mock(return_value=None)
+    mock_circuit.ready = asyncio.Future()
+
+    mock_circuit.ready.set_result(None)
+    dispatcher.select_circuit(mock_connection, mock_request)
+    await asyncio.sleep(0)
+
+    assert not dispatcher.on_socks5_udp_data.called

The "dummy mode" code seems like dead code to me (and this should not be in our production code at all IMO).

[Solomon1732]

@qstokkink Thanks a bunch! I'll work on adding your changes to my PR. I'll also remove dummy_mode while I'm at it.

[qstokkink]

@Solomon1732 No worries, I'm happy to help.

I would advise against removing dummy_mode in this PR though. Our testing infrastructure is complex, to say the least. You may end up with really nasty and annoying refactoring. Perhaps this would be better suited to a follow-up PR.

[drew2a]

@Solomon1732 @qstokkink, as an option, we can ignore all uncovered lines related to dummy_mode and remove/fix them in the next one.

@qstokkink, could you please create an issue regarding the potential removal of dummy_mode? I agree with your suggestions, but we should analyze how we use dummy_mode and whether we can safely remove it (I can perform the analysis).

[qstokkink]

@drew2a sure 👍 #7797

[Solomon1732]

@drew2a will do as you advise. Thanks a bunch! Yeah, it looks like it'll make this PR a hairy one.

[Solomon1732]

I have no idea how to fix the errors "final newline missing" and "Unexpected line ending format. There is 'CRLF' while it should be 'LF'." I followed both Codasity's article Error caused by incompatible line endings and Github's page Configuring Git to handle line endings but to no avail. I can't find any CRLF ending in any python file in the repo of my PR.

I even ran the following command as per Codasity's article above.:

find . -type f -name '*.py' -exec file {} \; | grep "CR line"

It found nothing.

[qstokkink]

@Solomon1732 are you using PyCharm? If so, there is a sneaky setting in the bottom right:

EDIT: If I'm suggesting fixes anyway:

• The Unused variable 'dl_state' can be solved by renaming the variable to _.
• The Redefining name 'fake_dlmgr' from outer scope (line 50) can be solved by either moving the fixture to conftest.py or redefining it as:

@pytest.fixture(name="fake_dlmgr")
async def fixture_fake_dlmgr(tmp_path_factory):

[Solomon1732]

@qstokkink I Visual Studio Code. Or I use Geany if I find VSC too cumbersome for this or that task.

[qstokkink]

@Solomon1732 IIRC VSC also has its line endings setting in the bottom-right corner somewhere.

[Solomon1732]

I finally fixed it somehow. I did an ugly hack of committing, squashing, and force pushing. It should look fine on the PR, though.

[qstokkink]

👍 whatever works. It seems our Codacy check did not run (for some reason) now though.

[Solomon1732]

Oh. Should the tests be reran, then? To ensure Codasity reports a solid result on this matter? Edit: Nevermind, They run anyway as part of more pushes to fix Codasity errors in my PR.

[drew2a]

Great work! I've added some comments, most of which are related to coding style. All of them come with suggestions, so if you agree with them, you could just apply the working version.

[Solomon1732]

@drew2a thanks! I applied all of your recommendations, so I marked all of them as "Resolved" instead of writing "done" and resolving each one.

Edit: Minor phrasing fixes.

[Solomon1732]

I suspect Codasity doesn't understand Python's @pytest.fixture. This is why it complains about "Redefining name 'fake_dlmgr' from outer scope (line 50)" in this PR for tests I added in test_dowload_manager.py.

[drew2a]

@Solomon1732

I suspect Codasity doesn't understand Python's @pytest.fixture. This is why it complains about "Redefining name 'fake_dlmgr' from outer scope (line 50)" in this PR for tests I added in test_dowload_manager.py.

Don't go down this rabbit hole (see our discussion with @qstokkink #7764 (review)) :)

Basically, you have two options:

1. Suppress the warning.
2. Follow @qstokkink's suggestion.

I'll approve the PR in both cases, as both options are correct. However, @qstokkink most likely will not approve the PR if you suppress the warning. So, my advice is to opt for number 2 and follow @qstokkink's suggestion.

[Solomon1732]

So, my advice is to opt for number 2 and follow @qstokkink's suggestion.

@drew2a I looked at it, and I will do it 😰 . I indeed don't want to go into that rabbit hole.

[Solomon1732]

I'm not sure why this test failed, or if it's my PR's code or not. May I get some help? I think it may has to do with #7798?

Prepare all required actions
Run ./.github/actions/timeout
  with:
    command: python ./scripts/exit_node/run_exit_node.py --testnet --fragile
    duration: 10s
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python[2](https://github.com/Tribler/tribler/actions/runs/7424222588/job/20203375875?pr=7466#step:6:2)_ROOT_DIR: /opt/hostedtoolcache/Python/[3](https://github.com/Tribler/tribler/actions/runs/7424222588/job/20203375875?pr=7466#step:6:3).8.18/x6[4](https://github.com/Tribler/tribler/actions/runs/7424222588/job/20203375875?pr=7466#step:6:4)
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
    VIRTUAL_ENV: /home/runner/.virtualenvs/.venv
    PYTHONPATH: ./src:
    HELPER_INDEX: 1000
    HELPER_BASE: 2000
Run timeout --foreground -s SIGKILL 10s python ./scripts/exit_node/run_exit_node.py --testnet --fragile || code=$?; if [[ $code -ne 124 && $code -ne 0 ]]; then exit $code; fi
  timeout --foreground -s SIGKILL 10s python ./scripts/exit_node/run_exit_node.py --testnet --fragile || code=$?; if [[ $code -ne 124 && $code -ne 0 ]]; then exit $code; fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
    VIRTUAL_ENV: /home/runner/.virtualenvs/.venv
    PYTHONPATH: ./src:
    HELPER_INDEX: 1000
    HELPER_BASE: 2000
Traceback (most recent call last):
  File "./scripts/exit_node/run_exit_node.py", line 226, in <module>
    run(main())
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "./scripts/exit_node/run_exit_node.py", line 220, in main
    await service.start(args)
  File "./scripts/exit_node/run_exit_node.py", line 133, in start
    config = make_config(options)
  File "./scripts/exit_node/run_exit_node.py", line [5](https://github.com/Tribler/tribler/actions/runs/7424222588/job/20203375875?pr=7466#step:6:5)1, in make_config
    config.tunnel_community.competing_slots = options.competing_slots
  File "pydantic/main.py", line 35[7](https://github.com/Tribler/tribler/actions/runs/7424222588/job/20203375875?pr=7466#step:6:7), in pydantic.main.BaseModel.__setattr__
ValueError: "TunnelCommunitySettings" object has no field "competing_slots"
Error: Process completed with exit code 1.

Edit: Based on #7798 (comment) it may indeed has to do with it.

[qstokkink]

It seems the following two lines can/should be removed:

tribler/scripts/exit_node/run_exit_node.py

Line 51 in 959169e

config.tunnel_community.competing_slots = options.competing_slots

tribler/scripts/exit_node/run_exit_node.py

Line 204 in 959169e

parser.add_argument('--competing_slots', '-c', default=20, type=int, help='Specifies the number of competing slots')

[drew2a]

@Solomon1732 @qstokkink, I'll fix it within a couple of hours.

The fix:

• Remove leftovers of competing slots configuration. #7804

[drew2a]

@Solomon1732 fixed.

[drew2a]

Looks good except for one small mistake in the tests, which can be easily fixed. I proposed a complete solution during the previous review round, but it appears to have been overlooked.

[drew2a]

Thank you for your patience.

[drew2a]

It could be merged if @qstokkink or @kozlovsky don't have any objections 👍 .