Problem uploading reports to docker-based single-instance codecov install

[andreiz]

Describe the bug
I am running codecov self-hosted on a single EC2 instance using docker-compose.yml. Logged in via Github and set up the test repo per tutorial. However, trying to upload reports fails.

Environment (please complete the following information):)

• System: codecov on EC2, CI is Github

CI log

/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov --enterprise-url https://codecov.bbbb.ccc/ -v do-upload -Z --git-service github
info - 2024-05-14 18:58:33,195 -- ci service found: github-actions
debug - 2024-05-14 18:58:33,198 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-05-14 18:58:33,201 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
warning - 2024-05-14 18:58:33,203 -- No config file could be found. Ignoring config.
debug - 2024-05-14 18:58:33,203 -- No codecov_yaml found
debug - 2024-05-14 18:58:33,205 -- Starting upload processing --- {"branch": "step2", "build_code": "9084726456", "build_url": "https://github.com/aaaaa/testrepo/actions/runs/9084726456", "commit_sha": "1d711a6f6b0a141f1499341aed98469db16bf83d", "disable_file_fixes": false, "disable_search": false, "enterprise_url": "https://codecov.bbbb.ccc/", "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": [], "files_search_root_folder": "/home/runner/work/testrepo/testrepo", "flags": [], "git_service": "github", "handle_no_reports_found": false, "job_code": "API workflow", "name": "9084726456", "network_filter": null, "network_prefix": null, "network_root_folder": "/home/runner/work/testrepo/testrepo", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": null, "report_code": "default", "slug": "bbbb/testrepo", "token": "f******************", "upload_file_type": "coverage"}
debug - 2024-05-14 18:58:33,205 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-05-14 18:58:33,205 -- Running xcode plugin...
warning - 2024-05-14 18:58:33,205 -- xcrun is not installed or can't be found.
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-05-14 18:58:33,205 -- Running gcov plugin...
warning - 2024-05-14 18:58:33,208 -- No gcov data found.
debug - 2024-05-14 18:58:33,208 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
info - 2024-05-14 18:58:33,209 -- Generating coverage.xml report in /home/runner/work/testrepo/testrepo
info - 2024-05-14 18:58:33,305 -- Wrote XML report to coverage.xml
debug - 2024-05-14 18:58:33,306 -- Collecting relevant files
info - 2024-05-14 18:58:33,315 -- Found 1 coverage files to report
info - 2024-05-14 18:58:33,315 -- > /home/runner/work/testrepo/testrepo/coverage.xml
debug - 2024-05-14 18:58:33,319 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-05-14 18:58:33,319 -- Sending upload request to Codecov
info - 2024-05-14 18:58:33,457 -- Your upload is now processing. When finished, results will be available at: https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d
debug - 2024-05-14 18:58:33,457 -- Upload request to Codecov complete. --- {"response": {"external_id": "0f7b6457-cd4f-44ad-bcac-c9f9b171621d", "created_at": "2024-05-14T18:58:34.208832Z", "raw_upload_location": "http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=codecov-default-key%2F20240514%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240514T185834Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ddd12c90eb7b6c20dda5f33f7256b87d2998d4ffb7424168fe64de4d24852ab7", "state": "", "provider": null, "upload_type": "uploaded", "url": "https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d", "ci_url": "https://github.com/bbbb/testrepo/actions/runs/9084726456", "flags": [], "env": {}, "name": "9084726456", "job_code": "API workflow"}}
debug - 2024-05-14 18:58:33,458 -- Sending upload to storage
warning - 2024-05-14 18:58:33,465 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-05-14 18:58:33,972 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-05-14 18:58:34,980 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 83, in <module>
  File "codecov_cli/main.py", line 79, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload.py", line 257, in do_upload
  File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line 104, in send_upload_data
  File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[1628] Failed to execute script 'main' due to unhandled exception!
Error: Codecov:
                        Failed to properly upload report: The process '/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov' failed with exit code 1

Additional context
Add any other context about the problem here.

[drazisil-codecov]

@andreiz

Is http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt reachable from the uploader?

Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.

[andreiz]

@andreiz

Is http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt reachable from the uploader?

Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.

If it is supposed to be reachable from the outside, it's not something I saw mentioned in the docs. I literally just cloned the https://github.com/codecov/self-hosted repo and then followed the Self-Hosted Configuration Guide and nothing there mentioned minio.

I was under the impression that the codecov API uses minio locally as the storage backend API, but I guess I am mistaken?

[drazisil-codecov]

Minio uses whatever the host is set for the storage service

https://github.com/codecov/codecov-api/blob/main/services/storage.py#L26

The value needs to be reachable both in the container and externally, as the report is uploaded directed to storage, bypassing Codecov.

[andreiz]

I got it running by opening up port 9000 in the firewall and configuring minio host to be the same as the main codecov one. But can codecov-gateway proxy external traffic to minio container instead and do SSL termination? If not, then how do I set up minio to run with SSL enabled?