llvm-cov report - no filenames specified (failed only on windows and works on linux) #300

LilyFoote · 2023-08-11T11:04:17Z

I've been trying to add coverage for the Rust part of a combined Python and Rust project. I've got it working for our linux CI builds, but trying the same thing for Windows is leading to this error:

Run cargo llvm-cov report --manifest-path=rust\Cargo.toml --codecov --output-path codecov.json -vvv
  
     Running `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-profdata.exe merge -sparse -f D:\a\kolo\kolo\python\rust\target\rust-profraw-list -o D:\a\kolo\kolo\python\rust\target\rust.profdata`
     Running `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-cov.exe export -format=text -use-color=0 -instr-profile=D:\a\kolo\kolo\python\rust\target\rust.profdata -ignore-filename-regex \\rustc\\[0-9a-f]+\\|^D:\\a\\kolo\\kolo\\python\\rust(\\.*)?\\(tests|examples|benches)\\|^D:\\a\\kolo\\kolo\\python\\rust\\target($|\\)|^C:\\Users\\runneradmin\\\.cargo\\(registry|git)\\|^C:\\Users\\runneradmin\\\.rustup\\toolchains($|\\)`
error: failed to generate report: process didn't exit successfully: `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-cov.exe export -format=text -use-color=0 -instr-profile=D:\a\kolo\kolo\python\rust\target\rust.profdata -ignore-filename-regex \\rustc\\[0-9a-f]+\\|^D:\\a\\kolo\\kolo\\python\\rust(\\.*)?\\(tests|examples|benches)\\|^D:\\a\\kolo\\kolo\\python\\rust\\target($|\\)|^C:\\Users\\runneradmin\\\.cargo\\(registry|git)\\|^C:\\Users\\runneradmin\\\.rustup\\toolchains($|\\)` (exit code: 1)
--- stderr
No filenames specified!

I have confirmed that the .profraw files are generated and are in the directory I expect them in:

Run dir rust\target
  

    Directory: D:\a\kolo\kolo\python\rust\target

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           8/11/2023 10:56 AM                debug
-a---           8/11/2023 10:55 AM           1048 .rustc_info.json
-a---           8/11/2023  9:00 AM            177 CACHEDIR.TAG
-a---           8/11/2023 10:56 AM        2377336 rust-1720-9197467843878644390_0.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-3488-9197467843878644390_0.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-4996-9197467843878644390_4.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-5820-9197467843878644390_4.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-5936-9197467843878644390_0.profraw

I'm at a loss how to further debug this.

The text was updated successfully, but these errors were encountered:

taiki-e · 2023-08-12T12:48:22Z

Thanks for the report. The report subcommand does not work by itself. Could you provide information on all the commands you have run?

Also, is it possible to provide the information needed to reproduce this, such as code, workflows, compiler version, etc.?

LilyFoote · 2023-08-14T14:26:52Z

Hi! Sorry for not giving enough detail before. Here's the github actions workflow file (with some irrelevant jobs omitted):

on:
  push:
    paths:
      - .github/workflows/python-package.yml
      - python/**
  pull_request_target:
    paths:
      - .github/workflows/python-package.yml
      - python/**

jobs:
  python-package:
    if:
      (github.actor != 'dependabot[bot]' && github.event_name == 'push') ||
      (github.actor == 'dependabot[bot]' && github.event_name == 'pull_request_target')
    strategy:
      matrix:
        python: ["3.8", "3.9", "3.10", "3.11", "pypy-3.9", "pypy-3.10"]
        django: ["3.2.0", "4.1.0", "4.2.0"]
      fail-fast: false
    runs-on: buildjet-2vcpu-ubuntu-2204	
    defaults:
      run:
        working-directory: ./python
    steps:
    - name: Checkout
      if: ${{ github.event_name != 'pull_request_target' }}
      uses: actions/checkout@v3

    - name: Checkout PR
      if: ${{ github.event_name == 'pull_request_target' }}
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}

    - name: Install Rust toolchain
      uses: dtolnay/rust-toolchain@stable
      with:
        components: llvm-tools-preview

    - name: Install cargo-llvm-cov
      uses: taiki-e/install-action@cargo-llvm-cov

    - uses: Swatinem/rust-cache@v2
      with:
        workspaces: python/rust

    - uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python }}
        cache: "pip"
        cache-dependency-path: python/dev-requirements.txt
    - run: python -m venv .venv
    - run: .venv/bin/pip install -r dev-requirements.txt
    - run: .venv/bin/pip install django~=${{ matrix.django }}
    - run: .venv/bin/pip install maturin
    - run: .venv/bin/maturin develop --profile dev --extras=test_generation
      env:
        RUSTFLAGS: -C instrument-coverage -C llvm-args=--instrprof-atomic-counter-update-all --cfg=coverage --cfg=trybuild_no_target
        CARGO_INCREMENTAL: 0
        CARGO_LLVM_COV: 1
    - run: .venv/bin/pip list
    - run: .venv/bin/coverage run -m pytest --ignore=tests/simplepoll
      env:
        PYTHONFAULTHANDLER: 1
        RUST_BACKTRACE: 1
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
    - run: .venv/bin/coverage combine
    - run: .venv/bin/coverage xml
    - run: pwd
    - run: head -n 100 coverage.xml
    - run: cargo llvm-cov report --manifest-path=rust/Cargo.toml --codecov --output-path codecov.json
      env:
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
      if: ${{ !contains( matrix.python, 'pypy') }}
    - run: |
        . .venv/bin/activate
        cargo llvm-cov --manifest-path=rust/Cargo.toml --all-features --workspace --codecov --output-path codecov-unit.json
      env:
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
      if: ${{ !contains( matrix.python, 'pypy') }}
    - name: codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        flags: linux,${{ matrix.python }},${{ matrix.django }}
        verbose: true
        files: codecov.yaml,python/coverage.xml,python/codecov.json,python/codecov-unit.json

  windows:
    if:
      (github.actor != 'dependabot[bot]' && github.event_name == 'push') ||
      (github.actor == 'dependabot[bot]' && github.event_name == 'pull_request_target')
    strategy:
      matrix:
        python: ["3.8", "3.9", "3.10", "3.11", "pypy-3.9", "pypy-3.10"]
      fail-fast: false
    runs-on: windows-latest
    defaults:
      run:
        working-directory: ./python
    steps:
    - name: Checkout
      if: ${{ github.event_name != 'pull_request_target' }}
      uses: actions/checkout@v3

    - name: Checkout PR
      if: ${{ github.event_name == 'pull_request_target' }}
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}
    - name: Install Rust toolchain
      uses: dtolnay/rust-toolchain@stable

    - uses: Swatinem/rust-cache@v2
      with:
        workspaces: python/rust
    - uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python }}
        cache: "pip"
        cache-dependency-path: python/dev-requirements.txt
    - run: python -m venv .venv
    - run: .venv/Scripts/pip install -r dev-requirements.txt
    - run: .venv/Scripts/pip install django -U
    - run: .venv/Scripts/pip install maturin
    - run: .venv/Scripts/maturin develop --profile dev --extras=test_generation
    - run: .venv/Scripts/coverage run -m pytest --ignore=tests/simplepoll
      env:
        PYTHONFAULTHANDLER: 1
        RUST_BACKTRACE: 1
    - run: .venv/Scripts/coverage combine
    - run: .venv/Scripts/coverage xml
    - run: pwd
    - run: head -n 100 coverage.xml
    - run: mv coverage.xml ..
    - name: codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        flags: windows,${{ matrix.python }}
        verbose: true

The python-package version collects and reports coverage information, but the windows version leads to the error above.

The code being tested is not straightforward to share, but I will try to create a smaller replication.

LilyFoote · 2023-08-14T15:31:46Z

Here's a minimised example:

https://github.com/LilyFoote/llvm-cov-windows-bug
https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5857596860/job/15879786951

taiki-e · 2023-08-16T03:15:12Z

@LilyFoote

Here's a minimised example

https://github.com/LilyFoote/llvm-cov-windows-bug

Thanks. You seem to be manually setting some of the environment variables that cargo llvm-cov show-env sets, would the same problem occur if you used source <(cargo llvm-cov show-env --export-prefix) instead of setting them manually?

LilyFoote · 2023-08-16T10:16:59Z

@taiki-e That seems to work for linux, but not for windows - I get a parser error:

Run source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust\Cargo.toml)
ParserError: D:\a\_temp\9f4104b9-187b-4c10-acb1-975c3498a176.ps1:2
Line |
   2 |  source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust …
     |         ~
     | The '<' operator is reserved for future use.

https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5877498670/job/15937744529

taiki-e · 2023-08-16T10:20:44Z

I get a parser error:

@LilyFoote Could you try with shell: bash?

LilyFoote · 2023-08-16T10:46:34Z

I got that running, but we still have the original error:

https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5877711598/job/15938334102

taiki-e · 2023-08-22T14:31:11Z

I hid comments unrelated to this issue as off-topic.

The issue @messense has encountered is misuse due to lack of clarity in the documentation or lack of warnings. The situation here will be improved by adding warnings for such cases. See (and subscribe) Emit runtime warning on cargo llvm-cov nextest run #302 and Emit warning when show-env is used together with cargo llvm-cov commands other than report/clean #305 if you are interested in the status of adding warnings.
The issue @detly has encountered is the lack of nextest archive support. This is tracked in Any plans to make this work with --archive-file ? #265. The PR supporting it (Use extract-to if using --archive-file with nextest #266) is stalled, but the implementation itself should work (with my patch), so if there is no response from the PR author, I may add a commit to that PR and merge it. See (and subscribe) these issue and PR if you are interested in the status of supporting it.

And one of the reasons this issue has received unrelated additional reports is that the error itself is unclear. This is related to the fact that the error is handled on the llvm-tools side, and reporting the error with some hint on the cargo-llvm-cov side before passing arguments to llvm-tools should improve the situation (#306).

As for the issue @LilyFoote has encountered (the original topic of this issue -- I think that is cargo-llvm-cov or llvm-tools bug), I will fork your reproduction repo later and look into it. I'm guessing it is probably a windows path issue, but I'm still not sure what it really is.

taiki-e · 2023-08-23T15:11:19Z

Windows issue has been fixed in 0.5.30 (#307).

Also, diagnostics improvements (#302, #305, #306) have been released in 0.5.28 and 0.5.29.

taiki-e · 2023-12-28T11:05:36Z

The issue @detly has encountered is the lack of nextest archive support. This is tracked in Any plans to make this work with --archive-file ? #265. The PR supporting it (Use extract-to if using --archive-file with nextest #266) is stalled, but the implementation itself should work (with my patch), so if there is no response from the PR author, I may add a commit to that PR and merge it. See (and subscribe) these issue and PR if you are interested in the status of supporting it.

Btw, this has been supported in v0.6.0.

taiki-e added C-question Category: A question needs-mcve Call for participation: This issue needs a Minimal Complete and Verifiable Example labels Aug 12, 2023

This comment was marked as off-topic.

Sign in to view

taiki-e mentioned this issue Aug 15, 2023

Emit runtime warning on cargo llvm-cov nextest run #302

Closed

This comment was marked as off-topic.

Sign in to view

taiki-e added O-windows Operating system: Windows and removed needs-mcve Call for participation: This issue needs a Minimal Complete and Verifiable Example labels Aug 15, 2023

This comment was marked as off-topic.

Sign in to view

taiki-e mentioned this issue Aug 22, 2023

Emit warning when show-env is used together with cargo llvm-cov commands other than report/clean #305

Closed

taiki-e changed the title ~~llvm-cov report - no filenames specified~~ llvm-cov report - no filenames specified (failed only on windows and works on linux) Aug 22, 2023

This was referenced Aug 23, 2023

cargo llvm-cov report --lcov fails: No filenames specified! #304

Closed

Always handle .dll as object file on Windows #307

Merged

taiki-e closed this as completed in #307 Aug 23, 2023

taiki-e mentioned this issue Jan 24, 2024

Combining coverage from unit & integration tests produces incorrect summary #276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llvm-cov report - no filenames specified (failed only on windows and works on linux) #300

llvm-cov report - no filenames specified (failed only on windows and works on linux) #300

LilyFoote commented Aug 11, 2023

taiki-e commented Aug 12, 2023 •

edited

LilyFoote commented Aug 14, 2023

LilyFoote commented Aug 14, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

taiki-e commented Aug 16, 2023

This comment was marked as off-topic.

LilyFoote commented Aug 16, 2023

taiki-e commented Aug 16, 2023

LilyFoote commented Aug 16, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

taiki-e commented Aug 22, 2023

taiki-e commented Aug 23, 2023

taiki-e commented Dec 28, 2023

llvm-cov report - no filenames specified (failed only on windows and works on linux) #300

llvm-cov report - no filenames specified (failed only on windows and works on linux) #300

Comments

LilyFoote commented Aug 11, 2023

taiki-e commented Aug 12, 2023 • edited

LilyFoote commented Aug 14, 2023

LilyFoote commented Aug 14, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

taiki-e commented Aug 16, 2023

This comment was marked as off-topic.

LilyFoote commented Aug 16, 2023

taiki-e commented Aug 16, 2023

LilyFoote commented Aug 16, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

taiki-e commented Aug 22, 2023

taiki-e commented Aug 23, 2023

taiki-e commented Dec 28, 2023

taiki-e commented Aug 12, 2023 •

edited