Unknown Badge on coveralls.io

[meherett]

I tried many options to fix this including deletion of the repo but still the issue was not resolved. I need help. https://coveralls.io/github/meherett/python-bip38

Thanks!

[ericmorand]

I have the exact same issue there: https://coveralls.io/github/NightlyCommit/html-source-map-rebase

[afinetooth]

Hi, @meherett and @ericmorand.

I see that you both have the same issue. You both appear to be using TravisCI, but we are receiving no git-related data with your coverage reports.

We require some minimum git-related data in an incoming parameter called [git](https://docs.coveralls.io/api-jobs-endpoint#:~:text=String-,git%0Arequired,Hash,-run_at), which you can read about here.

This information is typically picked up automatically from the CI environment, except under certain conditions, like a missing .git folder, or git not being installed on your runners, or a misconfiguration.

Can you both please share what Coveralls Integration you're currently using? We have seen some older Language Integrations fail to include some required data lately.

Can you also please tell me if there's anything about your environment that might be atypical, like no .git folder or git not being installed?

[afinetooth]

P.S. I wanted to share what the UI looks like with that missing info, just so you're aware if you see it again.

So, for meherett/python-bip38:

And for NightlyCommit/html-source-map-rebase:

In your case, @ericmorand, we're getting slightly more info from your pull_request builds, which could hold a clue.

[ericmorand]

@afinetooth thanks for taking the time to help us.

I did an interesting test with another one of my node packages: I triggered a "build" on Travis CI and it ended up missing the branch information that you mentioned. This is interesting because the previous time a build was triggered was 3 years ago and it was fine as you can see there:

https://coveralls.io/github/NightlyCommit/css-source-map-rebase

So definitely something changed in the course of these three years. I'm using Travis CI and the coveralls node package to handle the push data to Coveralls for me, so I'll start my investigation there and will let you know.

[ericmorand]

OK, I think the issue is coming from Travis CI. I executed the coveralls command locally - putting my repo token as the appropriately named environment variable, and it corrected the issue:

https://coveralls.io/gitlab/nightlycommit/html-source-map-rebase

Most probably, @afinetooth, as you suspected, something is missing in the Travis CI environment that I use to execute the automation - maybe git itself. Since @meherett seems to also use Travis CI, it may also explain his issue.

I'll investigate the Travis side now.

EDIT: OK, I found that coveralls is not agnostic to the environment where it is running. In other words, it knows if it is running on Travis, Jenkins, Circle CI...and act in consequence. I won't comment on the fact that this is a very debatable design decision, but the consequence of this is that when run on a Travis CI environment, coveralls always set the git id to "HEAD":

if (process.env.TRAVIS) {
    options.service_name = 'travis-ci';
    options.service_number = process.env.TRAVIS_BUILD_NUMBER;
    options.service_job_id = process.env.TRAVIS_JOB_ID;
    options.service_pull_request = process.env.TRAVIS_PULL_REQUEST;
    git_commit = 'HEAD';
    git_branch = process.env.TRAVIS_BRANCH;
  }

The problem is that this can't be overriden since coveralls will only consider its own COVERALLS_GIT_COMMIT env variable or fetch the commit hash from git itself if it was not able to resolve it before:

if (!git_commit || !git_branch) {
    const data = detectLocalGit();
    if (data) {
      git_commit = git_commit || data.git_commit;
      git_branch = git_branch || data.git_branch;
    }
  }

@afinetooth is it possible that at one point in the past, having HEAD as the commit id instead of the commit hash was working? I'm still puzzled that it worked 3 years ago and not working today even though coveralls package did not change in the course of these last three years - at least regarding the commit id resolution.

Anyway, at least in my case, the Coveralls app is not at fault. I'll open an issue in coveralls package issue tracker but since the package seems not maintained anymore, I think I'll have to fix and publish my own dedicated fork.

EDIT 2: I'll switch to https://github.com/coverallsapp/coverage-reporter. Works flawlessly. :)

https://coveralls.io/gitlab/nightlycommit/html-source-map-rebase

[afinetooth]

@ericmorand still catching up, but wanted to respond to this:

So definitely something changed in the course of these three years. I'm using Travis CI and the coveralls node package to handle the push data to Coveralls for me, so I'll start my investigation there and will let you know.

We've made a lot of changes to Coveralls in the past 3 years and while we've tried to make them all backwards-compatible, it's very possible that some legacy integrations, including language integrations like node-coveralls, have encountered some gaps.

For one thing re: node-coveralls, we now recommend a more up-to-date and well maintained fork of node-coveralls, coveralls-next. (You can find the recommendation in our updated docs on JavaScript & Node language integrations.)

Since that's a fork of node-coveralls it should be a drop-in replacement.

But if you really want to use the most up-to-date integration, which supports many coverage report formats, not just LCOV as node-coveralls does, consider using one of our official integrations, either our Universal Coverage Reporter, or one of our two extensions for popular CI services, which are based on it.

The coverage reporter is a cross-platform binary that can be used in almost any environment.

[afinetooth]

@ericmorand re:

OK, I found that coveralls is not agnostic to the environment where it is running.

Yes, aside from my recommendation above to drop-in coveralls-next for node-coveralls, I see you've discovered the "context-aware" features of node-coveralls, which is common to many Coveralls integrations, which sense their CI environment and try to auto-configure themselves for you.

But you're right, this can be a problem if the integration's env var selections override your own, when you know which you need to use. TBH, I cannot remember now what the precedence of the several different classes of Coveralls env var are in node-coveralls, just that the intent, these days, is to have user-provided COVERALLS_-prefixed env vars take final precedence over all others.

The problem is that this can't be overridden since coveralls will only consider its own COVERALLS_GIT_COMMIT env variable or fetch the commit hash from git itself if it was not able to resolve it before:

It does look like COVERALLS_GIT_COMMIT will be overwritten with the TRAVIS-related auto-config code in node-coveralls. I'm not sure if this is easily worked-around in coveralls-next, but one approach is to "trick" the integration into thinking you're using a different CI service, by using the env var COVERALLS_SERVICE_NAME and passing for its value anything but travis, travis-ci etc. (anything without the /travis/ pattern in the name).

[...] is it possible that at one point in the past, having HEAD as the commit id instead of the commit hash was working?

Yes, very possible.

Newer integrations are more flexible. For instance, the commit to use for a given CI build is now customizable in newer, official integrations. For instance, in the Coveralls GitHub Action, you can override the commit obtained from the environment with the git-commit input option.

I'll open an issue in coveralls package issue tracker [...]

Yes, probably not the best use of time. Please see my recommendations above.

To reiterate them in order of most to least convenience:

1. Replace node-coveralls with coveralls-next in your project's package.json and try again as-is. (You may actually want to rewind recent changes to see if it works in off-the-shelf mode.)
2. Opt for the official integration, Coveralls Universal Coverage Reporter. Usage is pretty straightforward. These two examples (one | two), where the selected CI services are being treated like any other service, such that the usage also applies for TravisCI, show slightly more fancy usage, including downloading the coverage reporter binary with checksums for extra security (one), and installing it on three different OS's on three different runners in the same workflow (two).

[ericmorand]

Thank you so much @afinetooth for such thorough explanations and directions. I ended up using the universal reporter and as you say, it is very straightforward. Plus it is the future direction so I'm very glad I made the switch.

Everything is now fixed and arguably working better than before. So again, thank you!

[afinetooth]

@ericmorand you're more than welcome!

I'm very glad to hear you're up-and-running with the Coverage Reporter.

That should be smooth sailing once you're set up, and what's nice about that integration is that it supports a range of coverage report formats, so you should be able to replicate the pattern for any new integrations, regardless of what languages your other projects are in.

[meherett]

Hi @afinetooth,

What about for this https://pypi.org/project/coveralls package? Is there any recommendation?

Here it's my .travis.yml config

language: python
dist: bionic
before_install:
  - python -m pip install --upgrade pip
install:
  - pip install -e .[tests,docs] tox-travis coveralls
matrix:
  include:
  - name: "3.9"
    python: 3.9
  - name: "3.10"
    python: 3.10
  - name: "3.11"
    python: 3.11
script: tox
after_success: if [ "${TRAVIS_PYTHON_VERSION}" == "3.11" ]; then coveralls; fi;

[ericmorand]

@meherett , I think you could easily move to the universal reporter too. Here is the GitLab CI manifest I use now, if this can help you:

.test-and-emit-coverage:
  extends: .test
  image: node:19
  only:
    - merge_requests
    - main
  script:
    - npm install nyc --global
    - curl -L https://coveralls.io/coveralls-linux.tar.gz | tar -xz -C .
    - nyc npm test
    - nyc report --reporter=text-lcov > coverage/lcov.info && ./coveralls report --repo-token=$COVERALLS_REPO_TOKEN

Based on your example, I think switching to the universal reporter would look something like this:

language: python
dist: bionic
before_install:
  - python -m pip install --upgrade pip
install:
  - pip install -e .[tests,docs] tox-travis
  - curl -L https://coveralls.io/coveralls-linux.tar.gz | tar -xz -C .
matrix:
  include:
  - name: "3.9"
    python: 3.9
  - name: "3.10"
    python: 3.10
  - name: "3.11"
    python: 3.11
script: tox
after_success: ./coveralls report

Not sure what the path to your LCOV report is, but assuming it is in a path considered by default by the universal reporter, it should work; if not, the path needs to be passed explicitely as second parameter to the ./coveralls command:

https://github.com/coverallsapp/coverage-reporter#examples

Note that the token should be included as environment variable by the Travis runner so you probably don't have to bother with it. As you can see, I'm not using Travis CI so I had to pass it explicitely.

[afinetooth]

Hi @meherett. Thanks, @ericmorand.

That Coveralls Language Integration is one of the most well-maintained and up-to-date community-created integrations, so as far as that class of integration goes it's a good choice.

That said, users of that integration still have issues, for, I believe, the same reasons users of other language integrations do, which is that we have made many changes to the API over the past 18 months, and, while we've tried to make them all backwards compatible, issues still arise.

For that reason, I would still recommend an Official Integration, and @ericmorand is right, the Universal Coverage Reporter supports coverage.py/pytest-cov format reports, so it's a good option for a Python project. The real advantage of using an official integration, though, is that we can support your entire tool chain and you don't need to seek support across two projects when you encounter issues.

One of our goals with our official integrations is to streamline the integration process for Coveralls and remove the need for community integration creators to continue maintaining their integrations and give them the option to contribute a parser to the Coverage Reporter and retire their project if they wish.

[nuxy]

Chiming into this thread.

Experiencing the same issue with a new repository using a build configuration that works with another Coveralls project I maintain. Same package releases, Travis CI instance configuration, etc.. One works, the other doesn't.

The only difference I see between the two is that with the latest project I initialized my coveralls.io with one branch develop, ran multiple Travis CI builds, then switched branches master which didn't take until I cleared the Cache.

Not sure if this makes any difference, just throwing it out there.

UPDATE

One works, the other doesn't.

Looking into this further it seems that builds for the other project stopped synchronizing in May.

Beyond that point the branch and commit messages are no longer defined. 😞

RESOLUTION

@afinetooth
It seems that the coveralls-next fork doesn't work either and the fork owner doesn't plan on furthing development outside of dependency updates. That said, it's probably a good idea to remove this option from Coveralls repo (Travis CI) installation options since it provides a false hope which incurs a lot of unnecessary debugging pain.

In the end, as recommended I integrated the coverage-reporter with the following changes and everything now works.

// travis.yml
jobs:
  include:
    - language: node_js
      node_js:
        - "16"
      install:
        - curl -L https://coveralls.io/coveralls-linux.tar.gz | tar -xz -C .
        - npm install
      script:
        - npm run test
      after_success: npm run coverage
      
// package.json
{
  "scripts": {
    "test": "nyc mocha --require test/mocha.env.js 'test/**/*.js'",
    "coverage": "nyc report --reporter=text-lcov > lcov.info && ./coveralls report lcov.info"
  }
}