My requests keep being rate limited, where's the limit and why do I keep hitting it?

[thoastbrot]

Hi,

I can't figure out how to debug this (I'm not familiar with go and logr), and I keep getting growing delays inbetween my requests. I wrote a small python wrapper to check multiple packages at once and it looks to me like github was rate limiting me, ending up with scorecard execution durations from 3, 5, 10, 30, 90, ... seconds, until I cancel requests by timeout.

I believe this doesn't match 5k messages per hour - although of course I don't know how (often) scorecard queries github for the queried information. I skipped the "Vulnerabilities" check already, as it queries another SaaS service.

I found this issue already, but the actual question wasn't answered... though it would help me to figure out how to print this log 😄

#2945

[spencerschrock]

GitHub has secondary rate limits that you may be triggering. Does the same thing occur if you only check one package at a time?
https://docs.github.com/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits

As for the logging, those messages should be enabled by default. Can you paste your output that you are seeing?

[thoastbrot]

I'm also some kind of github noob - my response is in the comment below 🤕

[thoastbrot]

No, it does not occur on a single run - but I would like (=need) to check multiple packages or know about the feasibility to do so.

Starting [Maintained]
Starting [Pinned-Dependencies]
Starting [Binary-Artifacts]
Starting [Vulnerabilities]
Starting [Packaging]
Finished [Maintained]
Finished [Pinned-Dependencies]
Finished [Binary-Artifacts]
Finished [Vulnerabilities]
Finished [Packaging]

RESULTS
-------
Aggregate score: 5.5 / 10

Check scores:
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
|  SCORE  |        NAME         |             REASON             |                                             DOCUMENTATION/REMEDIATION                                              |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Binary-Artifacts    | no binaries found in the repo  | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#binary-artifacts    |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Maintained          | 30 commit(s) out of 30 and 14  | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#maintained          |
|         |                     | issue activity out of 30 found |                                                                                                                    |
|         |                     | in the last 90 days -- score   |                                                                                                                    |
|         |                     | normalized to 10               |                                                                                                                    |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
| ?       | Packaging           | no published package detected  | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#packaging           |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Pinned-Dependencies | dependency not pinned by hash  | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#pinned-dependencies |
|         |                     | detected -- score normalized   |                                                                                                                    |
|         |                     | to 0                           |                                                                                                                    |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
| 0 / 10  | Vulnerabilities     | 14 existing vulnerabilities    | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#vulnerabilities     |
|         |                     | detected                       |                                                                                                                    |
|---------|---------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------|
scorecard --npm=$package   9.00s user 33.85s system 23% cpu 3:00.62 total

Is the output of a single run, which already ran for 30s. I get it - hitting the secondary rate limits (which I read of) would lead to the observed behavior, but I'm blind and I don't like that... by enabled by default, what exactly do you mean? I ran
time scorecard --verbosity=debug --npm=$package --checks=Maintained,Pinned-Dependencies,Binary-Artifacts,Vulnerabilities,Packaging

(and I know that show-results doesn't help me, verbosity might help me, but didn't yet)

Edit: one error passed through earlier on:

scorecard --format json --show-details --verbosity trace --output foo.json --npm=react
Error: check runtime error: Branch-Protection: internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by personal access token
2024/03/13 13:43:18 error during command execution: check runtime error: Branch-Protection: internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by personal access token

but right now I was looking at a stalled execution again, no logs for 8 minutes. My mix up of parameter style should be ugly, but not the cause (foo.json has been filled properly).

[thoastbrot]

retried using --output to file, but doesn't show me rate limiting messages either... also tried using the docker image (stable release) and a custom build, to no avail.

[spencerschrock]

so --verbosity has to do with our results and supports debug, info and warn.

The logging statements associated with rate limiting are their own thing and would be showing up if it was a rate limit thing.

the Branch-Protection error has to do with a known limitation with fine-grained vs classic tokens.

[spencerschrock]

I notice you're using the --npm flag, which uses the npm registry API. I'm curious if the rate limiting is happening on the npm side. That uses a different HTTP client and wouldn't have our ratelimiting code attached.

Does the problem go away if you use the --repo flag (of course this would mean already knowing the repo url)?

[thoastbrot]

I had another run at the topic. I discovered using another token type ("classic") seems to perform better than the PAT I configured earlier. Additionally, there seem to be notable differences between "latest", tagged docker container and my homebrew install (which should be 4.13.1), but I don't want to burn my token/account before having solved the logging issue.

Funnily enough, as I wrongly called docker from within python ("-e var=var" as single arg), scorecard complained correctly and logged that including a rate limiting message).