[skip-ci] GHA: Attempt fix exceeded a secondary rate limit #22550
Conversation
Frequent but intermittently, the stale issue and PR locking workflow generates the error: ``` You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID XYZ ``` According to upstream `dessant/lock-threads` issue 48, this seems to be coming from the GitHub side (bug/feature/limitation), since the action uses an official github API rate-limiting library. It's unlikely related to which style/syntax of github token is used, nor if the action is executed concurrently across multiple repos. According to the rate-limiting docs: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits it's possible the issue is caused due to an unknown aspect of the clause: ``` These secondary rate limits are subject to change without notice. You may also encounter a secondary rate limit for undisclosed reasons. ``` The same docs indicate Github Apps have enhanced rate-limits which scale with the org's repo count. Attempt to fix the intermittent failures by making use of a new, dedicated, org-specific, private "Stale Locking App" I recently created. This requires the addition of a new action to the workflow that obtains a short-lived token for passing to lock-threads. Note: Because both `vars.STALE_LOCKING_APP_ID` and `secrets.STALE_LOCKING_APP_PRIVATE_KEY` are defined at the containers-organization level, the Buildah and Skopeo re-use of this workflow should continue to function normally w/o change. Signed-off-by: Chris Evich <cevich@redhat.com>
|
Review note: I think this is the needed fix and done correctly. The only practical way to test it, is to merge it @edsantiago I know this isn't your forte, but if you could look over my commit message and comments, I'd appreciate the feedback. @ashley-cui @lsm5 I realize this is a lot for such a small diff, but PTAL once you gave time to consume the background details. |
| uses: actions/create-github-app-token@v1 | ||
| with: | ||
| # N/B: These are both defined at the containers-org level | ||
| app-id: ${{ vars.STALE_LOCKING_APP_ID }} |
There was a problem hiding this comment.
The string vars does not appear anywhere else in .github/workflows; the usual form is secrets.XX. Is vars a real thing, or did you mean secrets here?
There was a problem hiding this comment.
Sharp eye. This was surprising to me also (new feature maybe?). Yes it's intended and documented that it will consider org-level action-vars (which this is).
openshift-ci
bot
added
the
approved
Unf. the problem is intermittent and nobody has a clue as to exactly which limit is being breached (including github support). As for testing, I s'pose I could give this a try over on the |
not worth the pain imho. Besides if we never see any rate limit complaints ever again, we can assume it works :) |
openshift-ci
bot
added
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cevich, lsm5 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
LGTM as far as I can tell, but only merging will reveal the truth :) |
|
The truth is out there. /hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
openshift-merge-bot
bot
merged commit 9c24033
into
containers:main
|
Thanks guys. Now that it's merged, I can at least manually trigger it to confirm there's no typos or such... |
Frequent but intermittently, the stale issue and PR locking workflow generates the error:
According to upstream
dessant/lock-threadsissue 48, this seems to be coming from the GitHub side (bug/feature/limitation), since the action uses an official github API rate-limiting library. It's unlikely related to which style/syntax of github token is used, nor if the action is executed concurrently across multiple repos.According to the rate-limiting docs:
https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits it's possible the issue is caused due to an unknown aspect of the clause:
The same docs indicate Github Apps have enhanced rate-limits which scale with the org's repo count. Attempt to fix the intermittent failures by making use of a new, dedicated, org-specific, private "Stale Locking App" I recently created. This requires the addition of a new action to the workflow that obtains a short-lived token for passing to lock-threads.
Note: Because both
vars.STALE_LOCKING_APP_IDandsecrets.STALE_LOCKING_APP_PRIVATE_KEYare defined at the containers-organization level, the Buildah and Skopeo re-use of this workflow should continue to function normally w/o change.Does this PR introduce a user-facing change?