[RFC] Stale issues/pulls action

[dmarto]

Related to: #36253 #36399 #36462

My believe is that with the current action and settings, the backlog won't disappear.

The action tries to consume all open issues and PRs, on every run.
Even with the max hourly limit of 1000 operation, the action can't check and label all currently open items.
(can be seen in the debug runs from @maldridge)

As of now ~210 issues and ~418 PRs are supposed to be Stale.
(taking into account the change from #36609, https://github.com/void-linux/void-packages/issues?q=is%3Aopen+updated%3A%3C%3D2022-01-11+no%3Aassignee+-label%3Arequest)

At the current rates of new new issues/PRs, I believe they will never be marked, only very few (the newest of the oldest) items will get marked.

Also there seems is another issue with the action - it doesn't count PR review comments as updates for the action, example #34884. That PR got marked Stale, received a review, and the next day the action saw the changes, but didn't count them as comments and left the Stale label. Thus it seems, only real comments are counted.

---

I am suggesting to raise the operation limit to something like 400 (which should result in roughly 50-60, newly marked items per run), so there can be at least some progress. The other actions, are not doing many requests, so for the one random hour in the middle of the night this should be okay.

Multiple runs a day, can also be an option - but they will mark one or two new items, as currently the bulk of the operations is spent on fetching brand new items and checking current stale ones.

Of course an option will be to find or write a smarter action

• that can has better access pattern into the backlog (i.e not always starting from the newest issue/PR)
• possibly even split itself into two actions that can run in different hours (issues and pulls)

P.S. There seems to be a v5 of the action, but it doesn't address any of the concerned topics.

P.P.S https://github.com/Sonia-corporation/stale may be a solution, but I haven't looked too much into it.

[classabbyamp]

Also there seems is another issue with the action - it doesn't count PR review comments as updates for the action, example #34884. That PR got marked Stale, received a review, and the next day the action saw the changes, but didn't count them as comments and left the Stale label. Thus it seems, only real comments are counted.

this looks like a bug to me. The github API shows it was updated after the stale label was added (it's the time paper added their review comments):

$ gh pr view --json updatedAt 34884
{
  "updatedAt": "2022-04-10T11:45:44Z"
}

[dmarto]

indeed, but here is what the action tell us (on runs after the marking)

[#34884] Pull request #34884
  [#34884] Found this pull request last updated at: 2022-04-10T11:45:44Z
  [#34884] The option only-labels (​https://github.com/actions/stale#only-labels​) was not specified
  [#34884] └── Continuing the process for this pull request
  [#34884] Days before pull request stale: 90
  [#34884] The pull request is not closed nor locked. Trying to remove the close label...
  [#34884] ├── The close-pr-label (​https://github.com/actions/stale#close-pr-label​) option was not set
  [#34884] └── Skipping the removal of the close label
  [#34884] This pull request has a stale label
  [#34884] The option any-of-labels (​https://github.com/actions/stale#any-of-labels​) was not specified
  [#34884] └── Continuing the process for this pull request
  [#34884] This pull request has no milestone
  [#34884] └── Skip the milestones checks
  [#34884] This pull request has no assignee
  [#34884] └── Skip the assignees checks
  [#34884] This pull request is already stale
  [#34884] Checking for label on this pull request
  [#34884] Pull request marked stale on: 2022-04-10T02:05:37Z
  [#34884] Checking for comments on pull request since: 2022-04-10T02:05:37Z
  [#34884] Comments that are not the stale comment or another bot: 0
  [#34884] Pull request has been commented on: false
  [#34884] Days before pull request close: 14
  [#34884] Pull request has been updated: true
  [#34884] The option remove-stale-when-updated (​https://github.com/actions/stale#remove-stale-when-updated​) is: true
  [#34884] The stale label should not be removed
  [#34884] Stale pull request is not old enough to close yet (hasComments? false, hasUpdate? true)
  [#34884] 1 operation consumed for this pull request

the important lines are

..
  [#34884] Pull request has been commented on: false
..
  [#34884] Pull request has been updated: true
..

fast reference in the source of the action, _processStaleIssue -> _hasCommentsSince -> listIssueComments

it checks only for comments, and if we check what the API gives us, we can confirm

$ gh pr view --json comments 34884
{
  "comments": [
    {
      "author": {
        "login": "ericonr"
      },
      "authorAssociation": "MEMBER",
      "body": "Please mention in the commit that you're switching upstream to the version that has the newest releases.",
      "createdAt": "2022-01-06T04:39:54Z",
      "includesCreatedEdit": false,
      "isMinimized": false,
      "minimizedReason": "",
      "reactionGroups": [
        {
          "content": "THUMBS_UP",
          "users": {
            "totalCount": 1
          }
        }
      ]
    },
    {
      "author": {
        "login": "github-actions"
      },
      "authorAssociation": "NONE",
      "body": "Pull Requests become stale 90 days after last activity and are closed 14 days after that.  If this pull request is still relevant bump it or assign it.",
      "createdAt": "2022-04-10T02:05:36Z",
      "includesCreatedEdit": false,
      "isMinimized": false,
      "minimizedReason": "",
      "reactionGroups": []
    }
  ]
}

so it is not exactly a bug (as in code mistake/error), it just not a feature that the action offers

[0x5c]

so it is not exactly a bug (as in code mistake/error), it just not a feature that the action offers

The documentation states:

remove-stale-when-updated
Automatically remove the stale label when the issues or the pull requests are updated (based on GitHub issue field updated_at) or commented.
Default value: true

The action not acting like the documentation says it should is a bug

Edit: adding those relevant lines

  [#34884] Pull request has been updated: true
  [#34884] The option remove-stale-when-updated (​https://github.com/actions/stale#remove-stale-when-updated​) is: true
  [#34884] The stale label should not be removed

(no message removed between them)

• It detects the PR has been updated: Correct
• It states it should remove stale when a pr is updated: Correct
• It decides that the label should not be removed: ???

[0x5c]

Found the code where it doesn't do the checks it should do, and reported the bug actions/stale#715.
It does the proper check ( not updated AND not commented) to gate the logic that actually closes issues/PRs, so that PR might be softlocked into a stale but not closed state.

[dmarto]

My bad, I only checked the table in the action documentation; that didn't mention ..based on field updated_at.. so I assumed wrong, it is indeed a bug.

[0x5c]

I was able to come up with a fix so I made a PR in the stale repo actions/stale#717

[dmarto]

As of few hours ago, @0x5c fix was merged in upstream; with that, I think all point are covered now and we just have to wait for a new release of the action.

So, maybe we can close this issue or convert it into a tracking one for the supposed action update?

[0x5c]

There are more bugs that weren't mentioned here, but are much harder to fix in stalebot without a refactor. For example issues/PRs being in eternal stale limbo if assigned to someone after they got the stale label since the stale/unstale logic is after the "is this assigned" check skips the issue.

The workflow will still need to be updated to the v5 branch (currently using v4) since that's the version that will contain my fix when it arrives.

[github-actions]

Issues become stale 90 days after last activity and are closed 14 days after that. If this issue is still relevant bump it or assign it.