Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit and/or Cache API requests to avoid: 403 - API rate limit exceeded for installation ID ___ #266

Open
trinitronx opened this issue Nov 16, 2023 · 2 comments
Open

Limit and/or Cache API requests to avoid: 403 - API rate limit exceeded for installation ID ___ #266

trinitronx opened this issue Nov 16, 2023 · 2 comments

Comments

@trinitronx
Copy link

trinitronx commented Nov 16, 2023 •
edited

I just switched to using Jekyll 4.3.2 and tried to deploy a static site using this theme, which lists all current GitHub repositories on the generated static site page. To do this, it uses jekyll-github-metadata gem (version 2.16.0) to gather all repositories for my GitHub user.

Apparently, this results in a lot of API requests (for my user, over ~486!) and quickly hits the rate limit when using GITHUB_TOKEN in a GitHub Actions workflow.

Would it be possible to cache or limit API requests further so static site deployments will complete successfully in GitHub Actions?

For example: The jekyll build --verbose output shows each repo is queried for both contributors and releases, but the template doesn't even need this information 

  Logging at level: debug
    Jekyll Version: 4.3.2
Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config.yml
Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config_dev.yml
         Requiring: jekyll-github-metadata
To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
   GitHub Metadata: Initializing...
            Source: /mnt/terabyte/src/sites/trinitronx.github.io
       Destination: /mnt/terabyte/src/sites/trinitronx.github.io/_site
 Incremental build: enabled
      Generating... 
           Reading: /_layouts/default.html
       EntryFilter: excluded /LICENSE.neumorphism
       EntryFilter: excluded /README.md
       EntryFilter: excluded /docs
       EntryFilter: excluded /package.json
       EntryFilter: excluded /node_modules
       EntryFilter: excluded /.jekyll-cache
       EntryFilter: excluded /.github
       EntryFilter: excluded /.ruby-version
       EntryFilter: excluded /Gemfile
       EntryFilter: excluded /yarn.lock
       EntryFilter: excluded /.ruby-gemset
       EntryFilter: excluded /Gemfile.lock
       EntryFilter: excluded /.gitignore
       EntryFilter: excluded /gulpfile.js
           Reading: index.html
           Reading: _data/projects.yml
           Reading: _data/skills-frameworks.yml
           Reading: _data/skills-languages.yml
           Reading: _data/skills-tools.yml
           Reading: _data/timeline.yml
         Rendering: index.html
  Pre-Render Hooks: index.html
  Rendering Markup: index.html
Post-Convert Hooks: index.html
  Rendering Layout: index.html
   GitHub Metadata: Generating for trinitronx/trinitronx.github.io
   GitHub Metadata: Calling @client.list_repos("trinitronx", {:type=>"public", :accept=>"application/vnd.github.mercy-preview+json"})
   GitHub Metadata: Calling @client.releases("trinitronx/action-doctl")
   GitHub Metadata: Calling @client.contributors("trinitronx/action-doctl")
   GitHub Metadata: Calling @client.releases("trinitronx/ansible")
   GitHub Metadata: Calling @client.contributors("trinitronx/ansible")
   GitHub Metadata: Calling @client.releases("trinitronx/ansible-examples")
   GitHub Metadata: Calling @client.contributors("trinitronx/ansible-examples")


[... SNIP ...]
[... TOTAL number of releases + contributors API log lines = 486 ...]
[... SNIP ...]


   GitHub Metadata: Calling @client.releases("trinitronx/zengge-lightcontrol")
   GitHub Metadata: Calling @client.contributors("trinitronx/zengge-lightcontrol")
           Writing: /mnt/terabyte/src/sites/trinitronx.github.io/_site/index.html
  Writing Metadata: .jekyll-metadata
                    done in 177.699 seconds.
 Auto-regeneration: disabled. Use --watch to enable.
[18:31:02] Finished 'jekyll' after 2.97 min
@trinitronx trinitronx changed the title Limit and/or Cache API requests - 403 - API rate limit exceeded for installation ID ___ Limit and/or Cache API requests to avoid: 403 - API rate limit exceeded for installation ID ___ Nov 16, 2023
@trinitronx
Copy link
Author

trinitronx commented Nov 17, 2023 •
edited

Some ideas:

  • Simply allow a config option to disable these API calls for sites that don't need releases or contributors data gathered
  • Gather the data lazily if the releases or contributors are actually accessed

In case it helps identify the code paths involved for the API calls...

Expand for stacktrace when pressing Ctrl+C during the API call loop 

/home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `initialize': Interrupt
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `open'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `block in connect'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:117:in `timeout'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1016:in `connect'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:995:in `do_start'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:984:in `start'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:102:in `perform_request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:66:in `block in call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/adapter.rb:45:in `connection'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:65:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:61:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/rack_builder.rb:153:in `build_response'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:444:in `run_request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:200:in `get'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/sawyer-0.9.2/lib/sawyer/agent.rb:99:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:156:in `request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:84:in `paginate'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/client/releases.rb:15:in `releases'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `public_send'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `block in method_missing'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:92:in `save_from_errors'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `method_missing'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:123:in `block (3 levels) in owner_public_repositories'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:53:in `call_or_value'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:35:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:55:in `block in sanitize_resource'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each_with_object'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `sanitize_resource'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:38:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `block in sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `map'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:216:in `memoize_value'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:120:in `owner_public_repositories'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/forwardable.rb:238:in `public_repositories'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `public_send'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `[]'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/context.rb:197:in `lookup_and_evaluate'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/variable_lookup.rb:48:in `block in evaluate'

@aspiers
Copy link

aspiers commented Apr 15, 2024

I see something similar; it's querying releases and contributors for ALL repos I own, even though I'm only using Jekyll on a single one - why is it doing this?!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aspiers @trinitronx