Pagination offset too large (http status: 400)

[nappernick]

Environment

Github action
platform: linux, architecture: x86_64
node v16.16.0
sentry-cli 2.5.0 (curl -sL https://sentry.io/get-cli/ | bash)

How do you use Sentry?
Sentry SaaS (sentry.io)

Steps to Reproduce

SENTRY_RELEASE=$(_______) >> $_______
  
curl -sL https://sentry.io/get-cli/ | bash
sentry-cli releases -o org -p folder new $SENTRY_RELEASE
sentry-cli releases -o org -p folder set-commits --auto $SENTRY_RELEASE
sentry-cli releases -o org -p folder finalize $SENTRY_RELEASE
sentry-cli releases -o org -p folder deploys $SENTRY_RELEASE new -e $SENTRY_ENV
sentry-cli releases -o org -p folder files $SENTRY_RELEASE upload src
sentry-cli releases -o org -p folder files $SENTRY_RELEASE upload-sourcemaps lib --log-level=debug

Expected Result

Successfully upload sourcemaps, so we can deploy to development/production.

Actual Result

Finalized release master
Created new deploy _______ for '_______'
> Found 201 release files
> Bundled 201 files for upload
> Uploaded release files to Sentry
> File upload complete (processing pending on server)
> Organization: _______
> Project: _______
> Release: _______
> Dist: None
  DEBUG   2022-08-04 19:56:53.369529646 +00:00 sentry-cli version: 2.5.0, platform: "linux", architecture: "x86_64"
  INFO    2022-08-04 19:56:53.369617948 +00:00 sentry-cli was invoked with the following command line: "sentry-cli" "releases" "-o" "______" "-p" "______" "files" "master" "upload-sourcemaps" "lib" "--ignore-file" ".sentryignore" "--log-level=debug"
> Found 374 release files
> Analyzing 374 sources
> Rewriting sources
> Adding source map references

Full logs

DEBUG   2022-08-04 19:59:45.964989286 +00:00 request GET https://sentry.io/api/0/projects/______/______/releases/master/files/?cursor=100:200:0
DEBUG   2022-08-04 19:59:45.965000386 +00:00 using token authentication
DEBUG   2022-08-04 19:59:45.965009286 +00:00 retry number 0, max retries: 0
DEBUG   2022-08-04 19:59:45.965118489 +00:00 > GET /api/0/projects/______/______/releases/master/files/?cursor=100:200:0 HTTP/1.1
DEBUG   2022-08-04 19:59:45.965126589 +00:00 > Host: sentry.io
DEBUG   2022-08-04 19:59:45.965131789 +00:00 > Accept: */*
DEBUG   2022-08-04 19:59:45.965136889 +00:00 > Connection: TE
DEBUG   2022-08-04 19:59:45.965141990 +00:00 > TE: gzip
DEBUG   2022-08-04 19:59:45.965147290 +00:00 > User-Agent: sentry-cli/2.5.0
DEBUG   2022-08-04 19:59:45.965168490 +00:00 > Authorization: ***
DEBUG   2022-08-04 19:59:46.174047491 +00:00 < HTTP/1.1 400 Bad Request
DEBUG   2022-08-04 19:59:46.174104993 +00:00 < Server: nginx
DEBUG   2022-08-04 19:59:46.174116893 +00:00 < Date: Thu, 04 Aug 2022 19:59:46 GMT
DEBUG   2022-08-04 19:59:46.174124593 +00:00 < Content-Type: application/json
DEBUG   2022-08-04 19:59:46.174132293 +00:00 < Content-Length: 40
DEBUG   2022-08-04 19:59:46.174139594 +00:00 < Connection: keep-alive
DEBUG   2022-08-04 19:59:46.174145994 +00:00 < allow: GET, POST, HEAD, OPTIONS
DEBUG   2022-08-04 19:59:46.174152094 +00:00 < access-control-allow-methods: GET, POST, HEAD, OPTIONS
DEBUG   2022-08-04 19:59:46.174162794 +00:00 < access-control-allow-headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding, sentry-trace, baggage
DEBUG   2022-08-04 19:59:46.174170494 +00:00 < access-control-expose-headers: X-Sentry-Error, Retry-After
DEBUG   2022-08-04 19:59:46.174176594 +00:00 < access-control-allow-origin: *
DEBUG   2022-08-04 19:59:46.174182695 +00:00 < x-sentry-rate-limit-remaining: 39
DEBUG   2022-08-04 19:59:46.174188795 +00:00 < x-sentry-rate-limit-limit: 40
DEBUG   2022-08-04 19:59:46.174195295 +00:00 < x-sentry-rate-limit-reset: 1659643186
DEBUG   2022-08-04 19:59:46.174201495 +00:00 < x-sentry-rate-limit-concurrentremaining: 24
DEBUG   2022-08-04 19:59:46.174207395 +00:00 < x-sentry-rate-limit-concurrentlimit: 25
DEBUG   2022-08-04 19:59:46.174213395 +00:00 < vary: Accept-Language, Cookie
DEBUG   2022-08-04 19:59:46.174220796 +00:00 < content-language: en
DEBUG   2022-08-04 19:59:46.174226696 +00:00 < x-frame-options: deny
DEBUG   2022-08-04 19:59:46.174232696 +00:00 < x-content-type-options: nosniff
DEBUG   2022-08-04 19:59:46.174238696 +00:00 < x-xss-protection: 1; mode=block
DEBUG   2022-08-04 19:59:46.174251396 +00:00 < x-envoy-attempt-count: 1
DEBUG   2022-08-04 19:59:46.174257596 +00:00 < x-envoy-upstream-service-time: 176
DEBUG   2022-08-04 19:59:46.174263797 +00:00 < x-served-by: getsentry-web-default-production-5887496c5f-m24x8
DEBUG   2022-08-04 19:59:46.174269697 +00:00 < x-served-by: lb-11
DEBUG   2022-08-04 19:59:46.174275497 +00:00 < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
DEBUG   2022-08-04 19:59:46.174303498 +00:00 response status: 400
DEBUG   2022-08-04 19:59:46.174315698 +00:00 body: {"detail":"Pagination offset too large"}
error: API request failed
caused by: sentry reported an error: Pagination offset too large (http status: 400)
DEBUG   2022-08-04 19:59:46.182472301 +00:00 skipping update nagger because session is not attended
Error: Process completed with exit code 1.

I think the critical line is that the upload fails at cursor=100:200:0. We've tried applying flags for --ignore .spec.js --ignore .spec.js.map, then --ignore-file .sentryignore, to shrink the number of files that are bundled for upload (successfully dropped the file count from 402 to 374), but always fail with Pagination offset too large (http status: 400).

[nappernick]

Found an issue from 2019 that was closed, not much insight into possible next steps to debug.

[nappernick]

At this point, here's what we've tried:
Assumption When run locally, the same set of commands completes execution of Adding source map references with the cursor at 100:9, which led us to think it may be an issue with the memory available to the GH Action - when there's less available memory, the "pages" of releases are split up by sentry/cli into smaller chunks, meaning more pages. Locally only 9 pages are needed as there is more available memory.

1. reduce the number of files - --ignore-file .sentryignore, which dropped the number of release files from 402 to 372
2. check that all step deps are present and accurate (directory build succeeded & is available)
3. increase available memory for the action with NODE_OPTIONS: "--max-old-space-size=8192"

[kamilogorek]

2.4.0 introduced release files deduplication via checksum comparison, for this we need to fetch all files first.
Could you try to use 2.3.1 instead, and see if the problem still persists? It'd clarify whether this exact step is a direct cause.

[nappernick]

That worked beautifully. Sourcemaps are now uploaded as expected @kamilogorek

[nappernick]

@kamilogorek specifying the version back a few patches is a good temporary solution, do you have any advice on a solution that will let us keep up with new releases?

[Zozman]

I'm also seeing a similar issue on cli tool version 2.5.2:

08:42:52  + sentry-cli --version
08:42:52  sentry-cli 2.5.2
[Pipeline] sh
08:42:53  + sentry-cli releases --org our-org --project our-project files project-1.0.0 upload-sourcemaps /ui/sourcemapBundle --url-prefix '~/build'
08:42:53  > Found 1 release file
08:42:53  > Analyzing 1 sources
08:42:53  > Rewriting sources
08:42:53  > Adding source map references
08:43:52  error: API request failed
08:43:52    caused by: sentry reported an error: Pagination offset too large (http status: 400) 

Is there any additional information or is our only choice to downgrade our sentry-cli version?

[kamilogorek]

2.6.0 will introduce --no-dedupe flag #1334 that will allow to skip over this issue while using never versions of the CLI. The issue itself however is still to be fixed "properly". Will work on that now.

[kamilogorek]

The problem here is that we have some limits on how many files we can query for. This is happening only if the number of release files is over 20k right now - https://github.com/getsentry/sentry/blob/fcc197755625c4a6223d243df1e8811252225593/src/sentry/constants.py#L298-L303
I'll see what we can do about that, but I was surprised that someone can have 20k files in one release in the first place 😅

[Zozman]

The problem here is that we have some limits on how many files we can query for. This is happening only if the number of release files is over 20k right now - https://github.com/getsentry/sentry/blob/fcc197755625c4a6223d243df1e8811252225593/src/sentry/constants.py#L298-L303 I'll see what we can do about that, but I was surprised that someone can have 20k files in one release in the first place 😅

Normally 20k in 1 release would be weird but in our case our UI releases aren't just 1 package, but are divided into microservices that can be updated independently of each other (we don't have a release 1 and release 2, but component 1 release 1, component 2 release 1, etc). This works for us because we can only update what's needed while letting the stuff that doesn't stay cached on the client, but doesn't exactly play nice with Sentry's release system since it expects 1 name for a release. As a result because you HAVE to have a release for uploading a source map, we just upload all updates under the same release but just update what URL the map corresponds to since we have a unique URL per update.

[kamilogorek]

Got it, makes sense. I'll try to come up with alternative idea for deduping functionality soon.