How can I tackle deep pagination?

[mlissner]

I'm trying to gather a bunch of data from the API, but when I get to page 100, it starts failing. What's up with that? Is there a workaround?

[mlissner]

Unfortunately, deep pagination is a well known problem in the database world. The issue is that the deeper you go, the larger the result set the database needs to prepare, and when the result set gets large, performance tanks.

Our general advice for this is to use a parameter of some kind to slice the data.

Most (if not all) of our APIs allow you to filter on the id parameter. Therefore, the easy workaround is to:

1. Order by ID.
2. Starting at page one, paginate to page 100.
3. Get the last ID on page 100.
4. Add a filter for everything __gt that value and return to step 2. This filter would probably look something like &id__gt=XYZ.

This should work well for all of our DB-based APIs, but for search results, it might be a bit trickier. Still, the solution is the same:

1. Order by a field.
2. Start at page one, and paginate to page 100.
3. Get the value of that field on page 100.
4. Add a filter for everything greater than that value and return to step 2.

Filters in step four and ordering will be different than the DB-based APIs, but it's the same idea. If folks have examples of doing this in the wild, please add them to this discussion!

[mathchou]

I'm trying to do this with the docket-entries endpoint, but I see that for filters there is only:

"id": {
"type": "NumberFilter",
"lookup_types": [
"exact"
]

Is there any way a greater than filter could be added to this? Or is there a work-around in this case?

[mlissner]

You're right, this should be better. I just wrote some code to broaden a number of ID fields in our API:

#3745

Tests are running now. When they complete, it will be automatically merged into CL's code. From there, it is automatically deployed to our servers. All of that usually completes in about 15-20 minutes.

I hope this helps!

[mathchou]

Thank you!