How to upload using the RECAP API?

[mlissner]

Hi, I'm a legal tech organization that has a lot of PACER data. How do I share it with you using your APIs?

[mlissner]

The APIs are invite only, so if you are reading this and have not reached out to us via email already, please start there. After being approved as an uploader, we'll give your account upload access and you'll be able to send us content.

Initial Caveats

1. It's important that we have dockets and attachment pages before documents. If you just send us a PDF, but you do not send us the docket or attachment page the PDF is associated with, we won't know what case to associate with the PDF and we'll label the PDF as an "orphan document." Orphan documents are not public in any of our systems, and are simply set aside in a holding area until we learn what case they're a part of. Sometimes this means that when you upload a docket, it'll suddenly have a bunch of PDFs associated with it. Those were orphans, but now their parents (the docket sheet or attachment page) have been found.

2. The fourth digit of every PACER document ID is a boolean. It is always either zero or one. You need to normalize it to a zero before you upload. If you don't, it could be saved in our DB as a duplicate. Yes, we should do this normalization on our end, but we don't yet, because all uploaders are considered trusted and our resources are limited.

3. We assume docket sheets are coming from PACER and are therefore up to date. Therefore, if you upload old docket information, it could wipe out newer stuff in the system. To avoid this, always check if something is already in our system before uploading or always upload fresh content.

4. Our court IDs match the subdomains of PACER, with a few limited exceptions where we didn't do a great job (sorry, too late now). This code should help you convert from PACER subdomains to our IDs:

   pacer_to_cl_ids = {
       # Maps PACER ids to their CL equivalents
       "azb": "arb",  # Arizona Bankruptcy Court
       "cofc": "uscfc",  # Court of Federal Claims
       "neb": "nebraskab",  # Nebraska Bankruptcy
       "nysb-mega": "nysb",  # Remove the mega thing
   }
   
   # Reverse dict of pacer_to_cl_ids
   cl_to_pacer_ids = {v: k for k, v in pacer_to_cl_ids.items() if v != "nysb"}
   
   def map_pacer_to_cl_id(pacer_id):
       return pacer_to_cl_ids.get(pacer_id, pacer_id)
   
   def map_cl_to_pacer_id(cl_id):
       return cl_to_pacer_ids.get(cl_id, cl_id) 

The Set Up

With those caveats out of the way, let's get into the details. Let's start by downloading some HTML and a PDF from PACER so we have something to work with. This uses Juriscraper, our tool for scraping data.

Use poetry or your favorite env thing to get set up and install Juriscraper:

11:24:01::mlissner@rhyolite::~/Programming/discussion3178 
↪ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [discussion3178]:  
Version [0.1.0]:  
Description []:  
Author [mlissner <mlissner@xxx.com>, n to skip]:  n
License []:  
Compatible Python versions [^3.11]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] 
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): juriscraper
Found 1 packages matching juriscraper

Enter package # to add, or the complete package name if it is not listed []:
 [ 0] juriscraper
 [ 1] 
 > 0
Enter the version constraint to require (or leave blank to use the latest version): 
Using version ^2.5.52 for juriscraper

Add a package (leave blank to skip): 

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "discussion3178"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
juriscraper = "^2.5.52"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] 
11:24:34::mlissner@rhyolite::~/Programming/discussion3178 
↪ poetry install
Creating virtualenv discussion3178-EKTbvQ-O-py3.11 in /home/mlissner/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/c6/5e/45c907597575094440f089050b3f17df578c00b15e57420b6e8397794c2c/geonamescache-1.6.0-py3-nResolving dependencies... Downloading https://files.pythonhosted.org/packages/c6/5e/45c907597575094440f089050b3f17df578c00b15e57420b6e8397794c2c/geonamescache-1.6.0-py3-nResolving dependencies... Downloading https://files.pythonhosted.org/packages/c6/5e/45c907597575094440f089050b3f17df578c00b15e57420b6e8397794c2c/geonamescache-1.6.0-py3-nResolving dependencies... Downloading https://files.pythonhosted.org/packages/c6/5e/45c907597575094440f089050b3f17df578c00b15e57420b6e8397794c2c/geonamescache-1.6.0-py3-nResolving dependencies... Downloading https://files.pythonhosted.org/packages/5d/43/42780ffb72d13a369facb5e6ee669f10d3dc8987ad7a3313b0d1dec46aed/lxml-4.9.3-cp27-cp27m-macResolving dependencies... (6.5s)

Package operations: 35 installs, 0 updates, 0 removals

  • Installing attrs (23.1.0)
  • Installing certifi (2023.7.22)
  • Installing charset-normalizer (3.2.0)
  • Installing h11 (0.14.0)
  • Installing idna (3.4)
  • Installing outcome (1.2.0)
  • Installing sniffio (1.3.0)
  • Installing sortedcontainers (2.4.0)
  • Installing urllib3 (2.0.5)
  • Installing exceptiongroup (1.1.3)
  • Installing pysocks (1.7.1)
  • Installing requests (2.31.0)
  • Installing six (1.16.0)
  • Installing trio (0.22.2)
  • Installing wsproto (1.2.0)
  • Installing filelock (3.12.4)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2023.3.post1)
  • Installing regex (2023.8.8)
  • Installing requests-file (1.5.1)
  • Installing sgmllib3k (1.0.0)
  • Installing trio-websocket (0.10.4)
  • Installing tzlocal (5.0.1)
  • Installing webencodings (0.5.1)
  • Installing argparse (1.4.0)
  • Installing chardet (5.1.0)
  • Installing cssselect (1.2.0)
  • Installing dateparser (1.1.8)
  • Installing feedparser (6.0.10)
  • Installing geonamescache (1.6.0)
  • Installing html5lib (1.1)
  • Installing lxml (4.9.3)
  • Installing selenium (4.12.0)
  • Installing tldextract (3.6.0)
  • Installing juriscraper (2.5.52)

Writing lock file

You should now be able to use Juriscraper. Check if you like:

(discussion3178-py3.11) 11:25:43::mlissner@rhyolite::~/Programming/discussion3178 
↪ ipython
In [1]: from juriscraper.pacer import DocketReport

A Random Case We'll Use

For this exercise we'll use this randomly-selected bankruptcy case:

Universal Solar America, LLC, 2:23-bk-06491, (Bankr. D. Ariz.)

It has the following metadata:

• CourtListener's court id: arb
• PACER's court id (using code above): azb
• pacer_case_id: 807215
• CourtListener URL: https://www.courtlistener.com/docket/67806364/universal-solar-america-llc/
• CourtListener docket ID (from the URL): 67806364

It currently looks like this in CourtListener:

Getting Some Data

Using Juriscraper, we can get a case's full HTML by logging in, and then querying it:

In [1]: from juriscraper.pacer import DocketReport
In [2]: from juriscraper.pacer import PacerSession
In [3]: s = PacerSession(username='mlissner.flp', password='xxx!')
In [4]: s.login()
In [5]: r = DocketReport('azb', s)
In [6]: r.query(807215)  # The PACER case ID
In [7]: with open('docket.html', 'w') as f:
   ...:     f.write(r.response.text)

We now have docket.html with the contents from PACER. Great!

Let's download a PDF too. Let's do doc number 2, with pacer_doc_id of 024052806385

In [10]: response, status_msg = r.download_pdf(807215, '024052806385')

In [11]: response.content[0:50]
Out[11]: b'%PDF-1.6\n%\xe2\xe3\xcf\xd3\n33 0 obj\n<</Filter/FlateDecode/Leng'

In [12]: with open('3.pdf', 'wb') as f:
    ...:     f.write(response.content)

We now have a PDF called 3.pdf that we can open. Lovely.

Uploading Content (Finally)

Here's the process for uploading content:

1. Begin with docket HTML. If you don't have the original HTML from PACER, there's still hope; open a new discussion to resolve that issue.

   Assuming you do have the original HTML, what you should do is check if we have the docket. As explained above, this is important to prevent your old data from overwriting what might be our newer data. Here's how to check:

   ↪ curl --silent 'https://www.courtlistener.com/api/rest/v3/dockets/?pacer_case_id=807215&court=arb&fields=date_modified' | jq
   {
     "count": 1,
     "next": null,
     "previous": null,
     "results": [
       {
         "date_modified": "2023-09-22T15:03:40.615638-07:00"
       }
     ]
   }
   

   That uses the docket API to look up the docket, and filter just to its modification date, which shows that the modification date was yesterday. We just got the latest HTML now, so we know ours is newer and we can upload our content. Great. To upload our HTML, use the instructions in the RECAP API. Do:

   ↪ curl    \
       --form upload_type=1\
       --form "filepath_local=@docket.html"\
       --form court=arb\
       --form pacer_case_id=807215\
       'https://www.courtlistener.com/api/rest/v3/recap/'\
       --header 'Authorization: Token xxx' | jq
   
   {
     "id": 10637960,
     "court": "arb",
     "docket": null,
     "docket_entry": null,
     "recap_document": null,
     "date_created": "2023-09-23T11:58:45.787275-07:00",
     "date_modified": "2023-09-23T11:58:45.787288-07:00",
     "pacer_case_id": "807215",
     "pacer_doc_id": "",
     "document_number": null,
     "attachment_number": null,
     "status": 1,
     "upload_type": 1,
     "error_message": "",
     "debug": false
   }
   

   The JSON response you see there is an processing queue item that shows that your item was processed (there's no error_message, and status is 1).

   Now if you refresh the page, you see:

   

   Great!

   Now, if you found that your HTML was too old, there's something else you could do. You could get more granular, and check if the docket entry data was available in our system. To do that, you'd use the docket-entries API.

   I don't have the docket-entries endpoint memorized like I do the docket endpoint, so let's inspect it using an HTTP OPTIONS request:

   ↪ curl --silent -X OPTIONS 'https://www.courtlistener.com/api/rest/v3/docket-entries/' --header 'Authorization: Token xxx' | jq '.filters'
   {
     "id": {
       "type": "NumberFilter",
       "lookup_types": [
         "exact"
       ]
     },
     "entry_number": {
       "type": "BooleanFilter",
       "lookup_types": [
         "exact",
         "gte",
         "gt",
         "lte",
         "lt",
         "range",
         "isnull"
       ]
     },
     "date_created": {
       "type": "NumberFilter",
       "lookup_types": [
         "exact",
         "gte",
         "gt",
         "lte",
         "lt",
         "range",
         "year",
         "month",
         "day",
         "hour",
         "minute",
         "second"
       ]
     },
     "date_modified": {
       "type": "NumberFilter",
       "lookup_types": [
         "exact",
         "gte",
         "gt",
         "lte",
         "lt",
         "range",
         "year",
         "month",
         "day",
         "hour",
         "minute",
         "second"
       ]
     },
     "date_filed": {
       "type": "NumberFilter",
       "lookup_types": [
         "exact",
         "gte",
         "gt",
         "lte",
         "lt",
         "range",
         "year",
         "month",
         "day"
       ]
     },
     "pacer_sequence_number": {
       "type": "BooleanFilter",
       "lookup_types": [
         "exact",
         "gte",
         "gt",
         "lte",
         "lt",
         "range",
         "isnull"
       ]
     },
     "docket": {
       "type": "RelatedFilter",
       "lookup_types": "See available filters for 'Dockets'"
     },
     "recap_documents": {
       "type": "RelatedFilter",
       "lookup_types": "See available filters for 'Recap Documents'"
     },
     "tags": {
       "type": "RelatedFilter",
       "lookup_types": "See available filters for 'Tags'"
     }
   }
   

   It looks like I can filter on docket and entry_number, so I'll use those. Recall that the CL docket ID from this case is 67806364, and we're interested in entry number three. So:

   ↪ curl --silent 'https://www.courtlistener.com/api/rest/v3/docket-entries/?docket=67806364&entry_number=3' --header 'Authorization: Token xxx' | jq
   {
     "count": 1,
     "next": null,
     "previous": null,
     "results": [
       {
         "resource_uri": "https://www.courtlistener.com/api/rest/v3/docket-entries/365196629/",
         "id": 365196629,
         "docket": "https://www.courtlistener.com/api/rest/v3/dockets/67806364/",
         "recap_documents": [
           {
             "resource_uri": "https://www.courtlistener.com/api/rest/v3/recap-documents/372094780/",
             "id": 372094780,
             "tags": [],
             "absolute_url": "/docket/67806364/3/universal-solar-america-llc/",
             "date_created": "2023-09-18T16:03:25.428929-07:00",
             "date_modified": "2023-09-23T12:21:40.921614-07:00",
             "sha1": "",
             "page_count": null,
             "file_size": null,
             "filepath_local": null,
             "filepath_ia": "",
             "ia_upload_failure_count": null,
             "thumbnail": null,
             "thumbnail_status": 0,
             "plain_text": "",
             "ocr_status": null,
             "date_upload": null,
             "document_number": "3",
             "attachment_number": null,
             "pacer_doc_id": "024052806385",
             "is_available": false,
             "is_free_on_pacer": null,
             "is_sealed": null,
             "document_type": 1,
             "description": "Appearance (Notice of)"
           }
         ],
         "date_created": "2023-09-18T16:03:25.419671-07:00",
         "date_modified": "2023-09-23T12:21:40.908647-07:00",
         "date_filed": "2023-09-18",
         "time_filed": "15:39:02",
         "entry_number": 3,
         "recap_sequence_number": "2023-09-18.001",
         "pacer_sequence_number": 13,
         "description": "Notice of Appearance & Request Notice filed by EDWARD K. BERNATAVICIUS of UNITED STATES TRUSTEE on behalf of U.S. TRUSTEE.(BERNATAVICIUS, EDWARD) (Entered: 09/18/2023)",
         "tags": []
       }
     ]
   }
   

   That's great. It's got the entry, and it's fairly new, so in this case we won't do more, but if you discovered that entry two was missing, you could make a fake docket HTML file that just had that entry, and upload it as above. You just have to make your HTML close enough to PACER's that our parser will work.

2. So now we have the docket and the docket entry. Let's upload the PDF. Again, we follow the examples in the documentation. Thus:

   curl  \
       --form upload_type=3 \
       --form "filepath_local=@3.pdf \
       --form court=arb \
       --form pacer_case_id=807215 \
       --form pacer_doc_id=024052806385 \
       --form document_number=3 \
       'https://www.courtlistener.com/api/rest/v3/recap/'\
       --header 'Authorization: Token xxx' | jq
   {
     "id": 10638041,
     "court": "arb",
     "docket": null,
     "docket_entry": null,
     "recap_document": null,
     "date_created": "2023-09-23T12:30:37.562832-07:00",
     "date_modified": "2023-09-23T12:30:37.562848-07:00",
     "pacer_case_id": "807215",
     "pacer_doc_id": "024052806385",
     "document_number": 3,
     "attachment_number": null,
     "status": 1,
     "upload_type": 3,
     "error_message": "",
     "debug": false
   }
   

   Again, no error messages, so we're in business. A few things to note though:

   • The fourth digit of the pacer_doc_id is normalized to zero.
   • The court ID is normalized to arb instead of azb.
   • The upload_type is 3.

   If we refresh the page again, we get:

   

Conclusion

Lessons:

• Big picture, it's about uploading fresh HTML before doing PDFs. You can make fake dockets to refine the data you send.
• There are a number of other types of HTML you can upload, like attachment pages, or history reports. These are in the documentation or the API itself can be inspected via HTTP OPTIONS requests.
• The API can be filtered by field or value.
• There are three important API endpoints for you: dockets, docket-entries, and recap.

Thanks for contributing to RECAP!

Addendum Q&As

What about PACER attachments?

Do to these, you'll need to upload the docket, then the attachment page. Then you'll be able to upload the attachment itself by including the attachment_number parameter when uploading the PDF. The API docs have more details on this.

[gbronner]

Any chance that the restricted apis (recap, attorney) can respond to OPTIONS requests without being specially provisioned?

[mlissner]

We'd have to up update the code to do so, I fear, but I'd welcome a PR for that.