Deprecated TLS negotiation in URLLIB3 causing issues

[flooie]

In reviewing a number of scrapers today for problem sets, I noticed that there was a pattern for when many started showing. That pattern was around December 5th.

After investigating further it became clear that three court websites for atleast four courts still use TLSv1.2 - see below.

- https://www.la-fcca.org
- * SSL connection using TLSv1.2 / AES128-SHA

- https://www.jud.ct.gov"
- * SSL connection using TLSv1.2 / AES256-SHA256
- Warning: Ignores instruction to use SSLv3

- https://oag.ca.gov
- * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

urllib3 - which underlies requests deprecated the negotiations for older TLS standards when it switched to urllib3 > 2.0.

Four Louisiana Court of Appeals, the Supreme and Appellate courts of Connecticut we can resolve this by keeping urllib3 at the 1.26.18

For whatever reason - I havent found a simple solution for the California AG website. But writing a custom http adapter with a "legacy server connect" solves California AG. See below.

class CustomHttpAdapter (requests.adapters.HTTPAdapter):
    # "Transport adapter" that allows us to use custom ssl_context.

    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = urllib3.poolmanager.PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_context=self.ssl_context)


def get_legacy_session():
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    ctx.options |= 0x4  # OP_LEGACY_SERVER_CONNECT
    session = requests.session()
    session.mount('https://', CustomHttpAdapter(ctx))
    return session

Regardless - calag, conn, connctapp and lactapp_1 all have a path to resolve them.

I have a feeling that this change was triggered in Missouri appellate courts - and the errors built up and caused the courts to block us. Which needs a call to reach out.

[mlissner]

Some details would help? What's the issue, which court, stacktrace?

[flooie]

Read more about these changes here

Here

[sentry-io]

Sentry issue: COURTLISTENER-65H

[sentry-io]

Sentry issue: COURTLISTENER-64R

[sentry-io]

Sentry issue: COURTLISTENER-64T

[sentry-io]

Sentry issue: COURTLISTENER-6C1