Encode str bodies to Latin-1 instead of UTF-8

[pquentin]

Relates #3053 by fixing the immediate regression. We may introduce warnings in the future to help users not rely on this behavior. The changes are split in 3 commits (but squashing is fine if we merge this):

1. I've changed the default encoding of simple bodies from UTF-8 to Latin-1, and made the code more explicit and tested
2. Even though the formal name of the encoding is ISO-8859-1, we used latin-1 in enough places that I preferred to be consistent and updated the remaining occurrences of iso-8859-1 to latin-1.
3. Chunked encoding already used Latin-1, but I've changed the test to ensure it stays that way in the future.

[pquentin]

This is actually suspicious because 1.26.x used utf-8 here:

urllib3/test/with_dummyserver/test_chunked_transfer.py

Lines 53 to 75 in 3c01480

	def _test_body(self, data):
	self.start_chunked_handler()
	with HTTPConnectionPool(self.host, self.port, retries=False) as pool:
	pool.urlopen("GET", "/", data, chunked=True)
	header, body = self.buffer.split(b"\r\n\r\n", 1)
	assert b"Transfer-Encoding: chunked" in header.split(b"\r\n")
	if data:
	bdata = data if isinstance(data, bytes) else data.encode("utf-8")
	assert b"\r\n" + bdata + b"\r\n" in body
	assert body.endswith(b"\r\n0\r\n\r\n")
	len_str = body.split(b"\r\n", 1)[0]
	stated_len = int(len_str, 16)
	assert stated_len == len(bdata)
	else:
	assert body == b"0\r\n\r\n"
	def test_bytestring_body(self):
	self._test_body(b"thisshouldbeonechunk\r\nasdf")
	def test_unicode_body(self):
	self._test_body(u"thisshouldbeonechunk\r\näöüß")

[sethmlarson]

Weird, but there was no test case for non-ASCII characters?

[illia-v]

There are a few non-ASCII characters on line 75

[illia-v]

And changing data.encode("utf-8") to data.encode("latin-1") breaks the test in 1.26.x

test_unicode_body failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert ((b'\r\n' + b'thisshouldbeonechunk\r\n\xe4\xf6\xfc\xdf') + b'\r\n') in b'1e\r\nthisshouldbeonechunk\r\n\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f\r\n0\r\n\r\n'

[illia-v]

Related encoding happens on this line in 1.26.x

urllib3/src/urllib3/connection.py

Line 274 in 57181d6

chunk = chunk.encode("utf8")

[pquentin]

Thanks, good find! So we were encoding chunks with UTF-8 and other bodies with Latin-1, and 2.0 reversed that unintentionally. I'll fix that too.

[sethmlarson]

Adding my two cents, if it's encoding as UTF-8 and the world isn't broken we may want to call this a "feature" and encode every body as UTF-8. That's what the world is using nowadays anyways, so we're likely to be causing more issues by using latin-1 instead?

[pquentin]

I like the idea, but note that chunks are encoded as UTF-8 in 1.26.x but encoded as Latin-1 in 2.x. So we should only fix that part: encoding chunks as UTF-8?

[sethmlarson]

@pquentin Oh gotcha, yeah let's UTF-8 all the things!