Unable to do http calls to punycode-encoded emoji domains

[twisted-trac]

	@Sorunome reported
Trac ID	trac#10078
Type	defect
Created	2021-01-08 13:32:13Z

Despite being able to register punycode-encoded emoji domains on some TLDs (e.g. .ws), twisted is unable to make HTTP calls to those. (e.g. xn--9s9hnf.ws).

Just to clarify: I am not asking for emoji-urls, but punycode-encoded urls. Twisted currently claims that xn--9s9hnf.ws was an invalid hostname, and thus does not do any request there.

This is because twisted.web.client uses the python package idna over here to check if a domain is valid. Editing that to use text.encode("idna") and text.decode("idna") fixes this issue.

Said idna package has an issue here about not supporting emoji domains, and from the sounds of it they do not plan to add it.

As it is possible to just register such domains, and their DNS records etc. can just be queried, it should be expected that twisted can also make http calls to those.

Searchable metadata

trac-id__10078 10078
type__defect defect
reporter__Sorunome Sorunome
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__new new
resolution__None None
component__web web
keywords__None None
time__1610112733707088 1610112733707088
changetime__1610369981124200 1610369981124200
version__None None
owner__None None
cc__clokep

[john-parton]

@Sorunome Would be nice to have a minimal example. I'm not super familiar with Twisted. I could try and include it in my PR.

[Sorunome]

@Sorunome Would be nice to have a minimal example. I'm not super familiar with Twisted. I could try and include it in my PR.

Just any http call to an emoji domain would do

[john-parton]

I'll see what I can do. I have never written any Twisted code. My experience with Twisted is almost exclusively through scrapy.