This repository has been archived by the owner on Jan 3, 2024. It is now read-only.
Unable to get content of page from special characters (utf-8) URL #733
Comments
|
For now moved to using SeleniumBase for proxy functionality - no such issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hey,
I am currently in progress of switching from main Selenium to Selenium Wire to start using proxies for scraping activities.
Now everything works fine and as expected, except websites that have special characters in their addresses, like
https://männimetsa.ee/. Here I have actually two issues.page_sourceresults in empty content<html><head></head><body></body></html>. No matter what I do, if I implement waiting or not, this is what I get. No actual Exception in my try catch block is thrown. Just emptybody.request_interceptorI get this error:UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 1: ordinal not in range(128). Even if inside interceptor I just print out the request.path, its the error that gets thrown in my face. And printing request.path results in this text:https://männimetsa.ee/where thehttps://mpart is as active URL.Not sure if issue 1 is related to issue 2, but both issues only happen with special character website URLs. So I have a feeling that both issues are caused by the same main reason.
If I switch back to Selenium, I can get the content of website with no problems.
The text was updated successfully, but these errors were encountered: