Windows 10 - RuntimeError: Event loop is closed and UnicodeEncodeError: 'charmap' codec can't encode character #3
Comments
Voyz
added
bug
|
Thanks for reporting and doing some initial investigation into the issue! To clarify: I've built the library on Windows 10, therefore Databay should hopefully be supported on that platform 😅 Let's try to figure out what is wrong here.
|
|
@Voyz That's good to know that you want or plan for databay to work in Windows since I work in a corporate Windows environment. Even though my company does support containerization, unfortunately, our IT only supports containerized Java or .Net Core applications for now.
Too bad my OOP and async knowledge is very weak. Would love to help contribute to something like this. I did look at aiohttp's github issue for hints, but activity for this issue has been pretty stale. But, I think this is the official bug report from Python dot org. Here is an issue from httpx project, person submitted a possible workaround to use the selector event loop, instead of proactor event loop. EDIT: Here is the workaround posted at httpx's repo: Not sure where to inject this code, I will see if can play around with this later and report back. |
|
@pybokeh Naturally, I certainly aim for Windows support and I'm very glad you reported this issue. I'm sure we can figure this one out 😊 Great job on investigating that issue - let me know if using the code you mentioned helps. The code you have could be placed in the Alternatively, the sources you linked seem to suggest replacing Out of curiosity, could you check if swapping |
|
Hey @Voyz by adding those snippets to my blockchain_demo.py: When executing, I no longer get an error, but nothing happens, the script may run maybe a second or two, then return control back to the terminal prompt. Then I went to my local link.py and replaced: then ran my blockchain_demo.py (but commented out the lines from workaround above). I then get the following error message: I reverted my link.py to its original state. Then modified my local config.py to: Then executed my blockchain_demo.py and then get no error message, but no output also, it just returns control back to the terminal after about 1 or 2 seconds. Finally, instead of using the APSPlanner, I used the SchedulerPlanner in my blockchain_demo.py. I get the same "RuntimeError: Event loop is closed()" traceback, but this time, the traceback is repeated continuously every second until I do CTRL+C several times. Here's also an odd observation, when replacing But what is odd is that the syntax warning happens infrequently or seldomly. |
|
Thank you for doing all these tests 🙌. I'm thoroughly confused by the behaviour you're experiencing - especially that I am unable to reproduce any of it. I've tried to run the examples with following permutations on my Python:
Using:
Running both:
To no avail - the code executes fine in all instances and I can't nail down how to cause the event loop issue, or how to make the control flow to return to terminal upon setting the policy manually like you describe. Bugger. I've read through the links you've left - very useful - thought couldn't find any more indication there. I'm guessing this may be issue related so some particular hardware or Windows version? Dunno, but worth giving it a shot:
Really sorry about you running into these issues, I'm truly surprised I can't reproduce any of these. Appreciate your input trying to fix it, as this may be true for many other users too 👍 |
|
Hey @Voyz , below is my console output that shows my workflow, start-to-finish (I'm using cmder mini as my Windows terminal, so after the lambda symbol is the start of the terminal where I would enter the terminal commands). I am using my company's laptop and not on VPN, connected to home's network, using plain, vanilla Python, not Anaconda or miniconda3, version 3.8.4, 64-bit. I use Python's built-in venv module to create virtual environments. I deleted my streaming_dev virtual environment folder that I've made previously and recreated it as in below to start from scatch: I am not using VM, just native OS and hardware. Yes, that is strange that everything is fine with your setup. Curious, if others are experiencing the same issue as me. Can't think of anything else to share at the moment that might explain for the difference. |
pybokeh
changed the title
|
@Voyz |
|
Thanks for doing all these tests. I work with CMDER too and also tested both that and native CMD - things are working well on both ends, but it's good you did that test. Also thanks for providing all the environment info - can't see anything out of the ordinary there for now. I've run a few tests on 3.8.4 using the exact same code as you did, with a fresh venv and still not no avail. However, I'm starting to suspect that the What's more, I can get some confirmation of this theory, by running the following snippet: import asyncio
import json
import aiohttp
url = 'https://blockchain.info/ticker'
async def task():
async with aiohttp.ClientSession() as session:
async with session.get('https://blockchain.info/ticker') as response:
payload = await response.read()
print(json.loads(payload))
asyncio.run(task())Each time when run on 3.8.4 venv, it prints once, then throws the Another easy way you could verify this, would be to put a My observation therefore is:
If that hypothesis is true, we should aim to understand why does your script exit instead of running continuously. As of current version, both APSPlanner and SchedulePlanner treat In case of APSPlanner, blocking is handled externally by APS, and should happen within the Another method I could think of would be to try to understand how soon that while loop stops (if it indeed does). You could do that by replacing the last lines of blockchain demo with the following: planner = APSPlanner(link)
# planner.start()
import threading
import time
th = threading.Thread(target=planner.start)
th.start()
while True:
print(planner.running, th.is_alive())
time.sleep(0.1)What happens when you run this? What is being printed and does it change at some point? Does that Finally, I'd suggest trying to run some examples from the examples folder. Majority of these don't use |
|
@Voyz This is the output in my console: This was executed on the plain, vanilla Python 3.8.4 environment. I will do the rest of the stuff that you suggest later on today. |
|
Great, that's clear the APS starts and is running for a short while before shutting down after approx. 2 seconds like you reported previously. It doesn't fail to start completely. Additionally, I just wanted to confirm there's nothing strangely weird with while loops on your end, so we know that isn't the case either. Something must shut the APS down. It's strange that this behaviour would happen also with SchedulePlanner, given they use different scheduling logic. Could it be that threading is the cause? Or the fact we use asyncio? I'll prep a branch for you with no asyncio and so we can eliminate it from the list. You can revert back to the previous version of your script - without that extra threading and while loop code - so that we don't miss any output otherwise hidden behind that thread. This snipped also seems to confirm my theory - the I'd add one more suggestion to the previous ones: add Thanks in advance for running the additional tests I described above. I understand this must be annoying, so just wanna say I really appreciate you helping me find the culprit of this issue. And sorry for some delay with responses - I have a feeling our timezones are way out of sync, though I try to reply as soon as I see your comments. |
|
@Voyz and I also added breakpoint() right before the line: Then executed the script using pdb: However, if I run VS Code's debugger, I get output in it's debug console terminal. I can see that the Here's the output from VS Code's debug terminal: So it looks like VS Code's debug console terminal supports UTF-8 or is in UTF-8 mode. But, if I execute the script in VS Code in normal mode (not debug mode), my script terminates due to the I tried to set cmder's encoding to support utf-8 via DOS's chcp command, but didn't work: |
|
@Voyz |
|
That's fantastic info! It's the ₹ (Indian rupee) character (\u20b9) that seems to break the print. The response from that URL indeed contains that character. I find it hard to understand why does it break the execution only in some cases, but you must be right - the debugging mode sets different UTF encoding. That's a really good lead for me to follow though, so cheers! Just to double check, could you give it a shot with this URL instead: Your further tests showcased that there is something wrong with exception logging in APSPlanner. We only started seeing this message thanks for re-enabling APS's logger. In its design, APSPlanner should handle APS's exceptions logging, which clearly wasn't happening. Even more so, when you enable full DEBUG logging on Databay the exceptions from APS are hidden and not logged. This is clearly my mistake and I'll find a way to fix it ASAP. Great job helping me uncover this oversight 👍 I feel we dig through a good deal of little problems in this issue case. For now, please run your code with:
and without:
Also, I'll again bring up my theory that your code doesn't stop working due to the import sys
import asyncio
if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())This will also clear up the error messages and make it easier to debug the unicode problem. |
|
Here seem to be some first clues for handling that Unicode error: https://stackoverflow.com/a/16120218/3508719 if sys.stdout.encoding != 'cp850':
sys.stdout = codecs.getwriter('cp850')(sys.stdout.buffer, 'strict')
if sys.stderr.encoding != 'cp850':
sys.stderr = codecs.getwriter('cp850')(sys.stderr.buffer, 'strict')and another one here: https://stackoverflow.com/questions/878972/windows-cmd-encoding-change-causes-python-crash/3259271 chcp 65001
set PYTHONIOENCODING=utf-8
python example.py |
|
@Voyz Well good news! Finally, got the blockchain_demo.py working. Added: to the top of my local copy of config.py and then at the terminal, executed: Then executed the blockchain_demo.py: EDIT: |
|
That's fantastic news!! I'm happy we managed to find the culprit (and a few others along the lines!). And most importantly I'm happy you can run that example finally 😊 Thanks for doing all these tests mate, that's been a massive help! I'll create a PR with the I'm also looking into fixing the exception logging that has failed you here and prevented you from seeing the right message straight away. I'll keep you posted on updates on these issues. In the meantime I'm modifying the title of this Issue so that it's easier to find info on our hunt for the Unicode error, in case others stumble upon it in the future too. Well done and thanks a lot! |
Voyz
changed the title
|
Sorry just realised you mentioned you'd like to contribute - would you like to introduce the PR with the |
|
@Voyz You're welcome! It was fun and rewarding for me to help in any way I can. As for the PYTHONIOENCODING thing, I have no idea. I think it has to do with whenever things are being echoed or printed to standard output? or what about to file? I do want to test your other examples where we are sending output to file. Thanks for asking me to do the PR. But, it's ok,, please do the PR yourself. I'm just glad to be able to help out. |
|
Either way I'm happy that it helped. It seems like it would break for a file too. I'm also guessing that this possibly was also breaking the exceptions - although that's just a blind guess - so it could be both stdout and stderr that are affected by it. From what I read on the print('₹')and with open('unicode_test_file.txt', 'a') as f:
f.write('₹')Both should break as far as I understand. I'll let you know if I figure out why your setup might have had that set incorrectly. And I'd love to hear how it goes with other examples for you, so keep me posted 😊 Naturally, I'll do the PR, no worries - you helped out big time. Thanks for all the contribution towards fixing this problem! |
|
Good news - I managed to get to the bottom of this. There was indeed a bug in exception logging of APSPlanner, triggered by exceptions with a strange signature such as that of UnicodeEncodeError. My bad. Got that verified and fixed now. Most importantly - I finally found a way to reproduce this error! And not without struggle - there's a million methods to fix it out there, just about none to reproduce it. Eventually, this is what was needed: or alternatively in code: Which would suggest that your machine had it set to Either way - happy I managed to reproduce what was happening on your end finally. Thanks for the patience once again! I'll update when the fixes are released. |
|
@Voyz I found out what my problem was. I had looked at my local environment variables before, but overlooked that there is a With no Python-related environment set and then executing |
|
Fantastic you managed to figure this one out! Thanks for following up with the explanation - that explains everything. I nonetheless added a warning should someone have that encoding set incorrectly, as this may cause unexpected behaviour without being very verbose. |
asyncio Event loop policy, UnicodeEncodeError and planners' exception handling (fixes #3)
Not a bug, but perhaps it should be mentioned that databay will not work with Windows 10 due to a bug or the way asyncio chooses the default event loop mechanism, which throws an error, instead of just a warning.
So if running asyncio-dependent code on Windows machine, you will get an error similar below:
Per this SO question, possible workaround is to use asyncio loop's
run_until_complete()instead of justrun()(Update 1 by Voy - added more info)
To Reproduce
Example code (blockchain example):
Environment
Databay version: 0.1.2
Python version: 3.8.2
OS: Ubuntu 20.04 and Windows 10
(Update 2 by Voy - explaining the true issue)
The
RuntimeError: Event loop is closedis only half of the problem here, and in fact is not the cause for the code to stop working, but a side effect happening when program terminates and can be fixed easily. The code wouldn't execute due toUnicodeEncodeError, which we figure out later down the line.The text was updated successfully, but these errors were encountered: