Force the InVEST python application to run in UTF-8 mode #1268
Conversation
I'm hoping that this will allow us to capture more information about what's going on. RE:natcap#1167
…o bugfix/1167-gdal-systemerror
Also adding CPL_DEBUG (copied from GDAL logging example) to increase verbosity of the gdal logging, and mirroring the log levels that GDAL has. RE:natcap#1167
The user is re-running the model on the same workspace and while the error message is consistent, I'm not getting it in the same point of execution in the model. RE:natcap#1167
There is a possibility that the issue here is UTF-16 strings being provided to GDAL. RE:natcap#1167
…167-gdal-systemerror
Now we don't return if there are an invalid number of parameters, just log what we know and put in an obvious placeholder on what we don't know. RE:natcap#1167
It turns out this can be configured via an environment variable. RE:natcap#1167
There was a problem hiding this comment.
Neat, I like that the GDAL debug setting is now exposed in the workbench! The translated string looks correct, we don't need to do anything else with it. I just had one minor code style suggestion. Also looks like there are a couple of files in the diff that are unrelated to this issue?
| import hashlib | ||
| import inspect | ||
| import itertools | ||
| import logging | ||
| import os |
There was a problem hiding this comment.
Looks like there aren't any functional changes to this file?
There was a problem hiding this comment.
Whoops! Yes, these were left over from a few changes I made to pollinaton, and isort sorted the imports. Unpatched in 0565160
There was a problem hiding this comment.
Hi @phargogh , thanks for explaining all this in so much detail. I think this is a nice solution!
I'm not sure if the changes in utils.py are causing this somehow, but test_utils.py seems like it never properly exits. I re-ran the jobs once and they all seem to be hanging again.
| onChange={this.handleChange} | ||
| > | ||
| <option value="NORMAL" key="NORMAL">NORMAL</option> | ||
| <option value="DEBUG" key="DEBUG">DEBUG</option> |
There was a problem hiding this comment.
We should wrap the displayed strings to get translated <...>{t('NORMAL')}</...>
| "Skipping.") | ||
| return | ||
| except Exception as e: | ||
| LOGGER.exception(str(e)) |
There was a problem hiding this comment.
How do we get here, some improperly typed args or kwargs? If so, do we want to proceed to with logging, or should we return/raise here?
There was a problem hiding this comment.
It is super unclear because I'm having a tough time reproducing it on my system, but the user I'm working with on the forums was seeing something in len raising a UnicodeDecodeError. And because of the nature of this error, I'm having a really hard time actually logging the object itself, or even the type! It's very bizarre.
I added a note about this in c883dc1 and am now returning from the logging function. You're right, if we're seeing this same error, we probably won't be able to do much about the rest of the logging.
This will fix the tests hanging. RE:natcap#1167
Refactor of how we create the arguments for spawning the InVEST process for clarity and brevity. RE:natcap#1167 Co-authored-by: Emily Soth <43770515+emlys@users.noreply.github.com>
…vest into bugfix/1167-gdal-systemerror
|
@davemfish @emlys thank you both for your review and good comments! I have addressed them and fixed the builds hanging issue, but I'm still trying to understand why the user's getting the error she's getting. So I'll re-request your reviews once we've figured that out. |
Mamba isn't able to resolve our builds with python 3.11. RE:natcap#1167
This PR addresses issues arising from #1167, in which a cryptic
UnicodeDecodeErrorwould present in some users computers but we were unable to reproduce it locally.My best understanding about the problem is that there is a discrepancy in the user's locale and the user's filesystem encoding, which, on Windows, would cause python to use the locale's encoding as the filesystem encoding unless we specify otherwise. This would ultimately cause problems when filepaths made it down to GDAL, where paths must be UTF-8. Although I cannot pinpoint precisely which GDAL function is failing, I strongly suspect that it is this difference between expected and real encodings that is causing the error.
This is only an issue on Windows. Mac and Linux are unaffected by the bug, as these are both POSIX systems.
As a solution, I have the workbench forcing the python application to start up in UTF-8 mode on Windows. This does have a side effect of causing text files to be opened as UTF-8 files by default, but this should not be a problem for InVEST because we generally tell people to use UTF-8 anyways. UTF-8 mode is scheduled to become the default in another 4 years, with the release of python 3.15.
While debugging this, I also discovered how to enable GDAL's debug logging, so I have added a workbench setting to enable this in case it's useful in the future. Note that to make full use of this logging, the user will also need to lower the logging threshold to DEBUG, which is then passed to the python application as an environment variable.
@emlys since the GDAL setting adds a new string to translate, is there anything else I will need to do to make sure it is ready for future translation?
@davemfish @emlys I took a stab at implementing this new setting, but please let me know if there is a better way to do any of this!
Fixes #1167
Checklist