Allow UTF-8 with BOM for features.fea #3495
Conversation
| @@ -269,7 +269,7 @@ def make_lexer_(file_or_path): | |||
| fileobj, closing = file_or_path, False | |||
| else: | |||
| filename, closing = file_or_path, True | |||
| fileobj = open(filename, "r", encoding="utf-8") | |||
| fileobj = open(filename, "r", encoding="utf-8-sig") | |||
There was a problem hiding this comment.
what if the file is a regular utf-8 that does not not start with a BOM? Will this still work?
There was a problem hiding this comment.
Yes (as evident by the tests passing): this encoding just let Python skip the BOM mark if present in UTF-8 text, otherwise functionally it's the same as normal utf-8.
https://docs.python.org/3/library/codecs.html#encodings-and-unicode
On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file.
Do note that this is useful for opening text files (especially made by Windows Notepad). Saving as utf-8-sig is not recommended as it will add the BOM mark in which will break compatibility.
I would personally suggest read everything as utf-8-sig for greatest compatibility and save as utf-8 for standardisation.
There was a problem hiding this comment.
I would personally suggest read everything as utf-8-sig for greatest compatibility and save as utf-8 for standardisation.
SGTM.
There was a problem hiding this comment.
is this the only place where fontTools reads in human-written (potentially MS Notepad edited) text files? probably not. But sure let's merge this if it helps
There was a problem hiding this comment.
So far in the UFO building process, only features.fea had caused this problem. Other components are loaded in through plistlib which probably stripped out the BOM by default.
Fix googlefonts/fontmake#972