ImageFont and non latin characters in a path to font

[Jenyay]

What did you do?

>>> from PIL.ImageFont import truetype
>>> import os.path
>>> fname = 'C:\\temp\\Шрифты\\font.ttf'
>>> os.path.exists(fname)
True
>>> truetype(fname)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\jenyay\.virtualenvs\outwiker-lFR0QK22\lib\site-packages\PIL\ImageFont.py", line 261, in truetype
    return FreeTypeFont(font, size, index, encoding, layout_engine)
  File "C:\Users\jenyay\.virtualenvs\outwiker-lFR0QK22\lib\site-packages\PIL\ImageFont.py", line 144, in __init__
    self.font = core.getfont(font, size, index, encoding, layout_engine=layout_engine)
OSError: cannot open resource

If fname contains only Latin characters, then everything works.

What versions of Pillow and Python are you using?

Pillow 5.0.1
Windows 7, Windows 10.
In Linux is OK.

[radarhere]

Can't reproduce on MacOS either.

[hugovk]

Info from duplicate #3480:

Reproducible with Chinese characters with:

• OS: Windows 10 Pro 1803
• Python: 3.6.5
• Pillow: 5.3.0

But not in CentOS.

[hugovk]

Looks like FreeType's FT_New_Face doesn't support Unicode paths on Windows.

---

core.getfont is getfont in _imagingft.c (making it hard to debug on Windows).

getfont can raise if error is set, and they're all return values from FT_ calls: FT_New_Face, FT_New_Memory_Face, FT_Set_Pixel_Sizes, etc.

Pillow/src/_imagingft.c

Lines 268 to 302 in 080bfd3

	if (filename && font_bytes_size <= 0) {
	self->font_bytes = NULL;
	error = FT_New_Face(library, filename, index, &self->face);
	} else {
	/* need to have allocated storage for font_bytes for the life of the object.*/
	/* Don't free this before FT_Done_Face */
	self->font_bytes = PyMem_Malloc(font_bytes_size);
	if (!self->font_bytes) {
	error = 65; // Out of Memory in Freetype.
	}
	if (!error) {
	memcpy(self->font_bytes, font_bytes, (size_t)font_bytes_size);
	error = FT_New_Memory_Face(library, (FT_Byte*)self->font_bytes,
	font_bytes_size, index, &self->face);
	}
	}
	if (!error)
	error = FT_Set_Pixel_Sizes(self->face, 0, size);
	if (!error && encoding && strlen((char*) encoding) == 4) {
	FT_Encoding encoding_tag = FT_MAKE_TAG(
	encoding[0], encoding[1], encoding[2], encoding[3]
	);
	error = FT_Select_Charmap(self->face, encoding_tag);
	}
	if (filename)
	PyMem_Free(filename);
	if (error) {
	if (self->font_bytes) {
	PyMem_Free(self->font_bytes);
	}
	Py_DECREF(self);
	return geterror(error);

And "cannot open resource" is a FreeType error:

  FT_ERRORDEF_( Cannot_Open_Resource,                        0x01,
                "cannot open resource" )

https://github.com/aseprite/freetype2/blob/ae3afbc47184db5c7753318b36237af261c66b09/include/freetype/fterrdef.h#L61-L62

FT_New_Face is most likely, as it passes in the filename.

---

From https://stackoverflow.com/a/10075289/724176:

Since freetype is meant to be portable, it only uses fopen, which can only accept const char* filenames. So, either load file to memory (or memory map it) and then use FT_New_Memory_Face to create font or convert wchar_t pszFileName into 8-bit encoding, potentially losing characters due to impossible conversion.

On linux, you could attempt to use setlocale so fopen will accept UTF8 strings, convert wchar_t string to UTF8. However on windows it won't work. So either load file to memory, or convert pszFileName to 8-bit encoding, then pass it to FT_New_Face.

Both of these are similar issues and contain suggestions to load the file and instead pass it to FT_New_Memory_Face

• https://freetype.nongnu.narkive.com/XIf73fcO/ft-ft-new-face
• Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) Robmaister/SharpFont#89

---

And Pillow already does something like this. ImageFont.truetype's font parameter:

font – A filename or file-like object containing a TrueType font. Under Windows, if the file is not found in this filename, the loader also looks in Windows fonts/ directory.

If a path is passed in, it uses FT_New_Face. If bytes, it uses FT_New_Memory_Face.

---

I've not tested this on Windows (and it still works on Mac), please try something like this:

from PIL import ImageFont
from io import BytesIO

FONT_PATH = "方正黄草简体.ttf"

def open_filename(filename):
    font = ImageFont.truetype(filename)
    return font

def open_data(filename):
    with open(filename, "rb") as f:
        font_bytes = BytesIO(f.read())
    font = ImageFont.truetype(font_bytes)
    return font

# print(open_filename(FONT_PATH))
print(open_data(FONT_PATH))

Does that help?

One of the threads above suggests not to use FT_New_Memory_Face by default, as it loads the whole font into RAM rather than a single font, so we wouldn't want to do that in all cases.

However, perhaps it could be done on Windows when the filename contains Unicode/non-ASCII.

[jdhao]

@hugovk , it works on my Windows machine. Thanks~

[radarhere]

I've created PR #3785 to resolve this.