[cffLib] Question on moving Private from topDict to FontDict

[NightFurySL2001]

I have copied this code: #147, and now I want to move Private from topDict into a FDArray/FontDict as directly converting it to CID-key makes the font invalid. Example: before

<CFF>
    <CFFFont name="..">
        ...
        <Private>...</Private>
        <CharStrings>...</CharStrings>
    </CFFFont>
...
</CFF>

after:

<CFF>
    <CFFFont name="..">
        ...
        <FDSelect format="3"/>
        <FDArray>
            <FontDict index="0">
                <Private>...</Private>
            </FontDict>
        <CharStrings>...</CharStrings>
    </CFFFont>
...
</CFF>

How can I achieve this in Python?

[NightFurySL2001]

I have read the source files here:

fonttools/Lib/fontTools/varLib/cff.py

Line 64 in 21d22ae

if not hasattr(topDict, "FDArray"):

, and managed to edit the following code that parsed successfully (also should be noted that CharString in CharStrings need to add an attribute fdSelectIndex="0" for CID-keyed fonts):

import copy
from fontTools import ttLib
from fontTools.cffLib import (
	FDArrayIndex,
	FontDict,
    FDSelect,
)

font = ttLib.TTFont("fonts/in.otf")

cff = font['CFF '].cff.values()[0]
fontName = font["CFF "].cff.keys()[0]

cff.ROS = ('Adobe','Identity',0)
mapping = {name:("cid"+str(n) if n else ".notdef") for n,name in enumerate(cff.charset)}
charstrings = cff.CharStrings
charstrings.charStrings = {mapping[name]:v for name,v in charstrings.charStrings.items()}
cff.charset = ["cid"+str(n) if n else ".notdef" for n in range(len(cff.charset))]


#sample from https://github.com/fonttools/fonttools/blob/21d22ae30f88b76fab5575f5f2e882b142ee1c08/Lib/fontTools/varLib/cff.py#L64
if not hasattr(cff, "FDArray"):
    #format 3 FDSelect
    cff.FDSelect = FDSelect(None, None, 3)

    #create FDArray
    fdArray = cff.FDArray = FDArrayIndex()
    fdArray.strings = None
    fdArray.GlobalSubrs = cff.GlobalSubrs
    cff.GlobalSubrs.fdArray = fdArray
    if charstrings.charStringsAreIndexed:
        charstrings.charStringsIndex.fdArray = fdArray
    else:
        charstrings.fdArray = fdArray

    #set FDSelectIndex = 0 for every T2CharString in GlobalSubrs
    for item in charstrings.keys():
        charstrings[item].fdSelectIndex = 0
    
    #create FontDict in FDArray
    fontDict = FontDict() #default index 0
    fdArray.append(fontDict)
    #generate fontdict name
    fontDict.FontName = fontName +"-All"
    fontDict.FontMatrix = cff.FontMatrix
    
    #add Private from CFF topDict into FontDict
    fontDict.Private = copy.deepcopy(cff.Private)

    #remove Private from CFF topDict
    #cff.Private = None
    del cff.Private

font.save("fonts/test.otf", reorderTables=False)

The last remaining question is, how do I delete cff.Private (Private in TopDict object)? del cff.Private is not doing anything.

[justvanrossum]

I've used something like this in the past (adapted from some internal code, untested):

from fontTools.ttLib import TTFont
from fontTools import cffLib


def convertCFFToCIDKeyed(inPath, outPath):
    ttFont = TTFont(inPath)
    glyphOrder = ttFont.glyphOrder
    fontName = ttFont["CFF "].cff.keys()[0]
    topDict = ttFont["CFF "].cff.values()[0]
    assert topDict.Encoding == "StandardEncoding"
    assert topDict.charset == glyphOrder

    topDict.ROS = ("Adobe", "Identity", 0)  # ???
    fontMatrix = topDict.FontMatrix
    fontDict = cffLib.FontDict()
    fontDict.FontName = fontName
    fontDict.FontMatrix = fontMatrix
    fontDict.Private = topDict.Private
    del topDict.Private
    topDict.FDArray = cffLib.FDArrayIndex()
    topDict.FDArray.append(fontDict)

    topDict.FDSelect = cffLib.FDSelect()
    topDict.FDSelect.format = None
    topDict.FDSelect.gidArray = [0] * len(glyphOrder)
    topDict.CIDCount = len(glyphOrder)

    # We'll temporarily monkeypatch cffLib's getCIDfromName() so we can
    # provide our own glyph name -> CID mapping. (For now: GID == CID.)
    revGlyphOrder = {gn: gid for gid, gn in enumerate(glyphOrder)}
    def getCIDfromName(name, strings):
        return revGlyphOrder[name]
    original_getCIDfromName = cffLib.getCIDfromName
    try:
        cffLib.getCIDfromName = getCIDfromName
        ttFont.save(outPath)
    finally:
        cffLib.getCIDfromName = original_getCIDfromName

[NightFurySL2001]

What are the possible reasons that topDict.Private is still present (hasattr(topDict, "Private") == True) even after copy.deepcopy(topDict.Private) then del topDict.Private?

[justvanrossum]

Found it, and also why it did work in the context where my code came from.

There is magic __getattr__ behavior that lazily reads from a lower level data structure if an attr isn't found. The del xxx.Private doesn't delete the underlying data, so accessing the Private attribute happily rebuilds it.

The workaround is to do this:

del cff.Private
del cff.rawDict["Private"]

We should proably open an issue for this.

(The reason it worked for me in the original context of my code is because it was a font built from scratch, not read form disk, so rawData doesn't contain anything.)

[justvanrossum]

For reference:

fonttools/Lib/fontTools/cffLib/__init__.py

Lines 2571 to 2586 in 21d22ae

	def __getattr__(self, name):
	if name[:2] == name[-2:] == "__":
	# to make deepcopy() and pickle.load() work, we need to signal with
	# AttributeError that dunder methods like '__deepcopy__' or '__getstate__'
	# aren't implemented. For more details, see:
	# https://github.com/fonttools/fonttools/pull/1488
	raise AttributeError(name)
	value = self.rawDict.get(name, None)
	if value is None:
	value = self.defaults.get(name)
	if value is None:
	raise AttributeError(name)
	conv = self.converters[name]
	value = conv.read(self, value)
	setattr(self, name, value)
	return value

[justvanrossum]

I think replacing:

value = self.rawDict.get(name, None)

with:

value = self.rawDict.pop(name, None)

fixes the problem.

[NightFurySL2001]

Maybe we should call help from nearby related issue #1488? @madig @anthrotype

[NightFurySL2001]

My final code:

import copy
from fontTools import ttLib
from fontTools.cffLib import (
	FDArrayIndex,
	FontDict,
    FDSelect,
)

font = ttLib.TTFont("fonts/in.otf")
glyphOrder = font.getGlyphOrder()

cff = font['CFF '].cff.values()[0]
fontName = font["CFF "].cff.keys()[0]

cff.ROS = ('Adobe','Identity',0)
cff.Copyright = cff.Copyright.replace("\n", " ")
mapping = {name:("cid"+str(n) if n else ".notdef") for n,name in enumerate(cff.charset)}
charstrings = cff.CharStrings
charstrings.charStrings = {mapping[name]:v for name,v in charstrings.charStrings.items()}
cff.charset = ["cid"+str(n) if n else ".notdef" for n in range(len(cff.charset))]


#sample from https://github.com/fonttools/fonttools/blob/21d22ae30f88b76fab5575f5f2e882b142ee1c08/Lib/fontTools/varLib/cff.py#L64
if not hasattr(cff, "FDArray"):
    #format 3 FDSelect
    cff.FDSelect = FDSelect(None, None, 3)

    #create FDArray
    fdArray = cff.FDArray = FDArrayIndex()
    fdArray.strings = None
    fdArray.GlobalSubrs = cff.GlobalSubrs
    cff.GlobalSubrs.fdArray = fdArray
    if charstrings.charStringsAreIndexed:
        charstrings.charStringsIndex.fdArray = fdArray
    else:
        charstrings.fdArray = fdArray

    #set FDSelectIndex = 0 for every T2CharString in GlobalSubrs
    for item in charstrings.keys():
        charstrings[item].fdSelectIndex = 0
    
    #create FontDict in FDArray
    fontDict = FontDict() #default index 0
    fdArray.append(fontDict)
    #generate fontdict name
    fontDict.FontName = fontName +"-All"
    
    #add Private from CFF topDict into FontDict
    fontDict.Private = copy.deepcopy(cff.Private)

    #remove Private from CFF topDict
    #cff.Private = None
    del cff.Private
    del cff.rawDict["Private"]

font.save("fonts/test.otf", reorderTables=False)

The only problem left is that the font saved shows up as "not a valid font" on Windows, but using ttx to convert it to XML and then convert the XML back to a binary font file generated a valid font. Attached is an invalid font on Windows generated by the program above.
test.zip

[justvanrossum]

There is a difference in the CFF tables, but not one that TTX can show :( I'm trying to find what it is.

[justvanrossum]

One difference seems to be in topDict.Encoding: the valid font has "StandardEncoding" as its value (which is the default), but the invalid font has a list of 256 '.notdef' entries. Perhaps the TTX roundtrip action strips that field.

I would try to delete it in your script, both with regular del, and with rawDict:

    del cff.Encoding
    del cff.rawDict["Encoding"]

[NightFurySL2001]

I see... It seems weird that the first line of del cff.Encoding gives an attribute error:

Traceback (most recent call last):
  File "D:\Desktop\font-project\test_cff_cid_key.py", line 62, in <module>
    del cff.Encoding
AttributeError: Encoding

but printing cff.Encoding then delete it works; or just del cff.rawDict["Encoding"] without del cff.Encoding works too.

The font file is generated using Google's fontmake btw.

[justvanrossum]

FWIW, here's the bit of magic that makes the TTX roundtrip fix the problem:

fonttools/Lib/fontTools/cffLib/__init__.py

Lines 2681 to 2682 in 1b9731d

	if hasattr(self, "ROS"):
	self.skipNames = ['Encoding']

[NightFurySL2001]

After a bit of digging, I found this info in the CFF specs:

Encoding data is located via the offset operand to the Encoding operator in the Top DICT. Only one Encoding operator can be specified per font except for CIDFonts which specify no encoding. A glyph's encoding is specified by a 1-byte code that permits values in the range 0-255.

As the original font is name-keyed, it has specified an Encoding operator which is fine, but converting this to CID-keyed will require the removal of topDict.Encoding. I have posted the complete code below.

[justvanrossum]

Btw. I have opened #2725 based on this discussion.

[NightFurySL2001]

import copy
from fontTools import ttLib
from fontTools.cffLib import (
    FDArrayIndex,
    FontDict,
    FDSelect,
)

font = ttLib.TTFont("fonts/in.otf")
glyphOrder = font.getGlyphOrder()

cff = font["CFF "].cff.values()[0]
fontName = font["CFF "].cff.keys()[0]

cff.ROS = ("Adobe", "Identity", 0)

# delete encoding when converting to CID-keyed
del cff.rawDict["Encoding"]
mapping = {
    name: ("cid" + str(n) if n else ".notdef") for n, name in enumerate(cff.charset)
}
charstrings = cff.CharStrings
charstrings.charStrings = {
    mapping[name]: v for name, v in charstrings.charStrings.items()
}
cff.charset = ["cid" + str(n) if n else ".notdef" for n in range(len(cff.charset))]


# sample from https://github.com/fonttools/fonttools/blob/21d22ae30f88b76fab5575f5f2e882b142ee1c08/Lib/fontTools/varLib/cff.py#L64
if not hasattr(cff, "FDArray"):
    # format 3 FDSelect
    cff.FDSelect = FDSelect(None, None, 3)

    # create FDArray
    fdArray = cff.FDArray = FDArrayIndex()
    fdArray.strings = None
    fdArray.GlobalSubrs = cff.GlobalSubrs
    cff.GlobalSubrs.fdArray = fdArray
    if charstrings.charStringsAreIndexed:
        charstrings.charStringsIndex.fdArray = fdArray
    else:
        charstrings.fdArray = fdArray

    # set FDSelectIndex = 0 for every T2CharString in GlobalSubrs
    for item in charstrings.keys():
        charstrings[item].fdSelectIndex = 0

    # create FontDict in FDArray
    fontDict = FontDict()  # default index 0
    fdArray.append(fontDict)
    # generate fontdict name
    fontDict.FontName = fontName + "-All"

    # add Private from CFF topDict into FontDict
    fontDict.Private = copy.deepcopy(cff.Private)

    # remove Private from CFF topDict
    # cff.Private = None
    del cff.Private
    del cff.rawDict["Private"] #remove raw data to prevent rebuilding, see https://github.com/fonttools/fonttools/issues/2725

font.save("fonts/out.otf", reorderTables=False)