ValueError: cmap format 4 subtable overflowed; sort glyph order by unicode to fix.

[TakWolf]

https://github.com/fonttools/fonttools/blob/7fce3444046485846118c1eda86d4ed0a541a94c/Lib/fontTools/ttLib/tables/_c_m_a_p.py#L1029

    header = struct.pack(
             ^^^^^^^^^^^^
struct.error: 'H' format requires 0 <= number <= 65535

During handling of the above exception, another exception occurred:

https://github.com/fonttools/fonttools/blob/7fce3444046485846118c1eda86d4ed0a541a94c/Lib/fontTools/fontBuilder.py#L415

ValueError: cmap format 4 subtable overflowed; sort glyph order by unicode to fix.

What does this mean? What should I sort? Should I sort glyphOrder: list[str] as asc code_points? Why this need a sequence?

I found an article：https://zhuanlan.zhihu.com/p/20100737
The point is: https://learn.microsoft.com/zh-cn/typography/opentype/spec/cmap#windows-platform-platform-id--3

Fonts that support only Unicode BMP characters (U+0000 to U+FFFF) on the Windows platform must use encoding 1 with a format 4 subtable. This encoding must not be used to support Unicode supplementary-plane characters.

Fonts that support Unicode supplementary-plane characters (U+10000 to U+10FFFF) on the Windows platform must use encoding 10 with a format 12 subtable.

Is it related to this？

Because if I set this:

builder.setupCharacterMap(character_mapping, allowFallback=True)

Compile without error, but the fonts will not support windows: (Not a valid font file)

By the way, in my case, all cmap code_point are < 0xFFFF. I don't understand the essence of the problem right now.

[justvanrossum]

Format 4 cmap uses some sort of segmented format, which works best if the glyph order is somewhat sorted by unicode. It will allow to compactly store runs of sequential code points that map to sequential glyph IDs.

Unfortunately, there are cases with large cmaps and "unsorted" glyph IDs where this principle apparently degenerates (I'm not even sure of the details) and an overflow occurs. Having more of the glyph order sorted by unicode generally helps.

[justvanrossum]

Btw. the overflow does not mean there's a bug in fonttools but rather that this particular cmap/glyph order combination can not be represented with a format 4 cmap.

[TakWolf]

If I sort glyphOrder as code_point sequence, how many glyphs can be stored in theory? Is 65535 (Because 0x0000 ~ 0xFFFF)?

Now I have 43678 glyphs without sort, It is overflowed.

[behdad]

Does this patch help?

diff --git a/Lib/fontTools/ttLib/tables/_c_m_a_p.py b/Lib/fontTools/ttLib/tables/_c_m_a_p.py
index 6c00aaf63..6ecdf3807 100644
--- a/Lib/fontTools/ttLib/tables/_c_m_a_p.py
+++ b/Lib/fontTools/ttLib/tables/_c_m_a_p.py
@@ -1029,7 +1029,7 @@ class cmap_format_4(CmapSubtable):
         header = struct.pack(
             cmap_format_4_format,
             self.format,
-            length,
+            min(length, 0xFFFF),
             self.language,
             segCountX2,
             searchRange,

[TakWolf]

Amazing, it's effective!

I still haven't understood the principle. Does this mean that sorting is not necessary? (When conditions: all code points < 0xFFFF and glyphs length < 0xFFFF)

[behdad]

It's a hack that seems to work. I've seen other fonts do the same.

You shouldn't need sorting anymore.

@justvanrossum should we commit this?

[justvanrossum]

Not sure. Does ots accept it? Should the spec be adjusted? Does fonttools round trip this?

[behdad]

Good questions...

FWIW the kern table has similar issue and we already accept such fonts (calibri.ttf for example). Looks like ots-sanitize drops the kern table of it though, but claims success. @khaledhosny

[behdad]

In this code:

        format = 4 
        subTable_3_1 = buildCmapSubTable(cmapping_3_1, format, 3, 1) 
        try: 
            subTable_3_1.compile(self.font) 
        except struct.error: 
            # format 4 overflowed, fall back to format 12 
            if not allowFallback: 
                raise ValueError( 
                    "cmap format 4 subtable overflowed; sort glyph order by unicode to fix." 
                ) 
            format = 12 
            subTable_3_1 = buildCmapSubTable(cmapping_3_1, format, 3, 1)

I think the latter should have plat/enc of 3,10; not 3,1. No? That might be why Windows doesn't accept the font. @justvanrossum

[TakWolf]

This can just build 8px to accelerate the build speed.

def main():
    fs_util.delete_dir(path_define.build_dir)

    update_service.setup_ark_pixel_glyphs()

    for font_config in configs.font_configs:
        # Just build 8px
        if font_config.size != 8:
            continue

Here can block out some unnecessary formats.

https://github.com/TakWolf/fusion-pixel-font/blob/master/services/font_service.py#L289

If there is any other issues, please let me know at any time.

[behdad]

Humm. When I ttx dump the length field of the cmap4 in those fonts, I don't see 65535...

[TakWolf]

Yes, currently there are about 48000 glyphs. Do you need more than 65535 glyphs? Or code points > 65535?

It can simulate this situation through a simple mapping.

[behdad]

No this should be fine. I'm just surprised that none of the cmap tables seems to be 65535, which is the number my patch would have put in it. I'll debug later.

[behdad]

Ah okay I think I need to look into the ttc...

[behdad]

Neither fonttools decompile nor ots-sanitize like this cmap. Maybe we abort the idea.

@justvanrossum can you check my suggestion in #3230 (reply in thread) ? I can send a PR.

[TakWolf]

Does it mean that there is no way to solve this problem?
I found that min(length, 0xFFFF) not really solving the problem. This just truncates the data. The generated data is still not compliant. (Although fonts can be compiled)

[justvanrossum]

The immediate way to likely solve the problem is to sort your glyph order by unicode, as has been suggested from the start.

Elswhere in this discussion, Behdad suggested some fixes for the on-overflow fallback mechanism in FontBuilder that may solve this without reordering the glyph order. But this requires implementing and testing those suggestions.

[TakWolf]

Sorting is possible for single fonts. But for OTC, there are some shared glyphs with multiple code points mapping. They can't determine the order. In different child font, they have different orders. This is the difficult point.

[justvanrossum]

It doesn't need to be completely sorted, but sorting what you can may already make the difference. I'm not sure why it would matter that different child fonts would have different orders. Are you saing all fonts in the OTC have the same glyph order and should keep the same glyph order? If that's the case, you could try computing a single glyph order based on the cmap of one of them, and apply that new order to all, and see if it doesn't overflow.

[justvanrossum]

can you check my suggestion in #3230 (reply in thread) ? I can send a PR.

You may be correct there, if I read the cmap recommendations:

• If the font supports only characters in the Unicode Basic Multilingual Plane (U+0000 to U+FFFF): either platform 3, encoding 1; or platform 0, encoding 3. With either encoding, use a format 4 subtable.

• If the font supports any characters in a Unicode supplementary plane (U+10000 to U+10FFFF): either platform 3, encoding 10; or platform 0, encoding 4. With either encoding, use a format 12 subtable.