RepackerError and Traceback from uharfbuzz

[bobh0303]

Doesn't happen on recent releases, but with latest master I get:

WARNING: hb.repack failed to serialize 'GPOS', attempting fonttools resolutions ; the error message was: RepackerError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 191, in tryPackingHarfbuzz
    return writer.getAllDataUsingHarfbuzz(self.tableTag)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 673, in getAllDataUsingHarfbuzz
    return hb.repack_with_tag(str(tableTag), data, obj_list)
  File "src/uharfbuzz/_harfbuzz.pyx", line 1566, in uharfbuzz._harfbuzz.repack_with_tag
uharfbuzz._harfbuzz.RepackerError

followed by two further exceptions.

To duplicate, download ttf and fea from https://www.dropbox.com/sh/s38r0y2t2bjkg1o/AACKWYG8H_D_0iJ9P7Q0HEbOa?dl=0 and try

fonttools feaLib -o out.ttf Harmattan-Medium.fea Harmattan-Medium.ttf

[behdad]

cc @garretrieger

[n7s]

Just to be specific about the versions: fontTools 4.41.1 as well as 4.42.0 (installed with Harfbuzz 8.1.1 and uharfbuzz 0.37.2) don't produce the error so it's likely a change introduced since these releases. Thanks!

Also we haven't figured out how to set the USE_HARFBUZZ_REPACKER option to either value at build or run time. Until a fix is available, slow (the HB packer rather than the python fallback) is OK with us if it doesn't break a build. Any pointers?

[n7s]

the other exceptions are:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 443, in getData
    items[i] = packUShort(item.subWriter.pos - pos)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 869, in packUShort
    return struct.pack(">H", value)
struct.error: 'H' format requires 0 <= number <= 65535

and

Traceback (most recent call last):
  File "/usr/local/bin/fonttools", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/fontTools/__main__.py", line 31, in main
    runpy.run_module(mod, run_name="__main__")
  File "/usr/lib/python3.10/runpy.py", line 227, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/feaLib/__main__.py", line 78, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/fontTools/feaLib/__main__.py", line 74, in main
    font.save(output_font)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/ttFont.py", line 212, in save
    writer_reordersTables = self._save(tmp)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/ttFont.py", line 259, in _save
    self._writeTable(tag, writer, done, tableCache)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/ttFont.py", line 710, in _writeTable
    tabledata = self.getTableData(tag)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/ttFont.py", line 732, in getTableData
    return self.tables[tag].compile(self)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 157, in compile
    return self.tryPackingHarfbuzz(writer, hb_first_error_logged)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 208, in tryPackingHarfbuzz
    return writer.getAllData(remove_duplicate=False)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 701, in getAllData
    tableData = table.getData()
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 446, in getData
    overflowErrorRecord = self.getOverflowErrorRecord(item)
  File "/usr/local/lib/python3.10/dist-packages/fontTools/ttLib/tables/otBase.py", line 803, in getOverflowErrorRecord
    LookupListIndex = item.repeatIndex
AttributeError: 'OffsetToWriter' object has no attribute 'repeatIndex'

[anthrotype]

AttributeError: 'OffsetToWriter' object has no attribute 'repeatIndex'

this in particular looks like a regression from #3241, I'll look into this

[anthrotype]

we haven't figured out how to set the USE_HARFBUZZ_REPACKER option to either value at build or run time

basically one has to set that option in the TTFont.cfg dict before compiling with TTFont.save, however one problem might be that you do not have a TTFont instance yet because a new one is created in ufo2ft.
In fact I have a pending PR to allow setting these fonttools-related config options as ufo2ft compile parameters so that clients of ufo2ft like fontmake can pass them on: googlefonts/ufo2ft#607
I'll try to finish this soon

[anthrotype]

another option you have to avoid using hb-repacker with fonttools is making sure uharfbuzz is not installed in the environment you are building the font...

[anthrotype]

I fixed the AttributeError exception that was caused by changes in #3241, but I can keep this one open in case we want to investigate the reason for the harfbuzz RepackerError.
Note, the way hb-repacker is integrated in fonttools works such that, if uharfbuzz is available to import, an attempt is made to serialize GSUB/GPOS with it, if that fails for any reason, we fall back to the pure-python compilation. In this case, the latter was also failing with the AttributeError noted above.

[bobh0303]

Thanks! Confirmed we're again able to build, though with the fallback warning.

[garretrieger]

I'll look into the repacking failure on the harfbuzz side.

[garretrieger]

Reproduced and got logs from the repacker attempt. It fails trying to repacking the 16bit space (non extension lookups) so I suspect that our extension promotion logic might be miscounting and leaving too much stuff in the 16 bit space.

[garretrieger]

Ok tracked this down and found an approach to fix it:

• The issue is that there are so many extension subtables that the links from the lookup to the extension subtable is overflowing.
• Fortunately though I found that many of the extension subtables point to the same subtable and thus can be deduped. The current harfbuzz extension promotion mechanism doesn't attempt to dedup extension subtables when it's adding new ones.
• I'm working on a change to add the deduping logic which gets this case to pass harfbuzz repacker on the first attempt.

[garretrieger]

Fix just landed in harfbuzz.

[garretrieger]

Filed issue in uharfbuzz to get a release which includes the fix: harfbuzz/uharfbuzz#176

[garretrieger]

uharfbuzz has now pulled the fix into the latest release (v0.37.3), see: harfbuzz/uharfbuzz#176 (comment)