Parsing FlatBuffers with Python in 22.9.24 is broken

[stewartmiles]

Python's code generator was broken by #7529 (FYI @joshua-smith8).

Given the table Foo, when generating code for the object API FooT now skips flatbuffer file headers.

For example, before pull/7529, tests/monster_test_generated.py:

flatbuffers/tests/monster_test_generated.py

Lines 177 to 181 in 2ad4086

	@classmethod
	def InitFromBuf(cls, buf, pos):
	monster = Monster()
	monster.Init(buf, pos)
	return cls.InitFromObj(monster)

After:

flatbuffers/tests/monster_test_generated.py

Lines 178 to 183 in c92e78a

	@classmethod
	def InitFromBuf(cls, buf, pos):
	n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, 0)
	monster = Monster()
	monster.Init(buf, pos+n)
	return cls.InitFromObj(monster)

This is cool as FooT.InitFromBuf() interprets the buf argument in the same way as Foo.GetRootAs(), i.e it assumes buf points at the start of a packed flatbuffer file and skips the header. However, this has some issues as:

• It breaks the existing users of the object API who were skipping the header themselves. (I doubt there were many, e.g we were doing foo = FooT.InitFromObj(Foo.GetRootAs(buf)).
• InitFromObj is used to parse nested union tables which is broken as they end up skipping the header block:

flatbuffers/tests/monster_test_generated.py

Lines 38 to 48 in c92e78a

	def AnyCreator(unionType, table):
	from flatbuffers.table import Table
	if not isinstance(table, Table):
	return None
	if unionType == Any().Monster:
	return MonsterT.InitFromBuf(table.Bytes, table.Pos)
	if unionType == Any().TestSimpleTableWithEnum:
	return TestSimpleTableWithEnumT.InitFromBuf(table.Bytes, table.Pos)
	if unionType == Any().MyGame_Example2_Monster:
	return MonsterT.InitFromBuf(table.Bytes, table.Pos)
	return None

Since FooT.InitFromBuf() is now better? since it aligns with GetRootAs() perhaps the solution is to extend InitFromBuf() to support parsing nested tables?

[stewartmiles]

Oh also, smells like the tests for nested tables needs to be fixed to deal catch this case.

[stewartmiles]

Hey folks, 22.9.24 is still the latest on pypi and it's broken. Any chance someone (@dbaileychess @rw @aardappel) could build and publish a release?

[dbaileychess]

@stewartmiles Sorry for the delay, I just merged the fix.

I am trying to automate publishing to our package managers (including Pypi) on releases, do you mind if I take a bit to set this up before releasing again with this fix in?

[stewartmiles]

@dbaileychess no worries, we're all good we just pinned to the previous release of FlatBuffers. I'm primarily concerned about newcomers to the project who will pull the latest from pypi and have a broken experience. Is it possible to publish the previous version as a new version manually to at least unbreak head on pypi?

[stewartmiles]

If it's a pain, I get it .. just folks will likely have a bad experience and not join the cult of "Flat" :)

[dbaileychess]

Should be up now: https://pypi.org/project/flatbuffers/22.10.26/

[stewartmiles]

Sweet thanks @dbaileychess 😄