building reproducible tarballs

[bollwyvl]

Over on ipython, @Carreau has been using the retar script to post-process sdist tarballs to be SOURCE_DATE_EPOCH-aware.

As it has no dependencies, so if all the licensing is copacetic, what about adopting that behavior to complement the whl-based ones in flit?

Noted on jupyterhub/team-compass#502 (comment)

[takluyver]

It should be more or less reproducible anyway - we're using SOURCE_DATE_EPOCH and normalising file ownership and permissions when creating the sdist. But I don't think it's particularly easy to test this automatically (because you want to check that the results are the same across things like different platforms), so there may well be inconsistencies that have crept in. Fixes welcome!

flit/flit_core/flit_core/sdist.py

Lines 18 to 34 in 048c87c

	def clean_tarinfo(ti, mtime=None):
	"""Clean metadata from a TarInfo object to make it more reproducible.
	- Set uid & gid to 0
	- Set uname and gname to ""
	- Normalise permissions to 644 or 755
	- Set mtime if not None
	"""
	ti = copy(ti)
	ti.uid = 0
	ti.gid = 0
	ti.uname = ''
	ti.gname = ''
	ti.mode = common.normalize_file_permissions(ti.mode)
	if mtime is not None:
	ti.mtime = mtime
	return ti

flit/flit_core/flit_core/sdist.py

Lines 167 to 168 in 048c87c

	source_date_epoch = os.environ.get('SOURCE_DATE_EPOCH', '')
	mtime = int(source_date_epoch) if source_date_epoch else None

[gitpushdashf]

Looks like with SOURCE_DATE_EPOCH set, the tarball and wheel are both reproducible. And they match whether flit build or python -m build are used. Very nice!

[bollwyvl]

Very nice indeed, thanks for looking into it.

So perhaps all that's needed is a note about that, e.g.

Wheels built by flit are reproducible... wheels (which are zip files) include the modification...

amended to

_Wheels and source distributions built by flit are reproducible... wheels (which are .zip archives) and source distributions (which are tar.gz archives) include the modification _

Though have a re-build step on a different os/container might be interesting. I have found that windows has... problems.

[takluyver]

They won't be reliably reproducible between flit build and python -m build, because the former uses information from git or hg to decide what files to include, while the latter doesn't do that. There's more discussion about that discrepancy as part of #522.

[pradyunsg]

Note that you use one of those specifically (either build or flit), the distributions generated will be reproducible.

[pradyunsg]

Given that source tarballs built by flit are reproducible already, is there anything actionable here?

Update: yes, a documentation update. :)

[pradyunsg]

Ah, nvm me, I need to read things more carefully. 😅

[nanonyme]

I thought the main source of OS reproducibility issues was undefined file emitting order from directories which you just have to mitigate by sorting your input files by filenames.

[takluyver]

Yup, and we should be ensuring things are sorted, e.g.:

flit/flit_core/flit_core/common.py

Lines 87 to 94 in 3f1ed8b

	# Ensure we sort all files and directories so the order is stable
	for dirpath, dirs, files in os.walk(str(self.path)):
	for file in sorted(files):
	full_path = os.path.join(dirpath, file)
	if _include(full_path):
	yield full_path
	dirs[:] = [d for d in sorted(dirs) if _include(d)]

flit/flit_core/flit_core/common.py

Lines 442 to 447 in 3f1ed8b

	for dirpath, dirs, files in os.walk(data_directory):
	for file in sorted(files):
	full_path = os.path.join(dirpath, file)
	yield full_path
	dirs[:] = [d for d in sorted(dirs) if d != '__pycache__']