SOURCE_DATE_EPOCH 0 can't be represented in zip timestamps

[Apteryks]

Hello,

I wanted to package flit-core for Guix, but even flit-core requires flit to be built, according to pyproject.toml. Is there an older release of flit-core that could be built simply using setuptools or another way? That'd allow us to have a clear bootstrapping path to flit.

Ideally, perhaps flit-core could be made to always build with something else than itself to fix this bootstrapping problem.

Thanks for your consideration!

[takluyver]

flit_core can build itself, with nothing besides Python! See PR #441, where I'm adding a document specifically on this topic. 😁

[Apteryks]

Hey, that's great news! I tried it and stumbled upon:

ValueError: ZIP does not support timestamps before 1980

I presume this is because the date is set to Epoch 0, for reproducibility reasons. I tried setting all files mtime to ~1980, but in vain (unpacking the sdist, fixing up the mtimes and repacking it didn't help also). Strange.

[takluyver]

If the SOURCE_DATE_EPOCH environment variable is set when flit_core builds a wheel, it will use that for all the timestamps in the zip file, overriding any timestamps from the filesystem:

https://github.com/takluyver/flit/blob/9335bfbf23bf79b05b30c23e909c073a477e7f65/flit_core/flit_core/wheel.py#L50-L53

I wasn't aware of the 1980 limitation. Is it possible to set SOURCE_DATE_EPOCH to something that works with zip files? If not, we can limit it to use 1980 if the supplied time is before then.

[RonnyPfannschmidt]

@takluyver its a common practice in repeatable build distros like nix/guix to set the EPOCH to 0 as a default,

that commonly creates issues with zip, auto-updating to a zip supported minimum is strongly recommended

[takluyver]

OK, we can do that. I assume 1980-1-1 00:00:00 is the timestamp we should use in that case.

[takluyver]

Take a look at PR #448. 🙂

(I've taken the liberty of updating the title of this issue to be specifically about handling SOURCE_DATE_EPOCH=0)

[Apteryks]

Hi! Thanks for the update. It seems to work now:

(define-public python-flit-core
  ;; Use the latest commit of the 'zip-date-min-1980' branch, as it includes
  ;; necessary fixes to the yet unreleased bootstrapping procedure defined in
  ;; the flit_core/build_dists.py script.
  (let ((revision "0")
        (commit "47c1b48b307f4c89fe9178b1fca2282950e213cb")
        (version* "3.3.0"))
    (package
      (name "python-flit-core")
      (version (git-version version* revision commit))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/takluyver/flit")
                      (commit commit)))
                (file-name (git-file-name name version))
                (sha256
                 (base32
                  "01vg6ir0gla5zb8539rxn8zmczly3d6qh2mcfpjkihxqvqlrs6gr"))))
      (build-system python-build-system)
      (propagated-inputs
       `(("python-toml" ,python-toml)))
      (arguments
       ;; flit-core has a test suite, but it requires Pytest.  Disable it so
       ;; as to not pull pytest as an input.
       `(#:tests? #f
         #:phases
         (modify-phases %standard-phases
           (replace 'build
             ;; flit-core requires itself to build.  Luckily, a
             ;; bootstrapping script exists, which does so using just
             ;; the checkout sources and Python.
             (lambda _
               (invoke "python" "flit_core/build_dists.py")))
           (replace 'install
             (lambda* (#:key inputs outputs #:allow-other-keys)
               (let ((out (assoc-ref outputs "out"))
                     (site (site-packages inputs outputs))
                     (whl (car (find-files "." "\\.whl$"))))
                 (mkdir-p site)
                 (invoke "unzip" "-d" site whl "-x" "flit_core/tests/*")
                 ;; Byte compile the installed source files.
                 (invoke "python" "-m" "compileall"
                         "--invalidation-mode=unchecked-hash"
                         site)))))))
      (native-inputs
       `(("unzip" ,unzip)))
      (home-page "https://github.com/takluyver/flit")
      (synopsis "Core package of the Flit Python build system")
      (description "This package provides @code{flit-core}, a PEP 517 build
backend for packages using Flit.  The only public interface is the API
specified by PEP 517, @code{flit_core.buildapi}.")
      (license license:bsd-3))))

Produces:

$ tree /gnu/store/p9pn8ynaa6fyf7b9aka8ar8yarfzv11h-python-flit-core-3.3.0-0.47c1b48
/gnu/store/p9pn8ynaa6fyf7b9aka8ar8yarfzv11h-python-flit-core-3.3.0-0.47c1b48
├── lib
│   └── python3.8
│       └── site-packages
│           ├── flit_core
│           │   ├── buildapi.py
│           │   ├── build_thyself.py
│           │   ├── common.py
│           │   ├── config.py
│           │   ├── __init__.py
│           │   ├── __pycache__
│           │   │   ├── buildapi.cpython-38.pyc
│           │   │   ├── build_thyself.cpython-38.pyc
│           │   │   ├── common.cpython-38.pyc
│           │   │   ├── config.cpython-38.pyc
│           │   │   ├── __init__.cpython-38.pyc
│           │   │   ├── sdist.cpython-38.pyc
│           │   │   ├── versionno.cpython-38.pyc
│           │   │   └── wheel.cpython-38.pyc
│           │   ├── sdist.py
│           │   ├── versionno.py
│           │   └── wheel.py
│           └── flit_core-3.3.0.dist-info
│               ├── METADATA
│               ├── RECORD
│               └── WHEEL
└── share
    └── doc
        └── python-flit-core-3.3.0-0.47c1b48
            └── LICENSE

9 directories, 20 files

[Apteryks]

Hmm. Does flit really requires tomli ? Now the issue has moved to trying to build a minimal tomli, which requires flit, which requires tomli:

starting phase `unpack'
tomli-1.2.1/LICENSE
tomli-1.2.1/README.md
tomli-1.2.1/pyproject.toml
tomli-1.2.1/tomli/__init__.py
tomli-1.2.1/tomli/_parser.py
tomli-1.2.1/tomli/_re.py
tomli-1.2.1/tomli/py.typed
tomli-1.2.1/PKG-INFO
phase `unpack' succeeded after 0.0 seconds
starting phase `ensure-no-mtimes-pre-1980'
phase `ensure-no-mtimes-pre-1980' succeeded after 0.0 seconds
starting phase `enable-bytecode-determinism'
phase `enable-bytecode-determinism' succeeded after 0.0 seconds
starting phase `patch-usr-bin-file'
phase `patch-usr-bin-file' succeeded after 0.0 seconds
starting phase `patch-source-shebangs'
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'
* Getting dependencies for wheel...

Traceback (most recent call last):
  File "/gnu/store/9s1i1cc670jp801j9dj1wiwv5snp0pxl-python-pep517-bootstrap-0.9.1/lib/python3.8/site-packages/pep517/wrappers.py", line 284, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/gnu/store/9s1i1cc670jp801j9dj1wiwv5snp0pxl-python-pep517-bootstrap-0.9.1/lib/python3.8/site-packages/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/gnu/store/p9pn8ynaa6fyf7b9aka8ar8yarfzv11h-python-flit-core-3.3.0-0.47c1b48/lib/python3.8/site-packages/flit_core/buildapi.py", line 12, in <module>
    from .config import read_flit_config
  File "/gnu/store/p9pn8ynaa6fyf7b9aka8ar8yarfzv11h-python-flit-core-3.3.0-0.47c1b48/lib/python3.8/site-packages/flit_core/config.py", line 8, in <module>
    import tomli
ModuleNotFoundError: No module named 'tomli'

[Apteryks]

Perhaps I can s/tomli/toml/ to workaround this (they share the same API, right?), as toml has no bootstrap requirements else than the (already bundled) setuptools that is shipped with Python itself.

[takluyver]

flit_core can build itself without a TOML parser, but then it requires tomli to build anything else - including tomli itself. It should be possible to make this work by letting it import tomli from the source directory while building tomli. I've tried to explain this in a the bootstrapping doc on #441 .

I'm hoping that ultimately this little wrinkle will be ironed out by having a TOML parser in the standard library, but that will take a while.

[Apteryks]

Hello!

I was able to build it, but I'm a bit puzzled as to why pypa build cannot find flit_core >= 3.2.0 <4 (the files installed are listed above). Any idea? It can be worked around with -x (--skip-dependency-check).

$ PYTHONPATH=$PWD:$PYTHONPATH "python" "-m" "build" "--wheel" "--no-isolation"
* Getting dependencies for wheel...

ERROR Missing dependencies:
        flit_core>=3.2.0,<4
        tomli
$ PYTHONPATH=$PWD:$PYTHONPATH "python" "-m" "build" "--wheel" "--no-isolation" -x
* Building wheel...
Successfully built tomli-1.2.1-py3-none-any.whl

[takluyver]

I'm not sure about that either. I think this is the relevant code in build if you want to investigate:

https://github.com/pypa/build/blob/e770319e9d8a1b498b6f79cd1b63ac770db3d0be/src/build/__init__.py#L161-L173

But I'd expect you'd need to skip the dependency check when building tomli in any case, because the workaround to make tomli importable doesn't make it appear as an installed distribution. So if this only affects building tomli, I'd just accept that the error message is mistaken. If you see the same thing when building normal packages without this trick, it's worth figuring out what's happening.

[Apteryks]

I see this is fixed in 3.4.0. Thanks! It works.