Is the ZipArchiver supposed to create reproducible zips?

[func0der]

Hey guys,

I am trying to wrap my head around the process of creating reproducible zips.
I found a ZipArchiver in the composer/composer package and thought it to be my best option to success with this endeavor.

I came across an issue with it though, that I do not know is either purposely not taken care of or just a bug that you rarely stumble upon.

Premise
You create a zip file of files and folders that are 100% identical in their contents AND their attributes (mtime etc.) on two different machines.

Outcome
The machines create ZIPs with the same content, BUT they have different checksums.

Reason
The order in which the files reside in the ZIP are different. You can check that by running:

diff --color -u <(zipinfo file1.zip) <(zipinfo file2.zip)

Question
Is the ZipArchiver class supposed to create reproducible builds?

[stof]

ArchivableFilesFinder is indeed not applying any sorting in the Finder instance it uses, meaning that the order is undefined (it depends on how the filesystem gets iterated by the PHP filesystem iterators, which might not be stable across machines).

I think the ArchivableFilesFinder should apply some explicit sorting. What do you think @Seldaek ?

[func0der]

I created a quick PR for it: #10274

[func0der]

I do not want to steal the light from the sorting issues, but should there be functionality to change the modified date of files in the archiver, too?

I have created code for this, to normalize the files before ziping them. I do not know though, if we could generalize it for this purpose.
It would touch all files before publishing, which could mess up their state.

use Composer\Package\Archiver\ArchivableFilesFinder;
use RuntimeException;

/**
 * This class will set all files' modified and access time in a path to
 * a specific time. The time is either taken from git (if the path has a .git file in it)
 * or just sets it it to '1970-01-01 00:00:00'.
 */
class TimestampNormalizer
{
    private const FALLBACK_DATE = '1970-01-01 00:00:01';

    public function normalize(string $path): void
    {
        if ($this->isGitRepo($path)) {
            $date = $this->getGitInformation($path) ?? self::FALLBACK_DATE;
        } else {
            $date = self::FALLBACK_DATE;
        }
        $dateTimestamp = strtotime($date);

        if ($dateTimestamp === false) {
            throw new RuntimeException('Timestamp could not be generated');
        }

        $files = new ArchivableFilesFinder($path, [], false);

        /** @var \SplFileInfo $file */
        foreach ($files as $file) {
            if ($file->isFile()) {
                $filePath = $file->getRealPath();

                if ($filePath !== false) {
                    touch($filePath, $dateTimestamp, $dateTimestamp);
                }
            }
        }
    }

    private function isGitRepo(string $path): bool
    {
        $gitPath = $path . '/' . '.git';

        return file_exists($gitPath) && is_dir($gitPath);
    }

    /**
     * Inspired by sebastian/version.
     */
    private function getGitInformation(string $path): ?string
    {
        if (!is_dir($path . DIRECTORY_SEPARATOR . '.git')) {
            return null;
        }

        $process = proc_open(
            'git log -1 --date=format:"%Y-%m-%d %H:%M:%S" --format="%ad"',
            [
                1 => ['pipe', 'w'],
                2 => ['pipe', 'w'],
            ],
            $pipes,
            $path
        );

        if (!is_resource($process)) {
            return null;
        }

        $result = trim((string) stream_get_contents($pipes[1]));

        fclose($pipes[1]);
        fclose($pipes[2]);

        $returnCode = proc_close($process);

        if ($returnCode !== 0) {
            return null;
        }

        return $result;
    }
}

[Seldaek]

We do have some interest in reproducible zips so I'll probably be looking into this in the nearish future. IMO normalizing in such a way (using git or the release date) would be acceptable, but the question remains whether filemtime is the only problem left. I believe there may be issues with file access times too causing variability in the output. I haven't looked at this in ages though.