Facilitate reproducible builds by dropping a target-platform-configuration 'log'

[ewillink]

A number of distinguished committers, including some Tycho committers, have strongly advocated reproducible builds but https://github.com/eclipse-justj/justj.tools/discussions/10 identifies many problems and that Tycho could solve many of them.

The Tycho target-platform-configuration is responsible for locating required IUs. It could therefore drop a 'log' identifying

• the exact version/spelling of each loaded IU
• the exact location from which each IU was loaded

As a minimum, this information could enable a manual reproduction attempt to rewrite the target file with the actual locations. Better an automated reproduction attempt could redirect to successors of transient nightly/milestone repos.

[laeubi]

Firs be aware the "reproducible" as a vague term that includes many things one might want to consider. Beside that, you can use mirror-target-platform to archive a "target" to local repository, you might then want to use this as a whole (can be quite big) or you can use the artifacts.xml and/or content.xml. I'm just not sure it will help you with what you have in mind....

[ewillink]

Obviously total pathologiocal reproducibility will need to log version numbers of all patches in all software and even hardware revisions. I'm not being that pathologiocal; just hitting the low hanging fruit of the software dependencies.

artifacts.xml and/or content.xml contain details of the built artifacts. They contain no information about the artifacts that were used during the build.

For reprodibility, it is necressary to know that the build used e.g.

Fetching org.eclipse.swt_3.125.0.v20240227-1638.jar from https://download.eclipse.org/eclipse/updates/4.31-I-builds/I20240228-1800/plugins/ (18.42kB)

[laeubi]

For reprodibility, it is necressary to know that the build used e.g.

If your target is stable you don'T need that information as its can be reproduced anytime, if not you need to conserve your target as explained first and then you can reproduce your build anytime.

[merks]

Note that after tomorrow, this site will be deleted:

https://download.eclipse.org/eclipse/updates/4.31-I-builds/I20240228-1800

Then one would need to find a site with that IU in order to do a build. So keeping the site URL is not as helpful as might appear to be the case.

For BIRT I made an effort to lock in stable repositories in the target platform as a commit in the Git repository:

eclipse-birt/birt@00fe855#diff-8aeadc3b52a99524e2e4fe3b4ba4f178581a9b90d5713ecdeb57386a73238150

But as was noted previously, when doing a release build, the stable final repositories for all the dependencies are not always know at that point in time, which is why for BIRT I waited until the are known.

[laeubi]

Even if repositories are not know, if you have exact versions in your target (in contrast to 0.0.0) it doesn't matter what URL is used.

[ewillink]

Yes. Once something has gone, reproducibility is reduced, but a human / tool can search the successors of https://download.eclipse.org/eclipse/updates/4.31-I-builds/I20240228-1800 potentially locating exact IUs, but at least the closest match.

Yes in principle relengs can use precise versions, but I (and I suspect most relengs) do not have the time to update all the versions prior to every build. We need to know exactly what was used by our successful build. (In the old rmap days updating was a major pain but at least there was tooling support.)

Yes a total target platform copy could be taken and in the old days, the workspace of at least the latest build had a long life-time. The webmasters claim we have insufficient disk space so that the workspace now has a persistence of barely minutes. Keeping a total platform for each build on the off chance that it needs reproducing is likely to be frowned upon. I'm just asking for a 'directory' listing of the total platform so that it can be reconstructed from source-related repos.

[laeubi]

If you don't use exact versions your build is not reproducible and never will. Updating the versions is a matter of open the target platform and hit the "update" button so I think we have all the tooling one needs.

[ewillink]

Hm. Not aware of "Update". Experimenting with an "Update" button in the target editor and org.eclipse.license.feature.group I fail to get any update from any of 0.0.0, 1.0.0.v20131003-1638, 1.0.1.v20140414-1359 or 2.0.0. Obviously I misunderstand.