Tycho 2.7.0 release

[akurtakov]

We plan to release Tycho 2.7.0 during 4th week of February 2022. For details of new features and bugfixes, see release notes.
Please help by testing the SNAPSHOTS build. To use it, change your tycho version to 2.7.0-SNAPSHOT and add the following snippet to your pom.

<pluginRepositories>
    <pluginRepository>
      <id>tycho-snapshots</id>
      <url>https://repo.eclipse.org/content/repositories/tycho-snapshots/</url>
    </pluginRepository>
</pluginRepositories>

We plan to promote this release unless major regressions are found that can't be fixed at least one week before the release date.

Regards,
Tycho team

[laeubi]

We are about to release Tycho 2.7.0 soon, please check that all your issues are fixed and no regressions are found.

@mbooth101 #658 Mirroring remote artifacts into local cache loses P2 artifact properties (eg PGP, maven info...)
@habbit-baggins #652 Nested target feature needs to support windows path in project_loc expansion
@LorenzoBettini #643 tycho-surefire-plugin:integration-test does not execute any test
@parttimenerd #631 Tycho project hangs when importing it into eclipse
@vaclavHala #610 Slicer fails when missing dependency
@pzi-dsa #596 Artifact id of pomless aggregators should be configurable.
@vogella #567 Pomless product build fails in package phase
@beyerjsb #562 tycho-p2-extras-plugin::compare-version-with-baselines Bug in Tycho 2.2.0 .. 2.6.0 for features with root files
@jbeichter #560 Build fails when targeting JDK17
@HannesWell #553 Inconsistent build-qualifier computation between goals tycho-packaging-plugin:build-qualifier and package-feature
@dwagelaar #552 Tycho: requires 'osgi.ee; (&(osgi.ee=JavaSE)(version=11))' but it could not be found
@sratz #551 Provide tycho specific org.apache.maven.graph.GraphBuilder for project dependency resolution
@eNBeWe #543 tycho-packaging-plugin warns about not being threadsafe
@vogella #541 Launcher name ignored for Mac app
@henryju #539 Problem with dependency types and artifactory Maven plugin
@EaseRider #537 NullPointer when validating on pom-Dependencies with parent version in tagetplatform
@vogella #527 Allow parallel resolving of eclipse-repository

[habbit-baggins]

Hi, I don't see my issue (#652) fixed in 2.7.0-SNAPSHOT, but I think that is because the JARs in the repos for that version is from Feb 12, while the issue was fixed on the 15th. I think it was first on the 3.0 branch, and then added to the 2.7 branch (by 89ef155), so maybe the new build was not pushed to the Maven repos.

[laeubi]

@habbit-baggins thanks for mention this, there was a wrong branch name in the Jenkins file so snapshots are not deployed... I fixed that and hopefully a new snapshot should arise soon.
https://ci.eclipse.org/tycho/job/tycho-github/job/tycho-2.7.x/6/

[vogella]

Off topic, how did you create the above list of topics?

[laeubi]

I used github cli for fetching the necessary data:

gh issue list --search "is:closed is:issue milestone:2.7" --json author,number,title --template \
   '{{range .}}{{printf "@%v" .author.login}} {{printf "#%v" .number}} {{printf "%v\n" .title}}{{end}}

[vogella]

On topic: My little test build works fine with 2.7.0-SNAPSHOT

+1 for the release

I used github cli for fetching the necessary data:

cool

[habbit-baggins]

Semi-off topic: I found another problem with the nested target platforms, which appears to only happen while "bootstrapping" the build from a pristine Maven. Since it also happens in 2.6.0 (albeit slightly differently) and is not related to Windows support, it is not a regression. I have raised it as a separate issue #665 .

[laeubi]

Since it also happens in 2.6.0 (albeit slightly differently) [..], it is not a regression.

I would consider this a regression as it has never worked. I''' try to provide a fix for this for master and 2.7.0.

[HannesWell]

I just tested the snapshot with my companies build (using the tycho-build 2.7.0-SNAPSHOT extension I build locally)
and the initial dependency resolution process takes magnitudes longer than before. Previously the resolution run about ~30sec, maybe a minute but now it is resolving since half an hour and did not yet complete.

From the logs I have the impression that each dependency of each project is resolved from scratch.
Even when the previously cached dependencies were not used (did this change?) this would be too long.

[HannesWell]

I think I found the culprit here, in LocalArtifactRepository there are two "caches" / "indexes"

For example 'org.eclipse.swt' is stored at

.m2/repository/p2/osgi/bundle/org.eclipse.swt/3.117.0.v20210906-0842 but the metadata is written to .m2/repository/org/eclipse/swt/org.eclipse.swt/3.117.0-SNAPSHOT.

So we store the data but after loading everything is lost (at least at the next build) ...

That's good! But I also observed that the same artifact is downloaded multiple times even in the same build.
Furthermore fetching the dependencies seems to be quite slow even when there is no download print-out, there are only round about two lines per second saying [INFO] Fetching ... .

[laeubi]

Yeah I'm looking into this. Sad enough the whole downloading stuff in Tycho is a bit wrecked, e.g. it checks all the sites for an artifact and the mirror management is also bad, but we won't change this for the 2.7.0 release :-\

I'm just a bit uncertain if I should go the hard way and fix the whole indexing stuff or find a simple workaround... actually I would have released tycho 2.7 today but without this fixed it does not makes much sense...

My ultimate goal is that Tycho has some more smart handling and for example try different mirrors if the are faster than others, @merks has somewhere described a similar approach for Ommpf.

[HannesWell]

Yeah I'm looking into this. Sad enough the whole downloading stuff in Tycho is a bit wrecked, e.g. it checks all the sites for an artifact and the mirror management is also bad, but we won't change this for the 2.7.0 release :-\

I'm just a bit uncertain if I should go the hard way and fix the whole indexing stuff or find a simple workaround... actually I would have released tycho 2.7 today but without this fixed it does not makes much sense...

Sorry for bringing bad news so late and it's relatable to not change something so fundamental in the (intended) last release of a major stream. And thanks for your effort!

At least I now have a relatively minimal reproducer of the problem with most of the target platform we use the stripped main pom and a few trivial plug-ins with dependencies. Please find it attached:
tycho-2-7.zip

On my computer this build runs about 50sec in total while building the three eclipse-plugin type projects takes only about one second or less.
Printout with download progress does not happen in each build (not sure if that only happens if it takes a bit longer), but when you run it multiple times occasionally the download print-out even occurs multiple times for the same artifact in one build.

The problem here seems to be that it takes almost the same time to resolve each (transitive) dependency of each plug-in. Therefore the resolution times increases proportionally to the number of times a dependencies is referenced. This propably leads to the half an hour resolution time for our build where we have about ~70 plug-ins to build.
In the given example you can simply increase the resolution runtime by duplicating one of the plugin.x which all have the same dependencies.
When using Tycho 2.6.0 the given example builds in about 13sec in total.

My ultimate goal is that Tycho has some more smart handling and for example try different mirrors if the are faster than others, @merks has somewhere described a similar approach for Ommpf.

That sounds great. In general I have the impression that Oomph does a lot of things very well when it comes to the resolution of the target platform, at least compared to PDE Oomph's targlets are often faster and very stable unlike the regular IU-container targets. I have the intention to work on this in PDE in the next time and therefor wanted to have a deeper look into how Oomph does it and learn from it, but I have not yet started.

[sratz]

I also ran our build (723 maven projects) with Tycho 2.7.0-SNAPSHOT and I can confirm that 2.7.0 is orders of magnitude slower.

Most projects are fetching the same artifacts over and over again. But even if they don't things feel much slower.

[HannesWell]

Created a dedicated issue for this regression: #670

[HannesWell]

The M2E build fails as well with the current 2.7.0 snapshot: eclipse-m2e/m2e-core#589

[ERROR] Failed to execute goal org.eclipse.tycho:target-platform-configuration:2.7.0-SNAPSHOT:target-platform (default-target-platform) on project org.eclipse.m2e.archetype.common: Execution default-target-platform of goal org.eclipse.tycho:target-platform-configuration:2.7.0-SNAPSHOT:target-platform failed: Tycho build extension not configured for MavenProject: org.eclipse.m2e:org.eclipse.m2e.archetype.common:1.18.1-SNAPSHOT @ /home/jenkins/agent/workspace/m2e_PR-589/m2e-maven-runtime/org.eclipse.m2e.archetype.common/pom.xml -> [Help 1]

I suspect the cause to be that the target-platform-configuration:target-platform mojo does not respect tycho.mode=maven and skips its execution if configured.
I can have a look at that this evening.

[laeubi]

Maven mode is really something we should get rid of for tycho 3.0....

Could you create an issue for both regressions so we keep track of them and add the 2.7 Milestone + regression label?

[HannesWell]

Maven mode is really something we should get rid of for tycho 3.0....

Making it obsolete would be great, just removing it could lead to problems in some edgy cases (like m2e is at the moment).

Could you create an issue for both regressions so we keep track of them and add the 2.7 Milestone + regression label?

Yes I'm about to do that.

[laeubi]

Due to the regressions here I'd will delay the release until Monday 28.02.

One Important note: If you are testing Tycho 2.7.0 it might be that there are interference with older Tycho versions!
In that case one needs to delete the file ~/.m2/repository/.meta/p2-artifacts.properties to fix this if you are done with the Tycho 2.7.x tests.

As an alternative, one might want to specify a separate maven local repository for Tycho with -Dmaven.repo.local=~/.m2/test-tycho27x

[laeubi]

W fixed a major regression in the 2.7.0-SNAPSHOT I'd like to encourage everyone to re spin their test.

[HannesWell]

Looks good for me as well.
I can confirm that the resolution of the target-platform runs as fast as before and no more duplicated downloads happen in my case as well.

Thanks again Christoph. Great work!

[habbit-baggins]

I also confirm that both my synthetic test case and the real case behind it do work with the latest snapshot.

[beyerjsb]

Can confirm that the fix for #562 is still in place and working. Although I suspect it's not been in the vicinity of the regression anyway :-)

[beyerjsb]

Regarding performance when resolving: Our biggest project takes ~45 seconds to resolve with Tycho 2.6.0 and ~110 seconds with Tycho 2.7.0-20220226.132720-73 from the public repo. You can literally see from the printed lines that it's slower. Both of course pre-resolved, so no downloads should be included. Does this version not include the fix?

[laeubi]

The fix is included, but the performance heavily depends on how you project is structured and what features it use. If you have a chance to collect some performance statistics we might be able to elaborate here more. Do you already use -T1C in your builds? This should for Tycho 2.7.0 improve the overall performance.
The important part is, that you don't see multiple downloads of the same artifact on each build.

[laeubi]

Tycho 2.7.0 has been released and is available from Maven Central repository.

🆕 https://github.com/eclipse/tycho/blob/tycho-2.7.x/RELEASE_NOTES.md
🏷️ https://github.com/eclipse/tycho/tree/tycho-2.7.0
👔 https://projects.eclipse.org/projects/technology.tycho/releases/2.7.0
🙏 contributors who contributed patches for this release:

• Alexander Kurtakov
• Arthur Brugière
• Christian Dietrich
• Christoph Läubrich
• Fred G
• Hannes Wellmann
• Jakub Stroleny
• Joe Shannon
• Lorenzo Bettini
• Marc Miltenberger
• Marco Lehmann-Mörz
• Mat Booth
• Michael Keppler
• Mickael Istria
• Sebastian Ratz
• Vaclav Hala

and thanks and to everyone who helped us with testing the staged version.

Regards,
The Tycho Team

[laeubi]

We are about to release a bugfix for 2.7.0 so I'd like to encourage everyone helping us with testing and identify of new or additional issues: #819

[merks]

I guess you mean 2.7.1 because that's what the release notes seem to mention first? Is that version using Eclipse 4.23 such that building products that use PGP-signed content will work well?

[merks]

Testing 2.7.1 answers my own question. My build fails like this:

Installing org.eclipse.oomph.setup.installer.product 1.25.0.v20220331-1212.
Installation failed.
An error occurred during the org.eclipse.equinox.internal.p2.engine.phases.CheckTrust phase.
	session context was:(profile=DefaultProfile, phase=org.eclipse.equinox.internal.p2.engine.phases.CheckTrust, operand=, action=).
	NullPointerException
	Caused by: 
	java.lang.NullPointerException
There were errors. See log file: D:\Users\merks\oomph-master-1.24\git\org.eclipse.oomph\workspace\.metadata\.log

The log has this:

!MESSAGE An error occurred during the org.eclipse.equinox.internal.p2.engine.phases.CheckTrust phase.
!SUBENTRY 1 org.eclipse.equinox.p2.engine 4 0 2022-03-31 14:21:10.415
!MESSAGE session context was:(profile=DefaultProfile, phase=org.eclipse.equinox.internal.p2.engine.phases.CheckTrust, operand=, action=).
!SUBENTRY 1 org.eclipse.equinox.p2.engine 4 0 2022-03-31 14:21:10.415
!MESSAGE NullPointerException
!STACK 0
java.lang.NullPointerException
	at org.eclipse.equinox.internal.p2.engine.phases.CertificateChecker.buildPGPTrustore(CertificateChecker.java:292)
	at org.eclipse.equinox.internal.p2.engine.phases.CertificateChecker.checkCertificates(CertificateChecker.java:117)
	at org.eclipse.equinox.internal.p2.engine.phases.CertificateChecker.start(CertificateChecker.java:73)
	at org.eclipse.equinox.internal.p2.engine.phases.CheckTrust.completePhase(CheckTrust.java:63)
	at org.eclipse.equinox.internal.p2.engine.Phase.postPerform(Phase.java:254)
	at org.eclipse.equinox.internal.p2.engine.Phase.perform(Phase.java:105)
	at org.eclipse.equinox.internal.p2.engine.PhaseSet.perform(PhaseSet.java:50)
	at org.eclipse.equinox.internal.p2.engine.Engine.perform(Engine.java:80)
	at org.eclipse.equinox.internal.p2.engine.Engine.perform(Engine.java:48)
	at org.eclipse.equinox.internal.provisional.p2.director.PlanExecutionHelper.executePlan(PlanExecutionHelper.java:46)
	at org.eclipse.equinox.internal.provisional.p2.director.PlanExecutionHelper.executePlan(PlanExecutionHelper.java:27)
	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.executePlan(DirectorApplication.java:772)
	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.planAndExecute(DirectorApplication.java:762)
	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.performProvisioningActions(DirectorApplication.java:743)
	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.run(DirectorApplication.java:1066)
	at org.eclipse.tycho.p2.tools.director.DirectorApplicationWrapper$DirectorApplicationWrapperCommand.execute(DirectorApplicationWrapper.java:47)
	at org.eclipse.tycho.plugins.p2.director.DirectorMojo.execute(DirectorMojo.java:153)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

So the 2.7.1 version will not be able to build products containing PGP signed content so it will not be usable by EPP to build the 2022-06 products.

[laeubi]

So the 2.7.1 version will not be able to build products containing PGP signed content so it will not be usable by EPP to build the 2022-06 products.

Hi Ed,

the Discussion for the 2.7.1 is here, if you mention the version of P2 we should use there I could try to update it there. At best you could provide a minimal reproducer for this as an integration-test to demonstrate the issue.