Deploy a maven based p2-updatesite

[laeubi]

Target Jetty version(s)
10.x

Enhancement Description
Currently projects in the eclipse build custom update-sites from the jetty deployed artifacts, this has the following drawbacks:

• artifacts are replicated on different places
• there is a delay between the jetty-releases and the upgrades
• if the p2 supplier choose not to upgrade to the latest release, or does not include optional modules it might be necessary for consumer to fire up their own replicated making it even more worse

There is an upcoming feature in Tycho 2.4.x that allows to supply an update-site for maven-deployed projects like jetty with minimal effort and no maintenance cost by simply generating all necessary data from existing artifacts and create a maven deployable artifact that later can be consumed by p2/tycho to be used as an update-site.

I therefore like to propose jetty as an early adopter of this feature.

[laeubi]

FYI: @mickaelistria @akurtakov

[jonahgraham]

IIUC if this issue is fixed then Jetty will publish a p2 update site along with every release and that means that Eclipse Platform/WTP/RAP etc will be able to consume Jetty more easily?

[laeubi]

@jonahgraham yep, that's the goal of this. In fact this (hopefully) could replace any custom build/published p2-update sites on the long term. There might be some issues to discover as this feature is relative new and currently jetty would be the first provider but my hope would be that this kind of publishing once becomes the de-factor standard for maven-centric projects that like to help p2 consumers using their artifacts.

[akurtakov]

What is the status of this one ? Tycho 2.4.0 is released. If this p2 site will make it to 10.0.7 and released soon enough I would rather skip generation eclipse platform specific p2 repo but use the new general one instead. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=575313 where 10.0.6 is requested for CVE fix.

[laeubi]

@akurtakov afaik the idea was to wait for

• Issue #201 assemble-maven-repository respect skip from maven-deploy-plugin eclipse-tycho/tycho#204

but @mickaelistria has had some concerns (or at least wants to review it in more detail). If we can get this done and no more issues arise from either jetty or tycho side I think the only thing left would be a Tycho 2.4.1 release (or maybe even 2.5.0).

I think it would be good to generate a (local) version of the p2 repro and test if there is anything left from the consumer side (platform) we have not considered yet.

[mickaelistria]

Why is eclipse-tycho/tycho#204 a requirement for this one? Why is the assemble-maven-repository mojo really necessary? Wouldn't a target + a category.xml + a pom.xml be able to achieve the result right now?

[laeubi]

@mickaelistria see comments here, the goal is to not have additional artifacts like targets (that won't work anyway for 'pure' maven projects), category and so on for such projects as jetty that don't like to put additional effort in managing those things explicitly, but still provide a fully useful p2-site for eclipse-consumers while only maintain a standard maven-deploy.

[mickaelistria]

OK, but that is already achievable with a combination of target+category+pom file, isn't it?

[laeubi]

not that I'm aware of... all of current tycho mechanism still require to deploy the artifacts on a dedicated server or additional transformation and modification steps are required that make several assumptions (e.g. all artifacts are produced by tycho and are part of an update-site already).

The assemble-maven-repository mojo on the other hand:

1. do not require (even though it possible) any special configuration (category.xml), it could be used 'as-is' for any maven project that produces OSGi-Bundles of any kind as a starting point
2. do not require these bundles are build by tycho (so could be applied to bnd-maven or felix-bundle projects as well), do not require a resolvable target or anything eclipse-eco-system-specific as it is possible to use the current reactor produced artifacts
3. do not require any additional deployment site as all artifacts are deployed (and consumed) to maven central/nexus/any other target with just one new artifact deployed with a few kilobytes
4. do not require any maintenance, as the site automatically includes new modules (depending on the configuration), and can adapt to the current version using standard maven-techniques (no need to edit targets, categories, features,...)
5. do not require any additional deployment steps, everything is up-to-date and ready with the usual deployment of the other artifacts (no need to ask orbit/platform/any maintainer to upgrade to the latest release).

[mickaelistria]

not that I'm aware of... all of current tycho mechanism still require to deploy the artifacts on a dedicated server or additional transformation and modification steps are required that make several assumptions (e.g. all artifacts are produced by tycho and are part of an update-site already).

Did you see https://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-repository-plugin/remap-artifacts-to-m2-repo-mojo.html ?
Tycho is expected to keep GAV for artifact that come from a Maven repo (and not from a p2 repo) with pomDependencies=consider or Maven location in .target file; then these artifacts are aggregated and this mojo will just replace the p2 metadata to reference the artifact location in the Maven repo, so artifacts don't need to be redeployed.
The p2 metadata can then be published anywhere, including in a Maven repo.

[laeubi]

Did you see https://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-repository-plugin/remap-artifacts-to-m2-repo-mojo.html ?

As mentioned above this requires that all artifacts are:

1. build by tycho / (maybe works with pomDependencies=consider also but never used that) / (maybe also with tycho m2e target location but never tested that)
2. including them in an update-site to be build (requires additional build time + disk space)
3. requires additional modification steps e.g. delete all remapped artifacts from plugins folder, repack the update-site zip, (replace the attached artifact?) befor it actually could be deployed

Beside that, the remap-artifacts-to-m2-repo] places hard coded URLs into the update-site making it unusable with any maven-mirror approach (e.g. nexus) without further manual adjustments.

[mickaelistria]

As mentioned above this requires that all artifacts are:

1. build by tycho

Wrong

2. including them in an update-site to be build (requires additional build time + disk space)

Not so important

3. requires additional modification steps e.g. delete all remapped artifacts from plugins folder, repack the update-site zip, (replace the attached artifact?) befor it actually could be deployed

Wrong, the remap mojo is expected to take care of that.

Beside that, the remap-artifacts-to-m2-repo] places hard coded URLs into the update-site making it unusable with any maven-mirror approach (e.g. nexus) without further manual adjustments.

Yet it would be enough as a 1st iteration for Jetty to easily enough build a p2 repo; and doesn't require to wait for a new release of Tycho.
I'll try to submit a solution for Jetty based on that soon.

[laeubi]

Yet it would be enough as a 1st iteration for Jetty to easily enough build a p2 repo

I don't think so, the recent eclipse-infra outage has shown that backups/mirror are a crucial part of an reliable infrastructure.

Anyways when providing a solution please make sure it works with the current reactor build (no need for an additional deploy step) and no further manual configuration is required. Anyway I think it would be better to invest time/resources in building a reliable first-class solution instead just to show it is somehow possible with lots of more effort....

[mickaelistria]

I just discussed with @akurtakov and the actual requirement is not to have a p2 update-site; it's to have signed artifacts.
How is signing supposed to happen with assemble-maven-repository?
IMO, the solution is more to have Jetty actually produce signed artifacts and publishing them as regular Maven artifacts (for example with a different classifier). Once we have it, Platform can use pomDependencies=consider or whatever.

[akurtakov]

I don't like the different classifier solution. It's time to embrace PGP signatures.

[mickaelistria]

I don't like the different classifier solution. It's time to embrace PGP signatures.

OK, but before doing so, let it be clear about how PGP signatures are working in p2 now: anyone who install an artifact which has a PGP signature will see the "Trust" dialog and will be asked whether to trust the key or not. There is no official/verified trust chain we can rely on to automate a decision for users.

[laeubi]

I don't see how signed artifacts should have an additional classifier. As jetty is an eclipse project it could sign the main artifacts simply with the eclipse-web service, there seem to be some setup already in the pom but I think @janbartel or @joakime can tell more about that.

Anyway signing itself does not require any special technique, as proven in m2e where we sign artifacts from a different source.

It's time to embrace PGP signatures.

PGP signatures are a work in progress for the assemble-maven-repository see

• assemble-maven-repository should include PGP signatures in the P2 site eclipse-tycho/tycho#203

Beside that, neither tycho nor jetty are driven only by platform needs, so even if platform do not care about a jetty-p2 site other do and we should provide them with the best possible solution, as this could encourage other projects (eclipse-link has recently dropped p2 support for the 3.x mainline) to adopt this soloution.

We should simply accept the fact that p2 is not that "standard" as many at eclipse-platform think of and either provide better tools or drop p2 also on the long-term as it is a maintenance nightmare otherwise.

[laeubi]

There is no official/verified trust chain we can rely on to automate a decision for users.

The logical choice for this would be an official eclipse-PGP key and/or key server as suggested here eclipse-tycho/tycho#203 (comment)

[mickaelistria]

As jetty is an eclipse project it could sign the main artifacts simply with the eclipse-web service, there seem to be some setup already in the pom but I think @janbartel or @joakime can tell more about that.

Last time we discussed that, in the Architecture Council about 3 months ago IIRC, Jetty was clearly not willing to have jarsigned artifacts by default because jarsigning seems to have caused issue with bytecode manipulation or other stuff. Unless that changes, we'd need new artifacts.

as proven in m2e where we sign artifacts from a different source

But then, they become different artifacts from what's initially published by the project.

Beside that, neither tycho nor jetty are driven only by platform needs, so even if platform do not care about a jetty-p2 site other do and we should provide them with the best possible solution

As a reminder,
The best possible solution is already there: Tycho and Platform can consume any Maven artifact in their build without requiring a p2 repository. What's missing for Platform is the signature.
So maybe a p2 repo would hypothetically help others, but as long as this is not certain it's actually profitable (ie we have a clear request from another project), it's also not worth adding complexity and effort where it's not needed.

We should simply accept the fact that p2 is not that "standard" as many at eclipse-platform think of and either provide better tools or drop p2 also on the long-term as it is a maintenance nightmare otherwise.

I don't get this comment: I'm saying that a p2 repo is useless here because Tycho and Platform can leave without it; and then you're insisting on building a p2 repo for Jetty (without a solid user-story to confirm it's profitable); and then you're claiming that I am too conservative of p2. This is incoherent...

The logical choice for this would be an official eclipse-PGP key

Yes, and this key exists and it the root of a "web of trust", but PGP itself doesn't recommend automatic trust decisions (a path of trust is onw indication for a human decision of whether to trust or not), and it will still take multiple releases before we get something like that part of p2.

Anyway, I'm dropping from this issue/conversation as it's not related to our goals on Platform. For Platform, I see those possible solutions, that do not involve Jetty producing a p2 repo:

• Either we say Platform embraces PGP signatures for p2 metadata -as it- then
  ** Either update Tycho to embed PGP signatures from Maven into p2 metadata for the Jetty artifacts, or
  ** We keep having Platform building a p2 repo for Jetty, but adding its own PGP signatures prduced with Platform PGP key.
• or we stick to JarSigner and make Jetty adopt it better Publish some artifacts jarsigned with Eclipse certifiacte #6612

[laeubi]

But then, they become different artifacts from what's initially published by the project.

if they produce new artifacts the same happens + it consumes useless space on maven central... so for me this is "not profitable"

I don't get this comment

Simple as that: Drop P2 completely from platform/eclipse/tycho and consume/publish everything from standard sources (e.g. maven central), or stop complaining that there is no user story for maven hosted P2 sites...

For me this whole "there is no user story" / "complexity" / "effort" complain is completely ridiculous for me, obviously at least three people (including me), the jetty main contributors and some people from the eclipse (including Alexander) have found this useful and the whole work was contributed by me and I'm committed in help supporting this and improve the feature, and it is perfectly isolated to not put any burden on the tycho-core features.

So if platform decides there are "better" way, why not, use them, completely ignore this feature, waste hours of development time to convince people they are doing things "wrong" and producing copies of already available artifacts over and over again, I won't mind (as I don't mind about code-signing or PGP signatures at all for my daily work)... this feature is not to force platform to do/change anything, so why force others to adapt to platform "best practice".

[joakime]

Something else to consider.

Eclipse Jetty does not release on Eclipse Infrastructure, or use Eclipse Infrastructure to build / test.

We are often too bleeding edge for the Eclipse Infrastructure tooling (for build requirements), or far too aggressive for Eclipse Infrastructure tooling (for test requirements).

Example: we've had recent discussions on making the build + release process require JDK 16 (and JDK 17 once it's official), as we have components that embrace features only present in those JDKs. (like JVM built-in UnixSocket support). We would build to a target of JVM 11, but still require the build itself to be of a higher JVM. This comes up with some regularity in Eclipse Jetty. This specific case (JVM based UnixSocket) didn't trigger the minimum build JDK requirement this time, as we found a simple enough solution.

If we go with the classifier approach, how can we integrate jarsigner with our build/release without requiring the use of Eclipse Infrastructure for all parts of our build / test? This wholesale requirement hasn't been possible in the past, has this requirement changed since we last talked about this?

[laeubi]

This wholesale requirement hasn't been possible in the past

At laest for me, this is not a requirement of my request/proposal for maven based updatesites but a pure eclipse-platformm issue and I would never expect this as a perquisite to use this feature. If jetty easily can (what not seems to be the case) provide signed artifacts this would be an additional bonus to me...

Whether or not a specific signature is desired can (and should) be handled at the consumer side.

[joakime]

This wholesale requirement hasn't been possible in the past

At laest for me, this is not a requirement of my request/proposal for maven based updatesites but a pure eclipse-platformm issue and I would never expect this as a perquisite to use this feature. If jetty easily can (what not seems to be the case) provide signed artifacts this would be an additional bonus to me...

Whether or not a specific signature is desired can (and should) be handled at the consumer side.

When this topic has come up in the past, the Eclipse requirement was that only Eclipse Infrastructure jarsigning was allowed, and only with the Eclipse keys.
Has this requirement changed?

[laeubi]

Has this requirement changed?

There recently where some discussions on the mailinglist (scroll down a bit as I can't find a direct reference to the mailinglist post): https://www.eclipse.org/lists/cross-project-issues-dev/msg18150.html

edit: reading it again, actually signing is only required for sim-rel participation... so if (theoretical) platform would leave sim-rel platform would no longer require signed artifacts...

[mickaelistria]

From Platform POV, the whole story behind Jetty (re-)packaging is jar signing that is required by SimRel.
Having a p2 repo in Jetty that doesn't contain signed jar wouldn't be more useful than current state with Platform rebuilding a p2 repo for Jetty artifacts just to have them jarsigned with Eclipse certificate. Even if Jetty publishes a p2 repo without signed artifacts, then those artifacts would still need to be repackaged by Eclipse Platform to add Eclipse signatures. So Platform doesn't win anything here.
But it's not per se a reason why not working on this task; I just want to make it clear that this task has no benefit for Platform and wouldn't improve Platform build.

[joakime]

The last we heard, P2 is going away in favor of something Tycho has developed based on the global repository system metadata. (what this issue is about, right?)
We were led to believe that P2 is no more, Orbit requirement is no more, and all of what that used to imply.
No repackaging in orbit, no rebuild for P2, just use what exists, and validate from maven repo signatures with a web-of-trust cross signing of approved artifacts (like most large projects do. re: apache and the linux kernel).

In this ideal world, jarsigning is no longer a requirement, as any available artifact by the project can be cryptographically validated already (even artifacts that contain no classes, or isn't a jar file, even validated inside or outside of the JVM, even before its packaged up in an archive, or put on a download site).

The only wrinkle left to sort out is (re: orbit) 3rd party deps that don't have osgi manifests, but that's what I thought osgi-fragments are for.

[mickaelistria]

p2 is still here, but can be hidden inside Tycho so producer of OSGi bundles don't need to care about p2 specifically. So you're right, one can consume any OSGi artifact from Maven -with or without p2 metadata- into a Tycho build and reuse what's there.
One reason of Orbit is that same as the one we discuss here: jarsigning. If jarsigning were not required, many Orbit bundles could be dropped as long as they are OSGi bundles upstream.
The case of non-OSGi artifacts can be addressed in 2 ways: at build-time with Tycho pomDependencies=wrap or equivalent in .target file -relying on BND-; or at dev-time by creation of dedicated artifacts and MANIFEST, like Orbit is doing.

In this ideal world, jarsigning is no longer a requirement

This ideal world is not for tomorrow. I don't think we can expect this requirement to vanish before several months. The Eclipse community (or IDE working group) has not yet elaborated a plan to let this requirement go away and to specify what features alternatives should provide.

[laeubi]

This ideal world is not for tomorrow

But we can prepare for them today... For me its simply chicken-egg-problem, as there is no (widely used) project providing a p2 site that only contains PGP signatures there is not much spin in supporting this use-case, and as there is no support for this we are waiting for "several months" (the PGP discussion is already half a year old if I remember correctly)... and so we are waiting forever.

Even if Jetty publishes a p2 repo without signed artifacts, then those artifacts would still need to be repackaged by Eclipse Platform

This is not true, just take a look at the bnd-tools plugin at m2e, it is not 'repacked' but simply signed before published so that's what platform could also do before publish its update-site (!) but until this point no-one cares about signed or not signed content and the original jetty jars can be used.

[laeubi]

By the way as mentioned before, PGP signatures are next on my todo list for this feature once the deploy issue is fixed.

Platform could still decide not to use it, but I understood @akurtakov request here that he better sooner than later want to use the official ones... if PGP is the only issue holding us back I can give some more priority on this...

[mickaelistria]

@laeubi I think you should read archive of the various cross-project discussions about SimRel. It's something that's slow to change because of bureaucracy, fear of change, many participants, people who sets expectation being sometimes far away from those who use or those who develop the tool..., so every item takes a long time to reach consensus. Because Platform adds PGP won't remove the requirement on jarsigner and won't solve the issue.
It's not a technical issue, but a human/organizational one. And as long as those humans/organizations don't agree on the expectation, there is a huge risk that preliminary efforts would be vain.
I started a discussion about jarsigner on the eclipse-ide-wg@eclipse.org maling-list.

[laeubi]

I subscribed to the list right now lets see what happens there. I also opened

• make jar-signer webservice return the same result for the same artifact eclipse-cbi/org.eclipse.cbi#53

so maybe signing artifacts afterwards become more consistent.

[akurtakov]

So this one is included in 10.0.7, right? What is the url for the p2 repo?

[olamy]

look here https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-p2/

[laeubi]

What is the url for the p2 repo?

mvn:org.eclipse.jetty:jetty-p2:10.0.7:zip:p2site
mvn:org.eclipse.jetty:jetty-p2:11.0.7:zip:p2site

[akurtakov]

Thanks!

[olamy]

now because of this eclipse-tycho/tycho#1814
our build doesn't work even if we use a locked version of the maven plugin.
but it looks to not being a reproducible and using some dynamic osgi.
This is not acceptable as this affect our work...