@timothyjward As an OSGi expert, could you please at least verify if my assumption and approach for the fix is correct? I tested it locally in an Maven based OSGi-application, and it works fine. But getting feedback from you would be very beneficial.

@timothyjward As an OSGi expert, could you please at least verify if my assumption and approach for the fix is correct?

Hi - I think the extender requirements that you have added look fine, but the implementation seems to be missing osgi.serviceloader capabilities for the implementations that it wants to register. This is described in the specification at https://docs.osgi.org/specification/osgi.cmpn/7.0.0/service.loader.html#d0e106889 and has an example at https://docs.osgi.org/specification/osgi.cmpn/7.0.0/service.loader.html#d0e107018

For reference I think you would need entries for each of the implementations registered in https://github.com/eclipse/eclipse-collections/tree/master/eclipse-collections/src/main/resources/META-INF/services

There are a lot of these, so you may wish to consider having bnd generate all this for you (including META-INF/services files) using the @ServiceProvider annotation https://bnd.bndtools.org/chapters/240-spi-annotations.html#serviceprovider

@timothyjward
Thanks for the feedback. From the specification point of view, you are totally right that the osgi.serviceloader capabilities need to be added. Interestingly it doesn't need to be required at runtime. SPI Fly seems to inspect META-INF/services and process everthing in there, even without the capabilities.

I will check if I can use the @ServiceProvider annotation. That would even reduce the maintenance effort on the project side, as the provider configuration files don't need to be maintained manually anymore.

I adapted the request from @timothyjward and use @ServiceProvider for the generation of the SPI resources. This way also the provider configuration files are generated. I therefore also removed the generation of those resources.

This also revealed a current issue in the eclipse-collections bundle. The following files are currently included in META-INF/services but the related implementations do not exist anymore. I suppose they were dropped in the meanwhile, but the services files are still generated:

org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanBooleanMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanByteMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanCharMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanDoubleMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanFloatMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanIntMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanLongMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanShortMapFactory

org.eclipse.collections.api.factory.map.primitive.MutableBooleanBooleanMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanByteMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanCharMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanDoubleMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanFloatMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanIntMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanLongMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanShortMapFactory

org.eclipse.collections.api.factory.set.primitive.ImmutableBooleanSetFactory

@nikhilnanivadekar
Sorry that this PR now got that big. But it was necessary to be able to build an update site with Tycho once the meta information in the MANIFEST files get generated.

@fipro78 trying to understand the PR. Is it possible to logically squash the commits? Multiple commits are ok, but some of them are overwritten and reviewing would be cumbersome.

Also, if you wouldn't mind explaining what will be the new process for p2 update site and to contribute to the Simrel repo? I am not an expert in this area. So, I will need (a lot of) help as I learn. Thanks!

@fipro78 trying to understand the PR. Is it possible to logically squash the commits? Multiple commits are ok, but some of them are overwritten and reviewing would be cumbersome.

After some struggles, I hope I have correctly squashed the commits.

Also, if you wouldn't mind explaining what will be the new process for p2 update site and to contribute to the Simrel repo? I am not an expert in this area. So, I will need (a lot of) help as I learn. Thanks!

Well, with this PR the p2 update site is created from the p2-feature project, and not the p2-repository anymore. And the p2 update site is not created on every build, but needs to be triggered manually. That means, the Jenkins job that publishes the p2 update site needs to be changed to build the p2-feature and publish the results from the p2-feature/org.eclipse.collections.repository/target.

With regards to contribute to the Simrel repo, this PR actually has no impact. What this PR does is, it ensures that the artifacts, that are published to Maven Central, can also be used in an OSGi context. This was not really possible as of now, because the API retrieves services from the Impl, but they reside in different bundles, and therefore we have classloader issues. That is why the previous EBR approach in p2-repository creates a new wrapped bundle that combines API and Impl. The wrapping should be not needed anymore, if a Service Loader Mediator implementation is available in the runtime.

Whether Eclipse Collections become part of Simrel or will be included in Eclipse Orbit, is out of the scope of this PR. But without additional build steps, it should be easy to include the artifacts from Maven Central there. It would be only needed to consume those artifacts, as they are OSGi bundles.

Of course that need to be validated again, to be sure that it is really working. I suppose @merks asked whether there is a Maven repository to consume the artefacts before they are actually released, so it can be verified how easy it would be. And maybe he can then also give some more information with regards to including Eclipse Collections to Orbit or Simrel. In the end, the question is, who is responsible for the publishing of the p2 update site. And IMHO that depends on how easy it becomes to consume the artifacts from Maven Central, rather then needing to create a wrapped bundle.

@merks
Have I explained that correctly? Do you want to add something? Because Simrel and Orbit is definitely your domain. ;-)

PDE, with m2e's help, as well as Tycho, support specifying Maven dependencies in a *.target. Orbit uses that specifically here:

https://github.com/eclipse-orbit/orbit-simrel/blob/main/maven-osgi/tp/Maven.target

As background, this file is generated/updated automatically by looking at the *.target of various SimRel projects and looking at Maven Central for new versions. As part of that analysis, the following target is also considered:

https://github.com/eclipse-orbit/orbit-simrel/blob/main/maven-osgi/tp/other/MavenSupplement.target

So if Eclipse Collections publishes well-formed OSGi artifacts to Maven Central we can trivially include it in MavenSupplement.target so that it's generated into Maven.target and in the end, is available in these Orbit p2 repositories:

https://download.eclipse.org/tools/orbit/simrel/maven-osgi/
https://download.eclipse.org/tools/orbit/simrel/orbit-aggregation/

Note that is is a convenience for the consumers because the consumers can include such dependencies directly in their own *.target, in which case they are also responsible for PGP signing those artifacts, which Orbit also does as an additional convenience.

Once that's in place, the Eclipse Collection's SimRel contribution should be removed and any other project relying on those bundles, must contribute them via the p2 repository of their own SimRel contribution.

So I'm happy with this path forward. The overhead for Orbit is trivial and is just done once, after which updates will be automatically generated.

Hi @fipro78, thanks for spotting this. I don't believe these classes ever existed. IIRC, we had a discussion a long time ago around the benefit of having boolean to anything Map and decided it wasn't worth coding or generating. These must have been mistakenly added to the services.

I adapted the request from @timothyjward and use @ServiceProvider for the generation of the SPI resources. This way also the provider configuration files are generated. I therefore also removed the generation of those resources.

This also revealed a current issue in the eclipse-collections bundle. The following files are currently included in META-INF/services but the related implementations do not exist anymore. I suppose they were dropped in the meanwhile, but the services files are still generated:

org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanBooleanMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanByteMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanCharMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanDoubleMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanFloatMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanIntMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanLongMapFactory
org.eclipse.collections.api.factory.map.primitive.ImmutableBooleanShortMapFactory

org.eclipse.collections.api.factory.map.primitive.MutableBooleanBooleanMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanByteMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanCharMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanDoubleMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanFloatMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanIntMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanLongMapFactory
org.eclipse.collections.api.factory.map.primitive.MutableBooleanShortMapFactory

org.eclipse.collections.api.factory.set.primitive.ImmutableBooleanSetFactory

@motlin @nikhilnanivadekar @donraab
The @aQute.bnd.annotation.spi.ServiceProvider is an annotation with class time retention. That means there is not additional runtime dependency by using this annotation. It is only used to tell bndtools to generate the SPI resources and the necessary headers in the MANIFEST.

And is there a build result for the PR that could be consumed by @merks to verify if the changes can be consumed easily in the Eclipse environment?

@motlin
I have no idea why the Javadoc build fails now. I tested it locally and there it works. Maybe you have an idea on this?

@timothyjward @merks
I know you already had a look. But could you please have a second look at this PR to give your opinion on the changes? The project team has limited knowledge about OSGi and bndtools, and therefore it is hard for them to verify this PR.

The @aQute.bnd.annotation.spi.ServiceProvider is an annotation with class time retention.

I'm familiar with com.google.auto.service.AutoService which seems similar and also has class time retention. What's the difference between then? Does ServiceProvider also power the generation of manifest entries?

I have no idea why the Javadoc build fails now. I tested it locally and there it works. Maybe you have an idea on this?

The error message is:

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.eclipse.collections:eclipse-collections-code-generator-maven-plugin:jar:12.0.0-SNAPSHOT in repo.eclipse.org

There are two things I find odd about this.

• We ran mvn install on that module, so it ought to exist in .m2 and we shouldn't be looking for it in on the internet at all
• Even if we were looking for it on the internet, why are we looking in repo.eclipse.org rather than maven central?

Actually looking at the diffs, it seems that you changed mvn install to mvn verify in pull_request.yml and that could definitely be the cause. Do you remember the reason for that change?

@motlin

What's the difference between then? Does ServiceProvider also power the generation of manifest entries?
Yes, that is actually the main reason for using that annotation.

Actually looking at the diffs, it seems that you changed mvn install to mvn verify in pull_request.yml and that could definitely be the cause. Do you remember the reason for that change?

No, probably it was the last locally working state. I reverted it and you can see the error that comes up then.

By accident I noticed that I have pushed the old separate early-access-javadoc build file before, which succeeded. The difference there is that the generator is build before. I have adapted that now. Not sure if it existed before or if I added it with the previous state to get the workflow build to succeed. And it seems I got it fixed again.

What about the annotation? Is it possible to avoid it? Even compile-time-retention annotation dependencies show up as dependencies when searching sites like mvnrepository.com. We'd prefer to show up as having no dependencies.

What about the annotation? Is it possible to avoid it? Even compile-time-retention annotation dependencies show up as dependencies when searching sites like mvnrepository.com. We'd prefer to show up as having no dependencies.

Well technically yes. The annotation triggers generation. If you do everything manually, you can avoid it. I actually tried it at the beginning but got the feedback that I missed several things. If you look at the generated manifest file, you will see that the manual maintenance of it would be a nightmare.

The usage of the annotation with the other changes leads to a reduced maintenance effort on your side with regards to support OSGi and therefore Eclipse RCP. Without it, well only very experienced OSGi experts are able to find an issue if you change something in the future. And to be honest, there are not that much experts. A reason for the annotations.

BTW I am not aware of any issues with compile time dependencies. Why do you want to avoid them?

What about the annotation? Is it possible to avoid it? Even compile-time-retention annotation dependencies show up as dependencies when searching sites like mvnrepository.com. We'd prefer to show up as having no dependencies.

Well technically yes. The annotation triggers generation. If you do everything manually, you can avoid it. I actually tried it at the beginning but got the feedback that I missed several things. If you look at the generated manifest file, you will see that the manual maintenance of it would be a nightmare.

The usage of the annotation with the other changes leads to a reduced maintenance effort on your side with regards to support OSGi and therefore Eclipse RCP. Without it, well only very experienced OSGi experts are able to find an issue if you change something in the future. And to be honest, there are not that much experts. A reason for the annotations.

BTW I am not aware of any issues with compile time dependencies. Why do you want to avoid them?

The issue is perception. If a project perceives there is a dependency because it is reported by a service like mvnrepository, they will probably just decide not to even evaluate Eclipse Collections for usage.

We had a recent situation where someone was unaware of the dual-licensing of Eclipse Collections, and they reported the usage of the EPL 1.0 license as an issue to a project as a Category B license (no, EPL 2.0 doesn't fix this). EDL 1.0 (BSD 3) should have made this a non-issue for the project, but the project decided to not ask any questions, and just remove the dependency. I didn't discover this until after the dependency was already removed, so it was already too late to clarify the issue as a non-issue.

If this dependency is reported by an automated scanning tool, even if it is compile time only, then projects will likely (rightly or wrongly) decide to not use Eclipse Collections.

If the artifact does not come in the form of an OSGi bundle then there will also be those who decide not to use it or cannot use it. You just can't please everyone because everyone has different perceptions and needs.

VIATRA, a SimRel project, has a dependency on Collections:

I've needed to personally make updates to Collections' SimRel contribution:

https://github.com/eclipse-simrel/simrel.build/commits/main/collections.aggrcon

That's a bit dysfunctional.

Orbit is capable of wrapping a non-OSGi artifact as a bundle because m2e provides support for that:

https://github.com/eclipse-orbit/orbit-simrel/blob/b18af19c4bb300f9683bc779fcdbba38e2340ae6/maven-bnd/tp/MavenBND.target#L225-L269

That being said, somewhere the BND instructions need to be defined and maintained...

I also noticed that Collections is long (years) over due for review:

https://projects.eclipse.org/projects/technology.collections

If the artifact does not come in the form of an OSGi bundle then there will also be those who decide not to use it or cannot use it. You just can't please everyone because everyone has different perceptions and needs.

Agreed. Just answering the question from @fipro78. I can't see a technical issue, but could be a perception issue for the project.

I appreciate all the help we have gotten from folks like yourself and @fipro78 on the OSGi front. Thank you! We'd like to make sure projects dependent on OSGi are able to leverage EC without issue, so long as there is not an adverse impact to others using the project. Eclipse Collections had over 800K downloads last month from Maven Central. I can't tell how many downloads there were from Eclipse RCP projects.

It's hard for me to tell from the VIATRA project whether they will be able to use Eclipse Collections after 11.1 now, since we have baselined development on Java 11 for the 12.0 release. Most of the pom files in VIATRA seem to be pointing to EC 10.2 as a dependency. Does the VIATRA project still depend on Java 8?

It's hard for me to tell from the VIATRA project whether they will be able to use Eclipse Collections after 11.1 now, since we have baselined development on Java 11 for the 12.0 release. Most of the pom files in VIATRA seem to be pointing to EC 10.2 as a dependency. Does the VIATRA project still depend on Java 8?

According to the release notes, the 2.8.x versions of VIATRA will be the last on Java 8.

https://projects.eclipse.org/projects/modeling.viatra/releases/2.8.0

Hi @fipro78, spending some time trying to understand the details and implications of this PR. A few competing goals we are trying to manage in the project in some sort of priority order.

1. Zero runtime dependencies and preferable zero reported dependencies
2. Works with JPMS
3. Works with OSGi

I'm hoping this PR helps address both 2 and 3, where 2 will have many more potential users of Eclipse Collections who may be impacted.

Some of the improvements you have made in this PR look and sound very nice, and I'd like to just shoot some ideas about to see if there is not some way that we can have all three benefits, with minimal pain to existing or future users of Eclipse Collections.

I see we had a previous dependency on the bnd-maven-plugin version 5.1.1, and this has been upgraded to 6.4.0. The new compile dependency is on 6.4.1 of biz.aQute.bnd.annotation. This is because we can't use 7.0 until we migrate to JDK 17 if I understand correctly.

I went and looked in mvnrepository for the projects that currently depend on bnd annotaiton. There is very small list for 6.4.1 reported here:

https://mvnrepository.com/artifact/biz.aQute.bnd/biz.aQute.bnd.annotation/6.4.1/usages

We can see how the bnd annotation dependency shows up for Apache Log4J Parent in mvnrepository here (under provided and managed dependencies):

https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j/2.23.1

This is the look we'd like to avoid, since we can't update this page in anyway alleviate any dependency concerns folks may have about dependencies.

There is a mechanism that Woodstox (uses 6.4.0) uses for identifying the dependency as "provided" and "optional" which impacts the display in mvnrepository. @motlin @nikhilnanivadekar @prathasirisha @mohrezaei

https://github.com/FasterXML/woodstox/blob/master/pom.xml#L191
https://mvnrepository.com/artifact/com.fasterxml.woodstox/woodstox-core/6.6.2

I took a look at Apache Log4J parent pom, and see they have quite a few bnd and osgi related properties.

https://github.com/apache/logging-parent/blob/main/pom.xml

Questions:

1. Is it possible to limit the bndtools annotation processing to our code generation module or limit the dependency somehow to the bnd-maven-plugin? I wouldn't be surprised if either or both of these make no sense, but just want to understand if there is anyway to move the dependency elsewhere so it is not in display by either eclipse-collections or eclipse-collections-api modules.
2. Can we code generate once and check in the generated artifacts and comment out the annotations (but leave them in for future). I realize this is less ideal, but Eclipse Collections is at a rather mature state already, and won't be changing too much so the maintenance cost should hopefully be minimal (note: possible famous last words where I have to eat them later).

Sorry to step in late here, but isn't this a limitation of maven? Effectively, it has no designation for a dependency that is required during compilation, but not runtime (such as class retention annotations).

Wouldn't an alternate approach be to publish a different pom file than the one that's used to compile (without the dependency, because it's not a real dependency)? I'm no maven expert so I'm not sure how to achieve that automatically (I know it's possible manually, as projects that don't even use maven for compilation can publish poms).

@donraab

Yes this PR helps address both 2 and 3.

Yes I added 6.4.0 because 7.0.0 needs Java 17 for the execution. You can also build using Java 17 and produce Java 11 compatible artifacts, but that would be a bigger change to your build setup that I didn't wanted to start.

To your first question, I don't think that is possible, because the annotations need to be known by the compiler.

To your second question, I already mentioned that this is possible. But as I also already explained, that would introduce a high risk in the maintenance. If you change anything at any time and don't be serious you will break things without knowing or even knowing how to fix it. And the PR already mentions a place where you have missed to cleanup everything and have incorrect leftovers. No, blaming, just an example why the generation is better in keeping things clean.

@mohrezaei
I have no experience if it is possible to have different pom files for building and publishing. I also don't know if we could change the compile dependency to a provided dependency. From the documentation, provided is also a compile time dependency, but probably not reported on the maven central page.

But I just want to give you something to think about. What you currently discuss is that you want to use a open source project to help to reduce the technical depth, improve the the maintainability and improve the options to consume your library. But you want to hide this from the rest of the world! Even if it is just a compile time dependency that has technically no impact for consumers of your library. It is just something written in maven central. And it is some kind of credit to the project that you would use to improve your project. Have you ever thought about how the maintainers of that project would think about the effort you put in to hide the usage? I mean we are all working in the OSS space, why not showing that we benefit from each other?
But this is my personal opinion. Please don't feel offended by this statement. I just want you to think about it.

@fipro78 We have no intention of hiding anything. As I said before, this is a deficiency in maven: it erroneously reports a jar as a dependency.

We're trying to provide correct dependency information in our public facing interface.

I also just re-read maven docs, and I'm guessing from their perspective, this should have a provided <scope> (which, IMHO, is not still erroneous: it's not a dependency, period, instead of something provided by the environment).

There was a similar discussion here.

eclipse/microprofile-config#716

Different topic for different annotations, but maybe the provided scope could work for you.

There is a mechanism that Woodstox (uses 6.4.0) uses for identifying the dependency as "provided" and "optional" which impacts the display in mvnrepository. @motlin @nikhilnanivadekar @prathasirisha @mohrezaei

https://github.com/FasterXML/woodstox/blob/master/pom.xml#L191 https://mvnrepository.com/artifact/com.fasterxml.woodstox/woodstox-core/6.6.2

The Jackson Woodstox project uses provided and optional set to true with biz.aQute.bnd.annotation. The project does the same with osgi.core.

@mohrezaei @motlin What do you think about just adding these to the dependency as @fipro78 has it now?

That would be the closest to correct you can get with maven with a single pom.xml.

I don't remember the PR very well anymore but in #1175 @donraab you went through some effort to create META-INF/services without adding a dependency on auto-service. I have another project where I use auto-service annotations heavily and like it, but I don't worry about dependencies there. As @mohrezaei says, provided & optional is the closest we can get to expressing the truth with maven's limited set of dependency scopes. It will show up that way in search engines as discussed. Eclipse Collections would be going from 0 dependencies to 1 optional dependency. biz.aQute.bnd.annotation has permissive licenses include Apache 2.0. I feel like it's up to you @donraab!

I don't remember the PR very well anymore but in #1175 @donraab you went through some effort to create META-INF/services without adding a dependency on auto-service.

I can't unring this particular bell, but it looks like this PR does quite a bit of that anyway with some file removals.

I feel like it's up to you @donraab!

I'm leaning toward the "Let's do this, add the optional dependency, and do an 12.0.0.M4 release and test the heck out of this on multiple projects, in the JPMS and OSGi ecosystems". I'm reading through some of the work and research that @fipro78 has done on the issues he linked in the start of this conversation, and feel like we have a lot of ground we can cover catching up with current recommended practice in the Eclipse Project ecosystem. I can't thank him enough for wading through this. Thank you again!

I'd like to hear from @nikhilnanivadekar and @prathasirisha who both did work with getting EC in shape with JPMS and also with sorting through some of the issues we had with ServiceProvider in the 11.0 release that we addressed in the 11.1 release. I feel like 11.1 has been super stable, and I want to know that even with the optional dependency defined, we can continue to consume Eclipse Collections 12.0 without any changes required to clients.