Access to the tycho .cache directory is not properly synchronized #663

urld · 2022-02-18T11:07:32Z

I am just upgrading from tycho 2.2.0 to 2.6.0 and now my CI build (using maven parallel builds with -T 1C) sporadically fails with this error during dependency resolution:

[INFO] Computing target platform for MavenProject: com.foo:bar:1.0.0-SNAPSHOT @ ...
[ERROR] Internal error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: IOException while extracting 'stax2-api-3.1.4.jar' from /opt/jenkins/workspace/MYPIPELINE/.repository/com/example/somedependency/1.2.3/somedependency-1.2.3.jar: /opt/jenkins/workspace/MYPIPELINE/.repository/.cache/tycho/somedependency-1.2.3.jar/stax2-api-3.1.4.jar -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: IOException while extracting 'stax2-api-3.1.4.jar' from /opt/jenkins/workspace/MYPIPELINE/.repository/com/example/somedependency/1.2.3/somedependency-1.2.3.jar
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:120)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: IOException while extracting 'stax2-api-3.1.4.jar' from /opt/jenkins/workspace/MYPIPELINE/.repository/com/example/somedependency/1.2.3/somedependency-1.2.3.jar
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method)
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance (Constructor.java:490)
    at java.util.concurrent.ForkJoinTask.getThrowableException (ForkJoinTask.java:600)
    at java.util.concurrent.ForkJoinTask.get (ForkJoinTask.java:1006)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.resolveProjects (TychoMavenLifecycleParticipant.java:164)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.afterProjectsRead (TychoMavenLifecycleParticipant.java:105)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:264)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

...

Caused by: java.nio.file.FileAlreadyExistsException: /opt/jenkins/workspace/MYPIPELINE/.repository/.cache/tycho/somedependency-1.2.3.jar/stax2-api-3.1.4.jar
    at sun.nio.fs.UnixException.translateToIOException (UnixException.java:94)
    at sun.nio.fs.UnixException.rethrowAsIOException (UnixException.java:111)
    at sun.nio.fs.UnixException.rethrowAsIOException (UnixException.java:116)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel (UnixFileSystemProvider.java:215)
    at java.nio.file.spi.FileSystemProvider.newOutputStream (FileSystemProvider.java:478)
    at java.nio.file.Files.newOutputStream (Files.java:219)
    at java.nio.file.Files.copy (Files.java:3066)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.copyStreamToFile (DefaultBundleReader.java:172)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.extractZipEntries (DefaultBundleReader.java:151)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.getEntry (DefaultBundleReader.java:126)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getNestedJarOrDir (OsgiBundleProject.java:547)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getBundleClasspath (OsgiBundleProject.java:513)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.resolveClassPath (OsgiBundleProject.java:231)
    at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:173)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:147)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183)
    at java.util.stream.SpinedBuffer.forEach (SpinedBuffer.java:250)
    at java.util.stream.Nodes$SpinedNodeBuilder.forEach (Nodes.java:1270)
    at java.util.stream.Nodes$InternalNodeSpliterator$OfRef.forEachRemaining (Nodes.java:1105)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:484)
    at java.util.stream.ForEachOps$ForEachTask.compute (ForEachOps.java:290)
    at java.util.concurrent.CountedCompleter.exec (CountedCompleter.java:746)
    at java.util.concurrent.ForkJoinTask.doExec (ForkJoinTask.java:290)
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec (ForkJoinPool.java:1020)
    at java.util.concurrent.ForkJoinPool.scan (ForkJoinPool.java:1656)
    at java.util.concurrent.ForkJoinPool.runWorker (ForkJoinPool.java:1594)
    at java.util.concurrent.ForkJoinWorkerThread.run (ForkJoinWorkerThread.java:177)

somedependency-1.2.3.jar is a proprietary lib which includes a bunch of other third party jars.

It looks like a concurrency issue somewhere in the dependency resolution when using parallel builds and multiple plugins have the same dependency but in a different target platform?

The text was updated successfully, but these errors were encountered:

laeubi · 2022-02-18T11:08:58Z

@urld please try out the current tycho snapshot build to check if your issue is solved there.

urld · 2022-02-18T12:28:08Z

There seem to be other regressions with 2.7.0-SNAPSHOT. I get e few errors during validate-classpath":

Unresolved requirement: Import-Package: javax.inject; version="[1.0.0,2.0.0)"
Unresolved requirement: Require-Bundle: javax.servlet

I'll have to check later whats wrong here with my configuration.

laeubi · 2022-02-18T14:08:18Z

I am just upgrading from tycho 2.2.0 to 2.6.0

You best upgrade to 2.5 first then to 2.6 (with non parallel) and then 2.7

laeubi · 2022-03-08T18:30:20Z

@urld could you provide any more information on this or does it works now for you?

urld · 2022-03-08T18:39:24Z

I was busy with other issues that were introduced with my target platform upgrade, so i could not test this with 2.7.0 yet.
But the bug was also present in 2.5.0 so i reverted back to 2.2.0 for now. I will try to find some time to get my build running with 2.7.0 tomorrow.

montanero · 2022-03-11T10:10:58Z

I'm experiencing the same problem with Version 2.7.0

laeubi · 2022-03-11T10:13:32Z

You you provide a reproducer or provide an integration-test to demonstrate the issue?

montanero · 2022-03-11T10:16:10Z

Could be difficult, b/c the problem only occurs sporadically in a fairly complex build.

laeubi · 2022-03-11T10:21:52Z

The it is difficult to analyze as well... could you test if the current tycho snapshot build (3.0.0-SNAPSHOT) might has fixes for this issue?

urld · 2022-03-11T10:23:27Z

I cant test with 2.7.0 because the changes for mixed reactor builds require me to change too much in the current build structure of my project which i cant do right now because of deadlines.

But a hint for the reproducer:
I think it needs to look something like this:

root-module
    project-a
         dependency: thirdparty.jar
    project-b
         dependency: thirdparty.jar
    project-c
         dependency: thirdparty.jar
    ...

Where thirdparty.jar is a jar that packages many other jars.

The more child projects that depend on thirdparty.jar, the more likely the error is to happen when the target platforms for each project are computed in parallel in the beginning.
From the error message the parallel target platform computations all try to extract the same thirdparty.jar to analyze its contents and since they extract to the same cache dir, they run into a conflict on the filesystem.

Edit: The error never happens on Windows for me, probably because the filesystem is too slow or file locking works differently there.

laeubi · 2022-03-11T10:27:25Z

I cant test with 2.7.0 because the changes for mixed reactor builds require me to change too much

You should not ned to adjust anything for an existing build so maybe you can explain / open a seperate ticket for that issue?

From the error message the parallel target platform computations all try to extract the same thirdparty.jar to analyze its contents and since they extract to the same cache dir, they run into a conflict on the filesystem.

Thanks this seems to explain it I haven't noticed that this is actually an OS error rather than an application error....

laeubi · 2022-03-11T10:29:37Z

The interesting codepath is this with nested jars of the same name:

Caused by: java.nio.file.FileAlreadyExistsException: /opt/jenkins/workspace/MYPIPELINE/.repository/.cache/tycho/somedependency-1.2.3.jar/stax2-api-3.1.4.jar
    at sun.nio.fs.UnixException.translateToIOException (UnixException.java:94)
    at sun.nio.fs.UnixException.rethrowAsIOException (UnixException.java:111)
    at sun.nio.fs.UnixException.rethrowAsIOException (UnixException.java:116)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel (UnixFileSystemProvider.java:215)
    at java.nio.file.spi.FileSystemProvider.newOutputStream (FileSystemProvider.java:478)
    at java.nio.file.Files.newOutputStream (Files.java:219)
    at java.nio.file.Files.copy (Files.java:3066)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.copyStreamToFile (DefaultBundleReader.java:172)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.extractZipEntries (DefaultBundleReader.java:151)
    at org.eclipse.tycho.core.osgitools.DefaultBundleReader.getEntry (DefaultBundleReader.java:126)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getNestedJarOrDir (OsgiBundleProject.java:547)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getBundleClasspath (OsgiBundleProject.java:513)
    at org.eclipse.tycho.core.osgitools.OsgiBundleProject.resolveClassPath (OsgiBundleProject.java:231)
    at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:173)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:147)

urld · 2022-03-11T10:38:49Z

I cant test with 2.7.0 because the changes for mixed reactor builds require me to change too much

You should not ned to adjust anything for an existing build so maybe you can explain / open a seperate ticket for that issue?

I think the problem is on my side here. I have a pom dependency on bundles that have a pom dependency on an eclipse bundle via maven central with scope=provided, assuming that the downstream build will provide it via a target platform. But due to the change in the behavior in pomDependency=consider this no longer works. I will need to change the upstream projects to build against the correct eclipse target platform aswell instead of picking random outdated api bundles from maven central.
I will create a separate issue when i run into any issues there.

laeubi · 2022-03-11T10:40:28Z

Alright, if you still thinks this is a valid use case, please provide an integration-test to demonstrate the issue and we might support this use case again.

montanero · 2022-03-14T13:49:01Z

But a hint for the reproducer: I think it needs to look something like this:

root-module
    project-a
         dependency: thirdparty.jar
    project-b
         dependency: thirdparty.jar
Where `thirdparty.jar` is a jar that packages many other jars.

That matches exactly my project's structure, therefor it is plausible that we have a common bug cause here.

laeubi · 2022-03-14T15:32:31Z

What I'm wondering: Is the content of thirdparty.jar the same or is it just the same name? And how is the dependency declared, is it a Bundle-Classpath-Entry?

urld · 2022-03-14T16:44:33Z

In my case its the same content. Its declared as a pom dependency and the manifests refer to it via Require-Bundle.
All the nested jars are declared via Bundle-ClassPath and some packages are exported by the thirdparty manifest.

The software component comes from a proprietary vendor. I don't really know why they do it that way, since all the nested jars are publicly available open source libs. But since their other bundles are hard wired to the thirdparty.jar bundle I cant work around the issue.

…roperly synchronized Signed-off-by: Christoph Läubrich <laeubi@laeubi-soft.de>

laeubi · 2022-03-15T08:08:42Z

@urld @wimjongman @montanero I created a patch that should appear in about an hour in the snapshots (see this build) please try out the current tycho snapshot build and check if this fixes your problem

synchronized Signed-off-by: Christoph Läubrich <laeubi@laeubi-soft.de>

laeubi added the needs info label Mar 8, 2022

laeubi changed the title ~~IOException during parallel build~~ Access to the tycho .cache directory is not properly syncronized Mar 11, 2022

laeubi added bug and removed needs info labels Mar 11, 2022

laeubi self-assigned this Mar 14, 2022

laeubi changed the title ~~Access to the tycho .cache directory is not properly syncronized~~ Access to the tycho .cache directory is not properly synchronized Mar 15, 2022

laeubi added a commit to laeubi/tycho that referenced this issue Mar 15, 2022

Fix eclipse-tycho#663 - Access to the tycho .cache directory is not p…

aea8a19

…roperly synchronized Signed-off-by: Christoph Läubrich <laeubi@laeubi-soft.de>

laeubi closed this as completed in 359c962 Mar 15, 2022

laeubi added this to the 2.7.1 milestone Mar 30, 2022

laeubi added a commit that referenced this issue Mar 30, 2022

Fix #663 - Access to the tycho .cache directory is not properly

8e5c5c8

synchronized Signed-off-by: Christoph Läubrich <laeubi@laeubi-soft.de>

laeubi mentioned this issue Apr 26, 2022

concurrent extracting of nested jars fails #917

Closed

Bananeweizen mentioned this issue May 29, 2022

ProvisionException already exists in repository #978

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Access to the tycho .cache directory is not properly synchronized #663

Access to the tycho .cache directory is not properly synchronized #663

urld commented Feb 18, 2022

laeubi commented Feb 18, 2022

urld commented Feb 18, 2022

laeubi commented Feb 18, 2022

laeubi commented Mar 8, 2022

urld commented Mar 8, 2022

montanero commented Mar 11, 2022

laeubi commented Mar 11, 2022

montanero commented Mar 11, 2022

laeubi commented Mar 11, 2022

urld commented Mar 11, 2022 •

edited

laeubi commented Mar 11, 2022

laeubi commented Mar 11, 2022

urld commented Mar 11, 2022

laeubi commented Mar 11, 2022

montanero commented Mar 14, 2022

laeubi commented Mar 14, 2022

urld commented Mar 14, 2022

laeubi commented Mar 15, 2022 •

edited

Access to the tycho .cache directory is not properly synchronized #663

Access to the tycho .cache directory is not properly synchronized #663

Comments

urld commented Feb 18, 2022

laeubi commented Feb 18, 2022

urld commented Feb 18, 2022

laeubi commented Feb 18, 2022

laeubi commented Mar 8, 2022

urld commented Mar 8, 2022

montanero commented Mar 11, 2022

laeubi commented Mar 11, 2022

montanero commented Mar 11, 2022

laeubi commented Mar 11, 2022

urld commented Mar 11, 2022 • edited

laeubi commented Mar 11, 2022

laeubi commented Mar 11, 2022

urld commented Mar 11, 2022

laeubi commented Mar 11, 2022

montanero commented Mar 14, 2022

laeubi commented Mar 14, 2022

urld commented Mar 14, 2022

laeubi commented Mar 15, 2022 • edited

urld commented Mar 11, 2022 •

edited

laeubi commented Mar 15, 2022 •

edited