Duplicate resources found with Maven / Surefire patching test resources into the module path.

[michael-simons]

Find a simple reproducer attached (mpcg.zip): Maven / IntelliJ and friends will patch test classes and test resources into the main module when there's a module-info.java in the project (--patch-module mpcg=/Users/msimons/Projects/misc/mpcg/target/test-classes -).

When this happens, classgraph will discover resources multiple times.

I don't repeat the code from the reproducer here,

.
├── pom.xml
└── src
    ├── main
    │   ├── java
    │   │   ├── foo
    │   │   │   └── App.java
    │   │   └── module-info.java
    │   └── resources
    └── test
        ├── java
        │   └── foo
        │       └── AppTest.java
        └── resources
            └── stuff
                └── whatever.cypher

10 directories, 5 files

In this scenario,

ScanResult scanResult = new ClassGraph().acceptPaths("stuff").scan();
ResourceList allResources = scanResult.getAllResources().nonClassFilesOnly()

will give me whatever.cypher twice

[file:/Users/msimons/Projects/misc/mpcg/target/classes/stuff/whatever.cypher,
    file:/Users/msimons/Projects/misc/mpcg/target/test-classes/stuff/whatever.cypher]

This can be fixed by

<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>3.0.0-M7</version>
				<configuration combine.self="append">
					<!-- Add this and the test works… -->
					<useModulePath>false</useModulePath>
				</configuration>
			</plugin>

but this is maybe not desired to skip the module path completely.

I am unsure if its an Maven issue or Classpath issue, though.

cc @sbrannen you expressed interest in this on twitter.

[lukehutch]

Hey @michael-simons,

I'm sorry it took me so long to get to this bug.

Can you please attach some verbose logs here? I don't know where the duplicate paths are coming from.

I have no idea how module patching works. Does it effectively insert another module into the same module layer, with the patched files? If so, then yes, there are potentially two modules with the same files in them, so ClassGraph would already be doing the right thing, since getAllResources() is supposed to return all same-named resources across different classpath elements.

[Sineaggi]

I just hit this while trying to modularize https://github.com/webjars/webjars-locator-core

The test webjars are being patched in and then reported as resources. In this case however, the duplicates aren't real duplicates since the target/classes/META-INF/resources/* directory doesn't exit.

My maven target directory looks like this

[Sineaggi]

Does it effectively insert another module into the same module layer, with the patched files?

From what I can tell, target/classes/ is the "main" lookup location, and target/test-classes/ is part of the "patcher's" lookup location.

there are potentially two modules with the same files in them

This made me wonder, it looks like Intellij and maven are patching the main module, but also adding test-classes to the classpath. In that case, I would expect in both cases the path to be target/test-classes, one for the module and one for the classpath.

In this case it's buggy tho because it doesn't choose the correct full path for the patched module, it uses the main target/classes directory.

[lukehutch]

I had to guess a lot of things about how classfile visibility worked within modules, because it's so complex and was very poorly documented at the time I wrote module support into ClassGraph. So it's highly probable that there are behavioral mismatches between ClassGraph and whatever the JVM does.

I'm swamped right now trying to get a company off the ground, so I won't have a lot of time to look into this. Any help you can give to track down what's going on would be appreciated. Thanks!

[Sineaggi]

I've got what I think is a fix. Instead of constructing a URI like we do for classpath resources, we instead use the module reader (+ a bit of reflection) to call the ModuleReader#find which will give us a proper URI, handing patched resources.

I've opened a pr here #793

[lukehutch]

The PR looks great to me, but @Sineaggi left a comment that there might be a better fix, so I'll wait on further work on this. Thanks!

[lukehutch]

@Sineaggi any updates?

[Sineaggi]

@lukehutch sorry, I've not come back to this code in a while.

As it stands, I'm not sure that all of classgraph is fixed with this change. However, it's more correct with the change in #793, and I'm not sure it's possible to patch getClasspathElementURI to behave correctly in all cases (in a module the canonical classpath element uri is the main uri minus any patched uris).

[lukehutch]

I merged #793, since it looks like the right thing to do. @michael-simons please comment back here if this doesn't solve the problem.

[lukehutch]

@michael-simons I tried running your mpcg.zip reproducer project under JDK 21 with the latest ClassGraph (4.8.163). It runs without a problem from Eclipse (JUnit 5), even using the old version of ClassGraph that you specified (4.8.149). However, I can't seem to get your project running from the commandline using mvn test -- I get the error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project mpcg: Compilation failure
[ERROR] /tmp/mpcg/src/main/java/module-info.java:[2,27] module not found: io.github.classgraph

Any ideas?

[lukehutch]

I'll reopen this since I can't reproduce the original issue, so I can't verify that this is fixed on my end.