Internal APIs stricter access issue on Java11

[NingZhang-e]

JDK11, javassist 3.24.0-GA
Unit test:

public void testJava11() throws Exception {
    	ProxyFactory factory = new ProxyFactory();
        factory.setSuperclass(HashMap.class);
        
        HashMap e = (HashMap)factory.create(null, null, new MethodHandler() {
            @Override
            public Object invoke(Object self, Method thisMethod,
                    Method proceed, Object[] args) throws Throwable {
                return proceed.invoke(self, args);
            }
        });
    }

Output:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access using Lookup on javassist.util.proxy.DefineClassHelper (file:/C:/Opensource/javassist/target/classes/) to class java.util.HashMap
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.DefineClassHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
java.lang.RuntimeException: Class not in same package as lookup class: java.util.HashMap has no permission to define the class
	at javassist.util.proxy.ProxyFactory.createClass3(ProxyFactory.java:614)
	at javassist.util.proxy.ProxyFactory.createClass2(ProxyFactory.java:587)
	at javassist.util.proxy.ProxyFactory.createClass1(ProxyFactory.java:523)
	at javassist.util.proxy.ProxyFactory.createClass(ProxyFactory.java:449)
	at javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:789)
	at javassist.util.proxy.ProxyFactory.create(ProxyFactory.java:774)
	at test.javassist.proxy.ProxySimpleTest.testJava11(ProxySimpleTest.java:372)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:252)
	at junit.framework.TestSuite.run(TestSuite.java:247)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)
Caused by: javassist.CannotCompileException: Class not in same package as lookup class: java.util.HashMap has no permission to define the class
	at javassist.util.proxy.DefineClassHelper.toClass(DefineClassHelper.java:300)
	at javassist.util.proxy.DefineClassHelper$Java11.defineClass(DefineClassHelper.java:48)
	at javassist.util.proxy.DefineClassHelper.toClass(DefineClassHelper.java:263)
	at javassist.util.proxy.FactoryHelper.toClass(FactoryHelper.java:136)
	at javassist.util.proxy.ProxyFactory.createClass3(ProxyFactory.java:603)
	... 25 more

If change HashMap class to a customized class, this issue will be gone.
Does this mean no way to proxy Java internal class on Java11? Any other way could be implemented in javassist as one alternative solution?

[skybber]

Hi, we have the same problem in the HotswapAgent. It's not able to make proxy on jdk classes. I did some description of it there:
HotswapProjects/HotswapAgent#274

[skybber]

We've fixed this issue in HotswapAgent using ClassLoader.define(..) preferentially before using MethodHandles.Lookup().define, if ClassLoader.define(..) is accessible. It solved our problem with proxying JDK classes.

[NingZhang-e]

@skybber Nice fix! skybber/HotswapAgent@faa8af8

[chibash]

@skybber Thanks for your suggestion!

We've fixed this issue in HotswapAgent using ClassLoader.define(..) preferentially

Do you mean java.lang.ClassLoader#defineClass(..)? There seems to be no #define(..).
I expect ClassLoader#defineClass() will cause a WARNING message. Is this correct?
A reason to use Lookup#defineClass was to supress this message.

[skybber]

@chibash : you are right. I mean - defineClass(). It is protected by default, but it can be tested if the operation is allowed, just before the defineClassMethod.setAccessible(true);. The drawback of Lookup#defineClass is the impossibility to define proxy of JDK classes. In our case it throws javassist.proxy.proxyObject ClassNotFoundException.

[chibash]

Thank you. I suppose a good fix is to explicitly pass a Lookup object to ProxyFactory#create().
This problem is not a Javassist bug. Its real reason is that JDK 11 prohibits a reflective access
to a closed module such as jdk.internal. We're looking for a trick to bypass this restriction.

[chibash]

I've fixed the bug in a different way I mentioned above since it was a sort of Javassist bug.
@NingZhang-Ericsson, @skybber, could you try the latest commit?

[NingZhang-e]

It works well for me. 👍 Great thanks! @chibash
And I also tried ProxyFactory#createClass(Lookup), it also works for me. :-D

[skybber]

Looks better, but in my case the proxied class name starts with jdk.internal. not java. , could you extend it to allow jdk.* packages as well?

[skybber]

If I modify the line in ProxyFactory to:

        if (basename.startsWith("java.") || basename.startsWith("jdk.") || onlyPublicMethods)

it works. I didn't study the code too much... But is there any other solution how to fix it without enumerate any other possible package in which it could have the same problem?

[chibash]

@skybber your fix is correct. I committed it.
Which jdk.* class do you want to make a proxy for?

[skybber]

jdk.internal.URLClassPath, HotswapAgent use proxy to patch URLClassLoader.

[NingZhang-e]

Close the issue. Wait for next release.

[chibash]

@skybber there is no class named jdk.internal.URLClassPath. It should be jdk.internal.loader.URLClassPath.

Can you really make a proxy by Javassist for jdk.internal.loader.URLClassPath?
In my test code,

Class<?> cc = jdk.internal.loader.URLClassPath.class;

This statement throws a runtime exception.

[skybber]

Yes, we're using modified JVM, that allows enhanced redefinition https://github.com/TravaOpenJDK/trava-jdk-11-dcevm , it makes some exports for some jdk modules for bytecode instrumentation.

https://github.com/HotswapProjects/openjdk-jdk11u/commit/d104058f7d21c10d4f6afe94bdfc445494a2cd2e#diff-ffd1efbb5a6b442e36861f5ee0ede5c1R4296

[chibash]

I see. I was wondering whether the fix really solved your problem.

[nivertius]

We also encountered this bug, and using snapshot version from master fixes the problem. Are there any plans to release 3.24.1-GA with this fix?

[chibash]

What about releasing by the upcoming holiday season?
3.24.0 was released just a month ago.

[nivertius]

We'd really would like release as soon as possible, because this prevents us from upgrading to JDK 11.

[chibash]

I've made the 3.24.1 release. It will be uploaded to maven soon.