Threads created by Jetty no longer run as the Subject that Jetty was started as #6396
Comments
…t that Jetty was started as
|
We will look into this, but in the meanwhile is there any other information you can supply? Is the thread a request handling-handling thread or has another thread been spun up during request handling? Are you using jetty's SPNEGO integration? |
|
Also, please can you provide some links as to which of your tests that are failing as we're not familiar with your codebase so some navigation help would be useful, as would a stack trace or two. |
|
Just followed the repro steps (I used jetty-9.4.42) and whilst there is a lot of test output, I can't see anything that is an obvious error and no output that mentions SPNEGO. Edit: Ah! Sorry, I downloaded calcite-master instead of calcite-avatica-master! |
|
Yes, this is a request handling thread. Avatica server does not create threads on its own. Avatica SPNEGO implementation is using Jetty's SPNEGO integration, but the some of the Jetty classes are inherited and extended to work around some bugs/limitations in Jetty's SPENGO handling. This is where SPNEGO gets configured in Avatica: The failing tests are: Unfortunately the stack traces aren't telling a lot, all you can see in them is a generic Kerberos failure: 021-06-10 18:33:05,285 [Test worker] INFO - JDBC URL jdbc:avatica:remote:url=https://localhost:49984;authentication=SPNEGO;serialization=PROTOBUF;truststore=/Users/stoty/workspaces/apache-phoenix/calcite-avatica/server/build/avatica-test.jks;truststore_password=avaticasecret |
|
OK, as you're already subclassing jetty classes, can you subclass the If that works, I'll look at adding the ability to set a |
|
Copying my reply here from the PR:
|
|
Gentle reminder that we are happy to take issues, and even better PRs, for things like this. |
|
@joakime Thank you. Using that constructor works, even if it was a bit awkaward to figure out all the defaults. |
|
If the Jetty team agrees that the new behaviour is the correct one, then I guess there's not much left to do with this issue. |
|
For future reference, this is the custom ThreadFactory solution for Avatica: Thanks again for the prompt and very constructive help from the team. |
|
We too have just suffered from this problem (though not related to Avatica). We use Jetty as an Embedded WebServer in a Java Server using the Spark Java microframework. The “GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400)“ is an issue that can have many causes. Identifying the cause can be like looking for a needle in several haystacks! It took us a long time and much testing and experimentation to realise the root cause was not a Kerberos / SPNEGO issue, but down to the Subject security context in which Jetty had been launched being lost. i.e. because the security context is lost, the Subject cannot be found; thus the Subject’s credentials (in our case Kerberos Keys) are not available, and SPNEGO tokens cannot be authenticated as a consequence. We were able to replicate the issue with in throwaway project that launched Spark in the context of a simple non-Kerberos Subject. Once we had discovered that, it took us a little more hunting to prove that an updated Jetty Version was the cause. For others who might land on this thread: How did we narrow down the issue to a Jetty Version change? As the problem occurred on customer site, where we could not do debugging from the Eclipse IDE, we decided to patch the key JRE classes in the module java.security.jgss using the java 9+ setting --patch-module The patched classes did not change the behaviour: we just added log out and stack-traces so we could then use desk-checking to work out the route taken through the JRE security code, and thus why an older setup worked, but a newer one did not. The critical point in the JRE security code is: https://github.com/openjdk/jdk11u-dev/blob/c1411113b396f468963a1deacc3b57ed366e735a/src/java.security.jgss/share/classes/sun/security/jgss/GSSUtil.java#L332 We now have a similar function in our own code which allows us to test the subject / security context at key points, in particular before we even initiate the GSSContext. Having understood this, and proved that our code for Kerberos Authentication and to launch Spark / Jetty was identical, we then created a minimal viable Spark project, and used Maven to quickly upgrade and downgrade Jetty versions. Having done that, a quick google with “jetty Subject doAs” quickly led us to this thread Our workaround is to do the SPNEGO authentication explicitly in the Subject’s Security Context instead of starting Jetty in the Context. i.e the Authentication is directly in a Subject.doAs{} structure. Edit: As we use Jetty Indirectly via Spark, we use our own Kerberos authentication code rather than Jetty's. However, as the Kerberos problem is a follow-on problem, any Kerberos implementation will be affected likewise. |
|
@FlyingSheepOnSailfish thanks for the details. What did you do to fix your situation on Spark? Did you use a custom ThreadFactory as @stoty did? |
|
@FlyingSheepOnSailfish are you using the deprecated |
|
@joakime I only discovered this thread once we already had a viable workaround, and had even nailed down the exact Jetty version. So our workaround is different. I can now configure my application to do either of: @sbordet No, we use our own custom SPNEGO code. We use Jetty "under-the-hood" of the Spark micro-framework, so we are a little limited as to which parts of Jetty we can easily use / configure. We do have some classes over-loading the Spark classes interacting with Jetty, but not for SPNEGO authentication. I will look into the ConfigurableSpnegoAuthenticator, thanks for bringing that to my attention. As I have a viable workaround (plus some others above to investigate), and have identified the root cause I am happy: For me the issue is closed. I commented mainly for the benefit of others who may face the same issue. |
|
Having established that the problem was not really a Kerberos problem, but caused by the Subject context getting lost, we called the function below at several points in our code to narrow down where. This function was inspired by one in the JRE class GSSUtils. Essentially it asks the question "in the context of which subject, if any, are we running at this point?". public static boolean testSubjectFound(Subject subject, String spn) {
final AccessControlContext acc = AccessController.getContext();
try {
Boolean found = AccessController.doPrivileged(new PrivilegedExceptionAction<Boolean>() {
public Boolean run() throws Exception {
Subject accSubj = Subject.getSubject(acc);
if (accSubj != null) {
if (accSubj.equals(subject)) {
// we have found a subject, and it is our subject
System.out.println("[testSubjectFound] Expected subject found with SPN '" + spn + "'.");
return true;
} else {
// we have another subject
for (Principal p : accSubj.getPrincipals()) {
String accSPN = p.getName();
System.out.println("[testSubjectFound] Unexpected subject found with SPN '" + accSPN + "'.");
}
return false;
}
} else {
// no subject found!
System.out.println("[testSubjectFound] No Subject found! We are no longer in the context of '" + spn + "'. Authentication of tokens will fail!");
return false;
}
}
});
return found;
} catch (PrivilegedActionException pae) {
System.out.println("[testSubjectFound] Exception testing for subject!");
return false;
}
} |
|
Oh no. Those are currently deprecated in Java 17 and will be removed entirely in Java 18. |
|
@joakime tx, we have just moved from Java 8 to Java 11: that code snippet was inspired by Java 11 JRE code. I can look into what newer JREs do instead. I will have to read this 8-) |
|
@FlyingSheepOnSailfish do keep us informed with your findings after researching newer JREs. If there is something we should be doing better with our managed threads, then do ask! |
|
Here is the summary of some quick research into future JDKs and the changes they plan for Security Manager et al. https://openjdk.java.net/jeps/411
https://bugs.openjdk.java.net/browse/JDK-8267108 Meanwhile, OpenJDK 16 |
Jetty version
9.4.37.v20210219 and later 9.4.x versions
Java version/vendor
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
OS type/version
Darwin 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
Description
The thread creation change in #5859 has caused a regression.
The threads created by Jetty no longer run as the Subject that Jetty was started as.
This was discovered when trying to build Avatica with Jetty 9.4.37.v20210219
See https://issues.apache.org/jira/browse/CALCITE-4646
How to reproduce
./gradlew clean testThe text was updated successfully, but these errors were encountered: