High CPU in qtp HttpInput.nextInterceptedContent()
#7712
Comments
|
Are you using logback, or logback access perhaps? Or something similar that doesn't do Improper RequestLog usage is the most common reason for your high CPU. |
|
Note that a consumeAll fix was implemented in Jetty 9.4.40.v20210413 as part of Issue #6168 Also note that Java 1.8.0_162 expired in January 16, 2018 and should not be used in production right now. btw ... ... that is your security system testing for the Log4Shell bug. |
|
The latest release, 9.4.45.v20220203 has fixes for logback access and bad request log implementations as well (specifically to address high CPU issues) |
|
Since Jetty 9.4.7, several bugs were fixed in multiple code area whose impact could solve your reported issue.
Upgrading to the latest 9.4 release and try again. |
|
Hi Joachim, thank you very much for all your comments and details. Indeed this application seems to write @WebFilter(filterName = "RequestLogFilter", urlPatterns = { "/*" })
final class RequestLogFilter implements Filter {
private static final Logger LOG = getLogger(RequestLogFilter.class);
private static final Logger LOG_ERROR = getLogger(RequestLogFilter.class.getPackage().getName());
private static final String USER_SECURITY = "user";
private static final String USER_NOT_SET = "UserNotSet";
private static final String NO_USER_AGENT = "UserAgentNotSet";
private static final String ACCESS_LOGGED = "AccessLogged";
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
if (response instanceof HttpServletResponse && request instanceof HttpServletRequest) {
Stopwatch timer = Stopwatch.createStarted();
try {
chain.doFilter(request, response);
} catch (Exception e) {
LOG_ERROR.error("Error occurred on: {}", ((HttpServletRequest)request).getRequestURI(), e);
throw e;
} finally {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String user = firstNonNull((String)httpRequest.getAttribute(USER_SECURITY), USER_NOT_SET);
httpRequest.removeAttribute(USER_SECURITY);
if (httpRequest.getAttribute(ACCESS_LOGGED) == null && (LOG.isTraceEnabled() || LOG.isInfoEnabled())) {
// In case the incoming request has NOT been matched to a particular resource by JAX-RS runtime
// the access logging falls into responsibility of the web filter.
String remoteAddress = httpRequest.getRemoteAddr();
String protocol = httpRequest.getProtocol();
String httpMethod = httpRequest.getMethod();
if (LOG.isTraceEnabled()) {
LOG.trace("Received {} - \"{} {} {}\"", remoteAddress, user, protocol, httpMethod);
}
if (LOG.isInfoEnabled()) {
String requestURI = httpRequest.getQueryString() == null
? httpRequest.getRequestURI()
: httpRequest.getRequestURI() + "?" + httpRequest.getQueryString();
LOG.info("{}-{} - \"{} {} {}\" {} \"{}\" {} ms",
remoteAddress, user,
protocol, httpMethod, requestURI,
((HttpServletResponse)response).getStatus(),
firstNonNull(httpRequest.getHeader(USER_AGENT), NO_USER_AGENT),
timer.stop().elapsed(MILLISECONDS));
}
}
}
} else {
throw new ServletException(
format("Servlet error occurred in the filter: %s",
RequestLogFilter.class.getCanonicalName()));
}
}
@Override
public void init(FilterConfig filterConfig) {
// Do nothing init not a necessity
}
@Override
public void destroy() {
// Do nothing destroy not a necessity
}
}Can you see anything wrong in the above? In the mean time I will study the issues you linked to, in the hope I can use them to convince our development team to upgrade. Btw. I'm aware of the Java version. Fortunately this isn't a production system, however I'll investigate if we have any production systems still running with outdated java versions. |
|
Does your configured logback pattern include references to request parameters? Also note that a Filter based approach for RequestLog is not going to work reliably. Please look into using LOG.info("{}-{} - \"{} {} {}\" {} \"{}\" {} ms",
remoteAddress, user,
protocol, httpMethod, requestURI,
((HttpServletResponse)response).getStatus(),
firstNonNull(httpRequest.getHeader(USER_AGENT), NO_USER_AGENT),
timer.stop().elapsed(MILLISECONDS));You would have the following configuration ... // Output request log events to a slf4j logger (you can route slf4j to log4j2 or even log4j1 easily)
Slf4jRequestLogWriter logRequestLogWriter = new Slf4jRequestLogWriter();
// The name of the logger to write the events to, you can use a logger configuration to sift/split the events for this named logger
// into a separate file easily enough
logRequestLogWriter.setLoggerName("com.boekhold.accesslog");
// The output format string
String requestLogFormat = "%{client}a-%u - \"%r\" %s \"%{User-Agent}i\" %D";
// Tie together the output/writer and the format string together
CustomRequestLog customRequestLog = new CustomRequestLog(logRequestLogWriter, requestLogFormat);
// Set the RequestLog behavior on the server level (so that all requests/responses can be seen, regardless of servlet rules or errors)
server.setRequestLog(customRequestLog);See javadoc for |
I've searched our code base for any filters used, and cannot find any direct references to request body and/or parameters. However, a few of the filters delegate to 3rd-party libraries, and I cannot be sure what's happening inside those. Your point on Filters not being suitable for implementing an access.log is taken. I'll bring this info to the development team and hopefully they'll agree to migrate to the CustomRequestLog method. Thanks Joakim for all the explanations, details and suggestions. Since I don't control the code of this application, I won't be able to try out all of your suggestions immediately, but I'll try my hardest to convince the dev team to take action. I'll close this issue for now, but will update in a comment once/if we've confirmed the fix. |
The question was about your logback configuration, usually in the form of a |
|
Joakim, although "access logging" is done through logback, I don't think logback is involved in any way except to just write a string. The logic is fully contained inside that Here are the relevant bits of our <configuration scan="true" scanPeriod="5 seconds">
<appender name="ACCESS_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${app.log}/http/access.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>${app.log}/http/access_%i.log</FileNamePattern>
<MinIndex>1</MinIndex>
<MaxIndex>10</MaxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>20MB</MaxFileSize>
</triggeringPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<Pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] %msg%n</Pattern>
</encoder>
</appender>
<logger level="INFO" additivity="false" name="package.path.to.RequestLogFilter">
<appender-ref ref="ACCESS_LOG" />
</logger>
</configuration> |
|
logback-access has several bugs surrounding the assumptions of what is valid from the HttpServletRequest and HttpServletResponse. See: |
Jetty version(s)
9.4.7.v20170914, embedded
Java version/vendor
(use: java -version)java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
OS type/version
Windows Server 2012, Version 6.2 build 9200
Description
We have an internal application running, for which I observed high CPU usage while idle. Upon inspecting with JVisualVM/JMC, I noticed that a single "qtp" thread is consuming most CPU. The application is otherwise idle, no requests are currently coming in.
This thread seems to be looping inside
org.eclipse.jetty.server.HttpInput.nextInterceptedContent():The stacktrace of this thread is:
Line 467 is a call to
_content = _inputQ.poll();, so it appears thatnextInterceptedContent()never exits thewhileloop started at Line 434.The only way to recover from this is to shutdown and restart the application.
Upon further research, I have discovered that this issue is triggered during an automated PAN test that our internal security team is running nightly against "all open ports on all servers" inside our data center. The
access.logfile unfortunately does not provide enough detail to identify the exact test that is triggering this issue. Some possible candidates are:These are the only messages in our log files related to this specific thread, listed in order. The last line therefore seems to be the most likely cause for this issue.
The only other issue logged against Jetty that references
nextInterceptedContent()is #6973, however my issue does not seem to match the description of #6973.I'm trying to find evidence that this issue is fixed in newer (>9.4.7.v20170914) releases of Jetty, so that I can try to convince the team owning this application to upgrade to a newer version of the (internal) framework on which this application has been built (the latest version of this framework includes Jetty 9.4.43).
How to reproduce?
The text was updated successfully, but these errors were encountered: