-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dependency Check gets stuck when used #3408
Comments
ODC has a timeout that was by default 30 minutes. I've seen a few reports of this issue recently so I just increased the timeout to 60 minutes. You can change this locally using:
|
Tried that but it still gives the same problem. :/ When I use the -X flag for debug output it doesnt timeout, do you have any suggestions to why that may be? |
We are experiencing the same issue. |
Same issue here after upgrade from 6.1.7 to 6.2.0. |
We got this as last Maven debug output:
After that we get after some time
(unfortunately the debug log output is broken, I have to rerun it with |
OK. Now the build continues, but fails with the issue reported in #3410 . |
We are experiencing the same issue as the original reporter (using ODC 6.2.0 release on Windows server 2016, via maven plugin). ODC hangs and fails if used without -X. But seems to finish and exit cleanly when using -X. |
This morning (just now) I again have the problem of a stuck dependency check, this time executed via |
You can always increase this using:
or
|
@jeremylong In a not working pipeline it just keeps waiting and waiting untill it errors or I cancel the pipeline. |
Have tried increasing the timeout to 60, 120 and 180 minutes but it doesnt fix the problem |
I don't see this as a timeout setting related issue, something changed from 6.1.6 to 6.2.0 that fully hangs ODC. |
I think that something is leaking db connections and the when it hits 8 used connections, dbcp2 waits forever until something else kills it. I've reproduced this with a silly amount of dependencies (when testing a BOM). Maven plugin, issue only showed up with 6.2.0 I can see the issue if I set the connection pool to have a timeout in org.owasp.dependencycheck.data.nvdcve.DatabaseManager
The issue "goes away" if I put no limit on the connection pool size.
Silly amount of dependencies
|
We are facing the same issue. Scanning just stuck on NVD CVE Analyser Phase after upgrade to 6.2.0 We are also scanning a bit more than a silly amount of dependencies. |
I don't know if this is the reason, but I've noticed that the Connection and PreparedStatement are opened in a try-with-resources, but then the result set it often closed in a finally block. From what I understand this means that the order the those objects get closed goes
"A try-with-resources statement can have catch and finally blocks just like an ordinary try statement. In a try-with-resources statement, any catch or finally block is run after the resources declared have been closed." |
I have updated a couple of things with #3419 - However, I have not been able to re-produce the issue where we hang after the FP analyzer completes. Server: MySQL 8.0.25
My local maven repository contains way more then the above listed silly amount of dependencies:
So to me - the real issue is the dependency bundling analyzer. I have thoughts on how to resolve this as it is currently something like an O(N^2) operation (my big o notation/evaluation is a bit rusty) and I know we can split it up into chunks and parallelize the analysis. However, that is a different problem then this ticket - I digress... @sellersj as you have been able to reproduce the issue - does it actually go away with the changes made in #3419? Note that there are DB schema changes (added a transaction around two operations - but this only affects updating, not the DB reads that appear to cause things to hang). |
I’ll get back to you with a reproducible project. Sorry, I should have made
one right away.
My tests were using
H2
Maven plugin
Java 8
Maven 3.8.1
A clean database, update as a separate step before.
I did test it with the changes to the closing of ResultSet but it still
didn't work for me. I have not checked if my last test has the transaction
code.
If I change the DS pool to unlimited it would pass, and then if I removed
that setting it would pass. Like a cached result? Not sure.
This feels like a threading race condition in the DS pool to me, but that’s
a wild guess. I looked through the code and things seem clean and nothing
jumped out at me.
Thanks for digging into this.
…On Fri, Jun 4, 2021 at 8:22 AM Jeremy Long ***@***.***> wrote:
I have updated a couple of things with #3419
<#3419> - However, I
have not been able to re-produce the issue where we hang after the FP
analyzer completes.
Server: MySQL 8.0.25
Driver: mysql-connector-java-8.0.21.jar
Command:
$dependency-check.sh --disableCentral --disableOssIndex --connectionString "jdbc:mysql://localhost/dependencycheck?serverTimezone=UTC" -o . -l odc.log -s ~/.m2/repository/
My local maven repository contains way more then the above listed silly
amount of dependencies: Dependencies Scanned: 9846 (6753 unique). A
trimmed version of the output was:
[INFO] Analysis Started
[INFO] Finished Archive Analyzer (45 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (18 seconds)
[INFO] Finished Assembly Analyzer (6 seconds)
[INFO] Finished Node.js Package Analyzer (0 seconds)
[INFO] Finished Dependency Merging Analyzer (15 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (2 seconds)
[INFO] Finished CPE Analyzer (34 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (4 seconds)
[INFO] Finished RetireJS Analyzer (15 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (4 seconds)
[INFO] Finished Dependency Bundling Analyzer (26332 seconds)
[INFO] Analysis Complete (27066 seconds)
So to me - the real issue is the dependency bundling analyzer. I have
thoughts on how to resolve this as it is currently something like an O(N^2)
operation (my big o notation/evaluation is a bit rusty) and I know we can
split it up into chunks and parallelize the analysis. However, that is a
different problem then this ticket - I digress...
@sellersj <https://github.com/sellersj> as you have been able to
reproduce the issue - does it actually go away with the changes made in
#3419 <#3419>? Note
that there are DB schema changes (added a transaction around two operations
- but this only affects updating, not the DB reads that appear to cause
things to hang).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3408 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABCJTGRJR2WH4POEE4SLTTRDAQ5ANCNFSM45WZKREQ>
.
|
If you were using H2 - one thing that might have been causing these issue could be related to a fix I just pushed to the mysql branch. See #3416 (comment) The updates around the transactions only affected mysql. I likely need to replicate the transactions on those two calls in oracle, ms sql, etc. - as it can cause slightly weird (but not catastrophic) things to occur. Everything will still work correctly - just might have some unexpected duplicate rows in the database. |
I updated my test project with a reproducible (on my laptop) test. The logs from 6.2.0 and 6.2.1-SNAPSHOT (what ever is in nexus snapshot) are included. It hangs on "Cache miss for" so I kill the script and it writes the final 3 lines of "Resetting Indexed File" I'm not sure if you wanted to set the MaxWaitMillis setting to the connection pool. It might hide the error because the plugin resumes, but the process wouldn't hang indefinitely. |
I am unable to observe the issue on my machine. I've tested 6.2.0 and some of the newer snapshot version. |
I think that I've figured out the reason why I'm seeing this behaviour.
|
My setup is internal db (I guess that is H2?), clean ODC maven plugin install, empty local maven repo, java 8. I see 8 errors from the NVD CVE Analyzer after the 1 hour timeout which would suggest/confirm the theory of lockup of 8 threads. Maybe to reproduce this, you need the local cache misses (i.e. an empty local maven repo)? |
Here is a thread dump from the problem occurring on a 12 core machine. |
We just released 6.2.1 - this might fix part of the underlying issue. However, I suspect it will not fully resolve the hanging issue. |
Same issue with similar thread dump on 6.2.1 |
Still see the stuck issue on 6.2.1
|
One tricky way is to scan packed file (e.g zip all the jars). This way worked for me when using v6.2.1 command line version.
|
I can confirm this issue in My setup:
I purged the db and then let the check (homebrew install) run for way over an hour until it was finished and this was the resulting console output (shortenend):
Since I have 8 cores with 16 threads (hyper threading) and I see 16 connection errors @sellersj idea of having all available threads connecting to the DB and maxing out the maximum allowed parallel connections seems plausible. |
I'm seeing the hang in 6.2.0 and 6.2.1 also. Version 6.1.6 is working fine. |
Some more information: I'm hesitant sharing the complete log as it may contain sensitive information (didn't check it all). But I can share the last normal log statement plus the following exceptions. So this is the end of the log: The console output was the same as in my last post. Looking at the log as someone who doesn't know the codebase @sellersj idea sounds more and more to be the root cause. |
- until jeremylong/DependencyCheck#3408 is fixed - we should then rather set a configured version in jahia-parent/pom.xml
- until jeremylong/DependencyCheck#3408 is fixed - we should then rather set a configured version in jahia-parent/pom.xml
Hanging in 6.2.1. $ mvn test org.owasp:dependency-check-maven:check [DEBUG] Begin Analysis of '/home/somedude/.m2/repository/org/springframework/spring-web/5.2.3.RELEASE/spring-web-5.2.3.RELEASE.jar' (NVD CVE Analyzer) |
Seems 8 is a magical number here. The last library logged differs from one run to the next but it always hangs after 8 cache miss log statements with mvn -X option. [DEBUG] Starting NVD CVE Analyzer |
#3408 reusing the same connection before returning it to the pool
I ran into this issue with the gradle-plugin of dependency check when upgrading from 6.1.6 to 6.2.0. When, according to the lock, "Finished False Positive Analyzer (0 seconds)" has happened, the execution is stuck. So I did a thread dump via jvisualvm and there are two peculiarities where 8 threads, respectively, are waiting to lock monitors:
(Code base is of dependency check 6.2.0.)
(I can provide a full thread dump if needed.) As it appears to be a concurrency issue I looked at the changes between 6.1.6 and 6.2.0:
Another difference that may or may not contribute to this issue is the fact that in |
Thanks @sellersj for the PR. I'll push a release shortly. |
Hello, been trying to use dependency check on one of my projects, but when it gets to one of the modules and the NVD CVE Analyzer is about to start it gets stuck for about 30min and then this message pops up.
Been getting the same message on other projects aswell. I'm using the maven command "mvn org.owasp:dependency-check-maven:6.2.0-SNAPSHOT:check" to run it. Anyone know a fix to this?
EDIT : If i run the same command but with the -X flag (debug output) for some reason it doesnt get stuck
"mvn org.owasp:dependency-check-maven:6.2.0-SNAPSHOT:check -X"
The text was updated successfully, but these errors were encountered: