RetireJS checks frequently fail due to corrupt jsrepository.json file

[hakanai]

Describe the bug

I frequently get this stack trace failing the dependency check:

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
        Failed to initialize the RetireJS repo: `/Users/tester/.gradle/dependency-check-data/4.0/jsrepository.json` appears to be malformed. Please delete the file or run the dependency-check purge command and re-try running dependency-check.
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:719)
        at org.owasp.dependencycheck.Engine$analyzeDependencies$2.call(Unknown Source)
        at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:88)
        ... 85 more

I delete the file manually and that does make the next check pass, but then randomly it corrupts itself again in the future.

Version of dependency-check used

The problem occurs using version 5.3.2.1 of the Gradle plugin.

Log file

I'm unable to find any logs other than the stack trace I get in the build itself.

To Reproduce

To determine.

It happens frequently in our multi-project build, but not at all in one of my simpler single-project builds, so I'm thinking it might require a multi-project build to reproduce it, but haven't determined anything specific yet, and our own build is private so I can't just post it. :(

When I see the build running, I notice that the checks are running on multiple subprojects at the same time. This could be a hint - maybe one of them reads the file while another one is actively writing data into it?

Expected behavior

1. Any updates to any file should be done atomically so that it isn't possible for someone to find the file in a half-initialised state.

2. If the file is corrupt, it should be deleted and then treated as if it didn't exist.

Additional context

Add any other context about the problem here.

[jeremylong]

The log file would be the gradle build log file - so adding --debug to the build would include more details from ODC. Can you share the configuration of ODC in your build?

[brettcooper]

I am experiencing the same issue starting today. As suggested, I even removed the entire ~/.m2/repository/org/owasp/dependency-check-utils/5.3.2/ directory to try to fix the corrupted jsrepository.json file, but I get the same error for every build:

[INFO] --- dependency-check-maven:5.3.2:aggregate (dependency-check) @ oauth-service ---
[INFO] Checking for updates
[INFO] Skipping NVD check since last check was within 4 hours.
[INFO] Skipping RetireJS update since last update was within 24 hours.
[INFO] Check for updates complete (1549 ms)
[INFO] 

Dependency-Check is an open source tool performing a best effort analysis of 3rd party dependencies; false positives and false negatives may exist in the analysis performed by the tool. Use of the tool and the reporting provided constitutes acceptance for use in an AS IS condition, and there are NO warranties, implied or otherwise, with regard to the analysis or its use. Any use of the tool and the reporting provided is at the user’s risk. In no event shall the copyright holder or OWASP be held liable for any damages whatsoever arising out of or in connection with the use of this tool, the analysis performed, or the resulting report.


[INFO] Analysis Started
[INFO] Finished Archive Analyzer (0 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (0 seconds)
[ERROR] ----------------------------------------------------
[ERROR] .NET Assembly Analyzer could not be initialized and at least one 'exe' or 'dll' was scanned. The 'dotnet' executable could not be found on the path; either disable the Assembly Analyzer or add the path to dotnet core in the configuration.
[ERROR] ----------------------------------------------------
[WARNING] No lock file exists - this will result in false negatives; please run `npm install --package-lock`
[INFO] Finished Node.js Package Analyzer (0 seconds)
[INFO] Finished Dependency Merging Analyzer (0 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (4 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (0 seconds)
[ERROR] Exception occurred initializing RetireJS Analyzer.
[INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (1 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Analysis Complete (8 seconds)
[WARNING] 

One or more dependencies were identified with known vulnerabilities in oauth-service:

log4j-api-2.12.1.jar (pkg:maven/org.apache.logging.log4j/log4j-api@2.12.1, cpe:2.3:a:apache:log4j:2.12.1:*:*:*:*:*:*:*) : CVE-2020-9488


See the dependency-check report for more details.


[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:23 min
[INFO] Finished at: 2020-05-15T11:15:33-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.owasp:dependency-check-maven:5.3.2:aggregate (dependency-check) on project oauth-service: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR]         Failed to initialize the RetireJS repo: `/Users/brett/.m2/repository/org/owasp/dependency-check-utils/5.3.2/../../dependency-check-data/4.0/jsrepository.json` appears to be malformed. Please delete the file or run the dependency-check purge command and re-try running dependency-check.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[henrykuijpers]

Indeed, I just saw the very same issue happening here. Started googling it and found this Github issue. For us it happened in our Jenkins server, which has multiple executors that can execute builds.

I think a few builds were running in parallel and indeed one build was writing the file, while another was probably reading it in-the-middle-of-a-write.

Probably some locking is required while reading/writing the related json file? It should be impossible to read a partially written JSON file.

[jeremylong]

@brettcooper sorry for the late reply - your error has nothing to do with the H2 lock file - the error you are seeing is related to NodeJS and the fact that there is no package.lock file.

@henrykuijpers can't believe I did not put a lock file on the json repository. That is a very quick update and I'll have that implemented shortly.

[jeremylong]

@brettcooper sorry - I meant package-lock.json.

[henrykuijpers]

Thank you so much @jeremylong ! Looking forward to seeing the fix. :)