Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Errors #6546

Open
Teicu opened this issue Mar 25, 2024 · 13 comments
Open

Update Errors #6546

Teicu opened this issue Mar 25, 2024 · 13 comments
Labels

Comments

@Teicu
Copy link

Teicu commented Mar 25, 2024

Hello,

Any idea why I keep getting these errors? Thank you

[INFO] Checking for updates [INFO] NVD API has 242,601 records in this update [INFO] Downloaded 10,000/242,601 (4%) [INFO] Downloaded 20,000/242,601 (8%) [INFO] Downloaded 30,000/242,601 (12%) [INFO] Downloaded 40,000/242,601 (16%) [INFO] Downloaded 50,000/242,601 (21%) [INFO] Downloaded 60,000/242,601 (25%) [INFO] Downloaded 70,000/242,601 (29%) [INFO] Downloaded 80,000/242,601 (33%) [INFO] Downloaded 90,000/242,601 (37%) [INFO] Downloaded 100,000/242,601 (41%) [INFO] Downloaded 110,000/242,601 (45%) [INFO] Downloaded 120,000/242,601 (49%) [INFO] Downloaded 130,000/242,601 (54%) [INFO] Downloaded 140,000/242,601 (58%) [INFO] Downloaded 150,000/242,601 (62%) [INFO] Downloaded 160,000/242,601 (66%) [INFO] Downloaded 170,000/242,601 (70%) [INFO] Downloaded 180,000/242,601 (74%) [INFO] Downloaded 190,000/242,601 (78%) [INFO] Downloaded 200,000/242,601 (82%) [INFO] Downloaded 210,000/242,601 (87%) [INFO] Downloaded 220,000/242,601 (91%) [INFO] Downloaded 230,000/242,601 (95%) [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

@Teicu Teicu added the question label Mar 25, 2024
@Teicu
Copy link
Author

Teicu commented Mar 25, 2024

[INFO] Checking for updates
[INFO] NVD API has 242,601 records in this update
[INFO] Downloaded 10,000/242,601 (4%)
[INFO] Downloaded 20,000/242,601 (8%)
[INFO] Downloaded 30,000/242,601 (12%)
[INFO] Downloaded 40,000/242,601 (16%)
[INFO] Downloaded 50,000/242,601 (21%)
[INFO] Downloaded 60,000/242,601 (25%)
[INFO] Downloaded 70,000/242,601 (29%)
[INFO] Downloaded 80,000/242,601 (33%)
[INFO] Downloaded 90,000/242,601 (37%)
[INFO] Downloaded 100,000/242,601 (41%)
[INFO] Downloaded 110,000/242,601 (45%)
[INFO] Downloaded 120,000/242,601 (49%)
[INFO] Downloaded 130,000/242,601 (54%)
[INFO] Downloaded 140,000/242,601 (58%)
[INFO] Downloaded 150,000/242,601 (62%)
[INFO] Downloaded 160,000/242,601 (66%)
[INFO] Downloaded 170,000/242,601 (70%)
[INFO] Downloaded 180,000/242,601 (74%)
[INFO] Downloaded 190,000/242,601 (78%)
[INFO] Downloaded 200,000/242,601 (82%)
[INFO] Downloaded 210,000/242,601 (87%)
[INFO] Downloaded 220,000/242,601 (91%)
[INFO] Downloaded 230,000/242,601 (95%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
at org.owasp.dependencycheck.App.runScan(App.java:262)
at org.owasp.dependencycheck.App.run(App.java:194)
at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349)
... 7 common frames omitted
[INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours.
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[ERROR] One or more fatal errors occurred
[ERROR] Error updating the NVD Data
[ERROR] No documents exist

@crizulm
Copy link

crizulm commented Mar 25, 2024

Hey !
I am getting the same error

[INFO] Checking for updates [INFO] NVD API has 242.601 records in this update [INFO] Downloaded 10.000/242.601 (4%) [INFO] Downloaded 20.000/242.601 (8%) [INFO] Downloaded 30.000/242.601 (12%) [INFO] Downloaded 40.000/242.601 (16%) [INFO] Downloaded 50.000/242.601 (21%) [INFO] Downloaded 60.000/242.601 (25%) [INFO] Downloaded 70.000/242.601 (29%) [INFO] Downloaded 80.000/242.601 (33%) [INFO] Downloaded 90.000/242.601 (37%) [INFO] Downloaded 100.000/242.601 (41%) [INFO] Downloaded 110.000/242.601 (45%) [INFO] Downloaded 120.000/242.601 (49%) [INFO] Downloaded 130.000/242.601 (54%) [INFO] Downloaded 140.000/242.601 (58%) [INFO] Downloaded 150.000/242.601 (62%) [INFO] Downloaded 160.000/242.601 (66%) [INFO] Downloaded 170.000/242.601 (70%) [INFO] Downloaded 180.000/242.601 (74%) [INFO] Downloaded 190.000/242.601 (78%) [INFO] Downloaded 200.000/242.601 (82%) [INFO] Downloaded 210.000/242.601 (87%) [INFO] Downloaded 220.000/242.601 (91%) [INFO] Downloaded 230.000/242.601 (95%) [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

Any idea?

@nicollasteixeira
Copy link

This is happening practically daily and apparently no solution has been made available yet.

@defaultbranch
Copy link

this looks similar to #6547 ?

@shekharclover
Copy link

shekharclover commented Mar 26, 2024

i keep getting this isuue logs on console
[WARN] NVD API request failures are occurring; retrying request for the 10 time

@jeremylong
Copy link
Owner

Apparently the NVD API - which is not controlled by this project is having issues. Not much I can do.

@Teicu
Copy link
Author

Teicu commented Mar 26, 2024

Apparently the NVD API - which is not controlled by this project is having issues. Not much I can do.

Hey Jeremy, appreciate the heads up. By the way, is there any alternative method for updating? I'm keen on using this tool, I've been struggling for the past two days trying to scan a local file.

Cheers

@jeremylong
Copy link
Owner

@tarrinho
Copy link

tarrinho commented Mar 26, 2024

The strange thing is that if I do it inside a docker (linux) it breaks, but if I run in my MacOS machine, it works.

[INFO] Download Started for NVD CVE - Modified
[INFO] Download Complete for NVD CVE - Modified (7130 ms)
[INFO] Processing Started for NVD CVE - Modified
[INFO] Processing Complete for NVD CVE - Modified (3873 ms)
[INFO] Begin database maintenance
[INFO] Updated the CPE ecosystem on 133015 NVD records
[INFO] End database maintenance (8264 ms)
[INFO] Skipping RetireJS update since last update was within 24 hours.
[INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours.
[INFO] Begin database defrag
[INFO] End database defrag (3642 ms)
[INFO] Check for updates complete (32053 ms)
[INFO]

@Teicu
Copy link
Author

Teicu commented Mar 26, 2024

The strange thing is that if I do it inside a docker (linux) it breaks, but if I run in my MacOS machine, it works.

[INFO] Download Started for NVD CVE - Modified [INFO] Download Complete for NVD CVE - Modified (7130 ms) [INFO] Processing Started for NVD CVE - Modified [INFO] Processing Complete for NVD CVE - Modified (3873 ms) [INFO] Begin database maintenance [INFO] Updated the CPE ecosystem on 133015 NVD records [INFO] End database maintenance (8264 ms) [INFO] Skipping RetireJS update since last update was within 24 hours. [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [INFO] Begin database defrag [INFO] End database defrag (3642 ms) [INFO] Check for updates complete (32053 ms) [INFO]

that's pretty weird, I have this problem with my Mac

@nicollasteixeira
Copy link

@jeremylong Thanks for letting us know. I saw your comment about caching nvd data but how do I point out to dependency check that it should use these local vulnerabilities? Is it possible to download the vulnerabilities and whenever a scan is done it is based on this local database?

@vptp
Copy link

vptp commented Mar 26, 2024

Investigating this further, it appears to relate to this issue

I am having the same error where the client attempts to fetch the same index 5 times then gives up.
If I use curl to download the index that it is having trouble with, then manually parse the json with objectMapper.readValue(json, CveApiJson20::class.java), I get the exception:
Unrecognized field "cvssMetricV40" (class io.github.jeremylong.openvulnerability.client.nvd.Metrics), not marked as ignorable (3 known properties: "cvssMetricV30", "cvssMetricV31", "cvssMetricV2"])
The JSON I downloaded from NVD does indeed contain a cvssMetricV40 property.

It appears that any JSON parsing error in the client results in just retrying the download again until it gives up after 5 tries.

The issue linked has a PR to ignore any unknown properties in the JSON which would likely resolve this issue too.

@jeremylong
Copy link
Owner

I just merged #6554 - so if people are having an issue due to the cvssMetricsV40 - that will be fixed with the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants