Fix: Restoring lucene index build during startup & allow rebuild through UI #2200

syalioune · 2022-11-29T02:15:23Z

Description

Fix : Restoring lucene index build during startup by having a dedicated listener

The following features/fixes are introduced as part of this PR

Index healthcheck at startup using a servlet listener. Healthcheck consist of index directory existence check and corruption check using lucene API. Any failure trigger rebuild of corresponding index
A new configurable background task to perform health and consistency check. Healthcheck is the same one performed during startup. Concistency check consist of computing a delta ratio between DB and Lucene and comparing it to a configurable threshold. Any failure trigger rebuild of corresponding index
Any Lucene CorruptIndexException will trigger a rebuild of the corresponding index
A REST API (and corresponding GUI admin screen) is provided to manually rebuild the index if need be

#2235 is created to reduce as much as possible delta between DB and lucene that feature 2 above mitigate.

Addressed Issue

#2104

Additional Details

Related frontend PR DependencyTrack/frontend#338

Checklist

I have read and understand the contributing guidelines
This PR fixes a defect, and I have provided tests to verify that the fix is effective
This PR implements an enhancement, and I have provided tests to verify that it works as intended
~~- [ ] This PR introduces changes to the database model, and I have added corresponding update logic~~
~~- [ ] This PR introduces new or alters existing behavior, and I have updated the documentation accordingly~~

sonatype-lift · 2022-11-29T02:24:27Z

src/main/java/org/dependencytrack/search/IndexManager.java

+        CPE(Cpe.class),
+        VULNERABLESOFTWARE(VulnerableSoftware.class);
+
+        private Class<?> clazz;


ImmutableEnumChecker: enums should be immutable: 'IndexType' has non-final field 'clazz'

Suggested change

private Class<?> clazz;

private final Class<?> clazz;

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

nscuro

Thanks for the PR @syalioune! 🚀

A few suggestions and nitpicks 😛

src/main/java/org/dependencytrack/model/ConfigPropertyConstants.java

src/main/java/org/dependencytrack/resources/v1/SearchResource.java

src/main/java/org/dependencytrack/search/IndexManager.java

nscuro · 2022-12-06T10:26:57Z

@syalioune Please rebase with master for the build to work again. The failure cause was fixed in #2241.

…d listener A REST API is also exposed to allow index rebuild through the GUI. See DependencyTrack#2104 Automatic periodic consistency check with database are performed if enabled Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>

…d listener Takint into account review comments Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>

…d listener Fixing unit tests. Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>

sonatype-lift · 2022-12-06T14:34:31Z

src/main/java/org/dependencytrack/search/IndexManager.java

+                    checkIndex.close();
+                }
+            } catch (IOException e) {
+                throw new RuntimeException(e);


Finally: If you return or throw from a finally, then values returned or thrown from the try-catch block will be ignored. Consider using try-with-resources instead.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

nscuro

Thanks for yet again a stellar PR @syalioune! 🙌

…dencyTrack#2200) * Fix: Restoring lucene index build during startup by having a dedicated listener A REST API is also exposed to allow index rebuild through the GUI. See DependencyTrack#2104 Automatic periodic consistency check with database are performed if enabled Signed-off-by: Alioune SY <sy_alioune@yahoo.fr> * Fix: Restoring lucene index build during startup by having a dedicated listener Takint into account review comments Signed-off-by: Alioune SY <sy_alioune@yahoo.fr> * Fix: Restoring lucene index build during startup by having a dedicated listener Fixing unit tests. Signed-off-by: Alioune SY <sy_alioune@yahoo.fr> Signed-off-by: Alioune SY <sy_alioune@yahoo.fr> Fixes DependencyTrack#2104 Signed-off-by: mulder999 <nospam099-github@yahoo.com>

sonatype-lift bot reviewed Nov 29, 2022

View reviewed changes

nscuro mentioned this pull request Nov 29, 2022

moving indices to their own initializer #2201

Closed

3 tasks

syalioune force-pushed the fix/allowing_lucene_index_rebuild branch from 790b845 to 7b5b1cf Compare December 5, 2022 01:51

syalioune marked this pull request as ready for review December 5, 2022 01:55

nscuro requested changes Dec 5, 2022

View reviewed changes

syalioune added 2 commits December 6, 2022 13:21

Fix: Restoring lucene index build during startup by having a dedicate…

e798ba7

…d listener Takint into account review comments Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>

syalioune force-pushed the fix/allowing_lucene_index_rebuild branch from 880cbb6 to e798ba7 Compare December 6, 2022 12:21

nscuro mentioned this pull request Dec 6, 2022

Progressive inconsistencies between Dependency Track database and lucene indexes #2235

Open

2 tasks

Fix: Restoring lucene index build during startup by having a dedicate…

808fe10

…d listener Fixing unit tests. Signed-off-by: Alioune SY <sy_alioune@yahoo.fr>

sonatype-lift bot reviewed Dec 6, 2022

View reviewed changes

nscuro approved these changes Dec 6, 2022

View reviewed changes

nscuro merged commit c40c117 into DependencyTrack:master Dec 6, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jan 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Restoring lucene index build during startup & allow rebuild through UI #2200

Fix: Restoring lucene index build during startup & allow rebuild through UI #2200

syalioune commented Nov 29, 2022 •

edited

sonatype-lift bot Nov 29, 2022

nscuro left a comment

nscuro commented Dec 6, 2022

sonatype-lift bot Dec 6, 2022

nscuro left a comment

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

Fix: Restoring lucene index build during startup & allow rebuild through UI #2200

Fix: Restoring lucene index build during startup & allow rebuild through UI #2200

Conversation

syalioune commented Nov 29, 2022 • edited

Description

Addressed Issue

Additional Details

Checklist

sonatype-lift bot Nov 29, 2022

Choose a reason for hiding this comment

nscuro left a comment

Choose a reason for hiding this comment

nscuro commented Dec 6, 2022

sonatype-lift bot Dec 6, 2022

Choose a reason for hiding this comment

nscuro left a comment

Choose a reason for hiding this comment

syalioune commented Nov 29, 2022 •

edited