Fully working test on 33 CVEs dataset #342

sacca97 · 2022-11-03T09:39:54Z

HUGE PR.

updated to python 3.10
Implemented twin commits, closes Add feature to indicate if a commit has a "twin" #147
Using basemodel again (pydantic)
Fixed the logger, improved log aesthetics and readability
Various ptimizations:
- all commit IDs, timestamp, parent, message and changed files are collected with a single call to git executable.
- github and jira issues are collected using their respective APIs (GITHUB token required via .env file)
- Better word extraction and filtering based on tests conducted on the small dataset.
Closes:
backend container fails, missing spacy language model #339
Matching of sec-relevant keywords in JIRA pages misses some terms #341
Improve retrieval of JIRA issues #334
Compare "relevant" keywords between the advisory text and the commits messages #331
Backend cannot process request when too many candidates are provided #326
Change default behaviour so that all code files are considered #336

updating...

With this merge we could create a new release.

…default is set

…mmits

… and abstracted the commits eliminating some intra-dependencies

…python postgres class to dynamically create the insert statement based on the data

…ithout spacifying with the cli

…mmits

sacca97 added 30 commits October 10, 2022 16:32

keywords extraction test using spacy related to #331, fixes #329

dc06c76

added spacy en_core_web_sm to python.yml for gh workflows

bc66866

fixed json report generation to handle sets and file extensions when …

a389c11

…default is set

flake8 failure

ae9ddfb

initial lsh implementation

e57af9b

initial lsh implementation

516d35c

Merge branch 'twin-commits' of github.com:SAP/project-kb into twin-co…

56a7785

…mmits

add datasketch to requirements.txt

43cd0f2

rules refactoring, now every rule is a sub-class of the class Rule

a71ca7d

finished rules refactoring, added Minhash to Commit: reworked backend…

e7a55ac

… and abstracted the commits eliminating some intra-dependencies

code cleanup, report code refactoring

a2e315e

added find_twin method, testing phase

43ebadb

minor addons and cleanup

9a58018

Major refactoring, cleanup and optimization. WIP

64ce70a

continuing refactoring and fixes

c93db29

almost finished Git, raw_commit and exec refactoring and cleaning

efbc0e7

other refactoring, fixed logging level and refactored it

f3e4906

old files cleanup

72ae9f9

fixed typos, updated pytest files, modified db structure and updated …

f46304e

…python postgres class to dynamically create the insert statement based on the data

modified logger class to wrap the default python logger

93dd2bc

fixed logger, updated some default parameters

6d4fa1c

adjusted some imports to match new refactoring, adjusted some types

46df41a

re-fixed logger

4103d09

updated method names and tests, added verb extraction from text

d44ef50

testing small relevant extensions version

09ebb94

added automatic filtering for different file extension at git level w…

157879f

…ithout spacifying with the cli

minor fixes to raw commit class

56f4890

finished git rework, added setup.py to allow external imports

84b9952

updated setup.py

17c1ff4

updated makefile and .gitignore

ce29db8

sacca97 and others added 22 commits October 25, 2022 15:58

update to match get_hunks

f9c7eb6

update to python 3.10

bc32ed4

updated html report format

42ed7b9

now jira issues are accessed using XML API

12eb250

updated rule messages and fully functioning twin commits

f8bbba4

reworked communication with backend

528d4d0

fixed bug in retrieving preprocessed commits

db23d0c

skip github when fetching references, we already do it separately

48168b2

added github apis, fine-tuned word extraction

c04e9db

allow single-hunk commits

5d0a87c

code cleanup

3764a31

fine-tuned security words extraction

d2d6a89

kinda-final lsh implementation

9f6c261

back to full basemodel

b2bd8d8

adapted tests

4170ca8

minor changes

4623541

Merge branch 'main' into twin-commits

ff056ae

changed gh workflows for python 3.10

2bae2bc

Merge branch 'twin-commits' of github.com:SAP/project-kb into twin-co…

1d1d1d9

…mmits

Merge branch 'twin-commits' of github.com:SAP/project-kb into twin-co…

f2c8c2c

…mmits

updated github actions for python

0cbde17

updated github actions for python

ee7c1ad

sacca97 marked this pull request as ready for review November 3, 2022 10:29

copernico approved these changes Nov 3, 2022

View reviewed changes

copernico merged commit 7284e34 into main Nov 3, 2022

copernico deleted the twin-commits branch November 3, 2022 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fully working test on 33 CVEs dataset #342

Fully working test on 33 CVEs dataset #342

sacca97 commented Nov 3, 2022 •

edited

Fully working test on 33 CVEs dataset #342

Fully working test on 33 CVEs dataset #342

Conversation

sacca97 commented Nov 3, 2022 • edited

sacca97 commented Nov 3, 2022 •

edited