feat: Add binary version parsing for nodejs, php and python #6524

kovacs-levent · 2024-04-19T10:28:58Z

Description

Implement binary version parsing for at least key binaries. The feature should be extended upon, but first round, I'm going to be focusing stuff which matters for my use-case (PHP, NodeJS, Python)... I'm keeping Java binary parsing since it's already been implemented by laurentdelosieresmano in the related PR👍 Since go-dep-parser was moved to this Repo, I'm reopening the PR and adding Python dep parsing.

Related issues

Trivy is not scanning standalone java / nodejs / PHP binaries #1064 - It was prematurely closed due to inactivity and without any support from maintainers to implement the feature request

Related PRs

Parse java php nodejs binary version go-dep-parser#35 - Old PR for the same feature request from 3 years ago, stuck in review

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

CLAassistant · 2024-04-19T10:29:03Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Levente Kovacs seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

kovacs-levent · 2024-04-19T12:47:52Z

I signed the cla... Not sure why it's not triggering as signed. On my forked branch, every test passed, so if you trigger the tests it should be good.

kovacs-levent · 2024-04-19T13:32:30Z

Hmm, while all tests are passing, I do not see the new binary results when I build the trivy from source and run the same experiment as in #6457, I'm not sure why that is the case.

EDIT: Guess I realized that I not only need to implement the parsing, but the analyzer logic as part of fanal. I'll try that.

kovacs-levent · 2024-04-20T01:32:37Z

I have spent some time with this. Now python version is detected correctly by trivy and added into the sbom in a way that xeol detects it properly according to my use case.

I looked into implementing vulnerability scanning as well for standalone binaries, but I think at first only having it part of sboms is a step forward.

There are a few problems regarding implementing this with standalone binaries:

It's not an OS package, hence the whole PR. But that also means I can't implement the vuln scanning as part of the ospkg detector library. When I tried that, trivy started to correctly detect the vulns in the binaries, but it also discarded everything else due to conflicting OS versions (if I implement it as ospkg detector driver, then I just lost all dpkg vulns for example).
It's also not really a langpkg. but this could be solved with some extra code as part of the driver logic and without messing more with the core. However, none of the ecosystems in trivy-db are really fit for querying vulnerabilities in these standalone binaries. The best bet would be NVD I thought.
BUT NVD does not have an interface to query it in trivy-db. This means that I can't query NVD's information about the standalone binary, neither OS package sources.
A Get function could be implemented for NVD, but I suspect that the best way to query it is through CPE2.3 vectors. For standalone-binaries, this is easy to generate correctly I think, but afaik, trivy does not support CPE for complexity reasons (it's also not in the SBOMs).

The only way I see to implement this nicely is by adding CPE support and also putting together a Get function for the NVD vulnsrc linked in 3rd point. Other way: just hack together an NVD get interface since it'd be used only for the scanner as a fallback in case of generic binary is detected, not for regular vulnerability scanning.

Now that I know how to implement these things, I think I can put together the remaining parts (Java, PHP, NodeJS binary as part of package detection and SBOMs) in the upcoming days and the PR would be ready.

kovacs-levent · 2024-04-20T10:54:55Z

I have added support for PHP and Nodejs standalone binaries and it works like a charm with the sboms. I'm dropping support for Java because I found that it's harder to correctly determine the package type (JRE vs. JDK), and I won't go down the rabbithole since it's not part of my usecase.

Only issue I discovered yesterday is that this will break lots of integration tests which I'll have to adjust those accordingly. After I'll adjust integration tests, I think this PR can be merged.

DmitriyLewen

Hello @kovacs-levent
Thanks for this PR.

I left some comments. Some these comments same for nodejs, php and python.

Also parsers are very similar.
I think we can merge them to 1 package. Something like this:
parser:

├── executable
│   ├── exe.go
│   ├── nodejs
│       ├── parse.go
│       ├── parse_test.go
│       └── testdata
│   ├── python
│       ├── parse.go
│       ├── parse_test.go
│       └── testdata
│   └── php
│       ├── parse.go
│       ├── parse_test.go
│       └── testdata

about analyzers:
looks like we can add new logic here - https://github.com/aquasecurity/trivy/blob/main/pkg/fanal/analyzer/executable/executable.go

If a php/python/npm binary is found, use its parser and add the library.

wdyt?

DmitriyLewen · 2024-04-22T04:30:38Z

go.mod

@@ -124,6 +124,7 @@ require (
 	github.com/alecthomas/chroma v0.10.0
 	github.com/antchfx/htmlquery v1.3.0
 	github.com/apparentlymart/go-cidr v1.1.0
+	github.com/aquasecurity/go-dep-parser v0.0.0-20240213093706-423cd04548a5


go-dep-parser in Trivy now - https://github.com/aquasecurity/trivy/tree/main/pkg/dependency/parser

DmitriyLewen · 2024-04-22T04:57:07Z

pkg/dependency/parser/python/binary/exe.go

+		return nil, err
+	}
+
+	if bytes.HasPrefix(data, []byte("\x7FELF")) {


What about windows and macos formats?

DmitriyLewen · 2024-04-22T04:58:27Z

pkg/dependency/parser/python/binary/parse.go

+	}
+
+	// Python's version pattern is [NUL]3.11.2[NUL]
+	re := regexp.MustCompile(`^\d{1,4}\.\d{1,4}\.\d{1,4}[-._a-zA-Z0-9]*$`)


Can you add a comment with a link where you get this regex?

DmitriyLewen · 2024-04-22T04:59:31Z

pkg/dependency/parser/python/binary/parse.go

+
+	var libs []types.Library
+	libs = append(libs, types.Library{
+		ID: packageID(name, vers),


You use dependent.ID once. So you can just use it here.

Suggested change

ID: packageID(name, vers),

ID: dependency.ID(ftypes.PythonGeneric, name, version),

DmitriyLewen · 2024-04-22T05:08:31Z

pkg/dependency/parser/python/binary/parse.go

+		}
+	}
+
+	return "python", vers


Is there a way to determine package name?
Looks like it might be confusing if all detected binaries are named "python"

DmitriyLewen · 2024-04-22T05:14:46Z

pkg/fanal/analyzer/language/python/binary/binary_test.go

+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			a := pythonBinaryAnalyzer{}
+			fileInfo, err := os.Lstat(tt.filePath)


Looks like we can use 1 file and change names, permissions, etc. in test.
Then we can reduce the number of copies of test files.

DmitriyLewen · 2024-04-22T05:18:06Z

pkg/scanner/langpkg/scan.go

@@ -65,7 +65,6 @@ func (s *scanner) Scan(target types.ScanTarget, _ types.ScanOptions) (types.Resu
 		}

 		logger := log.WithPrefix(string(app.Type))
-


Looks like we don't need this change.

DmitriyLewen · 2024-04-22T05:44:33Z

pkg/dependency/parser/python/binary/exe.go

@@ -0,0 +1,82 @@
+// Ported from https://github.com/golang/go/blob/e9c96835971044aa4ace37c7787de231bbde05d9/src/cmd/go/internal/version/exe.go


It looks like Go 1.22 doesn't have this file. Can you please update the link and double check - we may need to add some changes.

DmitriyLewen · 2024-04-22T06:10:58Z

pkg/fanal/analyzer/language/python/binary/binary.go

+	pythonLibNameRegex := regexp.MustCompile("^libpython[0-9]+(?:[.0-9]+)+[a-z]?[.]so.*$")
+	pythonBinaryNameRegex := regexp.MustCompile("(?:.*/|^)python(?P<version>[0-9]+(?:[.0-9]+)+)?$")


will be great if you add some comment about these regex (e.g. link to docs)

DmitriyLewen · 2024-04-22T06:12:36Z

pkg/fanal/analyzer/const.go

+	TypePip       	  Type = "pip"
+	TypePipenv    	  Type = "pipenv"
+	TypePoetry    	  Type = "poetry"
+	TypePythonGeneric Type = "python"


looks like we can use executable for new binaries

kovacs-levent · 2024-04-27T10:27:52Z

Thanks for the feedback, I will make the changes when I have the time, this week’s been busy, but I’ll try to make progress when I have the time.

Add binary version parsing for java, nodejs, php and python

105f39e

kovacs-levent requested review from knqyf263 and DmitriyLewen as code owners April 19, 2024 10:28

Levente Kovacs added 5 commits April 19, 2024 12:34

Add parse test for Python binary

b6027e8

go mod tidy

5b3eec8

lint via gci + fix lint errors

3ead37c

Remove := because of linter

82ae5f5

Fix parser to grab correct match index

407ebd8

Levente Kovacs added 3 commits April 19, 2024 15:32

Normalize to binary go package

8e8443c

Implement standalone python binary scanning in fanal

526cd87

Python version parsing

1a1425f

kovacs-levent marked this pull request as draft April 20, 2024 08:37

Remove java and add nodejs and php support for standalone binaries

35a0a66

kovacs-levent changed the title ~~feat: Add binary version parsing for java, nodejs, php and python~~ feat: Add binary version parsing for nodejs, php and python Apr 20, 2024

DmitriyLewen reviewed Apr 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add binary version parsing for nodejs, php and python #6524

feat: Add binary version parsing for nodejs, php and python #6524

kovacs-levent commented Apr 19, 2024 •

edited

CLAassistant commented Apr 19, 2024

kovacs-levent commented Apr 19, 2024

kovacs-levent commented Apr 19, 2024 •

edited

kovacs-levent commented Apr 20, 2024 •

edited

kovacs-levent commented Apr 20, 2024

DmitriyLewen left a comment

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

DmitriyLewen Apr 22, 2024

kovacs-levent commented Apr 27, 2024

	ID: packageID(name, vers),
	ID: dependency.ID(ftypes.PythonGeneric, name, version),

		@@ -65,7 +65,6 @@ func (s *scanner) Scan(target types.ScanTarget, _ types.ScanOptions) (types.Resu
		}

		logger := log.WithPrefix(string(app.Type))

		@@ -0,0 +1,82 @@
		// Ported from https://github.com/golang/go/blob/e9c96835971044aa4ace37c7787de231bbde05d9/src/cmd/go/internal/version/exe.go

		pythonLibNameRegex := regexp.MustCompile("^libpython[0-9]+(?:[.0-9]+)+[a-z]?[.]so.*$")
		pythonBinaryNameRegex := regexp.MustCompile("(?:.*/\|^)python(?P<version>[0-9]+(?:[.0-9]+)+)?$")

feat: Add binary version parsing for nodejs, php and python #6524

Are you sure you want to change the base?

feat: Add binary version parsing for nodejs, php and python #6524

Conversation

kovacs-levent commented Apr 19, 2024 • edited

Description

Related issues

Related PRs

Checklist

CLAassistant commented Apr 19, 2024

kovacs-levent commented Apr 19, 2024

kovacs-levent commented Apr 19, 2024 • edited

kovacs-levent commented Apr 20, 2024 • edited

kovacs-levent commented Apr 20, 2024

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kovacs-levent commented Apr 27, 2024

kovacs-levent commented Apr 19, 2024 •

edited

kovacs-levent commented Apr 19, 2024 •

edited

kovacs-levent commented Apr 20, 2024 •

edited