feat: add support `environment.yaml` files #6569

DmitriyLewen · 2024-04-26T09:35:01Z

Description

Add support for detecting dependencies from environment.yaml files.
We do not have a vulnerability database for conda files, so environment.yaml files will only be scanned for packages (-f json --list-all-pkgs, SBOM formats).

Related issues

Close feat(conda): detect dependencies from environment.yml #6565

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

knqyf263 · 2024-04-26T10:31:46Z

integration/repo_test.go

+		{
+			name: "conda environment.yaml generating SPDX SBOM",
+			args: args{
+				command: "fs",
+				format:  "spdx-json",
+				input:   "testdata/fixtures/repo/conda-environment",
+			},
+			golden: "testdata/conda-environment-spdx.json.golden",
+		},


Since we already have an intermediate SBOM representation, I think either CycloneDX or SPDX is enough.

Removed SPDX in d5b0a56

knqyf263 · 2024-04-26T10:36:10Z

pkg/dependency/parser/conda/environment/parse.go

+	// Default format for files created using the `conda Export` command: `<Name>=<Version>=<Build>
+	// e.g. `bzip2=1.0.8=h998d150_5`
+	// But it is also possible to set only the dependency name
+	ss := strings.Split(dep.Value, "=")


It seems like we have to care about a variety of cases.
https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#package-match-specifications

numpy numpy 1.8* numpy 1.8.1 numpy >=1.8 numpy ==1.8.1 numpy 1.8|1.8* numpy >=1.8,<2 numpy >=1.8,<2|1.9 numpy 1.8.1 py27_0 numpy=1.8.1=py27_0

I updated logic - 32b7ae2
@knqyf263 I couldn't create the logic without regexp.
But if you have any ideas, please tell me.

knqyf263 · 2024-04-26T10:37:44Z

pkg/fanal/analyzer/language/conda/environment/environment_test.go

+			})
+
+			if tt.wantErr != "" {
+				require.Error(t, err)


nit: this line is not needed when using ErrorContains.

Removed - 6fc4535

knqyf263 · 2024-04-26T10:38:42Z

pkg/fanal/analyzer/language/conda/environment/environment.go

+	return res, nil
+}
+func (a environmentAnalyzer) Required(filePath string, _ os.FileInfo) bool {
+	return filepath.Base(filePath) == "environment.yaml" || filepath.Base(filePath) == "environment.yml"


We should define a const.

trivy/pkg/fanal/types/const.go

Lines 107 to 143 in 18da7b1

// Language-specific file names

const (

NuGetPkgsLock = "packages.lock.json"

NuGetPkgsConfig = "packages.config"

GoMod = "go.mod"

GoSum = "go.sum"

MavenPom = "pom.xml"

NpmPkg = "package.json"

NpmPkgLock = "package-lock.json"

YarnLock = "yarn.lock"

PnpmLock = "pnpm-lock.yaml"

ComposerLock = "composer.lock"

ComposerJson = "composer.json"

PyProject = "pyproject.toml"

PipRequirements = "requirements.txt"

PipfileLock = "Pipfile.lock"

PoetryLock = "poetry.lock"

GemfileLock = "Gemfile.lock"

CargoLock = "Cargo.lock"

CargoToml = "Cargo.toml"

ConanLock = "conan.lock"

CocoaPodsLock = "Podfile.lock"

SwiftResolved = "Package.resolved"

PubSpecLock = "pubspec.lock"

MixLock = "mix.lock"

)

done - a3254f9

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263

@DmitriyLewen I refactored it a bit so it can parse more cases. Please let me know if it looks wrong.

DmitriyLewen · 2024-05-02T03:33:44Z

Good solution!
LGTM.

Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com>

kutkopy · 2024-05-06T12:18:41Z

Hi @knqyf263 and @DmitriyLewen, I have noticed an issue with the current implementation. In particular pip-dependencies in an environment.yml-file (as show below) are not supported:

name: lib-build-env
channels:
  - conda-forge
  - fastchan
  - pytorch
dependencies:
  - example-conda-lib=1.2.3
  - pip:
      - example-pip-lib==3.2.1

And lead to the following error:

[conda] Unable to detect the dependency versions from `environment.yml` as those versions are not pinned. Use `conda env export` to pin versions.

DmitriyLewen added 8 commits April 26, 2024 12:56

fear(parser): add parser for environment.yaml

64bb191

refactor(parser): update toLibrary function

0894f34

test(parser): update/add tests

cc797be

feat(fanal): add environmentAnalyzer

26d3908

feat(detector): add CondaEnv to drivers

8f73c4b

feat(purl): add CondaEnv

d1acd53

refactor(fanal): mark CondaEnv al lock file

a263edb

test(integration): add tests for cyclonedx and spdx

5415d71

DmitriyLewen self-assigned this Apr 26, 2024

ci(semantic-pr): add conda

16a62e0

DmitriyLewen changed the title ~~feat(conda): add support environment.yaml files~~ feat: add support environment.yaml files Apr 26, 2024

fix linter error

18da7b1

knqyf263 reviewed Apr 26, 2024

View reviewed changes

DmitriyLewen added 5 commits April 27, 2024 09:28

test(integration): remove environment.yaml spdx test

d5b0a56

test(fanal): remove unneeded error check

6fc4535

refactor(fanal): update logic to detect version of deps

32b7ae2

refactor(fanal): use const for environment.yaml

a3254f9

docs: add conda page

3425472

DmitriyLewen marked this pull request as ready for review April 27, 2024 08:20

feat: parse more versions

3b49a17

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263 approved these changes Apr 29, 2024

View reviewed changes

knqyf263 added this pull request to the merge queue Apr 29, 2024

Merged via the queue into aquasecurity:main with commit e3bef02 Apr 29, 2024
17 checks passed

DmitriyLewen deleted the feat-conda/add-support-environment-yaml branch May 2, 2024 03:33

fl0pp5 pushed a commit to altlinux/trivy that referenced this pull request May 6, 2024

feat: add support environment.yaml files (aquasecurity#6569)

88ab745

Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support `environment.yaml` files #6569

feat: add support `environment.yaml` files #6569

DmitriyLewen commented Apr 26, 2024

knqyf263 Apr 26, 2024

DmitriyLewen Apr 27, 2024

knqyf263 Apr 26, 2024

DmitriyLewen Apr 27, 2024

knqyf263 Apr 26, 2024

DmitriyLewen Apr 27, 2024

knqyf263 Apr 26, 2024

DmitriyLewen Apr 27, 2024

knqyf263 left a comment

DmitriyLewen commented May 2, 2024

kutkopy commented May 6, 2024 •

edited

	// Language-specific file names
	const (
	NuGetPkgsLock = "packages.lock.json"
	NuGetPkgsConfig = "packages.config"

	GoMod = "go.mod"
	GoSum = "go.sum"

	MavenPom = "pom.xml"

	NpmPkg = "package.json"
	NpmPkgLock = "package-lock.json"
	YarnLock = "yarn.lock"
	PnpmLock = "pnpm-lock.yaml"

	ComposerLock = "composer.lock"
	ComposerJson = "composer.json"

	PyProject = "pyproject.toml"
	PipRequirements = "requirements.txt"
	PipfileLock = "Pipfile.lock"
	PoetryLock = "poetry.lock"

	GemfileLock = "Gemfile.lock"

	CargoLock = "Cargo.lock"
	CargoToml = "Cargo.toml"

	ConanLock = "conan.lock"

	CocoaPodsLock = "Podfile.lock"
	SwiftResolved = "Package.resolved"

	PubSpecLock = "pubspec.lock"

	MixLock = "mix.lock"
	)

feat: add support environment.yaml files #6569

feat: add support environment.yaml files #6569

Conversation

DmitriyLewen commented Apr 26, 2024

Description

Related issues

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 left a comment

Choose a reason for hiding this comment

DmitriyLewen commented May 2, 2024

kutkopy commented May 6, 2024 • edited

feat: add support `environment.yaml` files #6569

feat: add support `environment.yaml` files #6569

kutkopy commented May 6, 2024 •

edited