[CWS] bump security agent policies to v0.42.1 #14964
Merged
+2
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paulcacheux
added
changelog/no-changelog
[deprecated] qa/skip-qa - use other qa/ labels
safchain
approved these changes
Jan 5, 2023
amenasria
approved these changes
Jan 5, 2023
YoannGh
approved these changes
Jan 5, 2023
paulcacheux
added a commit
that referenced
this pull request
Jan 5, 2023
(cherry picked from commit 90277ad)
paulcacheux
added a commit
that referenced
this pull request
Jan 5, 2023
guyarb
added a commit
that referenced
this pull request
Jan 18, 2023
* [fargate] Make hostname resolution more reliable (#14746)
* [config/environment] Check AWS_EXECUTION_ENV in Fargate detection
* [util/fargate] Rely on features for ECS Fargate detection
* [fargate/detection] Rely on features to detect EKS
* [trace-agent/config] Call fargate.GetOrchestrator after loading config
* add unit-test for trace-agent config on fargate
* Add release note
* [cmd/trace-agent/config] Fix TestFargateConfig in macOS
Co-authored-by: Cedric Lamoriniere <cedric.lamoriniere@datadoghq.com>
* 7.41.0 CHANGELOG (#14675) (#14745)
* Updated Python to 3.8.16
* CWS: sync BTFhub constants (#14804)
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* [CSPM] respect verbose on compliance check cli cmd (#14750)
* CODEOWNERS: splitting files so USM can own its own files (#14789)
* config: test: Removed duplicated test (#14705)
* Running dockers in the kitchen test (#14589)
* ci: kitchen: Allow running dockers in kitchen test, and extend the filesystem
The PR introduce a way to run external dockers in the kitchen tests, without pulling them
As we cannot authenticate in the kitchen machines to dockerhub, we had to work around that
and we are pulling and saving the dockers in gitlab, uploading them to the remote machine
using kitchen, and then loading those dockers on the remote machine so they are available
for usage.
In the PR we added steps to install docker and docker compose on the kitchen machines.
The PR introduce an example test that runs dockers.
During the PR we faced the problem of "no space left on the device", to solve those errors
we have to extend the filesystem of the remote machines.
* Fixed cr comments
* Debugging the artifacts
* Debugging the artifacts
* Debugging the artifacts
* Debugging the artifacts
* revert artifacts
* Giving another try to dependencies
* Fixed path
* Fixed CR comment
* [CWS] Add tests for activity dump processes content (#14708)
* [CWS] Add two checks to avoid adding nodes with abnormal paths in activity dumps (#14698)
* [gitlab] Repack macOS JUnit tarball to include correct name and job URL (#14793)
* Bump golang.org/x/tools from 0.3.0 to 0.4.0 in /pkg/security/secl (#14710)
* Bump golang.org/x/tools from 0.3.0 to 0.4.0 in /pkg/security/secl
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.3.0...v0.4.0)
---
updated-dependencies:
- dependency-name: golang.org/x/tools
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Auto-generate go.sum and LICENSE-3rdparty.csv changes
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* [single-machine-performance] Introduce regression detector jobs (#14528)
* [WIP][single-machine-performance] Introduce regression detector jobs
This PR intends to introduce the Single Machine Performance regression detector
into Agent CI. This builds on work done in #14477 and is peer to #14438. The
Regression Detector is a CI tool that determines if a changed introduced into a
project modifies project performance in a way that is more than just random
chance with some statistical guarantee. The Regression Detector is not a
microbenchmarking tool and must operate on the whole Agent. This PR introduces
only 'throughput' as an optmization goal -- how quickly can the Regression
Detector produce load into the Agent -- but other goals are
possible. Regressions are checked per-experiment, please see `tests/regression`
for details about how to define an experiment.
The Regression Detector runs today in vectordotdev/vector project and is
influential in keeping that project's performance consistently high.
REF SMP-208
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Use static smp binary
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* different base sha calculation
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Try to clone the whole repo
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* baseline sha computation redux
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* specify region explicitly
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* use smp 0.6.3-rc3
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Wait for job to complete, output report, status
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* update job name
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Update smp, lading
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* remove \
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Use smp 0.6.4
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* diagnose why file_to_blackhole fails
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* just one test for now
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* set log level for smp
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* tweaks
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* debug
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* actually add datadog.yaml et all, .gitignore issue?
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* tidy up cases to initial trio, less file_to_blackhole which needs work
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* update smp, config tweak
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* override .gitignore
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Apply @GeorgeHahn's patches
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* enable other tests, tweak OTEL
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* more fiddling
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* tweaks
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* use markdown output report
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* use OTEL http
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* use smp 0.6.5-rc1
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* debug -> info
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* preserve output
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* remove stray tick
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* Update test/regression/README.md
Co-authored-by: Kylian Serrania <kylian.serrania@datadoghq.com>
* Update test/regression/README.md
Co-authored-by: Kylian Serrania <kylian.serrania@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Co-authored-by: Kylian Serrania <kylian.serrania@datadoghq.com>
* Split bundle params (#14702)
* Split BundleParams into ConfigParams and LogParams
* Move ConfigParams and LogParams to their own file
* Move WithXXX functions from BundleParams to config.Params
* Use constructors for config.Params
* Fix comp/core/log/params_test.go
* Make fields for log.Params unexported
* Make config.Params fields not exported.
* Fix package names in the security agent.
* Explain why `fx.Provide` is needed in bundle.go
* Remove configLoadSecurityAgent from NewSecurityAgentParams
* Add NewAgentParamsWithSecrets and NewAgentParamsWithoutSecrets
* CWS: sync BTFhub constants (#14815)
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* Check the package exists before creating package. Restore install script after packaging. (#14777)
* change networks slack channel (#14819)
* fix close_time value display in INFO log (#14744)
* Updates prometheusScrape to support tag_by_endpoint and collect_counters_with_distributions (#14805)
* Updates prometheusScrape to support tag_by_endpoint
* Adds release note
* Cleans release note
* Also adds support for `collect_counters_with_distributions`
* Updates release note to include the second added parameter
* Updates release note based on suggestion by @clamoriniere
* Migrating flare to a component (#14234)
Migrating flare to a component
This adds a 'flare' component and rework the flare package to be
compatible with fx app and non-fx app.
The flare generation now happens through a FlareBuilder which handles
all the logic of adding data to a flare. This FlareBuilder can be used
directly (by the flare package) or be received by each component when
they register a flare provider.
Migration workflow for each component would be to move their dedicated
code from the flare package to a flare provider.
Note: Until `cmd/systray/` is migrated to fx we can't start using the
flare component from other flare (on windows the systray can create
flare on it's own).
* Add netlink process monitor (#14706)
This monitor will read the netlink socket process events queue and run it on parallel worker (map to n cpu cores)
ProcessMonitor require root or CAP_NET_ADMIN capabilities
Aim to Subscribe() to process event Exec, Exit
With or without metadata process Any, Name, MAPfile
ProcessMonitor will subscribe to the netlink process events like Exec, Exit
and call the subscribed callbacks
Initialize() will scan the current process and will call the subscribed callbacks
callbacks will be executed in parallel via a pool of goroutines (runtime.NumCPU())
callbackRunner is callbacks queue. The queue size is set by processMonitorMaxEvents
Multiple team can use the same ProcessMonitor,
the callers need to guarantee calling each Initialize() Stop() one single time this maintain an internal reference counter
Netlink process subscription, socket connection is allowed only by one PID
* protocols: refactor tests to allow pre-post setups (#14817)
* protocols: refactor tests to allow pre-post setups
* Added temporary nolint for skippers
* Fixed bugs
* Escape path in get-acl command (#14818)
* ci: Add manual benchmark step for trace-agent (#14466)
* pkg/trace/config: Lower max tracer payload to 25 MB to better align with backend limits (#14782)
* Revert #14367 and use nano timestamp instead (#14825)
* Revert "Replace timestamp by increasing id to avoid configVersion matching different config changed in the same second"
This reverts commit f8e097de2aa3322670fcc6a6c8cfc5c1ed9d6239.
* Revert #14367 and use nano timestamp instead
* Disable by default remote-tagger in clc-runner mode (#14821)
* fix gofmt -s for pkg/collector/collector_demux_test.go (#14808)
* Improve debug logging in cloud foundry container tagger (#14803)
* Add logging around container retries
* Add trace log
* Change to debug and add release note
* Delete Improve-container-tagger-logging-e48b0fffbe8563d0.yaml
* Add timestamp id to events
* Make id more specific, use container String method
* Just print class
* Update pkg/cloudfoundry/containertagger/container_tagger.go
Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com>
* Address PR review
* Create event ID
Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com>
* [Serverless] Merge serverless/main to main. (#14826)
* [Serverless] change account (#14755)
* Aj/buffer cold start span data (#14664)
* wip dirty commit - trace being created but not flushed properly. No further traces appearing
WIP: more debugging. StopChan properly set up
feat: Starting coldstart creator as a daemon, and recieving data from two channels. Todo: spec
feat: Update specs to write to channels
feat: Merge conflicts resolved for tests
feat: Use smaller methods to handle locking
fix: pass coldstartSpanId to sls-init main
feat: Remove default
feat: Use Millisecond as Second is far longer than necessary
feat: No need to export ColdStartSpanId
fix: update units
feat: Directionality for lambdaSpanChan as well as for initDurationChan
fix: No need for the nil check, I need to stop javascripting my go
feat: ints
* feat: rebase missing changes from merge commits
* feat: update ints after moving accounts
* Empty commit to trigger ci
* [Serverless] Fix flaky integration tests and make them more easily maintainable. (#14783)
* Retry serverless integration test failures automatically. (#14801)
* [Serverless] Allow some keys to be option in serverless integration tests. (#14827)
* Ability to remove items from the json.
* Remove items from snapshot.
Co-authored-by: Maxime David <maxime.david@datadoghq.com>
Co-authored-by: AJ Stuyvenberg <astuyve@gmail.com>
* Allow Regression Detector pipeline to fail (#14828)
At present there's a race condition in the CI pipeline with regard to Regression
Detector: we rely on an artifact to be created by main pipeline merge but have
no way of making a hard dependency on that artifact. If that artifact is not
present then the Regression Detection job will be submitted and then immediately
fail. Absent a solution we allow the Regression Detector job to fail,
unfortunately making any actual regressions caught but also not contributing to
alert blindness in the meanwhile.
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
* [process-agent] Remove unused properties from AgentConfig (#14842)
* [process-agent] Remove unused properties from AgentConfig
* Fix tests
* 7.41.1 changelog (#14822) (#14824)
* Add do-not-merge github action (#14843)
* [CWS] remove useless resolver function (#14792)
* [kitchen] Work around bundler and ruby version issue in verifier (#14851)
Modifies the script used to run kitchen tests to run the verify phase twice, and adds a pre_verify lifecycle hook to install the dependency needed for system-probe kitchen tests.
Works around an issue (version mismatch between ruby and bundler) that started happening after the release of version 2.4.0 of bundler.
As long as this workaround is needed, we can't have Gemfiles in test suites, and instead need to manually install gems whenever needed.
* Add the 'test' build tag to the 'unit-tests' flavor
This tag is needed to run unit-test but was not printed by
'inv print-default-build-tags -b unit-tests'. When running tests from
a IDE or other we need the correct list of tags to be returned.
* flare: Added /opt/datadog-agent directory permissions to permissions.log (#14848)
* flare: Added /opt/datadog-agent directory permissions to permissions.log
system-probe internal files (sysprobe.socket, runtime compilation source files, prebuilt version, etc.) are located in /opt/datadog-agent
when getting a flare, we cannot know those files permissions (and if they exist).
* Take directories from configuration
* Fixed cr comments
* Fixed cr comments
* Fixed cr comments
* Update comp/core/flare/helpers/helpers.go
Co-authored-by: maxime mouial <hush-hush@users.noreply.github.com>
* [USM] protocol classification: add RabbitMQ classification (#14734)
* wip
* Fixed
* added support for amqp without tests
* added UT's for consumer and sender for rabbitmq
* removed redundant client and server
* added support to classify also protocol header of amqp
* removed redundant function
* test
* fixed most of the cr notes
* fixed all the cr notes
* add ut
* fixed licence issue
* fixed ci issue
* fixed event common protocol type number
* Revert update of github.com/DataDog/datadog-operator
* fixed all cr notes
* merged main
* fixed a cr note
* reverted datadog-operation
* update licence
* fixed ci issue
* merged main and updated ut
* fixed cr note
* added some UT's and support the latest classification uts update
* refactor the uts
* Added debug log
* Added debug log 2
* Added debug log 3
* Added pattern scanner
Co-authored-by: Guy Arbitman <guy20495@gmail.com>
* Handle environment variables without an equal sign (#14806)
* usm: protocols: Refactored server creation (#14869)
* Removed example docker tests (#14852)
* [CWS][SEC-5573] add custom CWS product (#14748)
* [CWS] add custom CWS product
* Add a debouncer to limit reloads
* Update URL regexp to detect for Datadog's URL
In the past we use to edit the regexp everytime Datadog would open a new
location. This commit allow the agent to detect for all present and
future locations as long as they follow the format of 2 letters + 1
digit. Example: 'us3.datadoghq.com'.
* system-probe: tasks: Save all dockers from docker-compose files in the protocols dir (#14873)
* system-probe: tasks: Save all dockers from docker-compose files in the protocols dir
* Fixed lint
* [process-agent] Move data scrubber and disallow list from pkg/process/config (#14863)
- Move these two fields in preparation for removal of pkg/process/config package.
- Use inclusive naming where possible - will rename the config param in the future.
- Update imports in pkg/security using the DataScrubber type.
* add `integration_profiling` config option (#14847)
Add a new option to enable profiling of python integrations. It's used only within the python integrations. See https://github.com/DataDog/integrations-core/pull/13576.
* Fix flaky TestKSMCheckInitTags unit-test (#14832)
* Fix flaky TestKSMCheckInitTags unit-test
* improve config.GetConfiguredTags testability
* update GetConfiguredTags function description
* Deleting Security Agent for Windows resources (#14833)
* deleting windows resources
* removing windows operations for security-agent.build task
* removing secagent for windows resources in omnibus, addressing python lint
* [process-agent] Remove orchestrator config from AgentConfig (#14867)
* [process-agent] Move data scrubber and disallow list from pkg/process/config
- Move these two fields in preparation for removal of pkg/process/config package.
- Use inclusive naming where possible - will rename the config param in the future.
- Update imports in pkg/security using the DataScrubber type.
* [process-agent] Remove orchestrator config from AgentConfig
- Further decouple config management in prep for removal of pkg/process/config.
- Remove orchestrator config, push it into the pod check and collector structs.
* Address review feedback
* [process-agent] Display system probe process module status in process agent info commands (#14880)
Updates the process agent status information displayed by the datadog-agent status, process-agent status and process-agent --info commands to display whether or not the system probe's process module is enabled
* tooling: Add invoke vscode devcontainer cmd (#14031)
* Add invoke vscode envcontainer cmd
* Update agent_dev_env.md
* fix typo in documentation
* adding err to exit SecAgent. fixes hanging if there's no API key (#14856)
* Replace hardcoded /proc path with config field (#14773)
Use the config field instead of hardcoding /proc. The config field should
be automatically detected to either /proc or /host/proc inside containers.
* usm: protocols: Added redis classification (#14886)
* usm: protocols: Added redis classification
* Fixed CR comment
* Fixed CR comment
* Fixed warning on centos
* [CWS] extract custom events package (#14230)
* [CWS] extract custom events package
* [CWS] extract selftest custom event
* [CWS] allow to specify a rate per rule through config
* post rebase
* add lint exception
* use the good sender
* [process-agent] Remove check intervals from pkg/process/config (#14878)
* [process-agent] Remove check intervals from pkg/process/config
- Remove check interval management from pkg/process/config package
- Never store intervals, just use config settings
- Generalize check for process and process RT check intervals
* Fix MacOS tests
* Address review feedback from @just-chillin
* flare: Ignore system probe dirs if they are empty (#14893)
* [CWS] increase exit event test timings (#14813)
* [CWS] fix rule id not sent for custom event (#14897)
* Adding return statment in GUI when an error is encountered
* [CI] Artifactory for Python (#14473)
* Introduce new E2E tests based on test-infra-definitions (#13643)
* manual check tracing uses new exhaustive tracing config option (#14892)
* manual check tracing uses new exhaustive tracing config option
Following up to https://github.com/DataDog/integrations-core/pull/13618, we now need to set both `integration_tracing` and `integration_tracing_exhaustive` config options to enable exhaustive tracing of integrations.
When manually running a check the increased overhead of exhaustive tracing (tracing all check methods) is acceptable. When continuous integration tracing is desired only the `integration_tracing` option should be set in order to keep the overhead minimal.
* update core agent check command
* fix sort order
* pkg/trace/traceutil: Add fast-path for NormalizeTags to reduce cpu usage (#14881)
* usm: remove the scenario of nil subprograms (#14909)
* usm: remove the scenario of nil subprograms
* Fixed CR comments
* Import order
* Fixed CR comments
* Bump datadog-api-client from 2.6.0 to 2.7.0 in /test/e2e/cws-tests (#14914)
Bumps [datadog-api-client](https://github.com/DataDog/datadog-api-client-python) from 2.6.0 to 2.7.0.
- [Release notes](https://github.com/DataDog/datadog-api-client-python/releases)
- [Changelog](https://github.com/DataDog/datadog-api-client-python/blob/master/CHANGELOG.md)
- [Commits](https://github.com/DataDog/datadog-api-client-python/compare/2.6.0...2.7.0)
---
updated-dependencies:
- dependency-name: datadog-api-client
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* usm: http2: improved functions (#14917)
* update profiling endpoint when the fips is enabled to avoid 404 (#14807)
* fix(fips): update profiling endpoint when the fips is enabled to avoid 404
Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@epita.fr>
* pkg/clusteragent/admission: introduce deployment patcher (#14500)
* [CWS] avoid using readonly map for eBPF test prog (#14780)
* [e2e] add codeowners for new e2e tests (#14865)
* DogstatsD component improvements (#14839)
* Inject defaultLogFile
* Move main.go inside command/command.go
* Move start command to subcommands/start
* Dogstatsd uses pkg/cli/subcommands/version/command.go for version command
* Use similar code for cfgpath compare to datadog-agent
* eval.Opts holds MacroStore and VariableStore (#14874)
* [fake-datadog] add docker compose (#14902)
* [fake-datadog] add docker compose
* [fake-datadog] add docker instructions
* usm: mongo: Added mongo classification (#14809)
* usm: mongo: Added mongo classification
* Fixed CR comment
* Fixed CR comment
* Fixed CR comment
* Fixed CR comment
* Update agent_dev_env.md (#14887)
Co-authored-by: Kaylyn <kaylyn.sigler@datadoghq.com>
* [CWS][SEC-6508] use tail call limit to increase the number of args/envs (#14796)
* use tail call limit to increase the number of args/envs
* do not validate process overflow events to avoid srubbing argv and timeout
* [notifications] Catch all image pull errors as infra failures (#14926)
Updates the regex to match infra failure logs when pulling images to include more patterns. The previous pattern didn't catch the following line:
WARNING: Failed to pull image with policy "always": context deadline exceeded (manager.go:203:7197s)
* Do not install the integrations downloader for python 2 (#14920)
* usm: classification: Shrink classification buffer to 24 bytes (#14925)
* config: usm: Added USM to system-probe.yaml.example file (#14908)
* setupConfig consumes 1 param instead of many, adding to SecAgent constructor (#14884)
* changing func signature of setupConfig
* setting security agent config file instead of merging because Viper only supports 1 config file per viper instance
* Revert "setting security agent config file instead of merging because Viper only supports 1 config file per viper instance"
This reverts commit 8e6736d5025db79e5c1f552a983f9050f86a2c5c.
* MergeConfigurationFiles is just for SecAgent
* undo moving sys probe and secagent merge
fix return of merge
* rename configMissingOK field to baseConfigMissingOK
* setting secagent config path and config load secrets params
* adding secagent bundle param test
* reverting renaming configMissingOK to baseConfigMissingOK
* params.configMissingOK should be false
* fixing test post bundle breaking into config and log components
* config params test copywrite info
* [e2e/ndm] add snmp test environment (#14768)
* [e2e/ndm] add snmpsim data folder
* [new e2e test] update test-infra-definition version
* [e2e] fix aws signature
* [e2e/ndm] add snmp test environment
* [e2e/ndm] simpliofy err return code
* [e2e/ndm] remove unused close function
* [e2e/ndm] actually parse flags
* [e2e] ndm: fix destroy
* [e2e/ndm] add copyright header
* [CWS] extract probe from event and activity dump manager (#14515)
* [CWS] extract TC resolver into own resolver
* no probe in event
* include tcresolver in usual resolvers
* fix test
* apply review suggestion
* apply review suggestion v2
* [corechecks/snmp] Add IP Addresses to NDM Metadata interfaces (IPv4) (#14823)
* {Dockerfiles/agent,trace-agent/config}: disable apm `max_memory` and `max_cpu_percent` by default (#14850)
* [pkg/otlp] Add a simple example on metric export (#14784)
* Bump github.com/vektra/mockery/v2 from 2.15.0 to 2.16.0 in /internal/tools (#14913)
* Bump github.com/vektra/mockery/v2 in /internal/tools
Bumps [github.com/vektra/mockery/v2](https://github.com/vektra/mockery) from 2.15.0 to 2.16.0.
- [Release notes](https://github.com/vektra/mockery/releases)
- [Changelog](https://github.com/vektra/mockery/blob/master/.goreleaser.yml)
- [Commits](https://github.com/vektra/mockery/compare/v2.15.0...v2.16.0)
---
updated-dependencies:
- dependency-name: github.com/vektra/mockery/v2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* gen mocks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Cacheux <paul.cacheux@datadoghq.com>
* usm: Reducing chances for mistakes in the protocol type values (#14816)
* usm: classification: split the functions and helpers to protocol-dedicated-files (#14924)
* usm: classification: split the functions and helpers to protocol-dedicated-files
* usm: classification: rename protocol-classification-helpers to protocol-classification
* [process-agent] Remove host info from AgentConfig (#14885)
* [process-agent] Remove host info from AgentConfig
* Fix info command per review feedback
* [process-agent] Remove remaining properties from AgentConfig (#14889)
* Ignore RemoteSamplingClient when marshaling agent config (#14927)
* Ignore RemoteSamplingClient when marshaling agent config
* Add release note
* pkg/obfuscate: fix panic due to missing logger (#14859)
Obfuscator.log was uninitialized which was causing agent panic
* Update github.com/lxn/walk version (#14905)
* gitignore runtime compiled hash files (#14764)
* Try ignoring runtime compiled hash files
* Build object files before linting
* [process-agent] Remove pkg/process/config package (#14904)
* [process-agent] Remove pkg/process/config package
* Address review feedback from @kkhor-datadog
- Revert back to using util.PathExists for simplicity
- Clean up code with early exits
* Review feedback from @sgnn7
* Bump github.com/avast/retry-go/v4 from 4.3.1 to 4.3.2 (#14935)
Bumps [github.com/avast/retry-go/v4](https://github.com/avast/retry-go) from 4.3.1 to 4.3.2.
- [Release notes](https://github.com/avast/retry-go/releases)
- [Commits](https://github.com/avast/retry-go/compare/4.3.1...4.3.2)
---
updated-dependencies:
- dependency-name: github.com/avast/retry-go/v4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/prometheus/procfs from 0.8.0 to 0.9.0 (#14934)
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.8.0 to 0.9.0.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.8.0...v0.9.0)
---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* [CWS Agent] Bugfixing SecAgent Params constructor (#14939)
* [USM] use per-cpu array map instead of in-stack buffer for classification (#14756)
* protocol classification: add per-cpu array map
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* Outsmart the verifier
* change map type on unsupported systems
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix runtime-compilation on older kernels + doc
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix array map
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* docs & refactor
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add missing editor flag to change map type
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* usm: Reverted #14925
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
Co-authored-by: Guy Arbitman <guy20495@gmail.com>
* [gitlab] Use DEB buildimage based on Ubuntu 14.04 instead of Debian 8 (#14929)
* Adding config option to disable delta profiles when profiling the Agent
* Fixed nil return instead of an error in DogStatsD file replay
* Removed sending API key as params in forwarder
* [CWS] remove now useless runtime files sync check (#14945)
* flags package to organize security agent subcommand flags (#14906)
* [CI] Improve visibility for `docker run` commands in the CI (#14899)
Add line breaks for docker run commands
* [CWS Agent] SecAgent command pkg to replace common pkg, moving status and version subcommands (#14907)
* adding command package, to replace common
* status and version subcommands
* Bump github.com/itchyny/gojq from 0.12.10 to 0.12.11 (#14938)
Bumps [github.com/itchyny/gojq](https://github.com/itchyny/gojq) from 0.12.10 to 0.12.11.
- [Release notes](https://github.com/itchyny/gojq/releases)
- [Changelog](https://github.com/itchyny/gojq/blob/main/CHANGELOG.md)
- [Commits](https://github.com/itchyny/gojq/compare/v0.12.10...v0.12.11)
---
updated-dependencies:
- dependency-name: github.com/itchyny/gojq
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Replacing TODOs in exposed comments with more meaningful comments (#14901)
* Revert "[agent] Support for running secrets backends with sha256 verification (#14529)" (#14940)
This reverts commit deb7fce8f668a4bca6697e76d0b77cb67d7f46f7.
* missing import in file with unsupported build flag (#14952)
* Bump golang.org/x/text from 0.5.0 to 0.6.0 (#14948)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.5.0...v0.6.0)
---
updated-dependencies:
- dependency-name: golang.org/x/text
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Download btfs for kitchen tests (#14587)
* Save btfs to dd-agent-omnibus s3 bucket
* Update folders to match new btfhub-archive names
* Download BTFs during kitchen-prepare task
* Add more details to error message
* Fix permissions
* Use btfs from dev box
* Update gitignore
* Check for bpftool compatability outside of generate_minimized_btfs
* Change x86-64 -> x86_64
* Fix generating minimized btfs
* Fix bpftool compatability check helper
* Fix python linting
* Fix python lint
* Only run BTF preparation outside of CI
* Explicitly indicate CI kitchen preparation
Co-authored-by: Hasan Mahmood <hasan.mahmood@datadoghq.com>
Co-authored-by: Bryce Kahle <bryce.kahle@datadoghq.com>
* [secrets] Fix getDDAgentUserSID to account for NT AUTHORITY\SYSTEM (#14941)
* [secrets] Fix getDDAgentUserSID to account for NT AUTHORITY\SYSTEM
* Address review feedback from @clarkb7
* usm: classification: removed redundant nolint (#14958)
* Bump wheel versions (#14918)
* Fixing system_probe.py on linux machine (#14959)
* document trace API v04, including response (#14868)
* [CWS] improve mount fallback (#14779)
* [CWS] improve mount fallback
* post review
* [CWS] bump security agent policies to v0.42.1 (#14964)
* [orchestration] Add Vertical Pod Autoscalers (#14669)
* [orchestration] Add Vertical Pod Autoscalers
We want to start collecting Vertical Pod Autoscalers from Kubernetes.
Co-authored-by: Kangyi LI <kangyi.li@datadoghq.com>
Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>
* [usm] Extract batching functionality into package (#14712)
* [Process Agent] Split Collector into Runner and Submitter (#14883)
* WIP
* Collector and submitter split, need to fix tests
* Rename receivers to `s`
* Delete components directory
* Add RT reporting to submitter
* Add `dropCheckPayloads` back into the submitter
* Move submitter tests to it's own file
* Delete component.go
* clean up comments and unused code
* Fix a couple tests
* Fix orchestrator tests
* Fix tests
* Fix copyright header
* Fix linter issues
* Use mocks in tests
* Fix import
* Fix data race in tests
* Fix data race in tests
* Update cmd/process-agent/collector.go
Co-authored-by: Ivan Ilichev <ivan@ivanilichev.com>
* Refactor `Submit` to not return an error
* Remove `init()` in favor of using mock config
* Remove `init()` in favor of using mock config
* Update mockery to use version 2.16 since they were updated in #14913
* Fix linter errors (again)
* Fix `TestPodCheck/enabled` failing due to the clustername package caching a bad cluster name
* Remove `forwarderRetryQueueMaxBytes`
Co-authored-by: Ivan Ilichev <ivan@ivanilichev.com>
* Bump Collector dependencies to v1.0.0-RC2/v0.68.0 (#14864)
* Bump Collector dependencies to v1.0.0-RC2/v0.68.0
* Revert InstrumentationLibraryMetadataAsTags changes
* Update collector test configuration error message
* Address PR comments
* Increase speed of generate_minimized_btfs jobs (#14585)
Co-authored-by: Bryce Kahle <bryce.kahle@datadoghq.com>
* Add dynamic way of determining eBPF helper availability on runtime compilation (#14685)
* Add KernelHeaderOptions type to prevent ebpf package dependency
* Add function to get available helpers on host
* Use dynamic method of finding available helpers
* Use static list for kernels with __BPF_FUNC_MAPPER macro
* Limit TestGetAvailableHelpers to kernels where it will work
* Fix udp bind for random ports (#14956)
* NDM: Add snmp.interface_status metric (#14797)
* NDM: Add snmp.interface_status metric
* update test
* Add reno
* Address review
* Rename metric
* Address review
* Add InterfaceStatus enum
* Remove iota and use explicit values
* NDM: Add snmp.device.[un]reachable metrics (#14649)
* NDM: Add snmp.device_up metric
* Address review
* update reno
* Address review
* fix import
* Improve log message (#14968)
Log the underlying error when GetUnitTypeProperties fails
* Use rv "0" when polling endpoint list (#13906)
Since this code path polls the endpoint list endpoint once every 60s by
default to update the internal stat in the agent, we don't really need
the consistency guarantees we implicitly get from the unset resource
version.
When the resource version is unset, the api-server needs to fetch all
endpoints from etcd, causing a costly round-trip that can potentially
result in a lot of data traffic. When setting resource version "0", all
requests are handled by the watch cache, meaning they will be much more
efficient and less costly.
For the most part, the actual returned data will be the same, but in
some cases where the API-servers are having a bad time, the data might
be a bit stall; but that is not very common. In that case, getting data
from the watch cache instead of not being able to list at all is
preferable.
The semantics are described in detail here;
https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list
Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <ougedal@palantir.com>
* Remove `CCA_IN_AD` flag and related unused code (#14955)
* remove CCA_IN_AD config flag
* PR feedback
* remove unused providers
* pr feedback
* epforwarder: add additional debug logging (#14161)
* Fix small typo in install XML. (#14687)
Causes Wix to throw error (although apparently non-fatal)
* CWS: sync BTFhub constants (#14986)
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* Revert "pkg/obfuscate: improve formatting and string parsing in the SQL obfuscator (#11967)" (#14976)
This reverts commit 8ab1d187421087d8ae746ec0dcca00f25918a9f0.
* [CWS] remove unsafe pointer from eval.Context (#14890)
* [CWS] remove unsafe pointer for eval.Context
* Add user context
* move perf helper to a perf file
* remove resolvers from event
* generate handlers
* add extra field handlers
* remove accessors from probe
* remove model mock
* fix unit and functional tests
* refactor model/field_handlers
* add helper for common object creation
* fix stress tests
* [workloadmeta/collectors/containerd] Collect image metadata (#14592)
* [util/containerd] Rename Image to ImageOfContainer
To be able to introduce a new Image func that gets an image just by image ID,
regardless of whether it's being used in container.
* [util/containerd] Add Image func
* [workloadmeta] Add GetImage func
* [config] Add option to enable image collection in workloadmeta
* [workloadmeta/collectors/containerd] Collect image metadata
* [CSPM] remove the hostSelector field not used anymore (#14770)
* [CSPM] remove the hostSelector field not used anymore
In a more global effort to remove the internal compliance DSL after
our move to rego, this commit removes one field where it is still
being used.
The hostSelector field has been put in place in order to make sure
we only run specific rules on hosts that match, in particular for
k8s nodes. However, the rule were not used anymore since the hosts
"master" labels are not properly set. We rely other side effects
(like process and file existence) to avoid running some rules on
bad nodes.
* [CSPM] remove k8s nodeLabels retrieval from compliance rules execution
Now that hostSelector fields have been removed, fetching the k8s node labels
is not required anymore and completely useless.
This PR just remove the nodeLabels fetching and all the subsequent
dependencies.
* [CWS] add tests for live process monitoring (#14944)
* [system-probe][NET-2899] fix race condition in ephemeral port checker (#14802)
* [NET-2899] use mutex to lock fields causing race condition in ephemeral port checker
* [NET-2899] gofmt on changed files
* [NET-2899] remove mutex, move racey code to sync.once func
* [CWS] restore SECL documentation generation (#14993)
* [CWS] fix event missing field resolver (#14992)
* fix missing fields resolver in some events (around policy eval CLI)
* do not emit event in policy eval output
* Add __TARGET_ARCH_ to runtime compilation flags (#14983)
* Add __TARGET_ARCH_ to runtime compilation flags
* Use append instead
* Re-delete http runtime asset hash file (#14982)
* Add CO-RE version of TCP Queue Length check (#14763)
* Add CO-RE version of TCP Queue Length check
* Fix version
* Fix generate BTF job
* Invert err check on CO-RE load
* Add helper for missing BTF check
* Bump golang.org/x/tools from 0.4.0 to 0.5.0 in /pkg/security/secl (#14996)
* Bump golang.org/x/tools from 0.4.0 to 0.5.0 in /pkg/security/secl
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/tools/releases)
- [Commits](https://github.com/golang/tools/compare/v0.4.0...v0.5.0)
---
updated-dependencies:
- dependency-name: golang.org/x/tools
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Auto-generate go.sum and LICENSE-3rdparty.csv changes
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
* Fix gateway lookup tests (#14951)
* [usm] Reduce HTTP test memory utilization (#15006)
* [CWS] mount fallback to pid 1 by default (#15007)
* [CWS][SEC-4020] parse args and envs from the new program stack pages (#13008)
* CWS: parse args and envs from the process stack
* remove useless function parameter
* get env vars offset from new program stack as well
* return from tailcall loop sooner
* use different kprobe to fix kernel function call order on CentOS 7
* [process-agent] Refactor conn rates with util/subscriptions (#14988)
* [process-agent] Refactor conn rates with util/subscriptions
* Update with a unit test for pub/sub
* Address feedback from @hmahmood
* [CWS] change programs to avoid mixing events between tests (#15012)
* [CWS] rework event json marshalling (#15010)
* externalize serialization
* a bit of cleanup
* refactor schema validators
* fix printfs
* re-enable policy eval event json
* fix trace dispatching
* fix deadcode
* fix validateProcessContextSECL error output
* [USM] protocol classification: add PostgreSQL classification (#14625)
* protocol classification: add per-cpu array map
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* Outsmart the verifier
* protocol classification: add per-cpu array map
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* Outsmart the verifier
* protocol classification: add PostgreSQL classification
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix licenses & set postgres port in test
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* test: fix port
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* test: use JoinHostPort instead of Sprintf
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* [USM] protocol classification: add Postgres detection
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* revert check_command fix
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* postgres: refactor check_command
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* change map type on unsupported systems
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix runtime-compilation on older kernels + doc
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix array map
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix merge
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* tests: add documentation
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* tests: add long query test
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* docs & refactor
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix licenses
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* refactor
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* postgres: try to classify from client start messages
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add missing Cgo defs
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add postgres docker image pulling
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add missing editor flag to change map type
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* remove unused import
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* case-insensitive check + docs
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* check on tmp buf
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* docs
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* try fixing verifier issue
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fix verifier issue
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* tests: fix docker-compose path
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* fixup! Merge remote-tracking branch 'origin/main' into guillaume.pagnoux/USMO-9-protocol-classification-posgres
* re-delete re-added files
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* go mod tidy
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add docs
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* remove redundant check
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* refactor server creation in tests
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* rename guards
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* specify postgres version in docker-compose
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* tests: skip when using NAT
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* split sql files
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* tests: add tests for all supported sql queries
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* move postgres struct to postgres-defs.h
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* remove redundant check
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* classify on command completion messages as well
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* add long response test
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
* re-enable query detection
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
Signed-off-by: Guillaume Pagnoux <guillaume.pagnoux@datadoghq.com>
Co-authored-by: Guy Arbitman <guy20495@gmail.com>
* [process-agent] Scaffold components for process agent (#14972)
* [process-agent] Scaffold components for process agent
* Addresss review comments from @ogaca-dd
* Addresss review comments from @ogaca-dd
* Change to use context.Context and reintroduce empty Component interface to suppress linting
* CWS: sync BTFhub constants (#15023)
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* [CWS] rework/cleanup `FieldHandlers` (#15015)
* remove probe from FieldHandlers
* cleanup `NewProcessResolver` resolvers dependency
* resolvers only need a link to the manager
* Update CODEOWNERS (#15024)
* Use sc query to gain information about the service before attempting to stop it. (#15028)
* [security-agent] remove redundant String() in compliance agent log (#15026)
* [invoke] Print summary of test failures at the end of inv test (#14682)
Updates the inv test command to print a summary of failed tests at the end of a run, across all modules and flavors that were tested, to more easily identify the list of failures, without having to visually parse the full job logs.
* [system-probe][NET-2891] Fix tcp retransmit count (#14740)
* [NET-2891] initial pass at changes to prebuilt code
* [NET-2891] use retrans_out for runtime compiled tcp_retransmit counter
* [NET-2891] runtime compiled version of tcp_retrans updates
* [NET-2891] remove debug comment
* [NET-2891] fix log
* [NET-2891] update bytecode
* [NET-2891] code review comments, regenerate license
* [NET-2891] newline
* [NET-2891] fix probe definitions
* [NET-2891] update comment
* [NET-2891] runtime compilation fixes
* [NET-2891] fix byte padding for args init
* [NET-2891] fix formatting
* testing debug logic
* more debug logic, added some config for the map
* [NET-2891] enable kretprobe and remove debug
* [NET-2891] disable bpf debug be default
* [NET-2891] update bytecode
* [NET-2891] make function as maybe unused
* [NET-2891] handle different paths of incremental vs absolute retransmit counters
* [NET-2891] use enum to track increment vs absolute retransmits
* [NET-2891] change enum values
* [NET-2891] move retrans code to runtime tracer
* pulled in new gitignore
* Revert "pulled in new gitignore"
This reverts commit b4b0df587aeb6b6f655ea90d7bc96ae250934170.
* remove runtime gen files, code review comments
* [NET-2891] use retransmit count none in runtime tracer
* [NET-2891] use retransmit_count_none in handle_tcp_stats
* [NET-2891] nit comments from code review
* [NET-2891] try to get runtime compilation working on 4.4 kernel
* usm: upgraded pgdriver version to indirectly upgrade mellium.in/sasl version due to a CVE ofound (#15030)
* usm: upgraded pgdriver version to indirectly upgrade mellium.in/sasl version due to a CVE ofound
* Fixed go.sum
* [CWS] fix signal test (#15025)
* [process-agent] Support dynamically enabling profiling for process agent from CLI (#14995)
Adds support for dynamically enabling profiling for the process agent from the CLI
* pkg/obfuscate: Fix parsing of sqlserver identifiers enclosed in square brackets (#15019)
* DBM-2010 Fix parsing of sqlserver literals enclosed in square brackets
* .gitlab: move APM benchmark job to manual only (#15036)
* fix datatype (#13791)
related to #13770
* [AD/prometheus] Ignore headless services (#15031)
* Fix stop service (#15035)
* Fix check conf directory
Durring the migration to component the hardcoded directory 'etc/confd'
for check configuration was removed.
* Fix shipping of 'version-history.json' and 'registry.json' in flares
When migrating to component the logic to include /opt/datadog-agent/run/
was handled as a file instead of a folder. This broke collecting
'version-history.json' and 'registry.json from it.
* Fix datadog.yaml file name in flare
* Force file permission to 644 within a flare
* auto instru: add rc provider (#15008)
* pkg/obfuscate: use github.com/outcaste-io/ristretto instead of github.com/dgraph-io/ristretto (#15005)
Migrate the usage of github.com/dgraph-io/ristretto to github.com/outcaste-io/ristretto
* [workloadmeta/kubelet] Parse image ID if name is a SHA256
We now try to parse the resolved image ID if the image in the pod's
container status is a SHA256. This seems to happen when pinning the
SHA256 in the container spec. This fixes an issue where `image:` filters
in DD_CONTAINER_INCLUDE/DD_CONTAINER_EXCLUDE would not be respected.
* pkg/trace/api: remove unused internal OTLP HTTP server (#14965)
* [pkg/trace/api] Remove unused OTLP HTTP server
* [pkg/trace] Remove protocol argument
* Remove unnecessary fmt.Sprintf
* Fix tests
* [CWS] cleanup last uses of `jsonschema_description` (#15050)
* [Serverless] Merge `serverless/main` to `main` (#14980)
* [Serverless] change account (#14755)
* Aj/buffer cold start span data (#14664)
* wip dirty commit - trace being created but not flushed properly. No further traces appearing
WIP: more debugging. StopChan properly set up
feat: Starting coldstart creator as a daemon, and recieving data from two channels. Todo: spec
feat: Update specs to write to channels
feat: Merge conflicts resolved for tests
feat: Use smaller methods to handle locking
fix: pass coldstartSpanId to sls-init main
feat: Remove default
feat: Use Millisecond as Second is far longer than necessary
feat: No need to export ColdStartSpanId
fix: update units
feat: Directionality for lambdaSpanChan as well as for initDurationChan
fix: No need for the nil check, I need to stop javascripting my go
feat: ints
* feat: rebase missing changes from merge commits
* feat: update ints after moving accounts
* Empty commit to trigger ci
* [Serverless] Fix flaky integration tests and make them more easily maintainable. (#14783)
* Retry serverless integration test failures automatically. (#14801)
* [Serverless] Allow some keys to be option in serverless integration tests. (#14827)
* Ability to remove items from the json.
* Remove items from snapshot.
* Do not expect spans when there is no spans object. (#14396)
* [Serverless] Improve stability of two tests. (#14895)
* Increase timeout while decreasing test time.
* Increase timeout in test.
* [Serverless] Consolidate log normalization to single file for integration tests. (#15004)
* Consolidate log normalization to single file.
* Save raw logs to a temp dir.
* Fix linting issues.
Co-authored-by: Maxime David <maxime.david@datadoghq.com>
Co-authored-by: AJ Stuyvenberg <astuyve@gmail.com>
* Fixes multiple problems with http processing/tagging on Windows. (#15022)
* Fixes multiple problems with http processing/tagging on Windows.
- There was an offset error in which the port was not properly computed
on ipv6 connections
- There was a problem with computing whether an ipv6 address was loopback or
not
- The fullpath indication (which is used to compute the key) was not
properly being computed. This led to the same tuple being used
as a different key, so transactions were not properly combined.
* fix grammar error in release notes
* Add the plumbing in the agent forwarder to submit container images and SBOM (#14962)
* Improve documentation for BundleParams (#15011)
* pkg/clusteragent/admission: add unit tests (#15044)
* [CWS] bump syscall table + extract into separate task (#15061)
* 5.19 -> 6.1
* switch syscall table generator from go generate to task
* extract linux version
* [gitlab] Temporarily disable SUSE Agent 5 upgrade tests (#15055)
* [corechecks/snmp] Add LLDP remote device IP address (#14946)
* [CWS] add discarders eBPF unit test (#14471)
* [CWS] add discarder retention ut
* add another test
* add a unit test task
* add trace param
* make eBPF test part of the CI
* fake time to speed up tests
* bump baloum version
* add more tests
* [CWS Agent] Moving SecAgent subcommands to new dir part 2 (#14915)
* moving flare command to subcommands dir
* consolidating and moving secagent config package
* moving runtime to subcommands dir
* moved check subcommand, updated compliance subcommand which is the entry point to check funcs
* moving compliance cmd to subcommand dir
* exporting CliParams and RunCheck in Check subcommand for Compliance tests
* fixing cluster agent entry point into the check subcommand
* Add `container_image` core check (#14567)
* Reorganize the specs for some kitchen test (#15027)
* [check command] Add `--instance-filter` option (#15034)
* Migrate systray to an fx.App (#14985)
Deprecate single-dash args and add double-dash args
Move code from cmd/systray to comp/systray
Update UAC manifest to requireAdministrator
Fix log file and add `system_tray.log_file` configuration option.
* epforwarder: update dbm samples endpoint prefix (#15053)
dbm-metrics-intake and dbquery-intake resolve to the same IPs. This change cleans up code so that we're only referencing one endpoint name.
* [process-agent] Refactor Check interface (#15063)
* [process-agent] Refactor Check interface
- Refactors Check interface to consolidate CheckWithRealTime features
- This will simplify integration with components in the future PRs since it eliminates casts
* Address feedback from @just-chillin
* usm: postgres classification: Reduced 5 seconds per test, 1m30s in total (#15070)
Improved the regex for which we are using to detect if the server is up and running, by that
we can spare the 'wait 5 seconds' in GetPGHandle
* CWS: sync BTFhub constants (#15074)
Co-authored-by: paulcacheux <paulcacheux@users.noreply.github.com>
* [DCA] Convert commands to Fx apps
* Extract magic strings into command.* constants
* [CWS] Add 4 tests, one for each kernel rate limiter algo (#15064)
* [CWS] remove useless callbacks (#15046)
* remove useless error check
* remove useless callback
* Add `SBOM` core check (#14989)
* Prevent check from running after it was unscheduled. (#15065)
* Prevent check from running after it was unscheduled.
If a check runs after it was unscheduled, in particular after it's
sender and samplers were removed, would create sender and samplers
again, leaking resources. This may happen if the check was cancelled
after it was put in the worker channel, but before worker called Run.
This change adjusts check_wrapper to make Cancel fully mutually
exclusive with Run, and adds a flag that would prevent Run from
executing the check after Cancel has completed.
* go fmt
* Update test helper
* Restrict flare file from being accessible by other users on Unix (#14862)
* pkg/clusteragent/admission/patch: poll rc on leadership switch (#15062)
* pkg/clusteragent/admission: add additional libconfig env vars (#15059)
* usm: classification: Split USM and NPM classifications (#15075)
USM does not need all classifiers, only those which we have dispatchers for (HTTP, and soon HTTP2)
* Python memory telemetry (#14757)
* Track memory used by the python arena allocator
pymalloc [1], Python built-in arena allocator is responsible for
handling small-sized allocations, while the rest goes through
the system malloc.
This patch tracks the amount of memory requested by pymalloc from the
operating system, allowing low cost, low granularity view into a
segment of python memory usage.
[1]: https://docs.python.org/3/c-api/memory.html#the-pymalloc-allocator
* inv -e rtloader.format
* Remove rtloader_mem.h from rtloader.h
This allows to call C malloc without warnings when we implement a
custom raw memory allocator for python.
* Add python raw allocator tracking.
Together with tracking pymalloc requests, this should give
comprehensive picture of memory allocated by the python interpreter.
* Make sure to call global malloc/free
In Pyraw allocator implementation, make sure to call global
malloc/calloc/realloc/free symbols, to avoid undesired interaction
with the rtloader-specific memory tracking (for example, call libc
free instead of RtLoader::free).
* Move all memory tracking to the same file
* Update Go naming to match C functions
pymalloc is now one of two tracked allocators, use pymem as umbrella.
* Add a note about new metrics to the docs
* Python memory telemetry supports py3 only
* Add releasenote
* Expand telemetry documentation.
* Update docs/dev/agent_memory.md
Co-authored-by: Kari Halsted <12926135+kayayarai@users.noreply.github.com>
* Update docs/dev/agent_memory.md
Co-authored-by: Kari Halsted <12926135+kayayarai@users.noreply.github.com>
* Update docs/dev/agent_memory.md
Co-authored-by: Kari Halsted <12926135+kayayarai@users.noreply.github.com>
* Update releasenotes/notes/pymem-telemetry-0f62acb520d80a1f.yaml
Co-authored-by: Kari Halsted <12926135+kayayarai@users.noreply.github.com>
* Update rtloader/three/three_mem.cpp
Co-authored-by: Scott Opell <me@scottopell.com>
* Improve metric description and remove outdated comment.
* Fix typo
* Add a comment about allocation size adjustments
Co-authored-by: Kari Halsted <12926135+kayayarai@users.noreply.github.com>
Co-authored-by: Scott Opell <me@scottopell.com>
* Add telemetry for number of contexts per origin (#15016)
* Add telemetry for number of contexts per origin
Report number of contexts at the end of flush for each container
sending dogstatsd metrics.
This PR relies on origin detection to provide a set of identifying
tags for each origin, and reports number of distinct contexts for each
tag set. While this may not fully identify individual origins when
running with low tagger cardinality, it accurately reflects the way
agent would aggregate metrics from different origins together if their
tags end up the same.
* Only enable per-origin stats if telemetry is enabled.
* [process-agent] Fix kitchen tests for process agent on main (#15072)
* include `functests` in `DD_PIPELINE_ID` for system probe and security agent functests (#15043)
* include `functests` in DD_PIPELINE_ID for system probe and security agent functests
* simpler/shorter pipeline_id
* [install_script] Backport removal of RPM signing key 4172A230 (#15082)
* [corechecks/snmp] LLDP resolve local interface (#14991)
* [CWS] fix rule in error reported twice (#15084)
* Add java package in our circle-ci image (#14665)
* Use DMI on EC2 Nitro instances to get host aliases
The Agent now leverage DMI information on Unix to get the instance ID on AWS EC2 when the metadata endpoint fails or
is not accessible. The instance ID is exposed throught DMI only on AWS Nitro instances.
This will not change the hostname of the Agent upon upgrading but will add to the list of host aliases.
* [CWS] add inode to pid context to detect exec loss (#14661)
* [CWS] add revision to pid context
* use inode instead of revision
* Fix post rebase
* Fix serializer tests flakiness (#15093)
* [RCM-632] Add UUID in request (#15088)
* Add org uuid field
* Add org uuid in request
* Remove generate file
* Comment exported method
* fix the receiver name consistency (#15068)
* Add limits to allocated dictionaries, prevent browser cross-site requests (#15067)
* pkg/trace/api: Move semantic conventions to separate internal package (#14963)
* [pkg/trace/api] Move semantic conventions to separate internal package
* Rename to shared
* Move tagContainersTags back to API package
* Rename package to 'header'
* Fix Windows build
* Factorize queue code duplicated at two places (#15098)
* Factorize the aggregating queue used by the SBOM and container image checks
* Mock time functions to make tests more reliable
* [single-machine-performance] Push agent containers to SMP ECR (#14438)
* [single-machine-performance] Push agent container to SMP ECR
This commit is an attempt to introduce pushing containers from Agent CI for
single-machine-performance's Regression Detector in our isolated
infrastructure. Much like we have done for vectordotdev/vector we intend to run
the Regression Detector on Agent changes, gi…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
DataDog/security-agent-policies@v0.42.0...v0.42.1
Motivation
Additional Notes
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Reviewer's Checklist
Triagemilestone is set.major_changelabel if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.changelog/no-changeloglabel has been applied.qa/skip-qalabel is not applied.team/..label has been applied, indicating the team(s) that should QA this change.need-change/operatorandneed-change/helmlabels have been applied.k8s/<min-version>label, indicating the lowest Kubernetes version compatible with this feature.