Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes for NR CSEC module support #819

Open
wants to merge 124 commits into
base: develop-k2-integration
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
b5711e3
initial changes for k2 integration
AnupamJuniwal Sep 21, 2022
20d5117
minor fix
AnupamJuniwal Sep 21, 2022
dfbcf0a
k2 agent would now init without any args
AnupamJuniwal Sep 22, 2022
f399c3a
initial implementation of securty config, removal of switch based k2 …
AnupamJuniwal Sep 27, 2022
c3232c9
minor fix with incorrect value for configuration mapper
AnupamJuniwal Sep 28, 2022
206160e
changes corresponding to latest k2 agent changes
AnupamJuniwal Sep 29, 2022
dd7bd2d
minor fixes
AnupamJuniwal Sep 29, 2022
7578dc0
updated integration source for k2
AnupamJuniwal Sep 30, 2022
857f3f9
Added customer id
AnupamJuniwal Sep 30, 2022
f0c30a1
initial changes for application id propagation
AnupamJuniwal Oct 1, 2022
42d621a
minor fix in application_id extraction
AnupamJuniwal Oct 1, 2022
e700fb2
minor fix
AnupamJuniwal Oct 1, 2022
1d5964b
guid propagation instead of application_id
AnupamJuniwal Oct 2, 2022
a2da1ba
Merge branch 'feature/k2_integration' into k2_integration
AnupamJuniwal Oct 2, 2022
163645b
This implements propagation of all possible k2 config in NR's config
AnupamJuniwal Oct 2, 2022
52e59e2
changes to populate dictionary of policy and changes to reflect secur…
AnupamJuniwal Oct 3, 2022
a609df8
This contains changes for policy propagation to k2
AnupamJuniwal Oct 6, 2022
30a280a
Merge branch 'feature/k2i/policy_propagation' into k2_integration
AnupamJuniwal Oct 6, 2022
a5dd418
updated config for logs upload and exception handling in security mod…
AnupamJuniwal Oct 7, 2022
5f226e8
Merge branch 'feature/k2i/logs_config_propagation' into k2_integration
AnupamJuniwal Oct 9, 2022
6d515e8
Changes for setting the trasnsaction id catcher by handing over a lam…
AnupamJuniwal Oct 9, 2022
34d610b
changes to send trace metadata long with transaction id from metadata…
AnupamJuniwal Oct 10, 2022
e02eacf
Merge 'feature/k2i/logs_config_propagation' into k2_integration
AnupamJuniwal Oct 10, 2022
385b966
This contains multiple changes:
AnupamJuniwal Oct 12, 2022
ded8f48
minor change to reflect k2 module changes
AnupamJuniwal Oct 12, 2022
5298430
Merge branch 'feature/k2i/add_linking_metadata_deprecate_old_k2_auth_…
AnupamJuniwal Oct 12, 2022
a2f385e
Addition of enforce flag
AnupamJuniwal Oct 26, 2022
2f46837
Merge branch 'main' into k2_integration
AnupamJuniwal Oct 26, 2022
a8a7a88
Addition of account id in linking metadata
AnupamJuniwal Dec 21, 2022
d925658
Temp changes
AnupamJuniwal Dec 23, 2022
a741c2d
Let the connect be called with refresh (When linking metadata is avai…
AnupamJuniwal Dec 23, 2022
0ec4fe4
agent would not connect with startup
AnupamJuniwal Dec 28, 2022
f5a98ba
removal of rest server endpoint config, use of SingletonAgentConfig i…
AnupamJuniwal Jan 25, 2023
524b315
removal of all security agent initialization logic
AnupamJuniwal Jan 25, 2023
46637fe
updates to newrelic config for security module
AnupamJuniwal Mar 14, 2023
5be1836
minor fix
AnupamJuniwal Mar 23, 2023
ac7e712
point to nr_adaptation with updated newrelic_security package
AnupamJuniwal Apr 17, 2023
a028082
updated to latest csec statup update
AnupamJuniwal Apr 17, 2023
49d7a99
Merge remote-tracking branch 'nr-public/main' into feature/nr_adaptation
AnupamJuniwal Apr 17, 2023
28d789a
Refactoring and relocation of security settings in core config
AnupamJuniwal Apr 28, 2023
1f780c2
initialising security agent before configuring nr apm hooks
AnupamJuniwal May 4, 2023
f0505f8
Merge branch 'main' into feature/nr_adaptation
AnupamJuniwal May 10, 2023
5fb7a42
fixes in config default values for security config for python 2.7 sup…
AnupamJuniwal May 17, 2023
11c0049
Will use dev branch for security agent
AnupamJuniwal May 26, 2023
1313de9
Merge branch 'k2io/nr-python-agent/develop' into develop-k2-integration
AnupamJuniwal Jun 1, 2023
093fd3a
Updated remote for pulling newrelic_security module
AnupamJuniwal Jun 1, 2023
88cb32e
Update install requires line.
umaannamalai Jun 1, 2023
7a43a7f
[Mega-Linter] Apply linters fixes
umaannamalai Jun 1, 2023
33b51fc
Testing install requires.
umaannamalai Jun 1, 2023
0c2cbbc
Merge conflicts.
umaannamalai Jun 1, 2023
7041783
Fixed k2 reference to Security Agent
AnupamJuniwal Jun 12, 2023
f192c24
Fix Testing Failures (#828)
TimPansino Jun 12, 2023
3def8b0
Fix pytest test filtering when running tox (#823)
hmstepanek Jun 12, 2023
90ccb4c
Validator transfer p3 (#745)
lrafeei Jun 14, 2023
668b0a9
Fix set output warning using new GHA syntax (#833)
TimPansino Jun 21, 2023
abb6405
Remove Python 2.7 and pypy2 testing (#835)
lrafeei Jun 21, 2023
ab92daf
Containerized CI Pipeline (#836)
TimPansino Jun 22, 2023
4422b95
Fix CI Image Tagging (#838)
TimPansino Jun 22, 2023
4da4612
Temporarily Restore Old CI Pipeline (#841)
TimPansino Jun 22, 2023
658f818
Rework CI Pipeline (#839)
TimPansino Jun 22, 2023
57720fd
Fix Tests on New CI (#843)
TimPansino Jun 23, 2023
a7dfe33
Instrument Redis waitaof (#851)
TimPansino Jun 26, 2023
33aa111
Ignore patched hooks files. (#849)
umaannamalai Jun 26, 2023
e707cc0
Fix local scoped package reporting (#837)
hmstepanek Jun 26, 2023
ab590a2
MSSQL Testing (#852)
TimPansino Jun 27, 2023
db07523
Exclude command line functionality from test coverage (#855)
lrafeei Jun 27, 2023
c2fd5e3
FIX: resilient environment settings (#825)
aaeabdo Jun 27, 2023
9883c2b
Replace drop_transaction logic by using transaction context manager (…
lrafeei Jun 28, 2023
998b035
Upgrade to Pypy38 for TypedDict (#861)
lrafeei Jun 30, 2023
66c2e19
Add profile_trace testing (#858)
umaannamalai Jun 30, 2023
e663c36
Add Transaction API Tests (#857)
lrafeei Jun 30, 2023
3bdb013
Add tests for jinja2. (#842)
umaannamalai Jun 30, 2023
6644846
Add tests for newrelic/config.py (#860)
hmstepanek Jun 30, 2023
ee92363
Fix starlette testing matrix for updated behavior. (#869)
TimPansino Jul 14, 2023
53fc51a
Correct Serverless Distributed Tracing Logic (#870)
TimPansino Jul 14, 2023
2f580af
Fix Kafka CI (#863)
TimPansino Jul 14, 2023
a7080e9
Change image tag to latest (#871)
hmstepanek Jul 17, 2023
56ea815
Add full version for pypy3.8 to tox (#872)
lrafeei Jul 18, 2023
a248688
Instrument RedisCluster (#809)
hmstepanek Jul 27, 2023
08eec5e
Ignore Django instrumentation from older versions (#859)
lrafeei Jul 28, 2023
4b3768b
Modify postgresql tests to include WITH query (#885)
lrafeei Aug 2, 2023
17f8937
Develop redis addons (#888)
lrafeei Aug 10, 2023
8ebe9a3
Add google firestore instrumentation (#893)
umaannamalai Aug 10, 2023
238b64d
Base Devcontainer on CI Image (#873)
TimPansino Aug 10, 2023
dc87bd3
Add support for redis v5. (#895)
umaannamalai Aug 15, 2023
7d76243
Use importlib.metadata first to avoid deprecation warnings (#878)
renanivo Aug 16, 2023
f1a673e
Fix Normalization Rules (#894)
TimPansino Aug 16, 2023
6a6228f
Fix database instance metric bug (#905)
hmstepanek Aug 18, 2023
62abb45
Add check for both path and file (#907)
hmstepanek Aug 21, 2023
399c81f
Update structlog instrumentation. (#865)
umaannamalai Aug 23, 2023
3988ecc
GraphQL Async Instrumentation Support (#908)
umaannamalai Aug 28, 2023
0baf8d5
Develop swap redis asyncio commits (#913)
lrafeei Aug 28, 2023
faaccd0
Increase days until stale. (#909)
umaannamalai Aug 28, 2023
b1be563
Pin anyio version to below 4.0.0 (#916)
lrafeei Sep 1, 2023
e371b02
Add redis.asyncio.Connection instrumentation (#919)
lrafeei Sep 12, 2023
eff66b5
Update testing matrix for supported packages. (#904)
umaannamalai Sep 14, 2023
17cd48d
Add sklearn instrumentation and ML model feature support (#921)
umaannamalai Sep 25, 2023
eef6916
Pin flask version for flask restx tests. (#931)
umaannamalai Oct 9, 2023
d577a69
Ignore new redis methods. (#932)
umaannamalai Oct 9, 2023
13e9891
Update CI Image (#930)
TimPansino Oct 9, 2023
43160af
Only get package version once (#928)
hmstepanek Oct 9, 2023
cc3e285
Cache Package Version Lookups (#946)
TimPansino Oct 19, 2023
5996de6
Fix Redis Generator Methods (#947)
TimPansino Oct 19, 2023
4721025
Automatic RPM System Updates (#948)
TimPansino Oct 23, 2023
2191684
Drop python 3.7 tests for Hypercorn (#954)
lrafeei Oct 30, 2023
b2512eb
Fix pyenv installation for devcontainer (#936)
TimPansino Nov 2, 2023
b12f7be
Remove duplicate kafka import hook (#956)
lrafeei Nov 2, 2023
b2e9e74
Handle 0.32.0.post1 version in tests (#963)
hmstepanek Nov 6, 2023
72aa6e8
Fix botocore tests (#973)
hmstepanek Nov 13, 2023
f939014
Package Version Performance Regression (#970)
TimPansino Nov 13, 2023
5cba84e
Merge remote-tracking branch 'refs/remotes/newrelic-public-fork/devel…
AnupamJuniwal Nov 14, 2023
a7e4870
Merge branch 'main' into develop-k2-integration
AnupamJuniwal Nov 14, 2023
3980127
Synthetics Info Header Support (#896)
TimPansino Nov 16, 2023
5eb1095
Fix CI Image Permissions for Non-Root Users (#969)
TimPansino Nov 16, 2023
fdaa4be
Add package_capturing.enabled setting (#982)
hmstepanek Nov 16, 2023
1986366
Revert "Synthetics Info Header Support (#896)" (#983)
TimPansino Nov 16, 2023
b6a9121
Remove accidental quote from api keys (#985)
TimPansino Nov 16, 2023
374bc67
Synthetics Info Header Support (#984)
TimPansino Nov 29, 2023
6e0b12e
Docker CGroups v2 Utilization Support (#980)
TimPansino Nov 30, 2023
55144b1
Testing for supported frameworks in Python 3.12 (#897)
lrafeei Nov 30, 2023
007be52
Remove all references to NR staging (#989)
TimPansino Dec 4, 2023
cd74bc4
Fix bug with Structlog CallsiteParameter processor (#990)
umaannamalai Dec 6, 2023
030cfc9
Update wrapt (#993)
TimPansino Dec 6, 2023
140bb54
Merge branch 'main' into develop-k2-integration
AnupamJuniwal Dec 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 26 additions & 0 deletions newrelic/config.py
Expand Up @@ -329,6 +329,13 @@ def _process_configuration(section):
_process_setting(section, "ca_bundle_path", "get", None)
_process_setting(section, "audit_log_file", "get", None)
_process_setting(section, "monitor_mode", "getboolean", None)
_process_setting(section, "security.agent.enabled", "getboolean", None)
_process_setting(section, "security.enabled", "getboolean", None)
_process_setting(section, "security.mode", "get", None)
_process_setting(section, "security.validator_service_url", "get", None)
_process_setting(section, "security.detection.rci.enabled", "getboolean", None)
_process_setting(section, "security.detection.rxss.enabled", "getboolean", None)
_process_setting(section, "security.detection.deserialization.enabled", "getboolean", None)
_process_setting(section, "developer_mode", "getboolean", None)
_process_setting(section, "high_security", "getboolean", None)
_process_setting(section, "capture_params", "getboolean", None)
Expand Down Expand Up @@ -3169,6 +3176,23 @@ def _setup_agent_console():
newrelic.core.agent.Agent.run_on_startup(_startup_agent_console)


def _setup_security_module():
"""Initiates k2 security module and adds a
callback to agent startup to propagate NR config
"""
try:
if not _settings.security.agent.enabled:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there error handling already in the security agent to handle security.mode or the security.validator_service_url settings not being configured by the user?

return
from newrelic_security.api.agent import Agent as SecurityAgent

# initialize security agent
security_agent = SecurityAgent()
# create a callback to reinitialise the security module
newrelic.core.agent.Agent.run_on_startup(security_agent.refresh_agent)
except Exception as k2error:
_logger.error("K2 Startup failed with error %s", k2error)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change the language here to use Security Agent instead of K2 to be consistent with other agents?


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _setup_security_module():
"""Initiates k2 security module and adds a
callback to agent startup to propagate NR config
"""
try:
if not _settings.security.agent.enabled:
return
from newrelic_security.api.agent import Agent as SecurityAgent
# initialize security agent
security_agent = SecurityAgent()
# create a callback to reinitialise the security module
newrelic.core.agent.Agent.run_on_startup(security_agent.refresh_agent)
except Exception as k2error:
_logger.error("K2 Startup failed with error %s", k2error)
def _setup_security_module():
def _security_agent_registration_callback():
"""Callback that initializes k2 security module and propagates NR config."""
try:
from newrelic.api.application import application_instance
application = application_instance(activate=False)
if not application or not application.settings or not application.settings.security.agent.enabled:
return
from newrelic_security.api.agent import Agent as SecurityAgent
# initialize security agent
security_agent = SecurityAgent()
security_agent.refresh_agent()
except Exception as k2error:
_logger.error("K2 Startup failed with error %s", k2error)
# create a callback to only initialize the security module after high security settings have been applied
newrelic.core.agent.Agent.run_on_registration(_security_agent_registration_callback)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For compliance I think we can't run any of this until high security mode has a chance to disable it. We'll need to wait until after connect and registration, and also use application instance settings instead of the global settings (which don't include server side fixups like high security).

I think this callback should do the trick, but that depends on if

  1. Your agent is a singleton and can be called like this multiple times with no issues.
  2. Your agent can handle being called this late into the import process.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should DEFINITELY add tests for this, somehow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above code suggestion will also need the application instance passed to the call to run_on_registration()


def initialize(
config_file=None,
environment=None,
Expand All @@ -3187,6 +3211,8 @@ def initialize(

_load_configuration(config_file, environment, ignore_errors, log_file, log_level)

_setup_security_module()

if _settings.monitor_mode or _settings.developer_mode:
_settings.enabled = True
_setup_instrumentation()
Expand Down
41 changes: 40 additions & 1 deletion newrelic/core/config.py
Expand Up @@ -279,6 +279,30 @@ class ApplicationLoggingLocalDecoratingSettings(Settings):
pass


class SecuritySettings(Settings):
pass


class SecurityDetectionSettings(Settings):
pass


class SecurityAgentSettings(Settings):
pass


class SecurityDetectionRCISettings(Settings):
pass


class SecurityDetectionRXSSSettings(Settings):
pass


class SecurityDetectionDeserializationSettings(Settings):
pass


class InfiniteTracingSettings(Settings):
_trace_observer_host = None

Expand Down Expand Up @@ -395,6 +419,12 @@ class EventHarvestConfigHarvestLimitSettings(Settings):
_settings.message_tracer = MessageTracerSettings()
_settings.process_host = ProcessHostSettings()
_settings.rum = RumSettings()
_settings.security = SecuritySettings()
_settings.security.agent = SecurityAgentSettings()
_settings.security.detection = SecurityDetectionSettings()
_settings.security.detection.deserialization = SecurityDetectionDeserializationSettings()
_settings.security.detection.rci = SecurityDetectionRCISettings()
_settings.security.detection.rxss = SecurityDetectionRXSSSettings()
_settings.serverless_mode = ServerlessModeSettings()
_settings.slow_sql = SlowSqlSettings()
_settings.span_events = SpanEventSettings()
Expand All @@ -412,7 +442,6 @@ class EventHarvestConfigHarvestLimitSettings(Settings):
_settings.transaction_tracer.attributes = TransactionTracerAttributesSettings()
_settings.utilization = UtilizationSettings()


_settings.log_file = os.environ.get("NEW_RELIC_LOG", None)
_settings.audit_log_file = os.environ.get("NEW_RELIC_AUDIT_LOG", None)

Expand Down Expand Up @@ -840,6 +869,16 @@ def default_host(license_key):
"NEW_RELIC_APPLICATION_LOGGING_LOCAL_DECORATING_ENABLED", default=False
)

_settings.security.agent.enabled = _environ_as_bool("NEW_RELIC_SECURITY_AGENT_ENABLED", False)
_settings.security.enabled = _environ_as_bool("NEW_RELIC_SECURITY_ENABLED", False)
_settings.security.mode = os.environ.get("NEW_RELIC_SECURITY_MODE", "IAST")
_settings.security.validator_service_url = os.environ.get("NEW_RELIC_SECURITY_VALIDATOR_SERVICE_URL", None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to provide wss://csec.nr-data.net as the default here.

_settings.security.detection.rci.enabled = _environ_as_bool("NEW_RELIC_SECURITY_DETECTION_RCI_ENABLED", True)
_settings.security.detection.rxss.enabled = _environ_as_bool("NEW_RELIC_SECURITY_DETECTION_RXSS_ENABLED", True)
_settings.security.detection.deserialization.enabled = _environ_as_bool(
"NEW_RELIC_SECURITY_DETECTION_DESERIALIZATION_ENABLED", True
)


def global_settings():
"""This returns the default global settings. Generally only used
Expand Down
27 changes: 26 additions & 1 deletion newrelic/newrelic.ini
Expand Up @@ -49,6 +49,32 @@ app_name = Python Application
# NEW_RELIC_MONITOR_MODE environment variable.
monitor_mode = true

# Indicates if attack detection security module is to be enabled
security.enabled = false

# To completely disable security set flag to false If the flag is
# set to false, the security module is not loaded. This property
# is read only once at application start.
security.agent.enabled = false


# security module provides two modes IAST or RASP
# RASP stands for Runtime Application Self Protection
# while IAST for Interactive Application Security Testing
# Default mode is IAST
security.mode = IAST


# web-protect agent endpoint connection URLs
security.validator_service_url = wss://csec.nr-data.net


# vulnerabilty detection flags
security.detection.rci.enabled = true
security.detection.rxss.enabled = true
security.detection.deserialization.enabled = true


# Sets the name of a file to log agent messages to. Whatever you
# set this to, you must ensure that the permissions for the
# containing directory and the file itself are correct, and
Expand Down Expand Up @@ -251,5 +277,4 @@ monitor_mode = true

[newrelic:production]
monitor_mode = true

# ---------------------------------------------------------------------------
1 change: 1 addition & 0 deletions setup.py
Expand Up @@ -152,6 +152,7 @@ def build_extension(self, ext):
"newrelic": ["newrelic.ini", "version.txt", "packages/urllib3/LICENSE.txt", "common/cacert.pem"],
},
extras_require={"infinite-tracing": ["grpcio", "protobuf"]},
install_requires=["newrelic_security @ git+https://github.com/newrelic/csec-python-agent.git@develop"]
)

if with_setuptools:
Expand Down