internal: Fixing _dd.p.dm decision maker collision on number 10.

[yuanyuanzhao3]

What does this PR do?

Fixing a collision in _dd.p.dm where number 10 was used both by data jobs and remote user rules.

An accompanying system test change PR is also made in DataDog/system-tests#2394.

** This is intended to be cherry-picked for patch release 1.63.1.**

Impact of the collision:
Small - feature won't be triggered unless allowed through a feature flag (currently disabled by default).

Motivation

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

[darccio]

Just for the record, parametric tests are failing due not being updated on main:

_ TestDynamicConfigSamplingRules.test_trace_sampling_rules_override_env[library_env0] _
[gw14] linux -- Python 3.9.19 /home/runner/work/dd-trace-go/dd-trace-go/venv/bin/python
self = <tests.parametric.test_dynamic_configuration.TestDynamicConfigSamplingRules object at 0x7f6970671a60>
library_env = {'DD_ENV': 'test_env', 'DD_INTERNAL_TELEMETRY_V2_ENABLED': '1', 'DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS': '0.2', 'DD_SERVICE': 'test_service', ...}
test_agent = <tests.parametric.conftest._TestAgentAPI object at 0x7f699d734100>
test_library = <utils.parametric._library_client.APMLibrary object at 0x7f699d6977f0>
    @parametrize(
        "library_env",
        [
            {
                **DEFAULT_ENVVARS,
                "DD_TRACE_SAMPLING_RULES": json.dumps([{"sample_rate": ENV_SAMPLING_RULE_RATE, "service": "*"}]),
            }
        ],
    )
    def test_trace_sampling_rules_override_env(self, library_env, test_agent, test_library):
        """The RC sampling rules should override the environment variable and decision maker is set appropriately.
    
        When RC is unset, the environment variable should be used.
        """
        RC_SAMPLING_RULE_RATE_CUSTOMER = 0.8
        RC_SAMPLING_RULE_RATE_DYNAMIC = 0.4
        assert RC_SAMPLING_RULE_RATE_CUSTOMER != ENV_SAMPLING_RULE_RATE
        assert RC_SAMPLING_RULE_RATE_DYNAMIC != ENV_SAMPLING_RULE_RATE
        assert RC_SAMPLING_RULE_RATE_CUSTOMER != DEFAULT_SAMPLE_RATE
        assert RC_SAMPLING_RULE_RATE_DYNAMIC != DEFAULT_SAMPLE_RATE
    
        trace = get_sampled_trace(test_library, test_agent, service="", name="env_name")
        assert_sampling_rate(trace, ENV_SAMPLING_RULE_RATE)
        # Make sure `_dd.p.dm` is set to "-3" (i.e., local RULE_RATE)
        span = trace[0]
        assert "_dd.p.dm" in span["meta"]
        # The "-" is a separating hyphen, not a minus sign.
        assert span["meta"]["_dd.p.dm"] == "-3"
    
        # Create a remote config entry with two rules at different sample rates.
        set_and_wait_rc(
            test_agent,
            config_overrides={
                "tracing_sampling_rules": [
                    {
                        "sample_rate": RC_SAMPLING_RULE_RATE_CUSTOMER,
                        "service": TEST_SERVICE,
                        "resource": "*",
                        "provenance": "customer",
                    },
                    {
                        "sample_rate": RC_SAMPLING_RULE_RATE_DYNAMIC,
                        "service": "*",
                        "resource": "*",
                        "provenance": "dynamic",
                    },
                ]
            },
        )
    
        trace = get_sampled_trace(test_library, test_agent, service=TEST_SERVICE, name="op_name")
        assert_sampling_rate(trace, RC_SAMPLING_RULE_RATE_CUSTOMER)
        # Make sure `_dd.p.dm` is set to "-10" (i.e., remote user rule)
        span = root_span(trace)
        assert "_dd.p.dm" in span["meta"]
>       assert span["meta"]["_dd.p.dm"] == "-10"
E       AssertionError: assert '-11' == '-10'
E         - -10
E         + -11
tests/parametric/test_dynamic_configuration.py:[62](https://github.com/DataDog/dd-trace-go/actions/runs/8836533580/job/24291020518#step:6:63)3: AssertionError

I think this is proof enough to assume the change is fine; it just needs to get updated in parametric-tests.

[yuanyuanzhao3]

The system tests

Just for the record, parametric tests are failing due not being updated on main:

_ TestDynamicConfigSamplingRules.test_trace_sampling_rules_override_env[library_env0] _
[gw14] linux -- Python 3.9.19 /home/runner/work/dd-trace-go/dd-trace-go/venv/bin/python
self = <tests.parametric.test_dynamic_configuration.TestDynamicConfigSamplingRules object at 0x7f6970671a60>
library_env = {'DD_ENV': 'test_env', 'DD_INTERNAL_TELEMETRY_V2_ENABLED': '1', 'DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS': '0.2', 'DD_SERVICE': 'test_service', ...}
test_agent = <tests.parametric.conftest._TestAgentAPI object at 0x7f699d734100>
test_library = <utils.parametric._library_client.APMLibrary object at 0x7f699d6977f0>
    @parametrize(
        "library_env",
        [
            {
                **DEFAULT_ENVVARS,
                "DD_TRACE_SAMPLING_RULES": json.dumps([{"sample_rate": ENV_SAMPLING_RULE_RATE, "service": "*"}]),
            }
        ],
    )
    def test_trace_sampling_rules_override_env(self, library_env, test_agent, test_library):
        """The RC sampling rules should override the environment variable and decision maker is set appropriately.
    
        When RC is unset, the environment variable should be used.
        """
        RC_SAMPLING_RULE_RATE_CUSTOMER = 0.8
        RC_SAMPLING_RULE_RATE_DYNAMIC = 0.4
        assert RC_SAMPLING_RULE_RATE_CUSTOMER != ENV_SAMPLING_RULE_RATE
        assert RC_SAMPLING_RULE_RATE_DYNAMIC != ENV_SAMPLING_RULE_RATE
        assert RC_SAMPLING_RULE_RATE_CUSTOMER != DEFAULT_SAMPLE_RATE
        assert RC_SAMPLING_RULE_RATE_DYNAMIC != DEFAULT_SAMPLE_RATE
    
        trace = get_sampled_trace(test_library, test_agent, service="", name="env_name")
        assert_sampling_rate(trace, ENV_SAMPLING_RULE_RATE)
        # Make sure `_dd.p.dm` is set to "-3" (i.e., local RULE_RATE)
        span = trace[0]
        assert "_dd.p.dm" in span["meta"]
        # The "-" is a separating hyphen, not a minus sign.
        assert span["meta"]["_dd.p.dm"] == "-3"
    
        # Create a remote config entry with two rules at different sample rates.
        set_and_wait_rc(
            test_agent,
            config_overrides={
                "tracing_sampling_rules": [
                    {
                        "sample_rate": RC_SAMPLING_RULE_RATE_CUSTOMER,
                        "service": TEST_SERVICE,
                        "resource": "*",
                        "provenance": "customer",
                    },
                    {
                        "sample_rate": RC_SAMPLING_RULE_RATE_DYNAMIC,
                        "service": "*",
                        "resource": "*",
                        "provenance": "dynamic",
                    },
                ]
            },
        )
    
        trace = get_sampled_trace(test_library, test_agent, service=TEST_SERVICE, name="op_name")
        assert_sampling_rate(trace, RC_SAMPLING_RULE_RATE_CUSTOMER)
        # Make sure `_dd.p.dm` is set to "-10" (i.e., remote user rule)
        span = root_span(trace)
        assert "_dd.p.dm" in span["meta"]
>       assert span["meta"]["_dd.p.dm"] == "-10"
E       AssertionError: assert '-11' == '-10'
E         - -10
E         + -11
tests/parametric/test_dynamic_configuration.py:[62](https://github.com/DataDog/dd-trace-go/actions/runs/8836533580/job/24291020518#step:6:63)3: AssertionError

I think this is proof enough to assume the change is fine; it just needs to get updated in parametric-tests.

Thanks for approving the PR and cherrypicking it into 1.63.1. The system tests change is in DataDog/system-tests#2394.

One question, should the golang.yml test filter be v1.63.1, v1.64.0-dev or something else?

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-04-26 18:24:22

Comparing candidate commit 2ba3f40 in PR branch yuanyuan.zhao/remote-rules-dm-collision-fix with baseline commit 4e7fb35 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 1 unstable metrics.