Add initial revision of oteltest #2487

pmcollins · 2024-05-01T16:09:49Z

This is an OpenTelemetry scenario runner for running integration tests and examples. It sets up and runs a scenario defined by each script in a given directory. Script authors define the dependencies, the environment variables, the executable part of the script + any client code, and oteltest sets up the environment, runs the script + client code, and sends the received telemetry back to the script, at which point it can perform assertions.

This was demoed at the Python SIG, and there was an initial PR in the core repo, but it has been recommended that this go into Contrib.

Addresses open-telemetry/opentelemetry-python#3872

pmcollins · 2024-05-01T16:48:39Z

Also, I published this package to pypi for my own testing purposes: https://pypi.org/project/oteltest/ . Not sure yet what the process is but am happy to transfer its ownership to this SIG.

util/oteltest/pyproject.toml

util/oteltest/README.md

util/oteltest/src/oteltest/__init__.py

util/oteltest/src/oteltest/private.py

Add missing docstrings Rename some public fields

util/oteltest/LICENSE

tammy-baylis-swi

Thank you @pmcollins !

xrmx

I think that oteltest may use a bit more of unit testing, the Telemetry class methods and some helpers in oteltest.private at least. I'm not asking to add them, just a thought.

pmcollins · 2024-05-09T13:33:19Z

I think that oteltest may use a bit more of unit testing, the Telemetry class methods and some helpers in oteltest.private at least. I'm not asking to add them, just a thought.

Totally agree. There's some testing that I want to add in later PRs but will add this now as well.

pmcollins · 2024-05-20T23:52:17Z

Thanks @xrmx -- I've added unit tests and missing telemetry functions where there was a lack of symmetry between the signals.

pmcollins · 2024-05-22T16:21:52Z

Hi all, any objections to me converting this to draft, opening a PR downstream, and closing this PR once that one has landed? (I'm happy to abort that process at any time to put this functionality here if folks would prefer)

Special thanks to @xrmx and @tammy-baylis-swi for the feedback!

xrmx · 2024-05-14T15:51:08Z

util/oteltest/src/oteltest/private.py

+
+    for req in oteltest_instance.requirements():
+        print(f"- Will install requirement: '{req}'")
+        run_subprocess([pip_path, "install", req])


Have you thought on installing all the requirements at once? e.g. requirements = " ".join(oteltest_instance.requirements())

That sounds like an excellent idea. Will try it.

xrmx · 2024-05-31T14:21:46Z

util/oteltest/src/oteltest/private.py

+
+    pip_path = script_venv.path_to_executable("pip")
+
+    run_subprocess([pip_path, "install", (wheel_file or "oteltest")])


oteltest version should probably be pinned to the same version of the one installed outside

Excellent point. Will fix.

This also should be installed after I've installed my requirements since they may have stricter dependencies on opentelemetry packages versions

BTW, there's no reason for us to require that an oteltest script define a class that formally implements the OtelTest base class. Longer term, I intend for the OtelTest base class to be just a convenience so your IDE can fill in the missing methods for you. But if you just have a class containing the name OtelTest, and it doesn't inherit from anything, but it does implement all of the OtelTest methods, I think we could easily consider that a suitable implementation. I'd like to not require that target scripts depend on oteltest at all.

(And we could allow methods to be missing as well)

util/oteltest/src/oteltest/__init__.py

xrmx · 2024-06-03T08:17:21Z

util/oteltest/src/oteltest/private.py

+        python_script_cmd.insert(0, v.path_to_executable(wrapper_script))
+
+    # pylint: disable=R1732
+    proc = subprocess.Popen(


With oteltest 0.11 I get the following:

- Running python script: test_process_runtime_sent_by_default.py Traceback (most recent call last): File "/home/rm/src/test-otel-python/venvtest/bin/oteltest", line 8, in <module> sys.exit(main()) File "/home/rm/src/test-otel-python/venvtest/lib/python3.10/site-packages/oteltest/main.py", line 42, in main run(args.script_dir, args.wheel_file, args.venv_parent_dir) File "/home/rm/src/test-otel-python/venvtest/lib/python3.10/site-packages/oteltest/private.py", line 37, in run setup_script_environment(temp_dir, script_dir, script, wheel_file) File "/home/rm/src/test-otel-python/venvtest/lib/python3.10/site-packages/oteltest/private.py", line 67, in setup_script_environment stdout, stderr, returncode = run_python_script( File "/home/rm/src/test-otel-python/venvtest/lib/python3.10/site-packages/oteltest/private.py", line 112, in run_python_script proc = subprocess.Popen( File "/usr/lib/python3.10/subprocess.py", line 971, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) PermissionError: [Errno 13] Permission denied: '/tmp/tmpcawnrbcp/test_process_runtime_sent_by_default/bin/'

The python_script_cmd is ['/tmp/tmp8frcymoc/test_process_runtime_sent_by_default/bin/', '/tmp/tmp8frcymoc/test_process_runtime_sent_by_default/bin/python', 'tests/integration/test_process_runtime_sent_by_default.py']

On the test class I have

def wrapper_command(self): return "opentelemetry-instrument"

The root cause I guess is load_test_class_for_script loading the imported class instead of the one defined in the test module 😅

Worked around by excluding the name of my class explicitly in is_test_class but that would probably require an argument in oteltest or some magic

I was hitting a similar problem (I think) where if the script name (minus ".py") was equal to an available package name, it would fail to load the script. I believe this is fixed now (just pushed here and published to pypi as well).

Nope, it's a different issue, my problem is with using an intermediate class that satisfies is_test_class checks but it's not a real test case.

Ah right. Will see if checking for is not ABC will fix that.

it will for an ABC class but I don't have an ABC class.

Oh, well I just added a check for not isabstract, which I think is good in any case. 😄

Would it be acceptable to mark your intermediate class as ABC? After this change, that should get it to not be recognized as a test class.

Nope, adding an "OtelTest" in value.__name__ clause works for me

util/oteltest/src/oteltest/private.py

xrmx · 2024-06-03T10:12:58Z

util/oteltest/src/oteltest/private.py

+    )
+    print_subprocess_result(stdout, stderr, returncode)
+
+    filename = get_next_json_file(script_dir, module_name)


Would be nice to have an option to avoid saving these files

Yes. Maybe should be off by default and on via a CLI switch. Noted for a later PR.

xrmx · 2024-06-03T10:21:33Z

util/oteltest/src/oteltest/telemetry.py

+
+def num_spans(telemetry) -> int:
+    out = 0
+    for req in telemetry.trace_requests:


It may make sense to have a flattened list of span serialized like the ConsoleSpanExporter we have in opentelemetry sdk. That would help a lot with writing asserts on specific attributes.

All for it. Noted for a later PR.

Catch base exception instead of keyboard interrupt Explicitly flush print statements Load script using spec file path to prevent name collisions Format source files Bump version to 0.12

util/oteltest/src/oteltest/sink/__init__.py

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

pmcollins added 2 commits April 29, 2024 10:51

Add initial revision of oteltest

89becd6

Add changelog entry

af1d5b8

pmcollins marked this pull request as ready for review May 1, 2024 16:38

pmcollins requested a review from a team as a code owner May 1, 2024 16:38

tammy-baylis-swi reviewed May 1, 2024

View reviewed changes

util/oteltest/pyproject.toml Outdated Show resolved Hide resolved

tammy-baylis-swi reviewed May 1, 2024

View reviewed changes

util/oteltest/README.md Outdated Show resolved Hide resolved

tammy-baylis-swi and others added 3 commits May 1, 2024 16:49

Merge branch 'main' into oteltest

7129b31

Fix README and project URLs

76f834f

Merge branch 'main' into oteltest

46a12c2

tammy-baylis-swi reviewed May 3, 2024

View reviewed changes

util/oteltest/src/oteltest/__init__.py Outdated Show resolved Hide resolved

tammy-baylis-swi reviewed May 3, 2024

View reviewed changes

util/oteltest/src/oteltest/__init__.py Outdated Show resolved Hide resolved

tammy-baylis-swi reviewed May 3, 2024

View reviewed changes

util/oteltest/src/oteltest/private.py Show resolved Hide resolved

tammy-baylis-swi and others added 3 commits May 3, 2024 10:30

Merge branch 'main' into oteltest

a193909

Add missing license headers

8387af7

Add missing docstrings Rename some public fields

Apply black

2cae6ef

tammy-baylis-swi reviewed May 3, 2024

View reviewed changes

util/oteltest/LICENSE Outdated Show resolved Hide resolved

pmcollins added 2 commits May 6, 2024 11:12

Update license

1b4f24c

Merge branch 'main' into oteltest

17c51db

tammy-baylis-swi approved these changes May 6, 2024

View reviewed changes

xrmx reviewed May 9, 2024

View reviewed changes

pmcollins added 5 commits May 14, 2024 12:39

start adding more tests

c195804

add tests and reorg

4f546b1

format with 'black'

12693fc

format with 'isort'

ac75454

fix lint

df31502

pmcollins marked this pull request as draft May 22, 2024 16:22

xrmx reviewed May 31, 2024

View reviewed changes

util/oteltest/src/oteltest/__init__.py Show resolved Hide resolved

xrmx reviewed Jun 3, 2024

View reviewed changes

util/oteltest/src/oteltest/private.py Outdated Show resolved Hide resolved

xrmx reviewed Jun 3, 2024

View reviewed changes

pmcollins added 2 commits June 3, 2024 14:51

End subprocesses on timeout + various

1473a52

Catch base exception instead of keyboard interrupt Explicitly flush print statements Load script using spec file path to prevent name collisions Format source files Bump version to 0.12

Check for isabstract in is_test_class()

10166d1

xrmx reviewed Jun 6, 2024

View reviewed changes

util/oteltest/src/oteltest/sink/__init__.py Outdated Show resolved Hide resolved

Update util/oteltest/src/oteltest/sink/__init__.py

794dd67

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial revision of oteltest #2487

Add initial revision of oteltest #2487

pmcollins commented May 1, 2024 •

edited

pmcollins commented May 1, 2024 •

edited

tammy-baylis-swi left a comment

xrmx left a comment

pmcollins commented May 9, 2024

pmcollins commented May 20, 2024

pmcollins commented May 22, 2024

xrmx May 14, 2024

pmcollins May 31, 2024

xrmx May 31, 2024 •

edited

pmcollins May 31, 2024

xrmx May 31, 2024

pmcollins May 31, 2024 •

edited

pmcollins May 31, 2024

xrmx Jun 3, 2024 •

edited

xrmx Jun 3, 2024 •

edited

xrmx Jun 3, 2024 •

edited

pmcollins Jun 3, 2024

xrmx Jun 3, 2024

pmcollins Jun 3, 2024

xrmx Jun 3, 2024

pmcollins Jun 3, 2024

xrmx Jun 5, 2024 •

edited

xrmx Jun 3, 2024

pmcollins Jun 4, 2024

xrmx Jun 3, 2024 •

edited

pmcollins Jun 4, 2024


		pip_path = script_venv.path_to_executable("pip")

		run_subprocess([pip_path, "install", (wheel_file or "oteltest")])

Add initial revision of oteltest #2487

Are you sure you want to change the base?

Add initial revision of oteltest #2487

Conversation

pmcollins commented May 1, 2024 • edited

pmcollins commented May 1, 2024 • edited

tammy-baylis-swi left a comment

Choose a reason for hiding this comment

xrmx left a comment

Choose a reason for hiding this comment

pmcollins commented May 9, 2024

pmcollins commented May 20, 2024

pmcollins commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xrmx May 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmcollins May 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xrmx Jun 3, 2024 • edited

Choose a reason for hiding this comment

xrmx Jun 3, 2024 • edited

Choose a reason for hiding this comment

xrmx Jun 3, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xrmx Jun 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xrmx Jun 3, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmcollins commented May 1, 2024 •

edited

pmcollins commented May 1, 2024 •

edited

xrmx May 31, 2024 •

edited

pmcollins May 31, 2024 •

edited

xrmx Jun 3, 2024 •

edited

xrmx Jun 3, 2024 •

edited

xrmx Jun 3, 2024 •

edited

xrmx Jun 5, 2024 •

edited

xrmx Jun 3, 2024 •

edited