Django app on K8s crashes when auto-instrumented via the opentelemetry-operator without explicitly setting the PYTHONPATH and DJANGO_SETTINGS_MODULE #2495
Labels
bug
Something isn't working
Comments
|
This issue was also discussed in the Python SIG meeting on May 02, 2024. Notes: https://docs.google.com/document/d/1CIMGoIOZ-c3-igzbd6_Pnxx1SjAkjwqoYSUWxPY8XIs/edit Between the meeting and discussion on this slack thread, it seems like the operator login needs to be fixed on how the pythonpath is handled. |
|
From the SIG meeting discussion, tagging @jeremydvoss since he encountered this issue as well and to take a look. |
srprash
added a commit
to aws-observability/aws-otel-python-instrumentation
that referenced
this issue
May 8, 2024
…179) *Issue #, if available:* open-telemetry/opentelemetry-python-contrib#2495 The issue is that users on K8s/EKS/ECS explicitly need to set their Django application container's working directory in the `PYTHONPATH` environment variable ([operator code](https://github.com/open-telemetry/opentelemetry-operator/blob/d42a0ce1166efe86c95f85268f265e338d0002f8/pkg/instrumentation/python.go#L55-L63) that handles the `PYTHONPATH`), otherwise the application fails to start with error messages related to the Django's settings module. *Analysis:* Python auto-instrumentation using the `opentelemetry-instrument` command doesn't have this issue, which is I believe is because of [this specific handling of Django app current directory](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/1ee7261ea7117fbd22e2262e488402213a874125/opentelemetry-instrumentation/src/opentelemetry/instrumentation/auto_instrumentation/__init__.py#L98-L102) in the auto-instrumentation startup. Since [this script is only run when using the `opentelemetry-instrument` command](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/1ee7261ea7117fbd22e2262e488402213a874125/opentelemetry-instrumentation/pyproject.toml#L34), auto-instrumentation done by the operator (using the [sitecustomize](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/opentelemetry-instrumentation/src/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py) I believe) doesn't set the current working directory in the PYTHONPATH. *Description of changes:* Solution is to insert the current working directory to the `sys.path` within the auto-instrumentation so that Django can find the application folder. We are adding to the `sys.path` and not `PYTHONPATH` because at this point the python process has already started and mutations to `PYTHONPATH` doesn't reflect in the `sys.path` which is what Django uses. *Testing:* Manually tested by deploying a [Django sample application](https://github.com/srprash/otel-python-k8s-samples/tree/main/django) to minikube without setting the `PYTHONPATH` in the container environment. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.
Steps to reproduce
Sample app code is here: https://github.com/srprash/otel-python-k8s-samples/tree/main/django
eval $(minikube docker-env)so that minikube can use the docker images from local repository.docker build -t django-app .kubectl apply -f k8s/deployment.yamlandkubectl apply -f k8s/service.yamlkubectl port-forward svc/django-service 8000http://127.0.0.1:8000/outgoing-http-callendpointWhat is the expected behavior?
What did you expect to see?
What is the actual behavior?
What did you see instead?
PYTHONPATHandDJANGO_SETTINGS_MODULE), only then the application works correctly and is instrumented.Additional context
instrumentation.opentelemetry.io/inject-python: "true"annotation to the application deployment manifest. Auto-instrumentation via theopentelemetry-instrumentcommand works fine (likely due to this code that handles the current working directory).The text was updated successfully, but these errors were encountered: