Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stacktrace frame vars reported as NaN #1886

Open
kruskall opened this issue Aug 9, 2023 · 5 comments
Open

stacktrace frame vars reported as NaN #1886

kruskall opened this issue Aug 9, 2023 · 5 comments
Assignees
@xrmx
Labels
8.12-candidate agent-python bug upstream Issue caused by an upstream package
Projects
APM-Agents (OLD)

Comments

@kruskall
Copy link
Member

kruskall commented Aug 9, 2023

Describe the bug:

As part of the ops KPI review we've been investigating the following errors in ecs logs:

decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Stacktrace: []v2.stacktraceFrame: v2.stacktraceFrame.Vars: Read: unexpected value type: 0, error found in #10 byte of ...|length": [NaN, NaN, |..., bigger context ...| redacted ...", "common_length": [NaN, NaN, NaN, NaN, NaN, NaN, 27.0, 76.0, 76.0, 76|...

vars is defined as a flat mapping of local variables but NaN is not a valid type.

To Reproduce

I don't have steps to reproduce, this is part of the ops KPI review and was observed in the ecs logs.

Environment (please complete the following information)

The error has been observed in the following agent versions:

  • apm-agent-python/6.18.0
  • apm-agent-python/6.15.1
  • apm-agent-python/6.13.2
  • apm-agent-python/6.12.0
  • apm-agent-python/6.10.0
  • apm-agent-python/6.7.2
  • elasticapm-python/5.10.1
  • elasticapm-python/5.10.0

This happens on multiple apm-server versions.

Additional context

none
@basepi
Copy link
Contributor

basepi commented Aug 10, 2023

As discussed in slack, we could probably fix this by adding a dep for simplejson; however, the performance implications are unclear. Alternatively, we could do allow_nan=False and drop the JSON when it raises.

@basepi basepi added bug upstream Issue caused by an upstream package labels Aug 10, 2023
@basepi basepi added this to the 8.11 milestone Aug 23, 2023
@basepi basepi removed this from the 8.11 milestone Feb 9, 2024
@aydasraf
Copy link

We are facing the same issue, and this is blocking the whole ML APM Adoption, any ideas if this fix has been decided to be implemented or any release expectations? Any insights would be extremely helpful .

@xrmx
Copy link
Member

xrmx commented Mar 21, 2024 •
edited

Will converting NaNs to null fix things for you? We can fix that shortly if that's sufficient.

@aydasraf
Copy link

@xrmx In our case, yes, it will be extremely helpful.

@xrmx xrmx self-assigned this Mar 22, 2024
xrmx added a commit to xrmx/apm-agent-python that referenced this issue Mar 22, 2024
@xrmx
utils: add simplejson based JSON handling
89e1d06 
Add an alternative json serialization and deserialization implementation
using simplejson configured to be JSON compliant by converting floats like
nan, +inf and -inf to null .
The idea is to permit users to use a different serialization if they
need a different behaviour.

Refs elastic#1886
@xrmx xrmx mentioned this issue Mar 29, 2024
Merged
xrmx added a commit to xrmx/apm-agent-python that referenced this issue Mar 29, 2024
@xrmx
utils: add simplejson based JSON handling
37e4f7f 
Add an alternative json serialization and deserialization implementation
using simplejson configured to be JSON compliant by converting floats like
nan, +inf and -inf to null .
The idea is to permit users to use a different serialization if they
need a different behaviour.

Refs elastic#1886
xrmx added a commit that referenced this issue Mar 29, 2024
@xrmx
Add ability to override default serialization (#2018)
3688340 
* utils: add simplejson based JSON handling

Add an alternative json serialization and deserialization implementation
using simplejson configured to be JSON compliant by converting floats like
nan, +inf and -inf to null .
The idea is to permit users to use a different serialization if they
need a different behaviour.

Refs #1886

* client: fix overriding of transport json serializer

We get a string from config so need to import it.
xrmx added a commit to xrmx/apm-agent-python that referenced this issue Apr 3, 2024
@xrmx
Add ability to override default serialization (elastic#2018)
8b733ec 
* utils: add simplejson based JSON handling

Add an alternative json serialization and deserialization implementation
using simplejson configured to be JSON compliant by converting floats like
nan, +inf and -inf to null .
The idea is to permit users to use a different serialization if they
need a different behaviour.

Refs elastic#1886

* client: fix overriding of transport json serializer

We get a string from config so need to import it.
@xrmx
Copy link
Member

xrmx commented Apr 3, 2024

With apm agent 6.22.0 you can change the behaviour by installing simplejson as dependency and the setting the following environment variable:

ELASTIC_APM_TRANSPORT_JSON_SERIALIZER=elasticapm.utils.simplejson_encoder.dumps

xrmx added a commit that referenced this issue Apr 4, 2024
@xrmx
Add ability to override default serialization (#2018)
6590cd2 
* utils: add simplejson based JSON handling

Add an alternative json serialization and deserialization implementation
using simplejson configured to be JSON compliant by converting floats like
nan, +inf and -inf to null .
The idea is to permit users to use a different serialization if they
need a different behaviour.

Refs #1886

* client: fix overriding of transport json serializer

We get a string from config so need to import it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xrmx xrmx

Labels
8.12-candidate agent-python bug upstream Issue caused by an upstream package
Projects
APM-Agents (OLD)
  
Planned
Milestone
No milestone
Development

No branches or pull requests
4 participants
@xrmx @basepi @aydasraf @kruskall