Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Issues correlating logs with traces with Google Cloud Run Serverless Env #2351

Open
Vlaquit opened this issue Nov 14, 2023 · 3 comments
Labels
bug unintended behavior that has to be fixed

Comments

@Vlaquit
Copy link

Vlaquit commented Nov 14, 2023

Hello, I'm trying to instrument my Cloud Run (google serverless) but I can't get the logs to correlate with the traces.

Describe what happened:
I have tried several options such as using this small library in contrib with logrus.
https://github.com/DataDog/dd-trace-go/blob/main/contrib/sirupsen/logrus/logrus.go
And the log does not show the same dd.trace_id than the traces UI.
Like on this picture:
image
What I find suspicious is that the trace_id seems very similar, so it can't come from a completely different trace in the code because if I've understood correctly it's randomly generated in the dd-trace lib.
In addition, I have the feeling that this ID is rounded off to end with ...00

Steps to reproduce the issue:
As I said, I've tried using bot DDContextLogHook library from contrib and the manual method of inserting the trace ID into the logs but I get the same result.
Here's my code snippet which I assume should work:

func (f *CloudRunFunction) SyncTransferJobs(w http.ResponseWriter, r *http.Request) {
	// ...
	span, ctx := tracer.StartSpanFromContext(
		r.Context(),
		OperationType,
		tracer.ResourceName(funcName),
		tracer.Tag("fileName", fileName),
	)
	defer func() { span.Finish(tracer.WithError(err)) }()
	logEntry := f.logEntry.WithFields(logrus.Fields{
		"function":    "sync_transfer_jobs",
		"dd.trace_id": span.Context().TraceID(),
		"dd.span_id":  span.Context().SpanID(),
	})

	logEntry.Info("Initializing cloud run function")

	// ...
}

I have logrus configured with the json formatter with this logger.SetFormatter(&logrus.JSONFormatter{})

Additional environment details (Version of Go, Operating System, etc.):
I am using go 1.19
The golang:1.19-buster Docker image

@Vlaquit Vlaquit added the bug unintended behavior that has to be fixed label Nov 14, 2023
@Vlaquit
Copy link
Author

Vlaquit commented Nov 14, 2023

Here's the result of another test I did:

image
The code:

func (f *CloudRunFunction) SyncTransferJobs(w http.ResponseWriter, r *http.Request) {
	// ...
	logEntry := f.logEntry.WithFields(logrus.Fields{
		"function":    "sync_transfer_jobs",
		"dd.trace_id": span.Context().TraceID(),
		"dd.span_id":  span.Context().SpanID(),
	})

	myLog := "Span context trace id: " + strconv.FormatUint(span.Context().TraceID(), 10)
	logEntry.Info(myLog)
	// ...
}

Can this be related to an issue with logrus or the datadog remapper ?

@gondyb
Copy link

gondyb commented Nov 14, 2023

Hello @Vlaquit, I'm not sure how logrus works internally, but make sure that your trace_id is sent as a string. Json encoding in go using encoding/json may lose precision on big integers, since the conversion is internally using float64. Let me know if that fixes the issue!

@Vlaquit
Copy link
Author

Vlaquit commented Nov 14, 2023

Thank you, it seems to work when I cast the TraceID to string! (BTW I forgot to check that the remapping was fine. The trace id reserved attribute remapper was not parsing the data.jsonPayload.dd.trace_id properly) 🙏🏻

However I had the same problem with the contrib/logrus lib, I think the bug could still occur when you want to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unintended behavior that has to be fixed
Projects
None yet
Development

No branches or pull requests

2 participants