Serialization of 64bit values are truncated #3837

kskem · 2021-11-03T07:45:00Z

In what area(s)?

/area runtime

What version of Dapr?

edge: output of git describe --dirty
starting Dapr Runtime -- version edge -- commit b1cad9a

Expected Behavior

Sending 64bit values with pub/sub are expected to arrive unchanged.

Actual Behavior

Receiving 64bit values are truncated in the trailing two decimal digits.
Best guess is that the serialized json has a max string-length for numbers.

Steps to Reproduce the Problem

I have posted a reproducible setup, based on Docker, with instructions:
https://github.com/kskem/dapr-serialization-issue

Recap of the problem:

owner_1            | Publishing record 1/5: {"int32":1778635775,"uint32":1778635775,"int64":637715205845980876,"uint64":637715205845980876,"base64long":"zJZQ2Jme2Qg=","base64int":"/9MDag=="}
replicaone_1       | Got event            : {"int32":1778635775,"uint32":1778635775,"int64":637715205845980900,"uint64":637715205845980900,"base64long":"zJZQ2Jme2Qg=","base64int":"/9MDag=="}

Release Note

RELEASE NOTE: FIX Bug in runtime.

The text was updated successfully, but these errors were encountered:

dapr-bot · 2021-12-03T07:51:49Z

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

real-zony · 2021-12-13T08:16:09Z

I have this problem too.

hueifeng · 2021-12-15T04:51:47Z

Any update?

SayHelloGsx · 2021-12-20T02:47:38Z

Yeah I got the same problem too. Could any one tell me how to fix it?

yaron2 · 2021-12-20T03:11:46Z

I triaged this to be fixed in 1.6.

cc @artursouza

artursouza · 2021-12-20T05:31:21Z

Is this a runtime or SDK issue?

yaron2 · 2021-12-20T05:55:35Z

Is this a runtime or SDK issue?

Runtime

hueifeng · 2021-12-20T06:29:27Z

Serialization is used in runtime.go, but when it encounters 64bit it automatically identifies Number as float64.

For example:

package main

import(
	jsoniter "github.com/json-iterator/go"
)

var name string

type Stu struct{
	Name string `json:"name"`
	Id int64
}

func main()  {
	Stu:=Stu{
		name:"张三",
		Id:637715205845980876
	}
	m1 :=make(map[string]interface{})
	var json= jsoniter.ConfigDefault
	var jsonStu,_=json.Marshal(stu)
	println(string(jsonStu))
	json.Unmarshal(jsonStu,&m1)

}

json.Unmarshal will use float64 to handle the problem, and you'll need to change the method to display the declaration.

var val map[string]interface{}
d := json.NewDecoder(bytes.NewBuffer(bdata))
d.UseNumber()
err = d.Decode(&val)

yaron2 · 2021-12-20T07:44:22Z

Related, possible: json-iterator/go#510

hueifeng · 2021-12-20T08:33:01Z

@yaron2 You can consider using the following code to solve.

package main

import (
	"strings"

	jsoniter "github.com/json-iterator/go"
)

type Stu struct {
	Name string `json:"name"`
	Id   int64
}

func main() {
	stu := Stu{
		Name: "张三",
		Id:   637715205845980876,
	}
	m1 := make(map[string]interface{})
	var json = jsoniter.ConfigDefault
	var jsonStu, _ = json.Marshal(stu)
	println(string(jsonStu))
	json.Unmarshal(jsonStu, &m1)

	decoder := jsoniter.NewDecoder(strings.NewReader(string(jsonStu)))
	decoder.UseNumber()
	d := make(map[string]interface{})
	decoder.Decode(&d)
	s2, _ := jsoniter.Marshal(d)
	println(string(s2))
}

Use the func (*Decoder) UseNumber method to deserialize the numeral type of json, instead of directly converting to float64, convert to json.Number, json.Number internal:

// A Number represents a JSON number literal.
type Number string

// String returns the literal text of the number.
func (n Number) String() string { return string(n) }

// Float64 returns the number as a float64.
func (n Number) Float64() (float64, error) {
    return strconv.ParseFloat(string(n), 64)
}

// Int64 returns the number as an int64.
func (n Number) Int64() (int64, error) {
    return strconv.ParseInt(string(n), 10, 64)
}

yaron2 · 2021-12-29T16:49:59Z

Thanks @hueifeng.

@artursouza after looking at this, I recommend we move away from jsoniter to the Go stdlib encoding/json library. It looks like since Go 1.14 performance is no longer a critical aspect of this library: json-iterator/go#455.

Moving to encoding/json will allow us to optimize for JSON consistency and stability whilst removing a dependency.

artursouza · 2021-12-29T23:55:53Z

Thanks @hueifeng.

@artursouza after looking at this, I recommend we move away from jsoniter to the Go stdlib encoding/json library. It looks like since Go 1.14 performance is no longer a critical aspect of this library: json-iterator/go#455.

Moving to encoding/json will allow us to optimize for JSON consistency and stability whilst removing a dependency.

LGTM

yaron2 · 2021-12-30T00:00:58Z

@hueifeng would you like to contribute this change?

hueifeng · 2021-12-30T06:08:59Z

I can do it.

artursouza · 2022-01-07T01:33:45Z

E2E tests are not passing in the PR, moving to 1.7

artursouza · 2022-01-14T05:13:35Z

Moving to 1.7 because we are cutting RC for 1.6.

shubham1172 · 2022-03-07T05:18:23Z

/assign

shubham1172 · 2022-03-17T18:29:31Z

Dapr/dapr does not have a problem. I tried putting telemetry around pub (pkg/grpc/api.go) and sub (pkg/runtime/runtime.go) code to check what data is being sent and received.

Example data sent from application:

{"int32":1041701776,"uint32":1041701776,"int64":637831257272814404,"uint64":637831257272814404,"base64long":"RAtjLCYI2gg=","base64int":"kBsXPg=="}

On Dapr side, while sending the data, the raw bytes from Marshaling the struct gets its int64 and uint64 data converted to float, and here it loses precision. Sample data after marshaling on publisher's end:

map[data:map[base64int:kBsXPg== base64long:RAtjLCYI2gg= int32:1.041701776e+09 int64:6.378312572728145e+17 uint32:1.041701776e+09 uint64:6.378312572728145e+17]

On Dapr side, while receiving the data, the raw bytes itself have incorrect data (as in shubham1172/dapr-serialization-issue's readme). Thus Unmarshaling with UseNumber here won't help. Sample data after unmarshalling:

map[data:map[base64int:kBsXPg== base64long:RAtjLCYI2gg= int32:1041701776 int64:637831257272814500 uint32:1041701776 uint64:637831257272814500]

The root cause is the json.Unmarshal in https://github.com/dapr/components-contrib/blob/master/pubsub/envelope.go
On adding the UseNumber way, it works as expected without data loss. I will create a PR and attach to this issue soon.

Should we create a separate item to move away from jsoniter to encoding/json?

dapr-bot · 2022-03-17T18:37:35Z

Please, track jsoniter migration separately. But it would be great to have a PR for that too. — Artur Souza

…

shubham1172 · 2022-03-21T18:03:17Z

I did some more research, and this will not be a breaking change.

The only change that this PR introduces is in the unmarshalling of the CloudEvent data from the PublishEventRequest in PublishEvent (api.go)

Earlier, this unmarshalling resulted in data loss while converting 64-bit values to 53-bits float (as per JSON). Now, it uses json.Number representation, an alias of string, to unmarshall the same without any data loss.

After unmarshalling the CloudEvent's data, it's packed into the PublishRequest as []byte (both HTTP and GRPC). This is the request that gets published on the actual pubsub component.

The same thing is being read as []byte by our subscribers, in beginPubSub (runtime.go)

Since the public API contracts are not changed, this won't affect anything external.

shubham1172 · 2022-03-21T18:38:20Z

Adding some more data here for comparison. All these are requests with CloudEvents in JSON format (other CloudEvent formats/rawPayloads are unaffected).

Before the PR:

// data received by PublishRequest
{"int32":1778635775,"uint32":1778635775,"int64":637715205845980876,"uint64":637715205845980876,"base64long":"zJZQ2Jme2Qg=","base64int":"/9MDag=="}

// data marshalled by PublishRequest
{"int32":1778635775,"uint32":1778635775,"int64":637715205845980900,"uint64":637715205845980900,"base64long":"zJZQ2Jme2Qg=","base64int":"/9MDag=="}

// data receievd by Subscriber
{"int32":1778635775,"uint32":1778635775,"int64":637715205845980900,"uint64":637715205845980900,"base64long":"zJZQ2Jme2Qg=","base64int":"/9MDag=="}

After the PR:

// data received by PublishRequest
{"int32":1228499251,"uint32":1228499251,"int64":637834811545047664,"uint64":637834811545047664,"base64long":"cFKFt2EL2gg=","base64int":"M2k5SQ=="}

// data marshalled by PublishRequest
{"int32":1228499251,"uint32":1228499251,"int64":637834811545047664,"uint64":637834811545047664,"base64long":"cFKFt2EL2gg=","base64int":"M2k5SQ=="}

// data received by Subscriber
{"int32":1228499251,"uint32":1228499251,"int64":637834811545047664,"uint64":637834811545047664,"base64long":"cFKFt2EL2gg=","base64int":"M2k5SQ=="}

shubham1172 · 2022-03-21T18:38:54Z

@mukundansundar can you please un-tag the PR as breaking-change?

kskem added the kind/bug Something isn't working label Nov 3, 2021

dapr-bot added stale Issues and PRs without response and removed stale Issues and PRs without response labels Dec 3, 2021

real-zony mentioned this issue Dec 15, 2021

jsoniter.Unmarshal deserialization int64 lose precision #4026

Closed

yaron2 added this to the v1.6 milestone Dec 20, 2021

yaron2 changed the title ~~Serialization of 64bit values are truncated with .net sdk~~ Serialization of 64bit values are truncated Dec 20, 2021

This was referenced Dec 30, 2021

Add json util to unmarshal numbers into int/float #4069

Closed

Add json util to unmarshal numbers into int/float #4092

Closed

Add json util to unmarshal numbers into int/float #4093

Closed

artursouza modified the milestones: v1.6, v1.7 Jan 7, 2022

yaron2 modified the milestones: v1.7, v1.6 Jan 7, 2022

artursouza removed this from the v1.6 milestone Jan 14, 2022

artursouza added this to the v1.7 milestone Jan 14, 2022

dapr-bot assigned shubham1172 Mar 7, 2022

shubham1172 mentioned this issue Mar 7, 2022

[WIP] Use encoding/json instead of json-iterator/go #4331

Closed

7 tasks

shubham1172 mentioned this issue Mar 14, 2022

[WIP] Use encoding/json instead of json-iterator/go and fix serialization bug #4372

Closed

7 tasks

shubham1172 mentioned this issue Mar 17, 2022

Fix serialization issue with 64 bit data type dapr/components-contrib#1597

Merged

3 tasks

shubham1172 mentioned this issue Mar 17, 2022

Move away from json-iterator/go to encoding/json #4395

Closed

mukundansundar mentioned this issue Mar 21, 2022

Long values are not sent correctly with PubSub dapr/java-sdk#686

Closed

yaron2 closed this as completed in dapr/components-contrib#1597 Mar 21, 2022

hueifeng mentioned this issue Jun 10, 2022

REQUEST: New membership for hueifeng dapr/community#179

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serialization of 64bit values are truncated #3837

Serialization of 64bit values are truncated #3837

kskem commented Nov 3, 2021 •

edited

dapr-bot commented Dec 3, 2021

real-zony commented Dec 13, 2021

hueifeng commented Dec 15, 2021

SayHelloGsx commented Dec 20, 2021

yaron2 commented Dec 20, 2021

artursouza commented Dec 20, 2021

yaron2 commented Dec 20, 2021

hueifeng commented Dec 20, 2021

yaron2 commented Dec 20, 2021

hueifeng commented Dec 20, 2021

yaron2 commented Dec 29, 2021 •

edited

artursouza commented Dec 29, 2021

yaron2 commented Dec 30, 2021

hueifeng commented Dec 30, 2021

artursouza commented Jan 7, 2022

artursouza commented Jan 14, 2022

shubham1172 commented Mar 7, 2022

shubham1172 commented Mar 17, 2022 •

edited

dapr-bot commented Mar 17, 2022 via email

shubham1172 commented Mar 21, 2022

shubham1172 commented Mar 21, 2022

shubham1172 commented Mar 21, 2022

Serialization of 64bit values are truncated #3837

Serialization of 64bit values are truncated #3837

Comments

kskem commented Nov 3, 2021 • edited

In what area(s)?

What version of Dapr?

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Release Note

dapr-bot commented Dec 3, 2021

real-zony commented Dec 13, 2021

hueifeng commented Dec 15, 2021

SayHelloGsx commented Dec 20, 2021

yaron2 commented Dec 20, 2021

artursouza commented Dec 20, 2021

yaron2 commented Dec 20, 2021

hueifeng commented Dec 20, 2021

yaron2 commented Dec 20, 2021

hueifeng commented Dec 20, 2021

yaron2 commented Dec 29, 2021 • edited

artursouza commented Dec 29, 2021

yaron2 commented Dec 30, 2021

hueifeng commented Dec 30, 2021

artursouza commented Jan 7, 2022

artursouza commented Jan 14, 2022

shubham1172 commented Mar 7, 2022

shubham1172 commented Mar 17, 2022 • edited

dapr-bot commented Mar 17, 2022 via email

shubham1172 commented Mar 21, 2022

shubham1172 commented Mar 21, 2022

shubham1172 commented Mar 21, 2022

kskem commented Nov 3, 2021 •

edited

yaron2 commented Dec 29, 2021 •

edited

shubham1172 commented Mar 17, 2022 •

edited