Closed
Description
Using ArrayEach
in concert with Set
leads to some weird behavior. Specifically that ArrayEach
provides slices over the original json object, such that the capacity of the slice is jsonEnd-(startOffsetOfSlice)
. This interacts poorly with Set
when a particular key does not exist in the object being mutated. Given this testcase, you can see the output shows the second JSON object becoming malformed upon setting a non-existent key in the first object.
func TestWhatHappens(t *testing.T) {
jsonObject := `[{"key1":"Foo","key2":"Bar","key3":"baz"},{"key1":"boo","key2":"dog","key3":"cat","key4":"mouse"}]`
var allSlices [][]byte
_, err := jsonparser.ArrayEach([]byte(jsonObject), func(value []byte, _ jsonparser.ValueType, _ int, err error) {
if err != nil {
t.Fatalf("failed: %s", err)
}
allSlices = append(allSlices, value)
})
if err != nil {
t.Fatalf("failed: %s", err)
}
newVal, err := jsonparser.Set(allSlices[0], []byte(strconv.Quote(`mouse`)), "key4")
if err != nil {
t.Fatalf("failed %s: ", err)
}
fmt.Println(string(newVal))
fmt.Println(string(allSlices[1]))
}
The suggested fix is to change this line:
Line 794 in f7e751e
to
value = append(data[:startOffset:startOffset], append(createInsertComponent(keys[depth:], setValue, comma, object), data[depthOffset:]...)...)
which truncates the capacity causing append
to allocate a new slice. Alternatively, the pattern in the path currently exists
flow can be used, this just happened to be a nifty one-liner
Activity
thempatel commentedon Apr 28, 2020
Follow-up: It was pointed out to me that the write side is probably fine to use the extra capacity, in which case, the
internalGet
function should likely restrict capacity on what it returns, up to you!thempatel commentedon Apr 29, 2020
FYI we ended up going with:
jsonparser/parser.go
Line 916 in f7e751e
[-]Set: allocate new slice when key does not exist[/-][+]internalGet: truncate capacity on return value[/+]fix issues buger#199
AllenX2018 commentedon May 18, 2020
Awesome! Thanks! I prefer to restrict the capacity of the slice returned by the
internalGet
function!thempatel commentedon May 18, 2020
Sweet!
thempatel commentedon May 18, 2020
I'm gonna go ahead and close this so it drops out of my list of things
Fix issue #199