Skip to content

internalGet: truncate capacity on return value #199

Closed
@thempatel

Description

@thempatel

Using ArrayEach in concert with Set leads to some weird behavior. Specifically that ArrayEach provides slices over the original json object, such that the capacity of the slice is jsonEnd-(startOffsetOfSlice). This interacts poorly with Set when a particular key does not exist in the object being mutated. Given this testcase, you can see the output shows the second JSON object becoming malformed upon setting a non-existent key in the first object.

func TestWhatHappens(t *testing.T) {
	jsonObject := `[{"key1":"Foo","key2":"Bar","key3":"baz"},{"key1":"boo","key2":"dog","key3":"cat","key4":"mouse"}]`
	var allSlices [][]byte
	_, err := jsonparser.ArrayEach([]byte(jsonObject), func(value []byte, _ jsonparser.ValueType, _ int, err error) {
		if err != nil {
			t.Fatalf("failed: %s", err)
		}
		allSlices = append(allSlices, value)
	})

	if err != nil {
		t.Fatalf("failed: %s", err)
	}

	newVal, err := jsonparser.Set(allSlices[0], []byte(strconv.Quote(`mouse`)), "key4")
	if err != nil {
		t.Fatalf("failed %s: ", err)
	}
	fmt.Println(string(newVal))
	fmt.Println(string(allSlices[1]))
}

The suggested fix is to change this line:

value = append(data[:startOffset], append(createInsertComponent(keys[depth:], setValue, comma, object), data[depthOffset:]...)...)

to

value = append(data[:startOffset:startOffset], append(createInsertComponent(keys[depth:], setValue, comma, object), data[depthOffset:]...)...)

which truncates the capacity causing append to allocate a new slice. Alternatively, the pattern in the path currently exists flow can be used, this just happened to be a nifty one-liner

Activity

thempatel

thempatel commented on Apr 28, 2020

@thempatel
Author

Follow-up: It was pointed out to me that the write side is probably fine to use the extra capacity, in which case, the internalGet function should likely restrict capacity on what it returns, up to you!

thempatel

thempatel commented on Apr 29, 2020

@thempatel
Author

FYI we ended up going with:

return value, dataType, offset, endOffset, nil

return value[:len(value):len(value)], dataType, offset, endOffset, nil
changed the title [-]Set: allocate new slice when key does not exist[/-] [+]internalGet: truncate capacity on return value[/+] on Apr 30, 2020
added a commit that references this issue on May 15, 2020
AllenX2018

AllenX2018 commented on May 18, 2020

@AllenX2018
Collaborator

Awesome! Thanks! I prefer to restrict the capacity of the slice returned by the internalGet function!

thempatel

thempatel commented on May 18, 2020

@thempatel
Author

Sweet!

thempatel

thempatel commented on May 18, 2020

@thempatel
Author

I'm gonna go ahead and close this so it drops out of my list of things

added a commit that references this issue on May 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @thempatel@AllenX2018

        Issue actions

          internalGet: truncate capacity on return value · Issue #199 · buger/jsonparser