Merge upstream again (#8)

* Support array remove with negative index.

* Allow replacement of null values

* Fix tabs

* Handle null paths and ops without panicking.

* Update stable version in Readme

* fix test panic

Without this change, the following code would panic:
```
package main

import (
        "fmt"

        jsonpatch "github.com/evanphx/json-patch"
)

func main() {
        original := []byte(`{"name": "John", "age": 24, "height": 3.21}`)
        patchJSON := []byte(`[
                {"op": "test", "path": "/name"}
        ]`)

        patch, err := jsonpatch.DecodePatch(patchJSON)
        if err != nil {
		fmt.Printf("decode %v\n", err)
		return
        }

        modified, err := patch.Apply(original)
        if err != nil {
		fmt.Printf("aplly %v\n", err)
		return
        }

        fmt.Printf("Original document: %s\n", original)
        fmt.Printf("Modified document: %s\n", modified)
}

```

* Add ability to disable non-standard negative indices support

* Add note about SupportNegativeIndices

* Fix typo in README

* limit how big array can grow

* Document the ArraySizeLimit global variable

* adding limit on extending array size

* Add doc for the ArraySizeAdditionLimit

* Deep copy the source to prevent the copy operation to create loops

* The "copy" operation should "add" the value to the target location,
instead of "set".

* Adding a configurable limit for accumulated size increase caused by the copy
operations in a patch.

* converting ArrarySizeError to a typed one

* Setting and resetting global configurations in the test case.

Previously, the global configurations are set in the init function in
the unit test. This can bite importers' unit tests if they don't prune
vendored tests.

* According to the RFC, the move operation should be functionally
identical to a remove followed by an add.

* Since that the "set" method is now only called to implement the
"replace" operation, and the "path" of "replace" must refer to an
existing location, we can greatly simplify the sanity check in "set".

Also, we removed the ArraySizeAdditionLimit, because according to the RFC, no
operation can increase the length of an existing array by more than 1.

* Removed the ArraySizeLimit.

It's not useful anymore because:
1. There used to be bugs in the library that allowed the "path" field of
the "move" and "copy" operations to refer to arbitrarily large index of
an existing array. Now the bugs are fixed, the two operations can at
most increase the length of any _existing_ array by 1, by appending to the
end.

2. The "add" and "replace" operation can create large new arrays.
However, for these two operations, the patch MUST contain the "value",
thus the caller of this library can cap the size by limiting the size of the
patch.

3. The "copy" operations can copy existing large array, but users can
control the size increase by setting AccumulatedCopySizeLimit.

* Expose Operation and methods, fix lint errors

* Update error messages

* Revert govet fixes

* Cleanup errors to help understand what failed. Fixes evanphx#75

* Add bits to better be a go module

* check lengths of maps and recurse over only one if it is necessary

* add benchmark tests for matchesValue

* Fix panic when SupportNegativeIndices is false

If SupportNegativeIndices is false then a negative indicie will cause a panic.
This regression was introduced in: fcd53ec
Before that change we always checked the validity of the negative index.

* Conform to RFC6902 replacement semantics.

A "replace" patch operation referencing a key that does not exist in the
target object are currently being accepted; ultimately becoming "add"
operations on the target object. This is incorrect per the
specification:

From https://tools.ietf.org/html/rfc6902#section-4.3 section about
"replace":

    The target location MUST exist for the operation to be successful.

This corrects the behavior by returning an error in this situation.

* Test for copying non-existent key.

* Test for testing non-existent and null-value keys.

* Test for copying null-value key.

* Equality comparison bug fix

* Fixed output of example

Made the output text in the examples match what the code does

* Typo in README

* Fix Issue evanphx#88, add more tests for the Equals() function.

* added some more Equals() tests

* fix little typo in README too

* copy go sources into v5

* update paths for v5

* correct module requirements

* also test v5 in travis

* update travis to most recent go versions

* Update instructions to use v5 dir.

* Resolve a couple of differences
t

* Return to panic

* allow replacing nil values

* update imports

* update gomod

* update gomod again

Co-authored-by: Evan Phoenix <evan@fallingsnow.net>
Co-authored-by: Cao Shufeng <caosf.fnst@cn.fujitsu.com>
Co-authored-by: Evan Phoenix <evan@phx.io>
Co-authored-by: Carlos Cobo <toqueteos@gmail.com>
Co-authored-by: Chao Xu <xuchao@google.com>
Co-authored-by: Sam Ward <github@ward.io>
Co-authored-by: Han Kang <hankang@google.com>
Co-authored-by: Eric Paris <eparis@redhat.com>
Co-authored-by: Brett Buddin <brett@buddin.org>
Co-authored-by: Igor Gubernat <igorgubernat@gmail.com>
Co-authored-by: Bård Aase <elzapp@gmail.com>
Co-authored-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>
Co-authored-by: jackhafner <jackhaf@gmail.com>
Co-authored-by: Benjamin Elder <ben@elder.dev>
Co-authored-by: atrakh <arnold@launchdarkly.com>

.travis.yml

@@ -1,8 +1,8 @@
 language: go
 
 go:
-  - 1.8
-  - 1.7
+  - 1.14
+  - 1.13
 
 install:
   - if ! go get code.google.com/p/go.tools/cmd/cover; then go get golang.org/x/tools/cmd/cover; fi
@@ -11,6 +11,9 @@ install:
 script:
   - go get
   - go test -cover ./...
+  - cd ./v5
+  - go get
+  - go test -cover ./...
 
 notifications:
   email: false

README.md

@@ -1,5 +1,5 @@
 # JSON-Patch
-`jsonpatch` is a library which provides functionallity for both applying
+`jsonpatch` is a library which provides functionality for both applying
 [RFC6902 JSON patches](http://tools.ietf.org/html/rfc6902) against documents, as
 well as for calculating & applying [RFC7396 JSON merge patches](https://tools.ietf.org/html/rfc7396).
 
@@ -11,11 +11,12 @@ well as for calculating & applying [RFC7396 JSON merge patches](https://tools.ie
 
 **Latest and greatest**: 
 ```bash
-go get -u github.com/evanphx/json-patch
+go get -u github.com/evanphx/json-patch/v5
 ```
 
 **Stable Versions**:
-* Version 3: `go get -u gopkg.in/evanphx/json-patch.v3`
+* Version 5: `go get -u gopkg.in/evanphx/json-patch.v5`
+* Version 4: `go get -u gopkg.in/evanphx/json-patch.v4`
 
 (previous versions below `v3` are unavailable)
 
@@ -25,6 +26,19 @@ go get -u github.com/evanphx/json-patch
 * [Comparing JSON documents](#comparing-json-documents)
 * [Combine merge patches](#combine-merge-patches)
 
+
+# Configuration
+
+* There is a global configuration variable `jsonpatch.SupportNegativeIndices`.
+  This defaults to `true` and enables the non-standard practice of allowing
+  negative indices to mean indices starting at the end of an array. This
+  functionality can be disabled by setting `jsonpatch.SupportNegativeIndices =
+  false`.
+
+* There is a global configuration variable `jsonpatch.AccumulatedCopySizeLimit`,
+  which limits the total size increase in bytes caused by "copy" operations in a
+  patch. It defaults to 0, which means there is no limit.
+
 ## Create and apply a merge patch
 Given both an original JSON document and a modified JSON document, you can create
 a [Merge Patch](https://tools.ietf.org/html/rfc7396) document. 
@@ -69,7 +83,7 @@ When ran, you get the following output:
 ```bash
 $ go run main.go
 patch document:   {"height":null,"name":"Jane"}
-updated tina doc: {"age":28,"name":"Jane"}
+updated alternative doc: {"age":28,"name":"Jane"}
 ```
 
 ## Create and apply a JSON Patch
@@ -151,7 +165,7 @@ func main() {
 	}
 
 	if !jsonpatch.Equal(original, different) {
-		fmt.Println(`"original" is _not_ structurally equal to "similar"`)
+		fmt.Println(`"original" is _not_ structurally equal to "different"`)
 	}
 }
 ```
@@ -160,7 +174,7 @@ When ran, you get the following output:
 ```bash
 $ go run main.go
 "original" is structurally equal to "similar"
-"original" is _not_ structurally equal to "similar"
+"original" is _not_ structurally equal to "different"
 ```
 
 ## Combine merge patches

cmd/json-patch/main.go

@@ -6,8 +6,9 @@ import (
 	"log"
 	"os"
 
-	jsonpatch "github.com/evanphx/json-patch"
 	flags "github.com/jessevdk/go-flags"
+
+	jsonpatch "github.com/launchdarkly/json-patch"
 )
 
 type opts struct {

errors.go

@@ -0,0 +1,38 @@
+package jsonpatch
+
+import "fmt"
+
+// AccumulatedCopySizeError is an error type returned when the accumulated size
+// increase caused by copy operations in a patch operation has exceeded the
+// limit.
+type AccumulatedCopySizeError struct {
+	limit       int64
+	accumulated int64
+}
+
+// NewAccumulatedCopySizeError returns an AccumulatedCopySizeError.
+func NewAccumulatedCopySizeError(l, a int64) *AccumulatedCopySizeError {
+	return &AccumulatedCopySizeError{limit: l, accumulated: a}
+}
+
+// Error implements the error interface.
+func (a *AccumulatedCopySizeError) Error() string {
+	return fmt.Sprintf("Unable to complete the copy, the accumulated size increase of copy is %d, exceeding the limit %d", a.accumulated, a.limit)
+}
+
+// ArraySizeError is an error type returned when the array size has exceeded
+// the limit.
+type ArraySizeError struct {
+	limit int
+	size  int
+}
+
+// NewArraySizeError returns an ArraySizeError.
+func NewArraySizeError(l, s int) *ArraySizeError {
+	return &ArraySizeError{limit: l, size: s}
+}
+
+// Error implements the error interface.
+func (a *ArraySizeError) Error() string {
+	return fmt.Sprintf("Unable to create array of size %d, limit is %d", a.size, a.limit)
+}

go.mod

@@ -0,0 +1,8 @@
+module github.com/launchdarkly/json-patch
+
+go 1.12
+
+require (
+	github.com/jessevdk/go-flags v1.4.0
+	github.com/pkg/errors v0.8.1
+)

go.sum

@@ -0,0 +1,4 @@
+github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA=
+github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=

merge.go

@@ -307,10 +307,8 @@ func matchesValue(av, bv interface{}) bool {
 		return true
 	case map[string]interface{}:
 		bt := bv.(map[string]interface{})
-		for key := range at {
-			if !matchesValue(at[key], bt[key]) {
-				return false
-			}
+		if len(bt) != len(at) {
+			return false
 		}
 		for key := range bt {
 			if !matchesValue(at[key], bt[key]) {
@@ -369,7 +367,8 @@ func getDiff(a, b map[string]interface{}) (map[string]interface{}, error) {
 				into[key] = bv
 			}
 		default:
-			return nil, fmt.Errorf("Unknown type:%T in key %s", av, key)
+			panic(fmt.Sprintf("Unknown type:%T in key %s", av, key))
+			//return nil, fmt.Errorf("Unknown type:%T in key %s", av, key)
 		}
 	}
 	// Now add all deleted values as nil

merge_test.go

@@ -1,6 +1,7 @@
 package jsonpatch
 
 import (
+	"fmt"
 	"strings"
 	"testing"
 )
@@ -447,6 +448,45 @@ func TestCreateMergePatchComplexAddAll(t *testing.T) {
 	}
 }
 
+// createNestedMap created a series of nested map objects such that the number of
+// objects is roughly 2^n (precisely, 2^(n+1)-1).
+func createNestedMap(m map[string]interface{}, depth int, objectCount *int) {
+	if depth == 0 {
+		return
+	}
+	for i := 0; i< 2;i++ {
+		nested := map[string]interface{}{}
+		*objectCount += 1
+		createNestedMap(nested, depth-1, objectCount)
+		m[fmt.Sprintf("key-%v", i)] = nested
+	}
+}
+
+func benchmarkMatchesValueWithDeeplyNestedFields(depth int, b *testing.B) {
+	a := map[string]interface{}{}
+	objCount := 1
+	createNestedMap(a, depth, &objCount)
+	b.ResetTimer()
+	b.Run(fmt.Sprintf("objectCount=%v", objCount), func(b *testing.B) {
+		for i := 0; i < b.N; i++ {
+			if !matchesValue(a, a) {
+				b.Errorf("Should be equal")
+			}
+		}
+	})
+}
+
+func BenchmarkMatchesValue1(b *testing.B)  { benchmarkMatchesValueWithDeeplyNestedFields(1, b) }
+func BenchmarkMatchesValue2(b *testing.B)  { benchmarkMatchesValueWithDeeplyNestedFields(2, b) }
+func BenchmarkMatchesValue3(b *testing.B)  { benchmarkMatchesValueWithDeeplyNestedFields(3, b) }
+func BenchmarkMatchesValue4(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(4, b) }
+func BenchmarkMatchesValue5(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(5, b) }
+func BenchmarkMatchesValue6(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(6, b) }
+func BenchmarkMatchesValue7(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(7, b) }
+func BenchmarkMatchesValue8(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(8, b) }
+func BenchmarkMatchesValue9(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(9, b) }
+func BenchmarkMatchesValue10(b *testing.B) { benchmarkMatchesValueWithDeeplyNestedFields(10, b) }
+
 func TestCreateMergePatchComplexRemoveAll(t *testing.T) {
 	doc := `{"hello": "world","t": true ,"f": false, "n": null,"i": 123,"pi": 3.1416,"a": [1, 2, 3, 4], "nested": {"hello": "world","t": true ,"f": false, "n": null,"i": 123,"pi": 3.1416,"a": [1, 2, 3, 4]} }`
 	exp := `{"a":null,"f":null,"hello":null,"i":null,"n":null,"nested":null,"pi":null,"t":null}`