json Changelog

not yet released

(nothing yet)

10.0.0

Backward incompatible and security-related change to parsing "lookup" strings.

This version restricts the supported syntax for bracketed "lookup" strings to fix a possible vulnerability (CVE-2020-7712). With a carefully crafted lookup string, command injection was possible. See #144 for a repro. If you use json (the CLI or as a node.js module) and run arbitrary user-provided strings as a "lookup", then you should upgrade.

For the json CLI, a "lookup" string is the 'foo' in:
```
  echo ...some json... | json foo
```
which allows you to lookup fields on the given JSON, e.g.:
```
  $ echo '{"foo": {"bar": "baz"}}' | json foo.bar
  baz
```
If one of the lookup fields isn't a valid JS identifier, then the JS array notation is supported:
```
  $ echo '{"http://example.com": "my-value"}' | json '["http://example.com"]'
  my-value
```
Before this change, json would effectively exec the string between the brackets as JS code such that things like the following were possible:
```
  $ echo '{"foo3": "bar"}' | json '["foo" + 3]'
  bar
```
This change limits supported bracket syntax in lookups to a simple quoted string:
```
  ["..."]
  ['...']
  [`...`]      # no variable interpolation
```
Otherwise generating an error of the form:
```
  json: error: invalid bracketed lookup string: "[\"foo\" + 3]" (must be of the form ['...'], ["..."], or [`...`])
```

9.0.6

[issue #107] Fix man page installation with npm install -g json.

9.0.5

[issue #112] Improve streaming (json -ga) performance for very long lines. For example, using a 35 MB JSON object on one line gave a 50x speed improvement. However, this is restricted to streaming of newline-separated JSON as opposed to adjacent JSON objects not separated by newlines ({"a":1}{"a":2}). The former case is expected to be much more common, and the latter may take a slight performance hit from this change.

9.0.4

[issue #108] Fix a crash on json foo.bar if "foo" is null.

9.0.3

[issue #82] Fix a race in -I/--in-place temp file creation. By https://github.com/inator

9.0.2

[pull #72] Correct examples in docs for conditional filtering.

9.0.1

[issue #71] Support -o json-tab and -o jsony-tab for TAB (i.e. \t) indentation of emitted JSON.

9.0.0

[issue #52] Fix termination on EPIPE in some cases.
Add -0, -2, -4 options to more conveniently set the JSON indentation without changing the mode.

[pull #64] Add -M, --items option for "itemizing" key/value pairs in an object for easy iteration. For example:

  $ echo '{"trent":{"age":38},
           "ewan": {"age":4}}' | json -M
  [
    {
      "key": "trent",
      "value": {
        "age": 38
      }
    },
    {
      "key": "ewan",
      "value": {
        "age": 4
      }
    }
  ]

  $ echo '{"trent":{"age":38},
           "ewan": {"age":4}}' | json -Ma key value.age
  trent 38
  ewan 4

  # List people that can vote.
  $ echo '{"trent":{"age":38},
           "ewan": {"age":4}}' | json -M -c 'this.value.age > 18' -a key
  trent

Thanks to AndrewO for providing this!

Backward incompatible change to -c CODE and -e CODE changing their implementation to use a JS function for processing rather than vm.runInNewContext. This is the technique for which the -C CODE and -E CODE options were added in version 7.0.0. Basically: This technique is obviously better because it is 10x faster, so it is being made the only supported way. -C and -E, then, become synonyms and may be removed in a later release.

Unfortunately this does mean a few semantic differences in the CODE, the most noticeable of which is that this is required to access the object fields:
```
  # Bad. Works with json < v9...
  $ echo '{"green": "eggs"}' | json-v8 -e 'green="ham"'
  {
    "green": "ham"
  }

  # ... does *not* work with json v9.
  $ echo '{"green": "eggs"}' | json -e 'green="ham"'
  {
    "green": "eggs"
  }

  # Good. Works with all versions of json.
  $ echo '{"green": "eggs"}' | json -e 'this.green="ham"'
  {
    "green": "ham"
  }
```
The old behaviour of -c and -e can be restored with the JSON_EXEC=vm environment variable:
```
  $ echo '{"green": "eggs"}' | JSON_EXEC=vm json -e 'green="ham"'
  {
    "green": "ham"
  }
```
See the notes on json 7.0.0 below for full details on the performance improvements and semantic changes.

8.0.0

[pull #70] Move from 'jsontool' to 'json' in the npm registry! Thanks to https://github.com/zpoley for graciously giving up the name, and to @izs for driving. npm install json FTW. Here after jsontool will stagnate at version 7.0.2.

7.0.2

[issue #68] Fix --keys, -k handling in streaming mode, i.e. json -gak.

7.0.1

[pull #60, issue #59] Fix not having a json on the PATH from 'npm install -g jsontool'.

7.0.0

[issue #49] New -C CODE and -E CODE options to replace -c CODE and -e CODE. The new options can be 10x or more faster. An example processing a large log of newline-separated JSON object as a stream:

$ ls -al big.log
-rw-r--r--  1 trentm  staff   156M Oct 25 23:31 big.log

$ time json -f big.log -gac 'this.audit' req.method req.url >/dev/null

real	0m21.380s
user	0m21.051s
sys	0m0.526s

$ time json -f big.log -gaC 'this.audit' req.method req.url >/dev/null

real	0m3.336s
user	0m3.124s
sys	0m0.295s

For comparison with jq (a C-based fast JSON processor tool):

$ time cat big.log | jq '.req.method, .req.url' >/dev/null

real	0m3.307s
user	0m3.249s
sys	0m0.136s

The speed difference in json is in how the given CODE is executed: the new implementation uses a JS function, while the -c/-e options use node.js's vm.runInNewContext. This change means some semantic changes to the given CODE. Some examples to show the semantic differences:

this is required to access the object fields:

 $ echo '{"foo": "bar"}' | json -e 'foo="baz"'
 {
   "foo": "baz"
 }
 $ echo '{"foo": "bar"}' | json -e 'this.foo="baz"'
 {
   "foo": "baz"
 }

 $ echo '{"foo": "bar"}' | json -E 'foo="baz"'   # doesn't work
 {
   "foo": "bar"
 }
 $ echo '{"foo": "bar"}' | json -E 'this.foo="baz"'
 {
   "foo": "baz"
 }

Explicit return is required with -C when using multiple statements:

 $ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; sum > 5'

 undefined:2
 return (sum = this.a + this.b; sum > 5)
                              ^
 SyntaxError: Unexpected token ;

 $ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; return sum > 5'
 {
   "a": 2,
   "b": 6
 }

Some operations on the input object are more as you'd expect:

 $ echo '["a", "b"]' | json -AE 'this.push("c")'
 [
   "a",
   "b",
   "c"
 ]

 $ echo '{"a":1,"b":2}' | json -E 'delete this.a'
 {
   "b": 2
 }

CODE is no longer run in a sandbox, so you can shoot yourself in the foot. Security warning: Don't run untrusted code with '-E' or '-C'.

 $ echo '{}' | json -C 'process.stdout.end()'
 [Error: process.stdout cannot be closed.]

 $ echo '{}' | json -c 'process.stdout.end()'

 vm.js:41
         return ns[f].apply(ns, arguments);
                      ^
 ReferenceError: process is not defined

Overall: (1) is a annoying, (2) is likely rare but at least is explicit, (3) is a minor win, (4) is something to just be aware of. The major win is a ~10x speed improvement!

See https://github.com/nfitch/node-test/blob/master/test/vmalterns.js for analysis on the perf of various options for running user-given code. Thanks to Nate for pushing me on this!

Major perf win on simple lookups, and with no behaviour change(!). Similarly this was achieved by avoiding vm.runInNewContext:
```
$ time json6 -f big.log -ga time >/dev/null   # v6

real	0m28.892s
user	0m26.839s
sys	0m2.295s

$ time json -f big.log -ga time >/dev/null    # v7

real	0m2.427s
user	0m2.289s
sys	0m0.212s
```
The changes for this have changed jsontool.parseLookup and jsontool.handleLookup incompatibly. However, using "jsontool.js" is experimental and extremely rare. Please contact me if this impacts you.
Support for node 0.11.x -- basically stop using util.puts.

Note that apparently vm.runInNewContext has changed in node 0.11.x such that the following no longer works:
```
$ echo '["a", "b"]' | json -A -e 'this[0]="A"'
[
  "A",
  "b"
]
```
The result with node 0.11.x is actually:
```
$ echo '["a", "b"]' | json -A -e 'this[0]="A"'
[
  "a",
  "b"
]
```
Using the new -E works:
```
$ echo '["a", "b"]' | json -A -E 'this[0]="A"'
[
  "A",
  "b"
]
```
All the more reason to use the new -E CODE.
Change to 4-space indents. 'make check' clean. No functional change.

Include project url (and my name) in json --version. Also show JSON formatted version info with -j. The point here isn't self-agrandization but to help differentiate from the other json tool out there (npm home json).

  $ json --version
  json 7.0.0
  written by Trent Mick
  https://github.com/trentm/json

  $ json --version -j
  {
    "version": "7.0.0",
    "author": "Trent Mick",
    "project": "https://github.com/trentm/json"
  }

Validation failures with a given filename will now show the filename, e.g.:

  $ json -nf foo.json
  json: error: "foo.json" is not JSON: Expected ',' instead of '"' at line 3, column 5:
              "baz": "car"
          ....^

Move json.1 to "man/man1" and set "directories.man" in package.json to have "man json" work after "npm install -g jsontool" with the coming npm 1.3.3 work.

6.0.0

[Backwards incompatibility, issue #55] Drop support for grouping of adjacent arrays (via -g, --group) separated by no space:

  # Before
  echo '["one"]["two"]' | json -g
  [
    "one",
    "two"
  ]

  # After
  $ echo '["one"]["two"]' | json -g
  json: error: input is not JSON: Syntax error at line 1, column 8:
          ["one"]["two"]
          .......^
  ["one"]["two"]

We still allow grouping of arrays separated by a newline:

  # Before and after
  $ echo '["one"]
  ["two"]' | json -g
  [
    "one",
    "two"
  ]

This was dropped because the current regex technique used for grouping can fail with a JSON string that contains ']['. Not worth it.

New -I/--in-place option for in-place editing of given files with:

  $ cat foo.json
  {"foo":1}

  $ json -I -f foo.json                   # format the file
  json: updated "foo.json" in-place
  $ cat foo.json
  {
    "foo": 1
  }

  $ json -I -f foo.json -e 'this.bar=42'  # add bar field
  json: updated "foo.json" in-place
  $ cat foo.json
  {
    "foo": 1,
    "bar": 42
  }

Note: I'd have loved to have used -i a la sed, but that is unfortunately taken in this tool.

5.1.3

Fix an issue with option parsing that resulted in this failing:
```
  json -f foo.json -- -1.someKey
```

5.1.2

[pull #43] Add '-n' as a short alias for '--validate'. (By Bill Pijewski, github.com/pijewski).

5.1.1

[issue #42] Fix an edge case where a blank line would be emitted for ... | json -ga -c COND where the COND resulted in no matches.
[issue #40] Improve "Lookups" section of docs to show how to lookup non-identifier keys.

5.1.0

[pull #39, issue #34] json -ga streams. (Largely by Fred Kuo, github.com/fkuo) This means you can use json with an input stream of JSON objects. E.g.:
```
  yes '{"foo":"bar"}' | json -ga
```
Limitations: As was already a limitation of the '-g' switch, this only supports JSON objects delimited by newlines or with no space:
```
  {"a":1}{"b":2}          # good
  {"a":1}\n{"b":2}        # good
  {"a":1} {"b":2}         # bad
```
Additionally, currently only a stream of objects is supported, not a stream of arrays. Such are the typical use cases I've encountered.

5.0.0

[backward incompatible, issue #35] Special case the output for a single lookup AND JSON output (i.e. -j or -o json*) to only output the value instead of the more general array or table that is necessary for multiple lookups. For objects:

  # Before:
  $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one
  {
    "one": "un"
  }

  # After:
  $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one
  "un"

  # Unchanged:
  $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one two
  {
    "one": "un",
    "two": "deux"
  }

For arrays:

  # Before:
  $ echo '["a", "b", "c"]' | json -j 0
  [
    "a"
  ]

  # After:
  $ echo '["a", "b", "c"]' | json -j 0
  "a"

  # Unchanged:
  $ echo '["a", "b", "c"]' | json -j 0 1
  [
    "a",
    "b"
  ]

The motivation for this change was (a) the WTF of the first example above and (b) issue #36 where one could no longer extract a single value using '-j' to explicitly get JSON string quoting.

4.0.1

[issue #36] Turn off coloring for inspect output (json -i, json -o inspect) if stdout is not a TTY.

4.0.0

Add --validate option to just validate (no processing and output)

  $ echo '{"foo" "bar"}' | json
  json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8:
          {"foo" "bar"}
          .......^
  {"foo" "bar"}
  $ echo '{"foo" "bar"}' | json --validate
  json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8:
          {"foo" "bar"}
          .......^
  $ echo '{"foo" "bar"}' | json --validate -q
  $ echo $?
  1

Add -f FILE option for specifying an input file (or files) instead of stdin:

  $ json -f foo.json
  {
    "foo": "bar"
  }
  $ json -f foo.json foo
  bar

[Backward incompatible] Move "auto-arrayification" to require explicitly using the "-g" or "--group" option:

  $ echo '{"one":"two"}{"three":"four"}' | json
  json: error: input is not JSON: Syntax error at line 1, column 14:
          {"one":"two"}{"three":"four"}
          .............^
  {"one":"two"}{"three":"four"}
  $ echo '{"one":"two"}{"three":"four"}' | json -g -o json-0
  [{"one":"two"},{"three":"four"}]

This is to avoid auto-arrayification accidentally making an invalid JSON object (with a missing comma) be transformed to a valid array:

  $ cat oops.json
  {
    "a": {
      "b": [
          {"foo": "bar"}
          {"foo": "bar"}
      ]
    }
  }
  $ cat oops.json | json3 -o json-0
  [{"a":{"b":[{"foo":"bar"},{"foo":"bar"}]}}]

Basically the jusitification for this breaking change is that the invariant of json validating the input JSON is more important than the occassional convenience of grouping.

Use 8 space indent for syntax error message on stderr instead of '\t'. Minor change. Tabs are evil.

3.3.0

Add -k|--keys option to output the input objects keys:

  $ echo '{"name": "trent", "age": 38}' | json -k
  [
    "name",
    "age"
  ]
  $ echo '{"name": "trent", "age": 38}' | json -ka
  name
  age

Drop jsontool v2 dependency. This had been added for the first few json3 releases to provide a json2 for comparison. json v3 is fairing well enough now to not bother.

3.2.0

Support negative array indeces (a la Python list indeces), e.g.:
```
  $ echo '["a", "b", "c"]' | json -- -1
  c
```

3.1.2

Update man page and move bulk examples from README to man page. Use ronn (the ruby one) instead of ronnjs: better and more reliable formatting. Add 'make docs' and 'make publish' (the latter to push to GH pages at http://trentm.com/json).
[issue #31] Fix error message for json -o.

3.1.1

[issue #32] Fix '-D' option processing so json -D/ works (no space).

3.1.0

[pull #29] Add '-D' option to set a delimiter for lookups (default is '.'), so that this example works:
```
$ echo '{"a.b": {"b": 1}}' | json -D / a.b/b
1
```
By Yaniv Aknin.

3.0.3

[issue #30] Fix lookup strings with multiple double-quotes.
[issue #28] Don't error on a multi-level lookup where one of the components is undefined. E.g., the following is no longer an error:
```
  $ echo '{"foo": "bar"}' | json not_foo.bar
```

3.0.2

[issue #27] Fix issue handling multi-level lookups (e.g. 'json foo.bar').

3.0.1

Fix a bogus 'json' dep.

3.0.0

Switched to json 3.x dev on master. "2.x" branch created for any necessary 2.x releases. See the 2.x changelog here.
[Backward incompatible] A significant change to 'jsony' default output and some use cases to increase utility. These changes necessitated a few backward incompatible changes. However, care was take to only break compat for (a) rare use cases and (b) where utility was much improved. See https://github.com/trentm/json/wiki/backward-incompat-json-3-changes for full details of backward incompatible changes.
New "conditional filtering" via the -c CODE option. If the input is an array, then -c will automatically switch to processing each element of the array. Example:
```
  $ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \
      | json3 -c 'age>21'
  [
    {
      "name": "trent",
      "age": 38
    }
  ]
```

Change -e CODE option to automatically switch to array processing if the input is an array. This matches the behaviour of the new -c CODE option. Example:

  $ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \
      | json3 -e 'age++' -o json-0
  [{"name":"trent","age":39},{"name":"ewan","age":5}]

New '-A' option to force -e and -c to process an input array as a single item.
Add ansidiff-based colored diffs for test suite failures.

[issue #26] Add support for escapes in the delimiter given by -d DELIM:

  $ echo '[{"one":"un","two":"deux"},{"one":"uno","two":"dos"}]' \
      | json -a -d'\t' one two
  un	deux
  uno	dos

2.2.1

Hack workaround for issue #24 to not get a spurious "process.stdout cannot be closed" from current node 0.6 versions. Note: currently this guard is only applied for node v0.6.0..v0.6.8 inclusive.

2.2.0

New "-e CODE" option to execute the given code on the input object; or, if '-a/--array' is given, then on each item in the input array. Execution is done before filtering.
```
  $ echo '{"age": 38}' | json -e 'this.age++'
  {
    "age": 39
  }
```

2.1.0

Improve error message when input is not JSON to include context and line and column position. This is implemented using a JSON parser from (https://github.com/douglascrockford/JSON-js). Example:
```
  $ echo "[1,,2]" | json
  json: error: input is not JSON: Unexpected ',' at line 1, column 4:
      [1,,2]
      ...^
  [1,,2]
```

2.0.3

Auto-arrayification: Drop support for arrayifying an array adjacent to an object. I.e. only arrayify adjacent objects or adjacent arrays.
Auto-arrayification: Change "arrayification" of adjacent arrays to be a single flat arrays of the input arrays' elements. Before:
```
  $ echo '[1,2][3,4]' | bin/json
  [
    [
      1,
      2,
    ],
    [
      3,
      4
    ]
  ]
```
and now:
```
  $ echo '[1,2][3,4]' | bin/json
  [
    1,
    2,
    3,
    4
  ]
```
This is expected to be more useful in practice.
Auto-arrayification: Allow JSON objects (or arrays) to be "arrayified" if not separated by any space. Previously a newline (at least) separation was required. So, for example, the following now works:
```
$ echo '{"a":1}{"b":2}' | bin/json -o json-0
[{"a":1},{"b":2}]
```
The rules for auto-arrayification then are: Objects and arrays only, separated by no space or space including a newline.
Fix stdout flushing in some cases.

2.0.2

Add node v0.6 support. Drop v0.2 and v0.3 support.

2.0.1

[issue#23] Fix output in '-a|--array' mode if one or more keys don't exist in one or more of the array items.

2.0.0

'-o | --output MODE' support. Supported modes:

jsony (default): JSON with string quotes elided
json: JSON output, 2-space indent
json-N: JSON output, N-space indent, e.g. 'json-4'
inspect: node.js `util.inspect` output

'-a|--array' for independently processing each element of an input array.
```
$ echo '[
{
  "name": "Trent",
  "id": 12,
  "email": "trent@example.com"
},
{
  "name": "Mark",
  "id": 13,
  "email": "mark@example.com"
}
]' | json -a name email
Trent trent@example.com
Mark mark@example.com
```
This example shows that '-a' results in tabular output. The '-d' option can be used to specify a delimiter other than the default single space, e.g.:
```
json -d, -a field1 field2
```
[Backward Incompatibility] This is a replacement for the experimental '*' syntax in the lookup strings (previously enabled via '-x|--experimental'). That syntax and option has been removed.
Add '--' option processing support and error out if an unknown option is given.
Support multiple top-level JSON objects as input to mean a list of these object:
```
$ echo '{"one": 1}
{"two": 1}' | ./lib/jsontool.js
[
  {
    "one": 1
  },
  {
    "two": 1
  }
]
```
This can be nice to process a stream of JSON objects generated from multiple calls to another tool or cat *.json | json. Rules:
- Only JS objects and arrays. Don't see strong need for basic JS types right now and this limitation simplifies.
- The break between JS objects has to include a newline. I.e. good:
```
{"one": 1}
{"two": 2}
```
  bad:
```
{"one": 1}{"two": 2}
```
  This condition should be fine for typical use cases and ensures no false matches inside JS strings.

1.4.1

[issue #9] Gracefully handle EPIPE (i.e. stdout being closed on json before it is finished writing).

1.4.0

[issue #19] Allow multiple lookup arguments:
```
  $ echo '{"one": 1, "two": 2}' | json one two
  1
  2
```
WARNING: This involve a backward incompatible change in the JS APIs jsontool.processDatum and jsontool.processDatumExperimental.

1.3.4

[issue #18] Fix json --version for standalone mode again (was broken in json 1.3.3).

1.3.3

WARNING: json --version is broken when running outside the source (or npm install'd) tree. I.e. this is a bad release for standalone.
[issue #17] Ensure stdout is flushed on exit.

1.3.2

[issue #16] Fix to use <regex object>.exec instead of using the regex object as a function -- no longer allowed in the v8 used in node v0.5.x.

1.3.1

Make "jsontool" require'able as a module. For example, you can now:
```
  $ npm install jsontool
  $ node
  > var jsontool = require('jsontool')
  > jsontool.parseLookup('a.b.c')
  [ 'a', 'b', 'c' ]
  > jsontool.parseLookup('my-key.0["bar"]')
  [ 'my-key', '0', '["bar"]' ]
  > jsontool.main(['', '', '--help'])
  Usage: <something generating JSON on stdout> | json [options] [lookup]
  ...
```
Currently other exported API is experimental and will likely change to be more generally useful (e.g. the current processDatum isn't all handy for module usage).

Note: For command-line usage, the main module has moved from "json" to "lib/jsontool.js". So, if you are not using npm, you can setup the json command via something like:
```
  alias json='.../json/lib/jsontool.js'
```

1.3.0

package.json and publish to npm as "jsontool" ("json" name is taken)
Add experimental support for '*' in the lookup. This will extract all the elements of an array. Examples:
```
  $ echo '["a", "b", "c"]' | json -x '*'
  a
  b
  c
  $ echo '[{"one": "un"}, {"two": "deux"}]' | json -x '*'
  {
    "one": "un"
  }
  {
    "two": "deux"
  }
  $ echo '[{"foo": "bar"}, {"foo": "baz"}]' | json -x '*.foo'
  bar
  baz
```
This is still experimental because I want to feel it out (is it useful? does it cause problems for regular usage?) and it is incomplete. The second example above shows that with '*', json can emit multiple JSON documents. json needs to change to support accepting multiple JSON documents.

Also, a limitation: How to extract multiple fields from a list of objects? Is this even a necessary feature? Thinking out loud:
```
  '*.{name,version}'      # a la bash. Josh likes it. What else do you need?
```
Add '-x|--experimental' option to turn on incomplete/experimental features.

1.2.1

[issue #12] Fix handling of output when result of lookup is undefined.

1.2.0

[issue #10] Fix for node v0.5.

1.1.9

[Issue 8] Don't emit a newline for empty output.

1.1.8

[Issue 7] Handle "HTTP/1.1 100 Continue" leading header block.
[Issue 4] Add a man page (using ronnjs).

1.1.7

[Issue 5] Fix getting a key with a period. E.g.:
```
  echo '{"foo.bar": 42}' | json '["foo.bar"]'
```
json is now doing much better lookup string parsing. Because escapes are now handled properly you can do the equivalent a little more easily:
```
  $ echo '{"foo.bar": 42}' | json foo\\.bar
  42
```

1.1.6

[Issue 6] Error exit value if invalid JSON.

1.1.4

[Issue 2] Fix bracket notation: echo '{"foo-bar": "baz"}' | json '["foo-bar"]'

(Started maintaining this log 19 March 2011. For earlier change information you'll have to dig into the commit history.)

Files

CHANGES.md

Latest commit

History