Should we consolidate the human-readable JSON efforts?

[octogonz]

I'd like to write comments in my .json files and not have GitHub's syntax highlighter go nuts with red error underlines.

When I searched for human-readable JSON options, I also found jsonc and Hjson. Are these efforts all essentially equivalent? If so should we merge these fragmented camps and make a de facto standard? Even better, what's involved to evolve the actual IETF standard?

The JSON parser that we use already supports comments just fine, but people keep complaining that (1) it's not "standard" (i.e. the world's worst parser JSON.parse() doesn't support it) and (2) that various syntax highlighters reject it.

I'm happy to rename thousands of file extensions and release major breaking versions of our tools... but I'm unsure which way to go:

1. Rename everything to .json5
2. Rename everything to .jsonc
3. Keep .json and go pester the various tools to accept comments, because even the inventor of JSON said he was cool with comments

Also, maybe someone should update the Wikipedia article to talk about these community efforts.

[octogonz]

@hogmoru @iclanton FYI

[jordanbtucker]

Thanks for the question. JSON5, jsonc, Hjson, and YAML all attempt to make JSON easier for humans to read and write. (Yes, YAML is a superset of JSON too.)

I think the main thing that stands out with JSON5 is that, like JSON, it's just JavaScript. You can copy a JSON5 document into a JavaScript document, and you won't get syntax errors. (You can say the same thing about jsonc, but it doesn't make writing JSON that much easier.)

Hjson and YAML are easy to write, but they add a lot of syntax that is not compatible with JavaScript.

Honestly, I think JSON5 should be the de facto standard for anything processed by humans, like configuration files. It just feels natural to write because it's just JavaScript. However, it's becoming more common to just export things from .js files, which would be fine except that different environments support different kinds of export syntax.

But as for machine processed data, like REST APIs, I think JSON is just fine. It's verbose, but its syntax is simple enough that it's easy to implement.

[octogonz]

Why did you put a number in there? Seems like if we adopt this file extension, in a few years we'll have to do another disruptive update to make everything .json6. I think .php eventually dropped the numbers to avoid this problem heheh.

[jordanbtucker]

Another good question. I can't speak for @aseemk, as he's the original creator, but for me, JSON5 indicates a specific, reliable target—ES5. JSON5 syntax will never go beyond the syntax of ES5. Sure, that means it will miss out newer features, but it also means that users don't need to worry about what version of JSONX is supported.

Personally, I don't mind having a JSON5, JSON6, JSON7, as it makes it clear what features are supported, and they would always be backward compatible.

Alternatively, we could use a .jsonx extension and have a version directive at the beginning of the file. Something like this maybe:

/* JSON6 */
{
  template: `JSON6 supports a limited version of template strings.`,
  unicode: `Plus code point escapes: 𠮷 === \u{20BB7}`,
  octal: 0o777,
  binary: 0b10110010,
}

The thing is, ES7, ES8, and ES9 haven't introduced any new JSON relevant syntax features. ES10 will have support for BigInt and the n number prefix, but there isn't really a specific need to implement that since there is technically no restriction on the length of numbers in JSON5. So JSON6 is the only new version of JSON that would have much value.

[octogonz]

Personally, I don't mind having a JSON5, JSON6, JSON7, as it makes it clear what features are supported, and they would always be backward compatible.

Imagine if your JavaScript source files had to use extensions .js5, .js6, .js7, etc. In a massive code base, people would find this endlessly annoying. That's why PHP got rid of .php3, .php4, php5, etc and converged on simply using .php for PHP files. From a syntax highlighter's perspective, the standards are mostly forwards compatible anyway.

Alternatively, we could use a .jsonx extension and have a version directive at the beginning of the file.

I don't think a directive is necessary. If .js files don't need directives to indicate their version, then neither should .jsonx, right?

[octogonz]

I'm liking the idea of JSON5. How difficult would it be to get the rfc8259 standard updated to incorporate this idea? Then we could just use the .json file extensions, which seems like the best long-term solution. 😄

[jordanbtucker]

I don't think a directive is necessary. If .js files don't need directives to indicate their version, then neither should .jsonx, right?

Good point. It could just be up to user to know which version of JSONX a library supports, especially since each version would be backward compatible. Plus, we could create transpilation tools to convert JSONX to a specific version.

How difficult would it be to get the rfc8259 standard updated to incorporate this idea?

I don't think we should. JSON has a different use case than JSON5. JSON is very widely used for machine-to-machine communication, and the fact that its syntax is relatively simple makes it easy to implement. That's why it's been so widely adopted.

JSON5 is for humans to read and write. Its best use case is configuration files. Although it can result in smaller payloads, it's more difficult to implement parsers and generators. I think we'd be better off using a different file extension for that.

Acutally, I have a proof of concept called JSONext, which is an extention of JSON that supports Next gen features. It's backward compatible with JSON5, and it supports some ES6 features. Its file extension is of course .jsonext.

[hogmoru]

I like the JSONext concept, but I think a .jsonx file extension would have a better chance at wide adoption than .jsonext.
The "X" alone is enough to mean "extended".

Wait... oh... oh, no...

JSONx is an IBM standard format to represent JSON as XML.

God dammit, IBM :(

[octogonz]

Acutally, I have a proof of concept called JSONext, which is an _ext_ention of JSON that supports Next gen features. It's backward compatible with JSON5, and it supports some ES6 features. Its file extension is of course .jsonext.

What are "Next gen features"? Why weren't they part of JSON5 as originally proposed?

I'm considering to choose JSON5 (over the other alternatives I mentioned) based on the following requirements:

• We want to write comments in our config files
• Maybe we'd benefit from a bit of syntax improvements that make the config files more readable
• We ship these files to paying customers, so we need it to be a "standard" that can be widely supported across many different editors and syntax highlighters
• The syntax should be simple and intuitive. The big appeal of JSON5 is that it is a subset of the ECMAScript grammar that everyone's already familiar with
• I would NOT want lots of syntax sugar or redundant notations for the same thing (as YAML does). It confuses people. It makes parser libraries larger. It makes it difficult to rewrite a file while preserving its spacing and layout.
• I would NOT want any executable expressions or "macros" in the file. It's just a data file. If we wanted executable code, we can use .js instead of .json5

Your earlier assertion that "JSON5 syntax will never go beyond the syntax of ES5" sounded really great. Representing simple strings/numbers in a text file is not a problem that requires a constantly evolving feature set.

[octogonz]

JSONx is an IBM standard format to represent JSON as XML.

God dammit, IBM :(

I found that VS Code's JSON language service already supports comments using a library called jsonc-parser that accepts "JSON with JavaScript style comments" (and no other syntax extensions beyond that).

The "jsonc" project that I mentioned above is for Go programming language, apparently hasn't been touched in 2 years, and it seems like hardly anyone is using it. Maybe we should propose to merge jsonc, jsonc-parser, and JSON5 into a single solution: It would use the JSON5 feature set and library, but with the .jsonc file extension. "JSONC" is a great name, because it nicely conveys the idea of "JSON for Config files."

The more I think about this, I love the concept of JSON5, but the name isn't very good because it implies that JSON5 is not a "final" solution to the problem. We have thousands of config files spread across many teams and repos and products. If we migrate away from .json, it's going to be a very disruptive change. It will take work to persuade everyone. The JSON5 name implies that .json5 will inevitably evolve into a .json6. People can be persuaded to migrate once, but nobody's going to agree to a "standard" that is actually an endless treadmill of churning file formats, that break compatibility with older tools.

[octogonz]

@aeschli FYI, who was involved with implementing jsonc in VS Code (microsoft/vscode#3641)

[jordanbtucker]

What are "Next gen features"? Why weren't they part of JSON5 as originally proposed?

As of now, JSONext supports JSON5 syntax plus several ES6 syntax features. The README gives examples. Some features weren't originally included in JSON5 because JSON5 targets ES5, not ES6. JSONext is meant to be an evolving version of JSON that adds new syntax features borrowed from the latest version of ES.

Your earlier assertion that "JSON5 syntax will never go beyond the syntax of ES5" sounded really great. Representing simple strings/numbers in a text file is not a problem that requires a constantly evolving feature set.

Glad to hear, because I don't see this ever changing for JSON5. My comments regarding JSON6, JSONX, and JSONext refer to other (potential) projects outside of JSON5.

• We ship these files to paying customers, so we need it to be a "standard" that can be widely supported across many different editors and syntax highlighters

JSON5 has an official specification. It is not codified in an RFC, but neither is YAML or Source Maps. It is used by popular libraries like Babel and config, among others. However, it is lacking an active ecosystem of good platform implementations, syntax hilighters, and linters. If I didn't have a full time job, I would be able to devote more time to building those. Any help is appreciated!

The "jsonc" project that I mentioned above is for Go programming language, apparently hasn't been touched in 2 years, and it seems like hardly anyone is using it. Maybe we should propose to merge jsonc, jsonc-parser, and JSON5 into a single solution.

I'm not opposed to this idea, but jsonc is already well established to mean JSON with comments, so changing jsonc to mean something else may present a challenge. BTW, the sourcegraph implementation is the most popular jsonc parser for Go, as far as I know, and it's actively maintained.

The more I think about this, I love the concept of JSON5, but the name isn't very good because it implies that JSON5 is not a "final" solution to the problem.

I don't think there really is a final solution. Software development changes very rapidly. New languages, tooling, IDEs, methodologies, and interfaces emerge every year. JSON5 is named so because it targets ES5, meaning it only borrows syntax from ES5, and it will always be compatible with ES5. As such, it is backward compatible with JSON and forward compatible with JavaScript.

I do see where you're coming from regarding the name and file extension, though. I didn't choose it, and I personally would have gone with something less specific.

Nobody's going to agree to a "standard" that is actually an endless treadmill of churning file formats, that break compatibility with older tools.

Maybe, but then you're locked into the situation we have now with JSON. It's an unchanging standard that won't break compatibility with older tools. Yet, we both came to this project looking for something better.

To sum things up. I'm not opposed to merging JSON5 into another JSON for Humans project, but I'd prefer that it is compatible with ECMAScript syntax and doesn't add any new data types over JSON. (No Date or RegExp support. I'm not even happy with JSON5 having Infinity and NaN.)

[aeschli]

@octogonz I did most of the work on the jsonc-parser. The syntax (json + comments) had been around for quite some years already , e.g in Sublime configuration files, that's why we picked it.
We always thought of it as VSCode internal thing and never had ambitions of making it a new standard.
Also, recently, it has lost importance in VSCode as we've added a settings UI. I'm also a fan of keeping syntaxes simple and well-defined and like the approach json5 has taken. Let's not add another flavor of JSON. I'm not aware of many requests to make the VSCode configuration file syntax richer, but if there were, we'd probably go with JSON5 as the format of our configuration files.

[octogonz]

@aeschli how do you feel about the .json5 file extension?

[aeschli]

I think .json5 is a good extension name, as the syntax is named that way and well known.

[octogonz]

Ok thanks everyone for helping me think through these considerations. It was really helpful. I'll go see if I can get people onboard with converting our config files to use .json5 format.

[geyang]

I'm here for the Infinity and NaN support that my numpy/pandas python server sends. There isn't a good alternative than to use a JSON format that supports these. and JSON5 is the closest thing to a standard.

Judging from the number of npmjs.org downloads, this one has the largest traction. Do we have any effort in actually making this a standard to be supported by JS engines?

[jordanbtucker]

Do we have any effort in actually making this a standard to be supported by JS engines?
— @episodeyang

Are you asking if there is any effort to have a global JSON5 object included in the ECMAScript spec? I'm not aware of any efforts. If you'd like to champion that, please see CONTRIBUTING.md in the ecma262 repo.

[mindplay-dk]

The obvious choice for a file extension of course is

.j5on

🤪

[DonaldTsang]

Requesting communications with HJSON.
Also @mindplay-dk .j5 would be even better

[d3x0r]

Re d3x0r/JSON6 This was my sort of conceptual fork of JSON5. It adds.. ( in addition to all of JSON5 extensions such as comments, unquoted identifiers). Internally it's a streaming processor, it will return a single token if given one. 123 for instance...

• _ in numbers
• bigint suffix n on number
• ISO 8601 numeric date format (number subtype instead of a string) (revives as a Date())
• 0 octal leadin, 0[xX|bB|oO] , +/- allowed as leadin to number,
• empty array elements, and trailing commas in objects and array.
• undefined which is parsed, but not generated.
• ", ' ,`` (backtick) quoted strings; which just save everything literally from open to close, saving newlines, if not escaped.

I forked that and created JSOX, which is really a thing that, although accepting all JSONX, adds a syntax feature which makes it incompatible with JSON and ES[x]... But, supports JS specific features that are somewhat meaningless in other langauges, such as reiving all Objects/Arrays/Strings that are tagged as a type with the same prototype, which can later be assigned directly to provide methods of that class of objects. It uses these extended type tags to represent ArrayBuffers and All typed Arrays (u8,i8, etc), which encode the binary data as base64 string, with a type tag before the string. It also adds an internal type that is a ref[/*path to reference*/] that allows specifying a path that is already exisitng in the current object being revived, allowing encoding of self-cyclic objects.

In these subsequent projects, based Similary with JSON5's method of assigning module loaders for a specific extension '.json5', and used 'json6' and 'jsox' where it appears in files.

[octogonz]

Hjson currently does have some major advantages, including multiline strings and round-trip parsing, not to mention how many languages it's been implemented in.

We've been using the jju parser, which does support "round-trip". In other words, your program can read a file, modify it, and write it back to disk, while preserving the original whitespace/comments. This page has an interactive demo of this capability.

The json5 package is described as a "reference implementation," so maybe it's goal is merely to specify the grammar as clearly as possible, and advanced features are left to serious implementors. If so, maybe the website should recommend these other implementations.

JSON5 supports multiline strings by escaping newlines.

If the goal is to deviate from JavaScript and invent the world's best config file format, I'd recommend XML. A huge committee spent years refining the design of XML. It has every feature imaginable, and is widely supported. :-)

[dqsully]

@octogonz are you suggesting everyone switch to JSON5, and if they need extra features they can just use the jju parser? I don't think that's a good solution for everyone because it alienates anyone using a different format.

Even if the goal is to convert everyone over to JSON5, you would need to provide a converter or a standardized interface across the different languages in order to ease migration. The parser I'm building could help people migrate to JSON5 eventually, even though I intend to support Hjson for the foreseeable future. If you have a better idea I would honestly love to know, and XML doesn't count :)

[d3x0r]

Round Tripping... that seems like a nice to have feature.

I thought in JSON6 and JSOX that I had done something like ```var parser = JSON6()and then usedparser.strinigfy(...)` or `parser.parse(...)` which then the parser could have internal context and save comments related to the data while still returning just the simple data to the program for most usage... although programs might like to know about comments for meta tag sort of things... which then the parser instance could have additional methods for accessing the AST of the parse (node tree)...

// sample ... 
var parser = JSON?( /* maybe an option object ... { keepComments : true, enableFeatureX: false }  */);
var array = parser.parse( "[ 'your typical JSON sort of string' ] " ); 
var string = parser.stringiify( { some: "other object" } );

I actually ended up just keeping the JSON6/JSOX objects as containers for methods, and added a begin(), which returns an object with different methods more appropriate for continuous streaming; JSON stringify ends at the end of the string, and cannot do partial evaluations...

var parser = JSOX.begin();
parser.on( "data", (value)=>{ console.log( "got some completed value... ", value ) };
parser.write( " [ 'some json content " );
parser.write( "split beteen buffers' ] ");
parser.write( "1234" );
parser.write(); // flush... will trigger thatn 1234 is the end of the simple number object

and var stringifier = JSOX.stringifier() provides an object with additional methods appropriate for stringification.

Of course providing simple 'parseandstringify` methods for drop-in replacement... behaving exactly as one would expect from those... if someone added a reasonable API extension I'd certainly entertain the thought of implementing it their way.

const JSON = JSOX;  // :) that's probably just me though.

[octogonz]

are you suggesting everyone switch to JSON5, and if they need extra features they can just use the jju parser?

I think so.

If we're all collaborating to build a house, it's useful to have both nails and screws. Sometimes the choice doesn't matter, but certain jobs really need a screw. And other jobs work best with a nail. I don't mind having both a screwdriver and a hammer in my toolbox.

What I /do/ mind is having to carry around 20 different screwdrivers in case of flathead screws, or Phillips screws, or screws with little square holes, or screws with little star-shaped holes. I'm sure those star-shaped screws have certain obscure benefits that their inventor was excited about, but it's just not worth the headache of me having to juggle 20 screwdrivers.

We could improve things by telling everyone to buy a complicated screwdriver. The complicated screwdriver comes with 20 different attachments, enabling it to work with every possible kind of screw. This is less of a hassle, but it's still a hassle. It doesn't address the real problem.

The right answer is simply to agree to use Phillips screws everywhere. Phillips might not be perfect, but it doesn't need to be: A standard merely needs to be reasonably good and widely adopted.

[RandomFractals]

interesting thread ... just an fyi, I ended up supporting json5, hjson & yml in my Data Preview 🈸 vscode ext., if you ever have the need to query, filter, transform or graph those data/config files: https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-preview

@octogonz best swiss army knife util I could come up with for dealing with these data formats :)

[RandomFractals]

wow! this escalated quickly. my take on it since I've done some work with json5 hjson & yaml: I think they are fine as they are and should stay that way. It's up to you which one you pick for your solution.

As far as json formats go, I prefer json5 over hjson || jsonc due to many benefits listed above.

[aseemk]

Just want to say I continue to appreciate your fantastic support & help as maintainer @jordanbtucker! ❤️

[d3x0r]

I threw together a new terminal application, and ran into a case that an editable span control returns '\xA0' which is a non-breaking space character.

' ' ' '

In an unquoted-keyword, using the standard-ish parsing rules, this could allow 'a\xA0identifier' to be unquoted; however, from a human perspective, this should be quoted...

So, should these human-readable JSON efforts side with the human and tread that the same as a normal space?

console.log( "{ setting\xa0 \xa0:\xa0 \xa0true }" );
{ setting   :   true }

that would result in a setting called 'setting ' only really it's 'setting '.

[DonaldTsang]

Question: should we contact these C/++ parsers for help (once we consolidated the syntax), or should we do our own thing?

• https://github.com/Tencent/rapidjson
• https://github.com/lemire/simdjson
• https://github.com/DaveGamble/cJSON
• https://github.com/lloyd/yajl
• https://github.com/zserge/jsmn

What about asking these Golang to include our standard?

• https://github.com/tidwall/gjson
• https://github.com/buger/jsonparser
• https://github.com/Jeffail/gabs

[scottdotjs]

@octogonz wrote:

3. Keep .json and go pester the various tools to accept comments, because even the inventor of JSON said he was cool with comments

Just coming to this issue I noticed the above is a link to Google+ which is dead. For posterity, via the Internet Archive, Douglas Crockford's post was:

Comments in JSON

I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.

Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.

[RandomFractals]

@DonaldTsang hey Don, great list of json parsers in the wild.

I'd say reach out to them and see who responds and willing to collabo on consolidating some of them.

also, a side note, I recently added .ndjson and .jsonl support to my Data Preview 🈸 vscode extension that over 28K devs use now. You guys should try it:

https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-preview&ssr=false

Let me know if you'd like to see any other major json formats heavily used by devs supported by that dev tool.

[dqsully]

Alright, I've finally got a proof of concept for a modular parser that could theoretically handle JSON, JSON5, Hjson, and pretty much anything else. At the moment I've only got booleans, strings, objects, comments, and whitespace implemented, and it doesn't store metadata or stringify anything yet, so there's still a lot of work left to do. Instructions to play around with it are in the README: https://github.com/dqsully/serde-js. I'm really excited to finally have something working, and I think this is going to turn out awesome.

[RandomFractals]

@dqsully that would be nice. Can you also add jsonc (JSON with comments) that vscode is using for their settings, and jsonl/ndjson to that wish list? :)

[dqsully]

@RandomFractals JSONC is definitely on the list. I think I can make ndjson/jsonl work too with an extra language feature, but I probably won't focus on those at first. I think I'll also add a streaming visitor to the eventual wishlist in order to support streaming ndjson/jsonl, among other formats.

Edit: any other feature requests or discussion about my project should be in an issue on my project's repository. I don't want to pollute this issue with off topic comments

[mathiasrw]

JSON5 should remain JSON5 and keep being compatible with ecmascript 5. If there is ever going to be a format that follows the developmend of ecmascript is should be called ESON.

[DonaldTsang]

@mathiasrw further evolutions should be considered since ESON is a good idea.

[mmkal]

JSON5 should remain JSON5 and keep being compatible with ecmascript 5. If there is ever going to be a format that follows the developmend of ecmascript is should be called ESON.

For anyone interested, I forked json5 into a strawman repo and published an npm package called eson-parser. Its usage is identical, but it supports multi-line strings delimited by backticks

example

{
  a: 1,
  b: 'two',
  c: `
    select *
    from users
  `,
  d: [
    true,
    "false",
  ],
}

Usage, after npm install eson-parser:

const ESON = require('eson-parser')

const obj = ESON.parse(`{
  a: 1,
  b: 'two',
  c: \`
    select *
    from users
  \`,
  d: [
    true,
    "false",
  ],
}`)

const str = ESON.stringify(obj)

There are very few changes from this repo - you can view them here (short version - add backticks and typescript types (to avoid the need for a separate types package)).

It's one day old so obviously using it for production right now would be very risky compared to going with json5. But I'd be interested to get any feedback on whether people think it's worth developing into its own project which can incorporate a subset of ecmascript features past es5, or if this is just a case of xkcd 927.

My hope was to not introduce another standard/any new syntax in the same way as JSON5 isn't - because it's strictly using a subset of existing ECMAScript features (as @octogonz talked about here). For me, the main reason for wanting this is as a way to express configs (which are mostly read and written by humans), with a better way of including multiline strings than the ambiguous newline-escaping trick. Off the top of my head, I can't think of many other ECMAScript features that would be needed, but I also have a tc39 proposal for automatically stripping margins from multiline strings, which I think would be a useful addition if it were accepted one day.

CC @jordanbtucker I believe jsonext has a very similar purpose, but there are no commits there since 2017. Is there a plan to revive that project at any point?

[d3x0r]

https://github.com/d3x0r/JSON6

Adds these vs JSON5

• Keyword undefined
• Objects/Strings back-tick quoted strings (no template support, just quotes); Object key names can be unquoted.
• Strings - generous multiline string definition; all javascript character escapes work. (\0, \x##, \u####, \u{} )
• Numbers - underscore digit separation in numbers, octal 0o and binary 0b formats; all javascript number notations.
• Arrays - empty members
• Streaming reader interface

Just finished coverage and automated test integration with travis.

[mathiasrw]

@d3x0r So awesome. Thank you!

[octogonz]

So is .json6 now a file extension alongside .json5? 🤔

After .php2, .php3, .php4, and .php5, it seems PHP gave up on massive bulk-renames and just used .php for everything.

[mathiasrw]

That is why we should use .eson

[octogonz]

JSON5 has now itself forked into JSON5 and JSON6. Whether that's for better or worse, we definitely are NOT consolidating anything. The question that I originally asked has been answered. 🤷‍♂️