Finalize JSON5

[seagreen]

The EMCA JSON specification states, "Because it is so simple, it is not expected that the JSON grammar will ever change."

Will this ever be a goal of JSON5?

I really like what I see of the project so far. It seems like a perfect solution to JSON being hard to write. However, looking through the issue list I see things like variables and datetimes being discussed which make me nervous. Not that those things are bad, they just don't fit with my use case of only needing "writeable JSON".

[aseemk]

Great question! cc @jordanbtucker to chime in too.

I think that's a reasonable goal. The reality is that we haven't shaken the boat with things like dates in all these years, so it's probably unlikely that we will at this point either.

So there could be some value in officially and formally "freezing" the spec... but I think there's also something to be said for learning from people's experiences and adapting to an ever-changing world.

How do others feel?

[seagreen]

Hey @aseemk, a few thoughts:

(1) If you freeze JSON5 that doesn't mean in 5+ years from now the lessons from JSON5 can't be taken and developed into a new spec (JSON22?).

JavaScript will continue to change slowly, and someday it might be good to have another spec for easy to write JSON to keep up with it. But I actually think that process will go better in some ways if the specs are separate! Otherwise you end up in a very awkward period where a program may accept .json5 documents, but you're not sure if you can use the new features or not.

Of course, this will end your ability to make small changes, which I admit is a real downside.

(2) If you think it's a good idea to move towards freezing JSON5 then it doesn't have to happen immediately. For instance, @jordanbtucker's issues are exactly the kind of thing that would be good to resolve first.

What do you think?

[aseemk]

@seagreen: That sounds like a good perspective to me. Thanks! 👍

[seagreen]

@aseemk: It's been half a year since I opened this issue. Do you think it's time to move towards finishing this thing? If so we should start making a list of blockers.

[seagreen]

Bump. YAML is terrible, the world needs JSON5, to get adoption from serious users it has to actually be finished.

[techtonik]

For me comments, trailing commas and non-quoted key names is a must.

Multiline stings with quoted ends are not convenient for copy/paste - Python-like triple quoted strings are better, but they lack indentation for readability.

Everything else potentially makes parsing slower.

Considering that JSON is cross-language, this - Infinity, -Infinity, NaN, and -NaN. - is not portable.

[seagreen]

@techtonik: Gotchya. I actually have no opinion on any of the specifics. I really like the general direction of JSON5, so I trust that whatever changes are made in the run up to finalization will be as classy as the rest of the spec.

The thing I do care about is that some kind of release does happen eventually. Even making a 1.x release and saying "nothing will change from this point except touchups" would go a long way, though of course I would prefer:

Because it is so simple, it is not expected that the JSON5 grammar will ever change.

[techtonik]

Comments alone are worthy of release - https://stackoverflow.com/questions/tagged/json?sort=votes

[jordanbtucker]

Please see the v1.0.0 TODO list. If you are able to contribute to the spec, please do so on the beta branch.

Beside completing the spec, I'm going to do a rewrite of the codebase because the current codebase is based on Crockford's implementation whereas the new codebase will be based on the ES5 implementation.

[jordanbtucker]

Please see the v1.0.0 Milestone. Many of the issues/PRs will be fixed with the codebase rewrite, and I'll close those once I've merged that in, however, the CLI enhancements still need to be written along with their tests.

[jordanbtucker]

v1.0.0-beta is live. Read the CHANGELOG to see the new features and fixes.

Use it in your projects with:

npm install --save json5@beta

Use it in your terminal with:

npm install -g json5@beta

Use it in your browser with:

<script src="https://unpkg.com/json5@beta/dist/index.js"></script>

[seagreen]

@jordanbtucker: Awesome! That was blazingly fast. Thank you for all your work on it.

My only comment would be that I look forward to this TODO being filled out. I'm having trouble telling whether JSON5 is meant to be a superset of JSON or not. This comment makes me think not, but the ABNF makes me think it is.

EDIT: I just want to say again how excited I am JSON5 is moving towards a 1.x release. This is seriously great.

[aseemk]

Amazing work @jordanbtucker!! :)

[jordanbtucker]

@seagreen Thanks for the commendation. Here's the relevant section from the WIP spec regarding line and paragraph separators.

Paragraph and Line Separators
Like JSON, JSON5 allows the Unicode code points U+2028 and U+2029 to appear unescaped in strings. Since ECMAScript 5.1 does not allow these code points in strings, JSON5 authors should avoid including them in JSON5 documents. JSON5 parsers should produce a warning when they are found unescaped in strings. JSON5 generators should escape these code points in strings.

So officially, JSON5 is a strict superset of JSON and a loose subset of ES5 (but only loose regarding line and paragraph separators). I updated that comment to reflect this now.

I haven't reviewed the ABNF with the new changes in mind, so I've added that to the TODO list.

[jordanbtucker]

I'm also curious to know what everyone thinks about the Date and RegExp support I've added in experimental branches.

[jordanbtucker]

I forgot to document a new feature in the CHANGELOG.

New: stringify checks for a toJSON5 method on objects and, if it exists, stringifies its return value instead of the object. toJSON5 overrides toJSON if they both exist.

This is mainly useful in two cases:

• When you want to convert an object to Infinity or NaN before stringifying.
• When you want JSON5 output to differ from JSON output because reasons.

[jordanbtucker]

The latest draft of the spec is up at https://json5.github.io/json5-spec/. Make sure you refresh the page to get the latest version. The date should be September 28, 2017.

I consider this the final version unless anyone finds anything wrong with it. Once finalized, it will say Standard instead of Draft, and the note about it being a draft will be removed.

I appreciate any feedback.

[aseemk]

@jordanbtucker: This is excellent work!! Thank you as always. =)

Minor bug: "A JSON5 parser transforms a JSON text into another representation" should be "JSON5 text", right?

[jordanbtucker]

@aseemk Good catch. I fixed that, added a JSON5Punctuator production that I missed, made it mobile friendly, and added a server script for development.

[jordanbtucker]

I've made several changes to the spec. Many of the changes are only formatting or rewording, but there are a few important changes.

• Reworded the introduction to state that JSON5 is a superset of JSON and removed the part about it being a subset of ES5 (json5/json5-spec@3b5c1f1)
  • Related to that, JSON5 doesn't add new data types to ES5, but it does add new data types to JSON, namely Infinity and NaN, so I removed that part too.
• Fixed an error regarding what characters must be escaped in strings (json5/json5-spec@932a7d4)
• Reworded the parts about Infinity and NaN and added an example (json5/json5-spec@9a6aa9c)
• Refactored some grammar productions (json5/json5-spec@f034fc0)
  • The diff is a little confusing, but essentially I added a JSON5Identifier, JSON5Null, and JSON5Boolean production to the lexical grammar and ensured that the syntactic grammar only references the JSON5 lexical grammar and not the ES5 lexical grammar.
• Addressed octal escapes in strings (json5/json5-spec@33aee99)

That last change also required a bug fix committed in c952a0d and 78b403a, which is live on npm at json5@beta.

[jordanbtucker]

A while ago, I ran the beta against the v0.5.1 test cases to make sure there were no regressions and fixed the bugs I found. Now, I've run v0.5.1 against the beta test cases to see if there were any changes I forgot to document. I found a few and updated the CHANGELOG. Here's a summary. (Subpoints are additional information not included in the CHANGELOG.)

• New: Unicode property names and Unicode escapes in property names are supported. (Unquoted object keys should support non-ASCII characters #1)
  • Example: { \u0061: true } and { a: true } are equivalent.
  • Character and escapes can be mixed, so { abc\u0064\u0065\u0066: true } and { abcdef: true } are equivalent.
  • The code points represented by the escaped character must still conform to IdentifierName, so the following will fail: { \u0000: true }
• New: In addition to the white space characters space \t, \v, \f, \n, \r, and \xA0, the additional white space characters \u2028, \u2029, and all other characters in the Space Separator Unicode category are allowed.
• New: In addition to the character escapes \', \", \\, \b, \f, \n, \r, and \t, the additional character escapes \v and \0, hexadecimal escapes like \x0F, and unnecessary escapes like \a are allowed in string values and string property names.
• New: When a character is not allowed in a string, stringify outputs a character escape like \t when available, a hexadecimal escape like \x0F when the Unicode code point is less than 256, or a Unicode character escape like \u01FF, in that order.
  • Example: '\u0061\u0009\u000F\u01FF' becomes 'a\t\x0F\u01FF'
• Fix: stringify properly handles the key argument of toJSON(key) methods. toJSON5(key) follows this pattern.
  • Raise your hand if you didn't know this was a thing. ✋
• Fix: stringify accepts Number and String objects as its space argument.
  • Related but different: Stringify throws if the value isn't an Object #154
• Fix: In addition to a function, stringify also accepts an array of keys to include in the output as its replacer argument. Numbers, Number objects, and String objects will be converted to a string if they are given as array values.
  • Raise your other hand if you didn't know this was a thing either. ✋

[aseemk]

Nice work! I raise both of my hands 🙌 — this is purposely a double-entendre. =)

[seagreen]

@jordanbtucker: This is amazingly fast progress. Also, thanks for the clarification that JSON5 is definitely a superset of JSON!

My only concern is the clarity of this line:

JSON5 generators should escape these code points in strings.

It depends a lot on the precise meaning of "should" here. If I write a program that consumes JSON5 created by a generator, can I rely on those code points being escaped (AKA that the JSON5 will not actually be a superset of JSON, but rather an overlapping set?) You may define how you're using "should" somewhere else in the spec, if so sorry I missed it!

[jordanbtucker]

Good question. The use of the word "should" has the same meaning as in RFC 2119.

SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

So, no you should not rely on this behavior, especially since the JSON spec does not recommend that generators escape those characters and JSON5 parsers must be backward compatible with JSON.

The old Bikeshed version of the spec referenced this RFC, but the Ecmarkup version doesn't. I'll be sure to add that reference back in.

[seagreen]

@jordanbtucker: Excellent😄 You've fully addressed all my concerns.

[jordanbtucker]

I just updated the spec with a clause about the keywords and a few other fixes and clarifications.

[jordanbtucker]

Apologies for the lack of updates lately. I've been busy with work and personal things. The reason I haven't released v1.0 yet is because I discovered a bug in the reference implementation, and I think it's important to release a working reference implementation along with the spec.

I've already fixed the bug, so v1.0 of the spec and reference implementation should be ready to be released after the READMEs, and CHANGELOGs are updated.

After v1.0 is released, I will continue work on refactoring parse to use Parser and Lexer classes, which will facilitate the development of modules that must inspect JSON5 code, e.g. an ESLint plugin.

[MartinMuzatko]

Hello. Having no idea about the entire standardization process of the w3c and tc39 etc.. Is this issue the ticket for us, to make JSON5 an official supported standard?

[jordanbtucker]

@MartinMuzatko At this point in time, there is no intention to make JSON5 a W3C or TC39 spec, and there isn't really a need to. The Source Maps spec is just a Google Doc, but it's in use by almost every modern transpilation software today, and I plan to implement it into JSON5 at some point too.

As long as everyone agrees with the official JSON5 spec, it will be a standard regardless of whether it's endorsed by W3C or TC39.

[hax]

@jordanbtucker Endorsement by TC39/W3C may be helpful to avoid many branches such as JSON6, JSON7, etc 😁

[jordanbtucker]

v1.0.0 has landed.

[aseemk]

Fantastic work as always @jordanbtucker!! 😃