[Question] writing json document using simdjson?

[cppchedy]

Hi @lemire,
simdjson v0.3 is a beast thanks for your great work(and also for all contributors).

I have a questions on the ability to write json document(starting from an empty doc/element). As of now (v0.3) I don't see support for this. so my questions is:

• is there any plans on supporting this?
• how much effort(time) does it take? (asking to evaluate a possible PR)

the current work around is to use simdjson + another lib, so it would be nice if I could only rely on simdjson?

thanks

[lemire]

Correct, it is not currently possible to write (serialize) with simdjson.

@jkeiser : can you take this question at your convenience?

[jkeiser]

A write API is an important feature for the reasons you mention, and it's on our minds--but we have some things we want to finish on the read side first (streaming and SAX particularly). So probably not soon. I expect another couple of releases to pass before we really jump into the problem.

We do welcome contributions, though! If you are interested, we can always talk it out, all of us here try to be helpful :) I imagine an initial stab isn't hard if it's designed right, but it depends entirely on your experience and engagement. Probably just a "document builder" API that builds a document in the current tape format. Then people can just use the current cout << document to write it outI think it could be done without SIMD, and with a single implementation.

[eranif]

Shame it is not supported. I am very impressed with the speed of reading, but I am afraid that only having the "Read API" is not sufficient for most C++ programs

[saidwho12]

I would love to have also a write API for this, otherwise for my use case I'll probably just have to use something like RapidJSON

[slidertom]

I'm also have same issue and thinking about something like: edit mode -> RapidJSON and read-only mode simdjson.
(e.g.: yyjson contains edit mode, but it's also a bit complicated, RapidJSON is the most clean in the edit mode)
simdjson looks like already contains some write operations like: simdjson::to_json_string for array.

[lemire]

We are openly calling for interested contributors. We don’t even have design documents at this point: what we want to do and how we are going to do it efficiently.

[SpeedyCraftah]

Would be nice if there was some kind of central discussion thread for this but this will have to do.
For now the best approach would probably be to use something like RapidJSON alongside simdjson (rapid for serializing, simdjson for reading), but of course it would be great if simdjson supported it.

I would love to help but more or less fear a lack of expertise in this type of area, but for a design something like this should likely at least be on par with RapidJSON in terms of performance, although I'm not sure how SIMD could be leveraged to accelerate this process, although that's probably the least of our problems right now.

It would probably also have to be separate from the core principles of the ondemand parser which is taking an iterative approach while an object builder would likely hold and have everything available immediately.

[lemire]

Would be nice if there was some kind of central discussion thread for this but this will have to do.

Good point. I have converted the issue to a discussion. It is not a good issue, as you point out, it is more of a discussion.

[lemire]

The following project https://github.com/getml/reflect-cpp is impressive...

Their API is...

enum class Color { red, green, blue, yellow };

struct Circle {
  float radius;
  Color color;
};

const auto circle = Circle{.radius = 2.0, .color = Color::green};

rfl::json::write(circle);

It could be used for our purposes. The downside is that it requires C++20 and only supports specific compilers. It is a minor issue.

[SpeedyCraftah]

Looks quite good, also has a lot of potential when it comes to improving performance by expecting the structure of the JSON since the object is just a struct and so the internal buffers can be allocated beforehand to expect said structure.

The downside is that it requires C++20 and only supports specific compilers

The most major compilers support this so that could be a reasonable drawback, although I'm not sure about C++20 since in most projects it's not enabled by default, and progress on its standardization is quite stagnant, although I'm sure most users wouldn't mind enabling C++20, but it can be annoying.

[niclar]

If one is prepared to depend on SIMD for json parsing I'd doubt c++20 would be a real issue.

[lemire]

If one is prepared to depend on SIMD for json parsing I'd doubt c++20 would be a real issue.

It should be easy to guard the implementation behind macro checks so that it is just magically unavailable for users who are not using C++20.

[lano1106]

I have looked into reflect-cpp. In terms of ease of use and elegance, it is probably unbeatable...

However, in terms of performance, I am not so sure...

The json serialization backend used by reflect-cpp is https://github.com/ibireme/yyjson

1. I am unsure if this can be avoided but it seems like memory allocation is performed by default.
2. The benchmark result is a bit misleading. yyjson times the time it takes to traverse its fully validated DOM structure and to write it into a string.

RapidJSON does character escaping and UTF-8 validation in its writer class which is included in the yyjson benchmark.

I still think that in terms of performance RapidJSON is superior because:

1. You can easily make sure that no memory allocation will occur
2. You can avoid the need to build a DOM structure. You can directly use the writer which exposes a function for each JSON component that you want to add to the output.

What is expensive is the escaping and the UTF-8 encoding/validation which makes it mandatory to check each output character one by one.

There is some simd usage but it only checks 16 bytes per check due to RapidJSON age... No AVX-512 usage there...

that being said... this discussion made me search for a faster JSON serializer than RapidJSON. I do not think that reflect-cpp or yyjson is the answer and I have come to the conclusion that the fastest json serializer is a handcrafted function that creates the json output itself without doing any char escaping or UTF8 validation with the help of some std::ostream object that performs no memory allocation (possible with boost) and/or with the help of the fmt facility (either standard or from https://github.com/fmtlib/fmt)

[jwl80303]

I'm late to the party here... I recently obtained a project at work that has me needing to create JSON programmatically from scratch - i.e. from an empty 'Document'.

I found 'rapidjson' (before discovering simdjson) and it seems to do the trick well enough. It'll do just fine. But that project seems to be 'on hiatus' and I'm reluctant to use it. So I read more and come across simdjson which seems (benchmarks) to run circles around rapidjson performance-wise.

So... I start my simdjson learning with having it ingest the JSON from a file that I had rapidjson create. All is fine - so far so good. Now I'm trying to take my learning to the next level by having simdjson create some JSON from scratch and..........OOPS, I'm running into a brick wall. And then I find this discussion that explains my brick wall - simdjson doesn't support creating JSON - bummer. Put me down as wishing that it did :-(

[SpeedyCraftah]

To be honest I was in a similar position, but I realised simdjson doesn't support creating JSON not too far after starting development, so I also integrated with rapidjson for creation. Obviously no fault of simdjson at all, I think there's just a good general assumption that JSON libraries support both creation and parsing. Maybe some day!

[lano1106]

I am currently using RapidJSON...

I am currently evaluating replacing the deserialization done by RapidJSON in my application with simdjson... My JSON packets are not that huge... 1KB up to an occasional 10KB but it processes millions of these small JSON packets every day.

yes its RapidJSON dev appears to have stopped but I am not sure that this is a bad thing... Once a project is feature complete and has been proven to be mature and stable, what else remains to be done?

This would definitely be a good addition to add serialization to simdjson... not so much to be much more efficient than all the others projects but simply in order to avoid its users to need drag over another library for the serialization...

[lemire]

I should stress that simdjson is a community driven project. There is no engineering team behind it, other that the community present here. So someone needs to set out to do the work if a feature is wanted.

Possibly, the best API I have seen so far is https://github.com/getml/reflect-cpp

We could provide something of the sort, but it will require engineering effort. To be clear, people need to work on it.

[lano1106]

I am taking good note that contributions are welcomed... I am unfortunately currently overwhelmed by my other professional commitments but if that eventually changes and I become a big simdjson user, I will be happy to roll up my sleeves and help the project...

[lano1106]

FYI, I think that the first time that I have heard about simdjson, it is by listening a Jason Turner video talking about it maybe a year ago...

back then, I did dismiss using it because I was using RapidJSON in SAX mode (for performance reasons) and I was not sure if simdjson was supporting this type of parsing... I didn't go far enough in the documentation back then to read about what On-Demand was...

By reading your papers, I now realize that you are addressing my concern in great length... and I must confess that writing code to use a streaming parse... It is really painful and creates somehow complex code that ends up being hard to maintain...

[lemire]

It is really painful and creates somehow complex code that ends up being hard to maintain...

Yes. Indeed.

[Steveantor]

I was scratching my head over the write/serialization API after started working with it
It's a real shame that the library doesn't support serialization yet,

rapidjson does it (despite it being slightly slower than simdjson but gets the job done)
so before comparing the performance with rapidjson, you should consider providing a full API first.

[lemire]

I was scratching my head over the write/serialization API after started working with it
It's a real shame that the library doesn't support serialization yet,

@FranciscoThiesen is working on this issue. Please refer to #1386 or get in touch with Francisco if you are available to help.