Location and lifetime of ondemand strings

[SpeedyCraftah]

I'm looking to parse a JSON object with the ondemand API, but I'm not sure when it comes to string_view's.
The documentation mentions that string_view's may either point to the source JSON string, or to an internal temporary buffer, is there a set of rules or specific operations which stores the string in one or the other?

I need the processing to have minimal memory allocations and copying, and so I'm not sure if I should copy these parsed strings to somewhere else, in my use case, the input JSON string will always outlive the next iterate call on another JSON string, so if string_view's were to be pointing to the input JSON string, this wouldn't be a problem, but since I may access different elements in the object and potentially even iterate another JSON string before needing the value of that string_view, I'm worried that if that string does end up in the temporary buffer before I manage to do things with the contents, I would need to copy it somewhere else immediately on reading it, which I want to avoid.

Thanks!

[lemire]

The documentation mentions that string_view's may either point to the source JSON string, or to an internal temporary buffer, is there a set of rules or specific operations which stores the string in one or the other?

If you specifically as for a raw JSON, then it points that the input (raw_json()). When you query a value (get_string()) then the string needs to be escaped before it is returned to you, so it is written on a string buffer within the parser.

In other words, if it is possible for us to just point at the input, then we do so. If you are asking for the string value of a JSON string, then, in general, it needs to be escaped (e.g, t\r\u0011 needs to be processed) and so we need a buffer.

so I'm not sure if I should copy these parsed strings to somewhere else, in my use case, the input JSON string will always outlive the next iterate call on another JSON string

You can store them directly into an std::string instance:

https://github.com/simdjson/simdjson/blob/master/doc/basics.md#storing-directly-into-an-existing-string-instance

[lemire]

is there any way to disable the special character encoding since I won't need it

We deliberately do not document it, but you can build simdjson with the UTF-8 validation disabled: https://github.com/simdjson/simdjson/pull/1957/files

Of course, then you are responsible to ensuring that the input is UTF-8.

[lemire]

You can definitively grab raw (unescaped) strings if that's what you need. We support that and it is obviously going to be faster.

[SpeedyCraftah]

You can definitively grab raw (unescaped) strings if that's what you need. We support that and it is obviously going to be faster.

I did have a look at .raw_json() on strings and it does give me references to unescaped strings, although wrapped in quotes but I can easily trim those out when it comes to string_views so I'm not concerned about that.

We deliberately do not document it, but you can build simdjson with the UTF-8 validation disabled:

I had a look at this and added a #undef SIMDJSON_UTF8VALIDATION after including the simdjson header (since I'm using it as a header dependency), but I don't seem to see a difference when calling .get_string() on a string which contains different escapes and it always unescapes them anyways as opposed to .raw_json() (e.g. "hello world\\n"), am I misunderstanding something?

[lano1106]

undefining SIMDJSON_UTF8VALIDATION will not do it as the header file is defining it if not def.

Instead define it to be 0

[lemire]

@lano1106 You are correct. You have to see SIMDJSON_UTF8VALIDATION to false (or 0) at the very start of the compilation cycle. Undefining it won't work.

[jkeiser]

It sounds like you want to keep the strings around from one document while you parse other documents? The parser.unescape() API lets you manage the buffer yourself when the parser's lifetime isn't working.

simdjson allocates a single buffer that is large enough to include all possible strings that could be dumped into it. This is probably what you'd want to do as well. You need to hold on to the original pointer so you can free it, of course, but you'll also need a pointer to where simdjson should write the next string.

uint8_t* const string_buf = malloc(ENOUGH_SPACE_TO_HOLD_ALL_THE_STRINGS);
uint8_t* string_buf_end = string_buf;

Anytime you are about to get a string_view, either by casting to string_view or using json_value.get_string(), you do parser.unescape(json_value, buf) instead. For example, if you have a JSON object with "name" and "description" fields, this will put both of them in there:

string_view name = parser.unescape(json_obj["name"], string_buf_end);
string_view description = parser.unescape(json_obj["description"], string_buf_end);

This will unescape name and description into string_buf, one after the other, and modifies string_buf_end to point after the description. Even if the parser is destroyed or used to parse a new document, the string_views remain valid as long as string_buf is valid and doesn't get overwritten.

We could make this easier, but I think this will do what you're after if you're really looking for the least number of allocations possible.

[SpeedyCraftah]

I see - thank you for the clarification! I'll take this approach if I cannot grab the string without the escape sequence decoding, which I don't think I need/will make use of as I replied above.

[jkeiser]

Just make sure that when you write the string back out to json, you write it raw: if you use a json writing library, by default it will escape the escape characters, turning \n into \\n.

[SpeedyCraftah]

I did forget that it happens both ways 😂 thank you very much for the reminder.

[jkeiser]

Yeah that would be a doozy, since you won't catch it with ordinary test cases, since you are expecting most input not to have escapes :) Happy to help!

[lemire]

On the plus side, it should be faster when writing without escaping.

[lemire]

@SpeedyCraftah You may enjoy this code written by @anonrig for the Node.js runtime environment:

https://github.com/nodejs/node/blob/431f32e30255daaea1a1b370350903aae39e7b1a/src/node_modules.cc#L195

It basically does what @jkeiser and I were discussing: it grabs raw JSON that does not need to be processed immediately and store it in a structure. It can be parsed later as JSON. As with everything @anonrig does, it is great code.

It is going to be stupidly efficient compared to the obvious alternative of deserializing->serializing->deserializing.

[SpeedyCraftah]

I never knew Node.js used simdjson under the hood, at least for some things, that's interesting.

It is going to be stupidly efficient compared to the obvious alternative of deserializing->serializing->deserializing.

Definitely, that sounds like a great idea to implement into my algorithm.

What I'm planning is for the JSON data to be converted into a more efficient structure/format so that it can be executed very efficiently many times and saved as opposed to playing around with JSON every time I want a value, so simdjson is perfect for this as it only allows a value to be consumed once anyways, with some arbitrary values being ignored instead of being parsed.

But as it seems there's also room for more optimisations by grabbing raw JSON that I don't need right now and only processing it when it is required, without having to keep the ondemand parser occupied with the entire document.

[lemire]

grabbing raw JSON that I don't need right now and only processing it when it is required

Yes. We don't document this approach thoroughly, but it is one the key strengths of simdjson.