Implement --stream

[MiSawa]

To really implement this without expand JSON in memory, we need one of followings

• generator feature so that we can yield to pass value into the VM while keeping the intermediate state of the parser,
• spawn a thread that reads input, and send the result into std::sync::mpsc::sync_channel(0) (or 1 so the reader thread can read one element ahead) to do a similar with above, or
• implement a JSON parser on our own (and perhaps yaml as well).

[MiSawa]

Actually generator wouldn't work since for example the A of visit_seq<A>(self, mut seq: A) itself may have a lifetime that is shorter than 'de or whatever so can't return a structure including this seq thing.

Also it's hard to do send/recv, since serde_json doesn't support providing DeserializeSeed for iteration mode (Deserializer::from_reader(stdin).into_iter()).
serde_yaml on the other hand is probably fine.

[MiSawa]

Though serde_yaml doesn't seem to parse until we supply everything till EOF??? So presumably they keep everything in memory??? 😞

[MiSawa]

serde-rs/json#304

[MiSawa]

Here's a code to see the feasibility. https://gist.github.com/MiSawa/4e4086bd649f4639f390b93ca214f7f0

[MiSawa]

Here's an implementation. Though it uses GATs in a non-essential part so require a nightly compiler to build. https://github.com/MiSawa/xq-stream-test