Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ability to stream arbitrary bytes to the underlying writer when in streaming mode #84

Open
elee1766 opened this issue Oct 26, 2023 · 1 comment

Comments

@elee1766
Copy link
Contributor

i have a use case where i would like part of my message to be json streamed encoded from reflection, but i would still like to use jx for the remaining encoding, as most requests are small and do not require any reflection.

currently, when the writer is in streaming mode, all calls to Write will cause an error.

i propose to simply flush the buffer and then pass the call to Write, which would allow now other json-encoders to stream into jx's encoder. i did this in a fork https://github.com/elee1766/jx and it seems to be working?

given that the package does not want to do any reflection based encoding, this seems like a good compromise?

@elee1766
Copy link
Contributor Author

elee1766 commented Oct 28, 2023

example detailing what i cannot figure out how to do

aka, streaming the "result" field of a jsonrpc response.

sometimes these responses can tens of GBS

package main

import (
	"bytes"
	"encoding/json"
	"log"

	"github.com/go-faster/jx"
)

type v struct {
	V string `json:"v"`
}

func main() {

	existingPackage()
	withIoWriter()

}

func existingPackage() {
	val := &v{V: "bar"}
	out := new(bytes.Buffer)
	e := jx.NewStreamingEncoder(out, 4096)

	// CURRENT
	e.Obj(func(e *jx.Encoder) {
		e.Field("jsonrpc", func(e *jx.Encoder) { e.Str("2.0") })
		e.Field("method", func(e *jx.Encoder) { e.Str("method") })
		e.Field("result", func(e *jx.Encoder) {
			// NOTE: in order to encode val, we must first buffer the ENTIRE message
			raw, _ := json.Marshal(val)
			// and pass that in
			e.Raw(raw)
		})
	})
	e.Close()
	ans := out.String()
	log.Println(ans)
	out.Truncate(0)
}
func withIoWriter() {
	val := &v{V: "bar"}
	out := new(bytes.Buffer)
	out.WriteString(`{`)
	out.WriteString(`"jsonrpc":"2.0",`)
	out.WriteString(`"method":"`)
	out.WriteString(`method`)
	out.WriteString(`",`)
	out.WriteString(`"result":`)
	// importantly, this can be streamed directly to network, instead of buffering. also, it adds a newline because it's an encoder, but that is a separate issue :)
	json.NewEncoder(out).Encode(val)
	out.WriteString(`}`)
	ans := out.String()
	log.Println(ans)
	out.Truncate(0)
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant