Support custom json and base64 encoders for Token and Parser #301

dcalsky · 2023-04-02T15:01:49Z

This PR adds support for Token and Parser using custom JSON and base64 encoders to encode/decode related data for performance and experimentation purpose.

By implementing the Base64Encoder interface with EncodeToString and DecodeString methods to create a custom base64 encoder, while implementing JSONEncoder with Marshal and Unmarshal methods to create a custom json encoder.

import (
	"github.com/bytedance/sonic"
	asmbase64 "github.com/segmentio/asm/base64"
)

type sonicJsonEncoder struct{}

func (s *sonicJsonEncoder) Marshal(v any) ([]byte, error) {
	return sonic.Marshal(v)
}

func (s *sonicJsonEncoder) Unmarshal(data []byte, v any) error {
	return sonic.Unmarshal(data, v)
}

type asmBase64Encoder struct{}

func (s *asmBase64Encoder) EncodeToString(data []byte) string {
	return asmbase64.StdEncoding.EncodeToString(data)
}

func (s *asmBase64Encoder) DecodeString(data string) ([]byte, error) {
	return asmbase64.RawURLEncoding.DecodeString(data)
}

Then Using WithJSONEncoder and WithBase64Encoder to create a Parser, and using WithTokenJSONEncoder and WithTokenBase64Encoder to create a Token.

// create the token and parser  with json encoder based on sonic and base64 encoder based on asm/base64
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.RegisteredClaims{}, jwt.WithTokenJSONEncoder(&sonicJsonEncoder{}), jwt.WithTokenBase64Encoder(&asmBase64Encoder{}))

parser := jwt.NewParser(jwt.WithJSONEncoder(&sonicJsonEncoder{}), jwt.WithBase64Encoder(&asmBase64Encoder{}))

The disadvantages of this PR are (Any suggestions are welcomed):

~~Introduce the third-party libs into the go module~~
due to almost all of functions and structs are under the same package, the optional methods naming might be confused, like WithJSONEncoder and WithTokenJSONEncoder

Resolves #168

oxisto · 2023-04-02T15:41:32Z

Very nice! I will do a review over the next days, but it’s basically the direction I envisioned. Just some general pointers: We want to avoid the inclusion of any third party libs, even if it’s only in the tests. I would suggest only testing the interface itself in the test without the actual sonic implementation.

dcalsky · 2023-04-02T15:55:33Z

Very nice! I will do a review over the next days, but it’s basically the direction I envisioned. Just some general pointers: We want to avoid the inclusion of any third party libs, even if it’s only in the tests. I would suggest only testing the interface itself in the test without the actual sonic implementation.

I've updated the PR according to your advice.

token_option.go

encoder.go

parser.go

oxisto · 2023-04-02T19:15:38Z

parser_option.go

@@ -125,3 +125,16 @@ func WithStrictDecoding() ParserOption {
 		p.decodeStrict = true
 	}
 }
+
+// WithJSONEncoder supports
+func WithJSONEncoder(enc JSONEncoder) ParserOption {


Missing docstring

parser_option.go

token.go

alam-chime · 2023-04-21T21:58:11Z

thanks @dcalsky
+1 to this PR

Co-Authored-By: Christian Banse <oxisto@aybaze.com>

oxisto · 2023-09-13T20:39:50Z

Thanks for the initial work on this @dcalsky. I had another go at this, basically splitting up encoding and decoding and also converting most of the interfaces into function types. This should be pretty good integrated in the existing code base now. It even supports both parsing JSON with and without JSON number (as long as the underlying JSON library supports it).

I wonder whether I could also make our "special" base64 options (strict, allow padding) work with external libraries as well. For now, they are disabled when an external decoder is used.

If we are happy with this approach, I want to finalise docs and examples.

cc @mfridman @dcalsky

oxisto · 2024-02-01T17:10:19Z

@mfridman Do we still want to pursue this? It did gather some interest from the community. If yes, I would do another cleanup pass and add docs and examples/tests.

mfridman · 2024-02-01T17:33:28Z

I'll review this in the next few days.

mfridman · 2024-02-05T14:45:48Z

It seems useful, albeit the number of requests for this was quite low.

I was mainly waiting to see what comes of golang/go#63397. If the native std lib could be improved then bringing your own encoding package becomes unnecessary?

I'm a bit on the fence on this one, but if we get it into a good enough shape I wouldn't block. My main hesitation is all the plumbing we have to do to support this in the current /v5 API.

dcalsky force-pushed the main branch 2 times, most recently from cd88b49 to 7c726ab Compare April 2, 2023 15:54

oxisto reviewed Apr 2, 2023

View reviewed changes

token_option.go Outdated Show resolved Hide resolved

oxisto reviewed Apr 2, 2023

View reviewed changes

feat: custom json and base64 encoders for Token and Parser

6207357

Co-Authored-By: Christian Banse <oxisto@aybaze.com>

oxisto force-pushed the main branch from 7c726ab to 6207357 Compare September 13, 2023 14:44

oxisto added 3 commits September 13, 2023 16:49

Slightly better way to handle useNumber

836be3f

Extract encoders/decoders to extra struct

d81e689

Also supports UseNumber

4c1e383

oxisto added 3 commits September 14, 2023 21:25

Support almost all base64 options

ea0051d

Added more Godoc

22b0855

supporting strict() for all libraries

441e06d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom json and base64 encoders for Token and Parser #301

Support custom json and base64 encoders for Token and Parser #301

dcalsky commented Apr 2, 2023 •

edited

oxisto commented Apr 2, 2023

dcalsky commented Apr 2, 2023

oxisto Apr 2, 2023

alam-chime commented Apr 21, 2023

oxisto commented Sep 13, 2023

oxisto commented Feb 1, 2024

mfridman commented Feb 1, 2024

mfridman commented Feb 5, 2024

Support custom json and base64 encoders for Token and Parser #301

Are you sure you want to change the base?

Support custom json and base64 encoders for Token and Parser #301

Conversation

dcalsky commented Apr 2, 2023 • edited

oxisto commented Apr 2, 2023

dcalsky commented Apr 2, 2023

oxisto Apr 2, 2023

Choose a reason for hiding this comment

alam-chime commented Apr 21, 2023

oxisto commented Sep 13, 2023

oxisto commented Feb 1, 2024

mfridman commented Feb 1, 2024

mfridman commented Feb 5, 2024

dcalsky commented Apr 2, 2023 •

edited