JSON decoding error parsing stream encoded with Jackson Smile

[eftche]

Affects: 5.1.12

With #24009 fixed and versions updated, there's still issues with decoding Smile-encoded stream. Looks like a missyncronization between use of generator and parser:

org.springframework.core.codec.DecodingException: JSON decoding error: Invalid shared name reference 6; 
only got 2 names in buffer (invalid content); nested exception is
com.fasterxml.jackson.core.JsonParseException: Invalid shared name reference 6; only got 2 names in buffer (invalid content)
 at [Source: UNKNOWN; line: -1, column: 6843033]
	at org.springframework.http.codec.json.Jackson2Tokenizer.tokenize(Jackson2Tokenizer.java:99)
	at reactor.core.publisher.FluxMapSignal$FluxMapSignalSubscriber.onNext(FluxMapSignal.java:137)

[eftche]

Test project attached:
test-spring-smile.zip

[poutsma]

I'm not sure it's a synchronisation issue. I disabled the USE_THREAD_LOCAL_FOR_BUFFER_RECYCLING feature for Jackson, which has been causing sync issues in the past, but it made no difference.

Maybe it was the "shared name reference" mentioned in the exception that put you on the track of synchronisation, but that is a feature of Smile.

Moreover, I was able to reproduce this issue without using any Spring code, see https://github.com/poutsma/jackson-smile-issue. It does use Reactor, but I suspect the issue is in Jackson itself.

@feutche As the original reporter, could you please report this Jackson issue here, possibly pointing to my repo?

[poutsma]

Closed as invalid, as the issue can be reproduced without Spring code.

[eftche]

Thanks for looking into it. Sorry I wasn't clear as I didn't mean multithreading/concurrency issue but more of maybe inconsistent use of generator and parser so their shared name buffers aren't in sync.

I still think it's the case - in your example (in your repo) - ObjectMapper.writeValuesAsBytes creates new SmileGenerator for every object (!) however there's a single Smile parser created for the whole flow (which is the way it should be, I guess). So obviously their _seenNames buffers are not in sync.
In this case the problem manifests itself when parser buffer overflows and being reset (with expectation that generator did the same, but it didn't, every new generator will only have 2 values in there, name0 and name1). However I've seen cases when backreference is just invalid and Json parser fails trying to assign value to a wrong field (that one drove me mad :), now at least I finally understand where it comes from!).

[eftche]

So I think this issue should be reopened, the problem is not in Jackson but in the way it's inconsistently used between encoder and decoder.

[poutsma]

Ok, reopening, but I won't have time to look into this until after my Christmas break.

[poutsma]

Thanks for spotting this. We are now using a Jackson SequenceWriter when streaming, which fixes this issue.