Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decodeFromBufferedSource throws JsonDecodingException on big JSON #2006

Closed
Artman12 opened this issue Aug 6, 2022 · 5 comments · Fixed by #2007
Closed

decodeFromBufferedSource throws JsonDecodingException on big JSON #2006

Artman12 opened this issue Aug 6, 2022 · 5 comments · Fixed by #2007
Assignees

Comments

@Artman12
Copy link

Artman12 commented Aug 6, 2022

Describe the bug
Trying to use new Okio streams, introduced recently and encountering JsonDecodingException when trying to decode big JSON via decodeFromBufferedSource().

decodeFromString() works fine.

To Reproduce

@Serializable
data class Node(
    val children: List<Node>?
)

fun test() {
        fun getNodes(count: Int, depth: Int) : List<Node> {
            val ret = mutableListOf<Node>()
            if (depth == 0) return ret
            for (i in 0 until count) {
                ret.add(Node(getNodes(1, depth - 1)))
            }
            return ret
        }

        val rootNode = Node(getNodes(1000, 10))

        val json = Serializer.json.encodeToString(rootNode)

        val buffer = okio.Buffer().apply {
            write(json.toByteArray())
        }

//        val decodedNode = Serializer.json.decodeFromString<Node>(json) // works fine
        val decodedNode = Serializer.json.decodeFromBufferedSource<Node>(buffer) // throws JsonDecodingException 
    }

Exception error:

kotlinx.serialization.json.internal.JsonDecodingException: Expected colon ':', but had 'EOF' instead at path: $.children[107].children[0].children[108]
    JSON input: .....hildren":[{"children":[{"children":[{"children":[{"children"

I've created github repository with reproducible example: https://github.com/Artman12/OkioTest

Expected behavior
decodeFromBufferedSource decodes JSON correctly.

Environment

  • Kotlin version: 1.7.10
  • Library version: 1.4.0-RC
  • Kotlin platforms: JVM, Native
@ihostage
Copy link

Hm 🤔 GH didn't close this automatically when #2007 was merged.

@qwwdfsad
Copy link
Member

qwwdfsad commented Aug 18, 2022

It closes automatically only when it is merged to master. The PR was merged to dev branch, master contains only sources for the latest release

@zacharee
Copy link

zacharee commented Dec 7, 2023

I think this came back or #2007 wasn't a full fix. With a JSON file that's about 800KB on JVM, I get the unexpected EOF error at seemingly random indices.

  • Kotlin 1.9.21.
  • Serialization 1.6.2.
  • Okio 3.6.0.

Example trace:

Exception in thread "AWT-EventQueue-0" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected EOF at path: $[389].timeMillis
JSON input: .....6406963009","status":true}}},{"timeMillis":1701922485310,"ce
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:598)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:596)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:405)
	at kotlinx.serialization.json.internal.ReaderJsonLexer.consumeKeyString(ReaderJsonLexer.kt:149)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:383)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenient(AbstractJsonLexer.kt:463)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenientNotNull(AbstractJsonLexer.kt:439)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeStringKey(StreamingJsonDecoder.kt:329)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:227)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:178)
	at dev.zwander.common.data.HistoricalSnapshot$$serializer.deserialize(HistoricalSnapshot.kt:9)
	at dev.zwander.common.data.HistoricalSnapshot$$serializer.deserialize(HistoricalSnapshot.kt:9)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:168)
	at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:538)
	at kotlinx.serialization.internal.CollectionLikeSerializer.readElement(CollectionSerializers.kt:80)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.json.internal.JsonStreamsKt.decodeByReader(JsonStreams.kt:43)
	at kotlinx.serialization.json.okio.OkioStreamsKt.decodeFromBufferedSource(OkioStreams.kt:63)

@sandwwraith
Copy link
Member

@zacharee Do you have a stable reproducer?

@zacharee
Copy link

zacharee commented Dec 7, 2023

I did some more testing and it looks like it isn't the JSON library, but some quirk in Okio's file writing/reading not properly syncing when using FileSystem.source() and FileSystem.sink(). Wrapping those operations in a Mutex fixes my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants