OutOfMemoryError happens when try to decode the wrong ByteArray

[sang-eun]

Describe the bug
Sometimes our server get wrong ByteArray. We hoped the server throw wrong format error immediately when we try to decode the message. However, protobuf decoder blocks until java heap space out of memory error happens.

From our debugging, message is keep pushBacked by reader.pushBackTag() and index never moves forward from this method.

private fun decodeTaggedListIndex(): Int {
        val protoId = if (index == -1) {
            // For the very first element tag is already read by the parent
            reader.currentId
        } else {
            reader.readTag()
        }

        return if (protoId == tagOrSize.protoId) {
            ++index
        } else {
            // If we read tag of a different message, push it back to the reader and bail out
            reader.pushBackTag()
            CompositeDecoder.DECODE_DONE
        }
    }

To Reproduce

// class 
@Serializable
@SerialName("FrontRequest")
data class FrontRequest (
    val requests: List<String>,
) {
    companion object {
        @OptIn(ExperimentalSerializationApi::class)
        fun fromProtobuf(bytes: ByteArray): FrontRequest
            = ProtoBuf.decodeFromByteArray(bytes)
    }
}


//test 
val payloadBase64 = "+kCbAII+8m6eL8J/nzj6GfwQwzzAa6gzw2uhO8FJ/AGbOv9vr2y/P8l1o1SWPfcylGOnaMFroz7BOak6xGykapQ6qG6Sa/M5lD+jb5RupzvFbvBvxGmnbZVt9WiSaqY6yWL1bpQ/8z3EbvdpwG6pacRj9GrEbfI8xmr1OpM+o2nHPvBqwT7wb8Fup2g="
val payload = Base64.getDecoder().decode(payloadBase64)
//keep working until java heap space error
val frontRequestReceived = FrontRequest.fromProtobuf(payload)

// result

Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3609)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.expand(ProtobufTaggedBase.kt:40)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.pushTag(ProtobufTaggedBase.kt:34)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.access$pushTag(ProtobufTaggedBase.kt:17)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:110)
	at com.dunamu.quot.realtime.front.request.FrontRequest$$serializer.deserialize(FrontRequest.kt:13)

Expected behavior
throw wrong format error immediately.

Environment

• Kotlin version: 1.8.21
• Library version: 1.5.1
• Kotlin platforms: JVM
• Gradle version: 7.5.1

[sandwwraith]

Similar problem in CBOR: https://youtrack.jetbrains.com/issue/KT-66282/Serialization-flaky-OutOfMemoryError-with-Cbor

[sandwwraith]

Thanks, reproduced on 1.9.21 + 1.6.3, too