CBOR Feature Drop: COSE #2412

JesusMcCloud · 2023-08-18T11:18:06Z

This PR obsoletes #2371 and #2359 as it contains the features of both PRs and many more.

Specifically, this PR contains all feature required to serialize and parse COSE-compliant CBOR (thanks to @nodh). While some canonicalization steps (such as sorting keys) still needs to be performed manually. It does get the job done quite well. Namely, we have successfully used the features introduced here to create and validate ISO/IEC 18013-5:2021-compliant mobile driving license data.

This PR introduces the following features to the CBOR format:

Serial Labels
Tagging of keys and values
Definite length encoding (this is the largest change, as it effectively makes the cbor encoder two-pass)
Globally preferring major type 2 for byte array encoding
Encodes null complex objects as empty map (to be COSE-compliant)

…ation

sandwwraith · 2023-08-21T12:54:59Z

Hi, thanks for your PR! Just as a quick heads-up so you know I've seen it — I'm going on vacation soon, so I'll be able to properly review it in September.

sandwwraith

Hi, I appreciate the amount of work you've done — especially for the tests. I didn't look into the implementation (yet), focusing on surface API for now.

Regarding failure on CI — it's connected to the fact that Long.toUnsignedString was added only in Android API 26. However, it is possible to use this declaration on Android with desugaring enabled, so there shouldn't be any problems. You have to crate a special annotation @SuppressAnimalSniffer (it already exists for core and json, but not for cbor, see e.g. here:

kotlinx.serialization/formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt

Line 92 in d192d24

@SuppressAnimalSniffer // Long.toUnsignedString(long)

) and use it in configuration here:

kotlinx.serialization/build.gradle

Line 205 in a87b0f1

switch (name) {

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/SerialLabel.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborArray.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Tags.kt

sandwwraith · 2023-09-22T14:07:11Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Tags.kt

+@SerialInfo
+@Target(AnnotationTarget.PROPERTY)
+@ExperimentalSerializationApi
+public annotation class ValueTags(@OptIn(ExperimentalUnsignedTypes::class) vararg val tags: ULong) {


Generally, it is recommended to propagate opt-in in the libraries (see here: https://kotlinlang.org/docs/opt-in-requirements.html#propagating-opt-in and here: https://kotlinlang.org/docs/jvm-api-guidelines-backward-compatibility.html#the-requiresoptin-annotation). The problem here is, while ULong is already stable, ULongArray which is automatically inserted by vararg is not.

I'm also not sure if it is very common for a property to have multiple tags. Have you considered using @Repeatable annotation instead of vararg (it works only with Kotlin 1.9.0 though, because of the #2099) or not providing such functionality at all?
I don't mind propagating ExperimentalUnsignedTypes, however, if this is necessary.

Interesting! To be frank, I did not even think about@Repeatable, because vararg seemed intuitive to use with minimum boilerplate when defining serializable data types using multiple tags, as the declaration directly reflects what's happening in the resulting CBOR byte string.

We discussed this internally and neither of us has a string opinion on either way of providing this feature. So if @Repeatable is preferred, we'll, of course, go with that!

Did you try it out? If it works for you, I think it may be better that way

https://youtrack.jetbrains.com/issue/KT-64246

After starting to refactor KeyTags and ValueTags to @Repeatable, I notices that this would make KeyTags and ValueTags behave differently from the @CborArray annotation, which requires the vararg syntax to make sense. IMO this change would therefore only make tagging inconsistent, resulting in API inconsistencies and more diverse (and hence a little more complex) code internally.

Therefore, I am now against this change. Of course, if you still prefer it we'll comply.

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

sandwwraith

I didn't look at the test thoroughly (to understand if there is a need for more test cases), but I highlighted all of the problematic moments from my perspective. Don't forget to press 'show hidden conversations' on Github :)

sandwwraith · 2023-12-11T16:20:51Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

+ * Note that `equals()` and `hashCode()` only use `value`, not `serialized`.
+ */
+@Serializable(with = ByteStringWrapperSerializer::class)
+public class ByteStringWrapper<T>(


NB: it seems you need to update apiDump since this class is no longer data

sorry, done!

sandwwraith · 2023-12-11T16:30:45Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

+@Serializable(with = ByteStringWrapperSerializer::class)
+public class ByteStringWrapper<T>(
+    public val value: T,
+    public val serialized: ByteArray = byteArrayOf()


I've skimmed through the diff and can't find any usages of this value. Is there any reason to be a val? In what cases will the user be interested in accessing it? Also, it is very easy to create inconsistent data with this approach (e.g. ByteStringWrapper(original.value, byteArrayOf(garbage))). If it is not necessary to have it, this class can be a value class. Or even no class would be needed, as this can be handled with yet another @SerialInfo annotation.

@nodh can you take care of this, please?

We'll use the CBOR serialization for ISO 18013-5 compliaent mobile driving licences. There verifiers need to check that (the digest of) the serialized byte values appear in another structure (signed by the issuer). So in that case its quite useful to have the bytes as they were serialized (and parsed) in the deserialized ByteStringWrapper object too.

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

sandwwraith · 2023-12-11T17:31:44Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

+    internal val writeValueTags: Boolean,
+    internal val verifyKeyTags: Boolean,
+    internal val verifyValueTags: Boolean,
+    internal val writeDefiniteLengths: Boolean,


This one is not documented as @param

sorry, done!

sandwwraith · 2023-12-11T17:48:23Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborArray.kt

+@SerialInfo
+@Target(AnnotationTarget.CLASS)
+@ExperimentalSerializationApi
+public annotation class CborArray(@OptIn(ExperimentalUnsignedTypes::class) vararg val tag: ULong)


vararg val tag is still undocumented though

sorry! done!

sandwwraith · 2023-12-12T17:38:22Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

            readProperties++
-            descriptor.getElementIndexOrThrow(elemName)
+            descriptor.getElementIndexOrThrow(elemName).also { index ->
+                if (cbor.verifyKeyTags) {


This block of duplicated code can be extracted to function (perhaps local)

sandwwraith · 2023-12-12T17:40:32Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

                } else {
+                    if (cbor.verifyKeyTags) {
+                        descriptor.getKeyTags(index)?.let { keyTags ->
+                            if (!(keyTags contentEquals tags)) throw CborDecodingException("CBOR tags $tags do not match declared tags $keyTags")


General observation (for all throw expressions): It is would be much easier to debug errors if they also included locations. Here we don't have a json path analog, but appending descriptor.toString() will already make the job easier significantly.

done where we have access to the descriptor (which we do not always have)

sandwwraith · 2023-12-12T17:52:18Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

            readByte()
        }
+
+        return (if (collectedTags.isEmpty()) null else collectedTags.toULongArray()).also { collected ->


IMO also creates unnecessary nesting here.

it needs to be transformed into an ULongArray for comparison and for returning it. the alternative of adding a temp variable just for two accesses seems a bit cumbersome to me

sandwwraith · 2023-12-12T17:54:25Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

+
+        return (if (collectedTags.isEmpty()) null else collectedTags.toULongArray()).also { collected ->
+            tags?.let {
+                if (!it.contentEquals(collected)) throw CborDecodingException(


Note that contentEquals has both nullable receiver and parameter. It results in behavior like this: intArrayOf(1,2,3).contentEquals(null) -> false. I'm not sure it is what you intended to have here.

We only want to compare if tags are actually set. Otherwise, we don't care.

I'll also add that comment to the code.

sandwwraith · 2023-12-12T17:54:53Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

@@ -633,9 +952,55 @@ private fun Iterable<ByteArray>.flatten(): ByteArray {

 @OptIn(ExperimentalSerializationApi::class)
 private fun SerialDescriptor.isByteString(index: Int): Boolean {
-    return getElementAnnotations(index).find { it is ByteString } != null
+    return kotlin.runCatching { getElementAnnotations(index).find { it is ByteString } != null }.getOrDefault(false)


Why runCatching is needed here?

it's only a theoretical possibility, but an IndexOutOfBoundsException could be thrown. I'll remove it.

well, such a call fails for the WASM target, even though we use runCatching. I have no explanation

develar · 2023-12-17T08:54:22Z

Globally preferring major type 2 for byte array encoding

It would be awesome, as currently it's somewhat surprising that the default encoded size of byte arrays is nearly twice as large.

JesusMcCloud · 2024-01-12T10:59:11Z

WASM still fails, because even runCatching does not catch a runtimeException for an expected IndexOutOfBoundsException when checking for tags. Help would be much appreachiated.

JesusMcCloud · 2024-01-12T14:33:57Z

WASM still fails, because even runCatching does not catch a runtimeException for an expected IndexOutOfBoundsException when checking for tags. Help would be much appreachiated.

Here are the details:

  Current test: testOpenPolymorphism
  Process output:
   wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:1
  
  
  RuntimeError: array element access out of bounds
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.Array.get (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[2207]:0x4247e)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.descriptors.SerialDescriptorImpl.getElementAnnotations (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[3974]:0x595a3)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.getKeyTags (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5369]:0x6b6e2)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.Preamble.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5180]:0x68fa4)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.Token.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5173]:0x68d56)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.CborWriter.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5294]:0x6a0c3)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.Cbor.encodeToByteArray (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5042]:0x6743b)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.CborPolymorphismTest.testOpenPolymorphism (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[6111]:0x79d2a)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.wasm.internal.testContainer$kotlinx.serialization.cbor test fun$CborPolymorphismTest test fun$testOpenPolymorphism test fun.invoke (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[6715]:0x8440e)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.test.TeamcityAdapter$test$lambda.invoke (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5605]:0x6f4a9)

EDIT: seems it's already been reported and there's really not much we can do: https://youtrack.jetbrains.com/issue/KT-59081/WASM-Cant-catch-index-out-of-bounds-and-divide-by-0-exceptions

JesusMcCloud and others added 28 commits August 18, 2023 09:02

CBOR basic value tagging

c2db1cf

CBOR: handle tagged keys

1edcec3

CBOR: make verifying and writing tags optional

04826be

CBOR: fix decoding value tags, add tests

8979a6e

CBOR: refactor Tagged → ValueTags

3e235dd

CBOR: Fixes (docs, whitespace, names, superflous params)

256e12d

CBOR: serialize cbor to tree

eaf5ef1

CBOR: tree fixes

d68c3a3

CBOR: frankensteined data tree

db1700b

CBOR: fix direct encoding of primitives

45f70e9

CBOR: write definite lengths

79bf170

CBOR: experiment with pruning null properties

b0b9974

CBOR: Catch errors on finding annotations

19f9227

CBOR: Implement labels to use as map keys

b2cc8bc

WIP: Ignore failing tests

c78f102

CBOR: Add option to write serial labels over names

2e84646

CBOR: Add annotation @CborArray to encode classes as arrays

b4e3942

CBOR: Add convenience class for wrapping byte strings during serializ…

a9d00c2

…ation

CBOR: Test combination of serial label with tags

2ef81c4

CBOR: Add option to always use compact byte string encoding

51bb184

CBOR: Fix potential issue with custom serializers

2c558f0

CBOR: Encode null complex object as empty map

1c6edde

CBOR: remove recursion and null pruning

337555e

CBOR: remove bogus explicitNulls

0d351c1

CBOR: streamline an simplify encoding

dbebe10

CBOR: minor cleanups

ad25978

CBOR: fix polymorphism

0e5e519

CBOR: document new features

36655cb

JesusMcCloud changed the base branch from master to dev August 18, 2023 11:18

JesusMcCloud mentioned this pull request Aug 18, 2023

CBOR: Use Serial Labels #2371

Closed

JesusMcCloud mentioned this pull request Aug 18, 2023

CBOR: Basic Support for Tags #2359

Closed

JesusMcCloud added 2 commits August 18, 2023 16:34

CBOR: run knit+ fix docs

f160d70

CBOR: update API dumps

751aa81

This was referenced Aug 18, 2023

Support definite-length Arrays and Maps in CBOR #1955

Open

CBOR tagged data can't be deserialized #1560

Open

sandwwraith reviewed Sep 22, 2023

View reviewed changes

nodh and others added 5 commits September 25, 2023 15:48

CBOR: Provide serializer for ByteStringWrapper

da25674

CBOR: Rename "SerialLabel" to "CborLabel"

68218a1

CBOR: Add documentation to CborLabel and CborArray

2c4ff1e

CBOR: Suppress warning about desugared API on Android

d4e2052

CBOR: remove Encoder class

d3039fb

JesusMcCloud requested a review from sandwwraith October 6, 2023 18:21

sandwwraith requested changes Dec 12, 2023

View reviewed changes

sandwwraith mentioned this pull request Dec 15, 2023

Support encoding/decoding ByteArrays in parameterized types as ByteStrings. #2383

Open

JesusMcCloud added 8 commits January 10, 2024 18:33

Merge 'upstream/dev' into feature/coseUpstream

f8d7dfd

CBOR: don't preallocate for preamble

91e97c9

CBOR: don't preallocate for data

4786990

CBOR: minor adjustments (missing docs, formatting, …)

b415365

CBOR: simplify encoding and comment data structures

dbb9d0e

CBOR: minor cleanups

77e89c7

run knit

4f8b680

update API dump

1137c42

JesusMcCloud requested a review from sandwwraith January 12, 2024 10:59

sandwwraith self-assigned this Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBOR Feature Drop: COSE #2412

CBOR Feature Drop: COSE #2412

JesusMcCloud commented Aug 18, 2023

sandwwraith commented Aug 21, 2023

sandwwraith left a comment

sandwwraith Sep 22, 2023

JesusMcCloud Sep 25, 2023

sandwwraith Dec 11, 2023

sandwwraith Dec 11, 2023

JesusMcCloud Jan 10, 2024

sandwwraith left a comment •

edited

sandwwraith Dec 11, 2023

JesusMcCloud Jan 12, 2024

sandwwraith Dec 11, 2023

JesusMcCloud Jan 10, 2024

nodh Jan 12, 2024

sandwwraith Dec 11, 2023

JesusMcCloud Jan 12, 2024

sandwwraith Dec 11, 2023

JesusMcCloud Jan 12, 2024

sandwwraith Dec 12, 2023

JesusMcCloud Jan 12, 2024

sandwwraith Dec 12, 2023

JesusMcCloud Jan 12, 2024

sandwwraith Dec 12, 2023

JesusMcCloud Jan 11, 2024

sandwwraith Dec 12, 2023

JesusMcCloud Jan 11, 2024

sandwwraith Dec 12, 2023

JesusMcCloud Jan 10, 2024

JesusMcCloud Jan 12, 2024

develar commented Dec 17, 2023

JesusMcCloud commented Jan 12, 2024

JesusMcCloud commented Jan 12, 2024 •

edited

CBOR Feature Drop: COSE #2412

Are you sure you want to change the base?

CBOR Feature Drop: COSE #2412

Conversation

JesusMcCloud commented Aug 18, 2023

sandwwraith commented Aug 21, 2023

sandwwraith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandwwraith left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

develar commented Dec 17, 2023

JesusMcCloud commented Jan 12, 2024

JesusMcCloud commented Jan 12, 2024 • edited

sandwwraith left a comment •

edited

JesusMcCloud commented Jan 12, 2024 •

edited