Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CBOR Feature Drop: COSE #2412

Open
wants to merge 43 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c2db1cf
CBOR basic value tagging
JesusMcCloud Jul 5, 2023
1edcec3
CBOR: handle tagged keys
JesusMcCloud Jul 5, 2023
04826be
CBOR: make verifying and writing tags optional
JesusMcCloud Jul 5, 2023
8979a6e
CBOR: fix decoding value tags, add tests
JesusMcCloud Jul 5, 2023
3e235dd
CBOR: refactor Tagged → ValueTags
JesusMcCloud Jul 5, 2023
256e12d
CBOR: Fixes (docs, whitespace, names, superflous params)
JesusMcCloud Jul 10, 2023
eaf5ef1
CBOR: serialize cbor to tree
JesusMcCloud Jul 18, 2023
d68c3a3
CBOR: tree fixes
JesusMcCloud Jul 19, 2023
db1700b
CBOR: frankensteined data tree
JesusMcCloud Jul 19, 2023
45f70e9
CBOR: fix direct encoding of primitives
JesusMcCloud Jul 19, 2023
79bf170
CBOR: write definite lengths
JesusMcCloud Jul 19, 2023
b0b9974
CBOR: experiment with pruning null properties
JesusMcCloud Jul 19, 2023
19f9227
CBOR: Catch errors on finding annotations
nodh Jul 25, 2023
b2cc8bc
CBOR: Implement labels to use as map keys
nodh Jul 13, 2023
c78f102
WIP: Ignore failing tests
nodh Jul 25, 2023
2e84646
CBOR: Add option to write serial labels over names
nodh Jul 13, 2023
b4e3942
CBOR: Add annotation @CborArray to encode classes as arrays
nodh Jul 13, 2023
a9d00c2
CBOR: Add convenience class for wrapping byte strings during serializ…
nodh Jul 13, 2023
2ef81c4
CBOR: Test combination of serial label with tags
nodh Jul 17, 2023
51bb184
CBOR: Add option to always use compact byte string encoding
nodh Jul 19, 2023
2c558f0
CBOR: Fix potential issue with custom serializers
nodh Jul 25, 2023
1c6edde
CBOR: Encode null complex object as empty map
nodh Jul 26, 2023
337555e
CBOR: remove recursion and null pruning
JesusMcCloud Aug 17, 2023
0d351c1
CBOR: remove bogus `explicitNulls`
JesusMcCloud Aug 17, 2023
dbebe10
CBOR: streamline an simplify encoding
JesusMcCloud Aug 17, 2023
ad25978
CBOR: minor cleanups
JesusMcCloud Aug 17, 2023
0e5e519
CBOR: fix polymorphism
JesusMcCloud Aug 18, 2023
36655cb
CBOR: document new features
JesusMcCloud Aug 18, 2023
f160d70
CBOR: run knit+ fix docs
JesusMcCloud Aug 18, 2023
751aa81
CBOR: update API dumps
JesusMcCloud Aug 18, 2023
da25674
CBOR: Provide serializer for ByteStringWrapper
nodh Sep 25, 2023
68218a1
CBOR: Rename "SerialLabel" to "CborLabel"
nodh Sep 25, 2023
2c4ff1e
CBOR: Add documentation to CborLabel and CborArray
nodh Sep 25, 2023
d4e2052
CBOR: Suppress warning about desugared API on Android
nodh Sep 25, 2023
d3039fb
CBOR: remove Encoder class
JesusMcCloud Sep 25, 2023
f8d7dfd
Merge 'upstream/dev' into feature/coseUpstream
JesusMcCloud Jan 10, 2024
91e97c9
CBOR: don't preallocate for preamble
JesusMcCloud Jan 11, 2024
4786990
CBOR: don't preallocate for data
JesusMcCloud Jan 11, 2024
b415365
CBOR: minor adjustments (missing docs, formatting, …)
JesusMcCloud Jan 11, 2024
dbb9d0e
CBOR: simplify encoding and comment data structures
JesusMcCloud Jan 11, 2024
77e89c7
CBOR: minor cleanups
JesusMcCloud Jan 11, 2024
4f8b680
run knit
JesusMcCloud Jan 12, 2024
1137c42
update API dump
JesusMcCloud Jan 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions build.gradle
Expand Up @@ -209,6 +209,9 @@ subprojects {
case "kotlinx-serialization-hocon":
annotationValue = "kotlinx.serialization.hocon.internal.SuppressAnimalSniffer"
break
case "kotlinx-serialization-cbor":
annotationValue = "kotlinx.serialization.cbor.internal.SuppressAnimalSniffer"
break
case "kotlinx-serialization-protobuf":
annotationValue = "kotlinx.serialization.protobuf.internal.SuppressAnimalSniffer"
}
Expand Down
65 changes: 65 additions & 0 deletions docs/formats.md
Expand Up @@ -13,6 +13,8 @@ stable, these are currently experimental features of Kotlin Serialization.
* [CBOR (experimental)](#cbor-experimental)
* [Ignoring unknown keys](#ignoring-unknown-keys)
* [Byte arrays and CBOR data types](#byte-arrays-and-cbor-data-types)
* [Definite vs. Indefinite Length Encoding](#definite-vs-indefinite-length-encoding)
* [Tags and Labels](#tags-and-labels)
* [ProtoBuf (experimental)](#protobuf-experimental)
* [Field numbers](#field-numbers)
* [Integer types](#integer-types)
Expand Down Expand Up @@ -161,6 +163,8 @@ Per the [RFC 7049 Major Types] section, CBOR supports the following data types:

By default, Kotlin `ByteArray` instances are encoded as **major type 4**.
When **major type 2** is desired, then the [`@ByteString`][ByteString] annotation can be used.
Moreover, the `alwaysUseByteString` configuration switch allows for globally preferring ** major type 2** without needing
to annotate every `ByteArray` in a class hierarchy.

<!--- INCLUDE
import kotlinx.serialization.*
Expand Down Expand Up @@ -218,6 +222,67 @@ BF # map(*)
FF # primitive(*)
```

### Definite vs. Indefinite Length Encoding
CBOR supports two encodings for maps and arrays: definite and indefinite length encoding. kotlinx.serialization defaults
to the latter, which means that a map's or array's number of elements is not encoded, but instead a terminating byte is
appended after the last element.
Definite length encoding, on the other hand, omits this terminating byte, but instead prepends number of elements
to the contents of a map or array. The `writeDefiniteLengths` configuration switch allows for toggling between the two
modes of encoding.


### Tags and Labels

CBOR allows for optionally defining *tags* for properties and their values. These tags are encoded into the resulting
byte string to transport additional information
(see [RFC 8949 Tagging of Items](https://datatracker.ietf.org/doc/html/rfc8949#name-tagging-of-items) for more info).
The [`@KeyTags`](Tags.kt) and [`@ValueTags`](Tags.kt) annotations can be used to define such tags while.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe ValueTags.Companion should be mentioned here as a source for pre-defined tags

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Writing and verifying such tags can be toggled using the `writeKeyTags`, `writeValueTags`, `verifyKeyTags`, and
`verifyValueTags` configuration switches respectively.

In addition, CBOR supports *labels*, which work just as `SerialNames`. The key difference is that labels are not strings,
but integer numbers. Labels can be assigned using the [`@CborLabel`](CborLabel.kt) annotation, while the
`preferCborLabelsOverNames` configuration switch can be used to prefer them over SerialNames in case both are present
for a property.


### Arrays

Classes may be serialized as a CBOR Array (major type 4) instead of a CBOR Map (major type 5).

Example usage:

```
@Serializable
data class DataClass(
val alg: Int,
val kid: String?
)

Cbor.encodeToByteArray(DataClass(alg = -7, kid = null))
```

will produce bytes `0xa263616c6726636b6964f6`, or in diagnostic notation:

```
A2 # map(2)
63 # text(3)
616C67 # "alg"
26 # negative(6)
63 # text(3)
6B6964 # "kid"
F6 # primitive(22)
```

When annotated with `@CborArray`, serialization of the same object will produce bytes `0x8226F6`, or in diagnostic notation:

```
82 # array(2)
26 # negative(6)
F6 # primitive(22)
```
This may be used to encode COSE structures, see [RFC 9052 2. Basic COSE Structure](https://www.rfc-editor.org/rfc/rfc9052#section-2).

## ProtoBuf (experimental)

[Protocol Buffers](https://developers.google.com/protocol-buffers) is a language-neutral binary format that normally
Expand Down
2 changes: 2 additions & 0 deletions docs/serialization-guide.md
Expand Up @@ -144,6 +144,8 @@ Once the project is set up, we can start serializing some classes.
* <a name='cbor-experimental'></a>[CBOR (experimental)](formats.md#cbor-experimental)
* <a name='ignoring-unknown-keys'></a>[Ignoring unknown keys](formats.md#ignoring-unknown-keys)
* <a name='byte-arrays-and-cbor-data-types'></a>[Byte arrays and CBOR data types](formats.md#byte-arrays-and-cbor-data-types)
* <a name='definite-vs-indefinite-length-encoding'></a>[Definite vs. Indefinite Length Encoding](formats.md#definite-vs-indefinite-length-encoding)
* <a name='tags-and-labels'></a>[Tags and Labels](formats.md#tags-and-labels)
* <a name='protobuf-experimental'></a>[ProtoBuf (experimental)](formats.md#protobuf-experimental)
* <a name='field-numbers'></a>[Field numbers](formats.md#field-numbers)
* <a name='integer-types'></a>[Integer types](formats.md#integer-types)
Expand Down
102 changes: 101 additions & 1 deletion formats/cbor/api/kotlinx-serialization-cbor.api
Expand Up @@ -5,9 +5,23 @@ public synthetic class kotlinx/serialization/cbor/ByteString$Impl : kotlinx/seri
public fun <init> ()V
}

public final class kotlinx/serialization/cbor/ByteStringWrapper {
public fun <init> (Ljava/lang/Object;[B)V
public synthetic fun <init> (Ljava/lang/Object;[BILkotlin/jvm/internal/DefaultConstructorMarker;)V
public final fun component1 ()Ljava/lang/Object;
public final fun component2 ()[B
public final fun copy (Ljava/lang/Object;[B)Lkotlinx/serialization/cbor/ByteStringWrapper;
public static synthetic fun copy$default (Lkotlinx/serialization/cbor/ByteStringWrapper;Ljava/lang/Object;[BILjava/lang/Object;)Lkotlinx/serialization/cbor/ByteStringWrapper;
public fun equals (Ljava/lang/Object;)Z
public final fun getSerialized ()[B
public final fun getValue ()Ljava/lang/Object;
public fun hashCode ()I
public fun toString ()Ljava/lang/String;
}

public abstract class kotlinx/serialization/cbor/Cbor : kotlinx/serialization/BinaryFormat {
public static final field Default Lkotlinx/serialization/cbor/Cbor$Default;
public synthetic fun <init> (ZZLkotlinx/serialization/modules/SerializersModule;Lkotlin/jvm/internal/DefaultConstructorMarker;)V
public synthetic fun <init> (ZZZZZZZZZLkotlinx/serialization/modules/SerializersModule;Lkotlin/jvm/internal/DefaultConstructorMarker;)V
public fun decodeFromByteArray (Lkotlinx/serialization/DeserializationStrategy;[B)Ljava/lang/Object;
public fun encodeToByteArray (Lkotlinx/serialization/SerializationStrategy;Ljava/lang/Object;)[B
public fun getSerializersModule ()Lkotlinx/serialization/modules/SerializersModule;
Expand All @@ -16,17 +30,103 @@ public abstract class kotlinx/serialization/cbor/Cbor : kotlinx/serialization/Bi
public final class kotlinx/serialization/cbor/Cbor$Default : kotlinx/serialization/cbor/Cbor {
}

public abstract interface annotation class kotlinx/serialization/cbor/CborArray : java/lang/annotation/Annotation {
public abstract fun tag ()[J
}

public synthetic class kotlinx/serialization/cbor/CborArray$Impl : kotlinx/serialization/cbor/CborArray {
public synthetic fun <init> ([JLkotlin/jvm/internal/DefaultConstructorMarker;)V
public final synthetic fun tag ()[J
}

public final class kotlinx/serialization/cbor/CborBuilder {
public final fun getAlwaysUseByteString ()Z
public final fun getEncodeDefaults ()Z
public final fun getIgnoreUnknownKeys ()Z
public final fun getPreferCborLabelsOverNames ()Z
public final fun getSerializersModule ()Lkotlinx/serialization/modules/SerializersModule;
public final fun getVerifyKeyTags ()Z
public final fun getVerifyValueTags ()Z
public final fun getWriteDefiniteLengths ()Z
public final fun getWriteKeyTags ()Z
public final fun getWriteValueTags ()Z
public final fun setAlwaysUseByteString (Z)V
public final fun setEncodeDefaults (Z)V
public final fun setIgnoreUnknownKeys (Z)V
public final fun setPreferCborLabelsOverNames (Z)V
public final fun setSerializersModule (Lkotlinx/serialization/modules/SerializersModule;)V
public final fun setVerifyKeyTags (Z)V
public final fun setVerifyValueTags (Z)V
public final fun setWriteDefiniteLengths (Z)V
public final fun setWriteKeyTags (Z)V
public final fun setWriteValueTags (Z)V
}

public final class kotlinx/serialization/cbor/CborKt {
public static final fun Cbor (Lkotlinx/serialization/cbor/Cbor;Lkotlin/jvm/functions/Function1;)Lkotlinx/serialization/cbor/Cbor;
public static synthetic fun Cbor$default (Lkotlinx/serialization/cbor/Cbor;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lkotlinx/serialization/cbor/Cbor;
}

public abstract interface annotation class kotlinx/serialization/cbor/KeyTags : java/lang/annotation/Annotation {
public abstract fun tags ()[J
}

public synthetic class kotlinx/serialization/cbor/KeyTags$Impl : kotlinx/serialization/cbor/KeyTags {
public synthetic fun <init> ([JLkotlin/jvm/internal/DefaultConstructorMarker;)V
public final synthetic fun tags ()[J
}

public abstract interface annotation class kotlinx/serialization/cbor/CborLabel : java/lang/annotation/Annotation {
public abstract fun label ()J
}

public synthetic class kotlinx/serialization/cbor/CborLabel$Impl : kotlinx/serialization/cbor/CborLabel {
public fun <init> (J)V
public final synthetic fun label ()J
}

public abstract interface annotation class kotlinx/serialization/cbor/ValueTags : java/lang/annotation/Annotation {
public static final field BASE16 J
public static final field BASE64 J
public static final field BASE64_URL J
public static final field BIGFLOAT J
public static final field BIGNUM_NEGAIVE J
public static final field BIGNUM_POSITIVE J
public static final field CBOR_ENCODED_DATA J
public static final field CBOR_SELF_DESCRIBE J
public static final field Companion Lkotlinx/serialization/cbor/ValueTags$Companion;
public static final field DATE_TIME_EPOCH J
public static final field DATE_TIME_STANDARD J
public static final field DECIMAL_FRACTION J
public static final field MIME_MESSAGE J
public static final field REGEX J
public static final field STRING_BASE64 J
public static final field STRING_BASE64_URL J
public static final field URI J
public abstract fun tags ()[J
}

public final class kotlinx/serialization/cbor/ValueTags$Companion {
public static final field BASE16 J
public static final field BASE64 J
public static final field BASE64_URL J
public static final field BIGFLOAT J
public static final field BIGNUM_NEGAIVE J
public static final field BIGNUM_POSITIVE J
public static final field CBOR_ENCODED_DATA J
public static final field CBOR_SELF_DESCRIBE J
public static final field DATE_TIME_EPOCH J
public static final field DATE_TIME_STANDARD J
public static final field DECIMAL_FRACTION J
public static final field MIME_MESSAGE J
public static final field REGEX J
public static final field STRING_BASE64 J
public static final field STRING_BASE64_URL J
public static final field URI J
}

public synthetic class kotlinx/serialization/cbor/ValueTags$Impl : kotlinx/serialization/cbor/ValueTags {
public synthetic fun <init> ([JLkotlin/jvm/internal/DefaultConstructorMarker;)V
public final synthetic fun tags ()[J
}

@@ -0,0 +1,86 @@
package kotlinx.serialization.cbor

import kotlinx.serialization.*
import kotlinx.serialization.builtins.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*

/**
* Use this class if you'll need to serialize a complex type as a byte string before encoding it,
* i.e. as it is the case with the protected header in COSE structures.
*
* An example for a COSE header data class would be:
*
* ```
* @Serializable
* data class CoseHeader(
* @CborLabel(1)
* @SerialName("alg")
* val alg: Int? = null
* )
*
* @Serializable
* data class CoseSigned(
* @ByteString
* @CborLabel(1)
* @SerialName("protectedHeader")
* val protectedHeader: ByteStringWrapper<CoseHeader>,
sandwwraith marked this conversation as resolved.
Show resolved Hide resolved
* )
* ```
*
* Serializing this `CoseHeader` object would result in `a10143a10126`, in diagnostic notation:
*
* ```
* A1 # map(1)
* 01 # unsigned(1)
* 43 # bytes(3)
* A10126 # "\xA1\u0001&"
* ```
*
* so the `protectedHeader` got serialized first and then encoded as a `@ByteString`.
*
* Note that `equals()` and `hashCode()` only use `value`, not `serialized`.
*/
@Serializable(with = ByteStringWrapperSerializer::class)
public class ByteStringWrapper<T>(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: it seems you need to update apiDump since this class is no longer data

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, done!

public val value: T,
public val serialized: ByteArray = byteArrayOf()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've skimmed through the diff and can't find any usages of this value. Is there any reason to be a val? In what cases will the user be interested in accessing it? Also, it is very easy to create inconsistent data with this approach (e.g. ByteStringWrapper(original.value, byteArrayOf(garbage))). If it is not necessary to have it, this class can be a value class. Or even no class would be needed, as this can be handled with yet another @SerialInfo annotation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nodh can you take care of this, please?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll use the CBOR serialization for ISO 18013-5 compliaent mobile driving licences. There verifiers need to check that (the digest of) the serialized byte values appear in another structure (signed by the issuer). So in that case its quite useful to have the bytes as they were serialized (and parsed) in the deserialized ByteStringWrapper object too.

) {
override fun equals(other: Any?): Boolean {
sandwwraith marked this conversation as resolved.
Show resolved Hide resolved
if (this === other) return true
if (other == null || this::class != other::class) return false

other as ByteStringWrapper<*>

return value == other.value
}

override fun hashCode(): Int {
return value?.hashCode() ?: 0
}

override fun toString(): String {
return "ByteStringWrapper(value=$value, serialized=${serialized.contentToString()})"
}

}


@OptIn(ExperimentalSerializationApi::class)
public class ByteStringWrapperSerializer<T>(private val dataSerializer: KSerializer<T>) :
KSerializer<ByteStringWrapper<T>> {

override val descriptor: SerialDescriptor = dataSerializer.descriptor

override fun serialize(encoder: Encoder, value: ByteStringWrapper<T>) {
val bytes = Cbor.encodeToByteArray(dataSerializer, value.value)
encoder.encodeSerializableValue(ByteArraySerializer(), bytes)
}

override fun deserialize(decoder: Decoder): ByteStringWrapper<T> {
val bytes = decoder.decodeSerializableValue(ByteArraySerializer())
val value = Cbor.decodeFromByteArray(dataSerializer, bytes)
return ByteStringWrapper(value, bytes)
}

}