Skip to content

Commit

Permalink
Merge #277
Browse files Browse the repository at this point in the history 
277: Add a `Bytes` type for more efficient byte sequences r=jonasbb a=jonasbb

The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
#[serde_as(as = "Bytes")]
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26

Co-authored-by: Jonas Bushart <jonas@bushart.org>
  • Loading branch information
@bors @jonasbb
bors[bot] and jonasbb committed Mar 12, 2021
2 parents 5f68bb2 + 1b9c5a6 commit a1b8cda
Show file tree
Hide file tree
Showing 9 changed files with 560 additions and 11 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.md
View file
Expand Up @@ -50,6 +50,21 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
},
```

* The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the `serde_as` system.

```rust
#[serde_as(as = "Bytes")]
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation (see [serde-bytes#14][serde-bytes-complex]).
2. Implementation for arrays of arbitrary size (Rust 1.51+) (see [serde-bytes#26][serde-bytes-arrays]).

[serde-bytes-complex]: https://github.com/serde-rs/bytes/issues/14
[serde-bytes-arrays]: https://github.com/serde-rs/bytes/issues/26

## [1.6.4] - 2021-02-16

### Fixed
Expand Down
1 change: 1 addition & 0 deletions Cargo.toml
View file
Expand Up @@ -56,6 +56,7 @@ ron = "0.6"
serde-xml-rs = "0.4.1"
serde_derive = "1.0.75"
serde_json = {version = "1.0.25", features = ["preserve_order"]}
serde_test = "1.0.124"
version-sync = "0.9.1"

[[test]]
Expand Down
53 changes: 53 additions & 0 deletions src/de/const_arrays.rs
View file
Expand Up @@ -2,6 +2,7 @@ use super::*;
use crate::utils::{MapIter, SeqIter};
use serde::de::*;
use std::collections::{BTreeMap, HashMap};
use std::convert::TryInto;
use std::fmt;
use std::mem::MaybeUninit;

Expand Down Expand Up @@ -146,3 +147,55 @@ macro_rules! tuple_seq_as_map_impl_intern {
}
tuple_seq_as_map_impl_intern!([(K, V); N], BTreeMap<KAs, VAs>);
tuple_seq_as_map_impl_intern!([(K, V); N], HashMap<KAs, VAs>);

impl<'de, const N: usize> DeserializeAs<'de, [u8; N]> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<[u8; N], D::Error>
where
D: Deserializer<'de>,
{
struct ArrayVisitor<const M: usize>;

impl<'de, const M: usize> Visitor<'de> for ArrayVisitor<M> {
type Value = [u8; M];

fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_fmt(format_args!("an byte array of size {}", M))
}

fn visit_seq<A>(self, seq: A) -> Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
array_from_iterator(SeqIter::new(seq), &self)
}

fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
where
E: Error,
{
v.try_into()
.map_err(|_| Error::invalid_length(v.len(), &self))
}

fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: Error,
{
v.as_bytes()
.try_into()
.map_err(|_| Error::invalid_length(v.len(), &self))
}
}

deserializer.deserialize_bytes(ArrayVisitor::<N>)
}
}

impl<'de, const N: usize> DeserializeAs<'de, Box<[u8; N]>> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<Box<[u8; N]>, D::Error>
where
D: Deserializer<'de>,
{
<Bytes as DeserializeAs<'de, [u8; N]>>::deserialize_as(deserializer).map(Box::new)
}
}
164 changes: 164 additions & 0 deletions src/de/impls.rs
View file
Expand Up @@ -4,6 +4,7 @@ use crate::rust::StringWithSeparator;
use crate::utils;
use crate::utils::duration::DurationSigned;
use serde::de::*;
use std::borrow::Cow;
use std::collections::{BTreeMap, BTreeSet, BinaryHeap, HashMap, HashSet, LinkedList, VecDeque};
use std::convert::From;
use std::fmt::{self, Display};
Expand Down Expand Up @@ -702,3 +703,166 @@ where
Ok(Option::<U>::deserialize_as(deserializer)?.unwrap_or_default())
}
}

impl<'de> DeserializeAs<'de, &'de [u8]> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<&'de [u8], D::Error>
where
D: Deserializer<'de>,
{
<&'de [u8]>::deserialize(deserializer)
}
}

// serde_bytes implementation for ByteBuf
// https://github.com/serde-rs/bytes/blob/cbae606b9dc225fc094b031cc84eac9493da2058/src/bytebuf.rs#L196
//
// Implements:
// * visit_seq
// * visit_bytes
// * visit_byte_buf
// * visit_str
// * visit_string
impl<'de> DeserializeAs<'de, Vec<u8>> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<Vec<u8>, D::Error>
where
D: Deserializer<'de>,
{
struct VecVisitor;

impl<'de> Visitor<'de> for VecVisitor {
type Value = Vec<u8>;

fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("a byte array")
}

fn visit_seq<A>(self, seq: A) -> Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
utils::SeqIter::new(seq).collect::<Result<_, _>>()
}

fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
where
E: Error,
{
Ok(v.to_vec())
}

fn visit_byte_buf<E>(self, v: Vec<u8>) -> Result<Self::Value, E>
where
E: Error,
{
Ok(v)
}

fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: Error,
{
Ok(v.as_bytes().to_vec())
}

fn visit_string<E>(self, v: String) -> Result<Self::Value, E>
where
E: Error,
{
Ok(v.into_bytes())
}
}

deserializer.deserialize_byte_buf(VecVisitor)
}
}

impl<'de> DeserializeAs<'de, Box<[u8]>> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<Box<[u8]>, D::Error>
where
D: Deserializer<'de>,
{
<Bytes as DeserializeAs<'de, Vec<u8>>>::deserialize_as(deserializer)
.map(|vec| vec.into_boxed_slice())
}
}

// serde_bytes implementation for Cow<'a, [u8]>
// https://github.com/serde-rs/bytes/blob/cbae606b9dc225fc094b031cc84eac9493da2058/src/de.rs#L77
//
// Implements:
// * visit_borrowed_bytes
// * visit_borrowed_str
// * visit_bytes
// * visit_str
// * visit_byte_buf
// * visit_string
// * visit_seq
impl<'de> DeserializeAs<'de, Cow<'de, [u8]>> for Bytes {
fn deserialize_as<D>(deserializer: D) -> Result<Cow<'de, [u8]>, D::Error>
where
D: Deserializer<'de>,
{
struct CowVisitor;

impl<'de> Visitor<'de> for CowVisitor {
type Value = Cow<'de, [u8]>;

fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("a byte array")
}

fn visit_borrowed_bytes<E>(self, v: &'de [u8]) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Borrowed(v))
}

fn visit_borrowed_str<E>(self, v: &'de str) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Borrowed(v.as_bytes()))
}

fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Owned(v.to_vec()))
}

fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Owned(v.as_bytes().to_vec()))
}

fn visit_byte_buf<E>(self, v: Vec<u8>) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Owned(v))
}

fn visit_string<E>(self, v: String) -> Result<Self::Value, E>
where
E: Error,
{
Ok(Cow::Owned(v.into_bytes()))
}

fn visit_seq<V>(self, seq: V) -> Result<Self::Value, V::Error>
where
V: SeqAccess<'de>,
{
Ok(Cow::Owned(
utils::SeqIter::new(seq).collect::<Result<_, _>>()?,
))
}
}

deserializer.deserialize_bytes(CowVisitor)
}
}
39 changes: 28 additions & 11 deletions src/guide/serde_as.md
View file
Expand Up @@ -17,17 +17,18 @@ The basic design of the system was done by [@markazmierczak](https://github.com/
5. [Re-exporting `serde_as`](#re-exporting-serde_as)
2. [De/Serialize Implementations Available](#deserialize-implementations-available)
1. [Big Array support (Rust 1.51+)](#big-array-support-rust-151)
2. [Bytes / `Vec<u8>` to hex string](#bytes--vecu8-to-hex-string)
3. [`Default` from `null`](#default-from-null)
4. [De/Serialize with `FromStr` and `Display`](#deserialize-with-fromstr-and-display)
5. [`Duration` as seconds](#duration-as-seconds)
6. [Ignore deserialization errors](#ignore-deserialization-errors)
7. [`Maps` to `Vec` of tuples](#maps-to-vec-of-tuples)
8. [`NaiveDateTime` like UTC timestamp](#naivedatetime-like-utc-timestamp)
9. [`None` as empty `String`](#none-as-empty-string)
10. [Timestamps as seconds since UNIX epoch](#timestamps-as-seconds-since-unix-epoch)
11. [Value into JSON String](#value-into-json-string)
12. [`Vec` of tuples to `Maps`](#vec-of-tuples-to-maps)
2. [`Bytes` with more efficiency](#bytes-with-more-efficiency)
3. [Bytes / `Vec<u8>` to hex string](#bytes--vecu8-to-hex-string)
4. [`Default` from `null`](#default-from-null)
5. [De/Serialize with `FromStr` and `Display`](#deserialize-with-fromstr-and-display)
6. [`Duration` as seconds](#duration-as-seconds)
7. [Ignore deserialization errors](#ignore-deserialization-errors)
8. [`Maps` to `Vec` of tuples](#maps-to-vec-of-tuples)
9. [`NaiveDateTime` like UTC timestamp](#naivedatetime-like-utc-timestamp)
10. [`None` as empty `String`](#none-as-empty-string)
11. [Timestamps as seconds since UNIX epoch](#timestamps-as-seconds-since-unix-epoch)
12. [Value into JSON String](#value-into-json-string)
13. [`Vec` of tuples to `Maps`](#vec-of-tuples-to-maps)

## Switching from serde's with to `serde_as`

Expand Down Expand Up @@ -300,6 +301,21 @@ value: [[u8; 64]; 33],
"value": [[0,0,0,0,0,...], [0,0,0,...], ...],
```

### `Bytes` with more efficiency

[`Bytes`]

More efficient serialization for byte slices and similar.

```ignore
// Rust
#[serde_as(as = "Bytes")]
value: Vec<u8>,
// JSON
"value": [0, 1, 2, 3, ...],
```

### Bytes / `Vec<u8>` to hex string

[`Hex`]
Expand Down Expand Up @@ -516,6 +532,7 @@ This includes `BinaryHeap<(K, V)>`, `BTreeSet<(K, V)>`, `HashSet<(K, V)>`, `Link

The [inverse operation](#maps-to-vec-of-tuples) is also available.

[`Bytes`]: crate::Bytes
[`chrono::DateTime<Local>`]: chrono_crate::DateTime
[`chrono::DateTime<Utc>`]: chrono_crate::DateTime
[`chrono::Duration`]: https://docs.rs/chrono/latest/chrono/struct.Duration.html
Expand Down

0 comments on commit a1b8cda

Please sign in to comment.