Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve performance of decoding #62

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Improve performance of decoding #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
870590e
Use lookup table for decoding
taiki-e Jul 18, 2021
3b75b32
Inline val function
taiki-e Jul 18, 2021
ff1d115
Use chunks_exact instead of chunks
taiki-e Jul 18, 2021
4fd0ed8
Use decode_to_slice in Vec::from_hex
taiki-e Jul 18, 2021
3e2aa25
Inline decode_to_slice function
taiki-e Jul 18, 2021
f3251bd
Fix build errors
taiki-e Jul 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
65 changes: 48 additions & 17 deletions src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@
#[cfg(feature = "alloc")]
extern crate alloc;
#[cfg(feature = "alloc")]
use alloc::{string::String, vec::Vec};
use alloc::{string::String, vec, vec::Vec};

use core::iter;
use core::{iter, u8};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uhm I do not think that you should import the u8 module, where u8::MAX is deprecated.

The u8 type itself has an associated constant u8::MAX.

Suggested change
use core::{iter, u8};
use core::iter;

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used u8 module because the u8::MAX associated constant requires a relatively new compiler.

See also: #55

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have a point. Personally, I do not really care about MSRV (because I want to use the new features and updating the compiler is just a simple rustup update) so I did not think much about it.

#58 (comment)


mod error;
pub use crate::error::FromHexError;
Expand Down Expand Up @@ -172,16 +172,47 @@ pub trait FromHex: Sized {
fn from_hex<T: AsRef<[u8]>>(hex: T) -> Result<Self, Self::Error>;
}

const fn val(c: u8, idx: usize) -> Result<u8, FromHexError> {
match c {
b'A'..=b'F' => Ok(c - b'A' + 10),
b'a'..=b'f' => Ok(c - b'a' + 10),
b'0'..=b'9' => Ok(c - b'0'),
_ => Err(FromHexError::InvalidHexCharacter {
c: c as char,
const __: u8 = u8::MAX;

// Lookup table for ascii to hex decoding.
#[rustfmt::skip]
static DECODE_TABLE: [u8; 256] = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this static and not const?

Suggested change
static DECODE_TABLE: [u8; 256] = [
const DECODE_TABLE: [u8; 256] = [

If this table were const, one could make the val function const: playground

Copy link
Author

@taiki-e taiki-e Aug 28, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lookup table is relatively large, so using const may bloat the binary size by inlining.

IIRC, the standard library uses static for that use-case. For example: https://github.com/rust-lang/rust/blob/1.54.0/library/core/src/unicode/unicode_data.rs

See also https://stackoverflow.com/questions/52751597/what-is-the-difference-between-a-constant-and-a-static-variable-and-which-should

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, benchmarking (on my machine) shows that changing this to const does not change the performance.

// 1 2 3 4 5 6 7 8 9 a b c d e f
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 0
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 1
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 2
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, __, __, __, __, __, __, // 3
__, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 4
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 5
__, 10, 11, 12, 13, 14, 15, __, __, __, __, __, __, __, __, __, // 6
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 7
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 8
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 9
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // a
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // b
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // c
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // d
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // e
__, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // f
];

#[inline]
fn val(bytes: &[u8], idx: usize) -> Result<u8, FromHexError> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fn val(bytes: &[u8], idx: usize) -> Result<u8, FromHexError> {
const fn val(bytes: &[u8], idx: usize) -> Result<u8, FromHexError> {

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new implementation of val function cannot be const

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the table were const and not static, this function could be constant: playground

let upper = DECODE_TABLE[bytes[0] as usize];
let lower = DECODE_TABLE[bytes[1] as usize];
if upper == u8::MAX {
return Err(FromHexError::InvalidHexCharacter {
c: bytes[0] as char,
index: idx,
}),
});
}
if lower == u8::MAX {
return Err(FromHexError::InvalidHexCharacter {
c: bytes[1] as char,
index: idx + 1,
});
}
Ok((upper << 4) | lower)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this bitshift + or do?

Suggested change
Ok((upper << 4) | lower)
// upper and lower are only 4 bits large, so of the 8 bits only the first 4 are used.
// this merges the two 4 bit numbers into one 8 bit number:
//
// upper: 0 0 0 0 U U U U
// lower: 0 0 0 0 L L L L
// result: U U U U L L L L
Ok((upper << 4) | lower)

(does not hurt to explain?)

}

#[cfg(feature = "alloc")]
Expand All @@ -194,10 +225,9 @@ impl FromHex for Vec<u8> {
return Err(FromHexError::OddLength);
}

hex.chunks(2)
.enumerate()
.map(|(i, pair)| Ok(val(pair[0], 2 * i)? << 4 | val(pair[1], 2 * i + 1)?))
.collect()
let mut out = vec![0; hex.len() / 2];
decode_to_slice(hex, &mut out)?;
Ok(out)
}
}

Expand Down Expand Up @@ -309,6 +339,7 @@ pub fn decode<T: AsRef<[u8]>>(data: T) -> Result<Vec<u8>, FromHexError> {
/// assert_eq!(hex::decode_to_slice("6b697769", &mut bytes as &mut [u8]), Ok(()));
/// assert_eq!(&bytes, b"kiwi");
/// ```
#[inline]
pub fn decode_to_slice<T: AsRef<[u8]>>(data: T, out: &mut [u8]) -> Result<(), FromHexError> {
let data = data.as_ref();

Expand All @@ -319,8 +350,8 @@ pub fn decode_to_slice<T: AsRef<[u8]>>(data: T, out: &mut [u8]) -> Result<(), Fr
return Err(FromHexError::InvalidStringLength);
}

for (i, byte) in out.iter_mut().enumerate() {
*byte = val(data[2 * i], 2 * i)? << 4 | val(data[2 * i + 1], 2 * i + 1)?;
for (i, (data, byte)) in data.chunks_exact(2).zip(out).enumerate() {
*byte = val(data, 2 * i)?;
}

Ok(())
Expand All @@ -340,7 +371,7 @@ fn generate_iter(len: usize) -> impl Iterator<Item = (usize, usize)> {
// the inverse of `val`.
#[inline]
#[must_use]
const fn byte2hex(byte: u8, table: &[u8; 16]) -> (u8, u8) {
fn byte2hex(byte: u8, table: &[u8; 16]) -> (u8, u8) {
taiki-e marked this conversation as resolved.
Show resolved Hide resolved
let high = table[((byte & 0xf0) >> 4) as usize];
let low = table[(byte & 0x0f) as usize];

Expand Down
2 changes: 0 additions & 2 deletions tests/version-number.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
#![allow(non_fmt_panic)]

#[test]
fn test_readme_deps() {
version_sync::assert_markdown_deps_updated!("README.md");
Expand Down