Skip to content

Releases: amazon-ion/ion-rust

v1.0.0-rc.4

31 May 20:06
@zslayton zslayton
v1.0.0-rc.4
80088b4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.0-rc.4 Latest
Latest

This release closes out the list of known blocking issues for version 1.0. It includes substantial changes to the experimental streaming reader and writer APIs, but only relatively small changes to the Element API being stabilized in ion-rs v1.0.

Breaking changes to the Element API

The minimum Rust version has been bumped to 1.67

Details

This allowed us to benefit from the stabilization of the ilog10 operation, which greatly simplified much of the code for our Decimal and Int types

Ints and Decimal coefficients are now limited to the i128 range

Details

The Ion data model does not impose any limitation on the range of integers that it can represent. Previously, the Int and Coefficient types would fall back to heap-allocated space to enable the representation of arbitrarily-sized integers. However, in practice there has been no call to support integers that require 17 or more bytes to represent. This simplification allowed us to remove our dependency on BigInt and to remove many branches from the codebase. We saw reading benchmark improvements in the 3-6% range across the board.

Element encoding methods have been replaced

Details

The following Element methods have been removed:

  • write_as (encode data as Ion and write it to an io::Write impl)
  • to_binary (encode data as binary Ion and return a newly allocated Vec<u8>)
  • to_text (encode data as text Ion and return a newly allocated String)

These methods did not offer a means to configure the way the data was encoded beyond a coarse-grained choice of format. In particular, there was no room in their signatures to allow users to specify a version of Ion to use, which will become necessary when Ion 1.1 is released.

To address this, we have added types that represent the available Ion encodings:

  • ion_rs::v1_0::Binary
  • ion_rs::v1_0::Text

v1_0 refers to the Ion specification version, not the crate's version.

These types can be passed to methods to specify an encoding to use. They also serve as entry points to a builder API for the new WriteConfig type which allows users to specify how their data is encoded.

Together with Element's new encode_as and encode_to methods, users will be able to fully specify how their data is encoded with a list of settings that will grow over time. WriteConfig's builder-style API gives us an evolution path.

encode_as

use ion_rs::v1_0::{Binary, Text};

let element: Element = Element::string("hello");

// Encode the element as binary Ion.
let binary_buffer: Vec<u8> = element.encode_as(Binary)?;
assert_eq!(element, Element::read_one(binary_buffer)?);

// Encode the element as text Ion, further specifying that the text should be generously spaced ("pretty").
let text_ion: String = element.encode_as(Text.with_format(TextFormat::Pretty))?;
assert_eq!(element, Element::read_one(text_ion)?);

Notice that using encode_as with a text encoding results in a String while using a binary encoding results in a Vec<u8>.

encode_to

use ion_rs::v1_0::{Binary, Text};

let element: Element = Element::string("hello");

// Encode the element as binary Ion and write it to an `io::Write` implementation
let mut buffer = Vec::new();
element.encode_to(&mut buffer, Binary)?;
assert_eq!(element, Element::read_one(buffer)?);

// Encode the element as pretty text Ion and write it to an `io::Write` implementation
let mut buffer = Vec::new();
let text_ion: String = element.encode_to(Text.with_format(TextFormat::Pretty))?;
assert_eq!(element, Element::read_one(text_ion)?);

Changes that do not affect the Element API

The experimental IonReader and IonWriter traits have been replaced

Details

The IonReader and IonWriter APIs mimicked the stateful streaming IonReader/IonWriter APIs used in ion-java. Each method call would modify the state of the reader or writer, potentially changing the operations that were legal to call next. On the reading side, it was also (nearly) impossible to return to data that had already been visited, which made handling struct fields (which can arrive in any order) painful.

Because development of the IonReader and IonWriter traits predated the availability of GATs, there were also many places in the API where it was necessary to use Box<dyn> to abstract over different encodings and layers of abstraction. Box<dyn> requires heap allocation to function, which very negatively affected throughput.

These traits have been replaced by a Reader type and a Writer type that are generic over the encoding you wish to use.

Reader

Details

A reader instance only offers a few methods, the most central of which is next(). Each call to next() returns a LazyValue representing the next top-level value in the stream.

let ion_data = "1 foo::true 2024T";
let mut reader = Reader::new(ion_data);

while let Some(value) = reader.next()? {
  println!("It's a(n) {:?}", value.ion_type());
}

In the above example, the reader visits each value in the stream but--because value is a LazyValue--does not read the value. A LazyValue can tell you the value's data type, whether it's null, its annotations, and upon request, its data.

The old IonReader trait had read_TYPE methods for each Ion type (read_bool, read_int, etc). These methods would fail if the reader was not positioned on a value of the correct type or if the value was null, requiring applications to inspect its state ahead of time. Once it was confirmed, however, there was not a way to avoid having to check the Result wrapping the read_TYPE method's output.

The StreamItem enum's Null and (non-null) Value variants were distinct to allow users to call read_TYPE with the confidence that the value returned was non-null.

This combination of characteristics required unwieldy code like the following, which recursively reads and counts the values at all levels of depth in the stream:

    fn read_all_values<R: IonReader<Item = StreamItem>>(reader: &mut R) -> IonResult<usize> {
        use IonType::*;
        use StreamItem::{Nothing, Null as NullValue, Value};
        let mut count: usize = 0;
        loop {
            match reader.next()? {
                NullValue(_ion_type) => { // null values get their own code path
                    count += 1;
                    continue;
                }
                Value(ion_type) => {
                    count += 1;
                    // We need to match against the IonType of this value to know what read_* method to call
                    match ion_type {
                        String => {
                            // Each scalar read method has a `?` that handles both invalid data and
                            // the case where the reader is on a valid value of an unexpected type.
                            let _string = reader.read_str()?;
                        }
                        Symbol => {
                            // This demonstration code would read the value and discard it for timing purposes.
                            let _symbol_id = reader.read_symbol()?;
                        }
                        Int => {
                            let _int = reader.read_i64()?;
                        }
                        Float => {
                            let _float = reader.read_f64()?;
                        }
                        Decimal => {
                            let _decimal = reader.read_decimal()?;
                        }
                        Timestamp => {
                            let _timestamp = reader.read_timestamp()?;
                        }
                        Bool => {
                            let _boolean = reader.read_bool()?;
                        }
                        Blob => {
                            let _blob = reader.read_blob()?;
                        }
                        Clob => {
                            let _clob = reader.read_clob()?;
                        }
                        Null => {
                            // Matching against the IonType requires us to handle `Null` even though our StreamItem
                            // variants make that impossible.
                        }
                        // Reading a container requires you to mutate the state of the reader and then continue the loop,
                        // which can be difficult for developers to mentally model.
                        Struct | List | SExp => reader.step_in()?,
                    }
                }
                Nothing if reader.depth() > 0 => {
                    reader.step_out()?;
                }
                _ => break,
            }
        }
        Ok(count)
    }

In contrast, when you call LazyValue::read()?, it returns a ValueRef--an enum of the possible types it can return. Here's updated code that does the same thing as the IonReader code above:

    fn count_value_and_children<D: Decoder>(lazy_value: &LazyValue<D>) -> IonResult<usize> {
        use ValueRef::*;
        // Calling `read()` on the lazy value returns a `ValueRef`
        let child_count = match lazy_value.read()? {
            // For the container types, we can...
Read more

Contributors

zslayton, nirosys, and 3 other contributors
Assets 2

v1.0.0-rc.3

26 Feb 17:29
@desaikd desaikd
v1.0.0-rc.3
5af0fe6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.0-rc.3

What's Changed

  • Fixes bug in reading shared symbol table import by @desaikd in #714
  • Adds implementation of finish for IonWriter by @desaikd in #720

Experimental (feature gated) changes

Lazy writer implementation

  • Fixes ivm-after-nop in the lazy reader by @zslayton in #708
  • Removes kludge import in integration test by @zslayton in #710
  • Implements writing length-prefixed and delimited structs by @zslayton in #709
  • Implements writing FlexSym annotation sequences by @zslayton in #711

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

Contributors

zslayton and desaikd
Assets 2

v1.0.0-rc.2

08 Feb 17:20
@zslayton zslayton
v1.0.0-rc.2
9f9a494
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.0-rc.2

Changes that affect users upgrading from v1.0.0-rc1

This release does not introduce any breaking changes for users of v1.0.0-rc1. However, it does include a variety of bug fixes and performance improvements.

Upcoming breaking changes

In order to support the upcoming Ion v1.1, the read_* and write_* methods in the Element API need to be able to specify additional options. For example: readers need to be able to specify a Catalog implementation to use and writers need to be able to specify which version of Ion to produce. @desaikd has been working on future-proof new WriteConfig and ReadConfig types that will be added to those methods as arguments in an upcoming rc3.

At this time, this is the only remaining API change to address before v1.0.0 can be cut.

  • Adds implementation of writer configuration by @desaikd in #685
  • Adds implementation for reader builder with catalog by @desaikd in #700

Experimental (feature gated) changes

Text impl of the Lazy Reader API

The lazy reader is now available for both text and binary Ion 1.0, offering a more ergonomic API that is both faster and exposes fewer
illegal states that could result in surprising errors. In the near future, the existing readers (which are themselves feature gated) will
be replaced by the lazy reader.

  • Initial raw lazy text reader (top-level nulls, bools, ints) by @zslayton in #609
  • Adds support for floats to the LazyRawTextReader by @zslayton in #612
  • Adds LazyRawTextReader support for comments by @zslayton in #613
  • Adds LazyRawTextReader support for reading strings by @zslayton in #614
  • Adds LazyRawTextReader support for reading symbols by @zslayton in #616
  • Adds LazyRawTextReader support for reading lists by @zslayton in #617
  • Adds LazyRawTextReader support for structs by @zslayton in #619
  • Adds LazyRawTextReader support for reading IVMs by @zslayton in #620
  • Initial impl of a LazyRawAnyReader by @zslayton in #621
  • Adds lazy reader support for reading annotations by @zslayton in #622
  • Adds lazy reader support for timestamps by @zslayton in #623
  • Lazy reader support for s-expressions by @zslayton in #627
  • Adds lazy reader support for decimals by @zslayton in #628
  • Adds lazy reader support for blobs by @zslayton in #629
  • Adds lazy reader support for long strings by @zslayton in #630
  • Adds lazy reader support for reading clobs by @zslayton in #638
  • Adds ion-tests integration for the lazy reader by @zslayton in #639
  • Incorporates pending feedback from lazy reader PRs by @zslayton in #642

"Lazy" writer

Work is underway to offer an improved writer API that has ergonomics and safety improvements that parallel the lazy reader.
The name is a placeholder; there's nothing especially lazy about the writer's implementation.

Ion v1.1 prototype implementation

Text reader (incl. macro evaluation)

Binary writer

  • Skeleton impl of binary 1.1 writer by @zslayton in #688
  • Implements reading/writing FlexInt, FlexUInt by @zslayton in #690
  • Implements reading/writing FixedInt/FixedUInt by @zslayton in #694
  • Implements writing v1.1 nulls, bools, ints, floats by @zslayton in #695
  • Implements writing v1.1 strings, symbols, and SIDs by @zslayton in #696
  • Implements writing v1.1 decimals by @zslayton in #697
  • Implements writing v1.1 timestamps by @zslayton in #699
  • Implements writing delimited and length-prefixed sequence types by @zslayton in #701
  • Implements writing blobs and clobs by @zslayton in #704

serde support

Full Changelog: v1.0.0-rc.1...v1.0.0-rc.2

Contributors

zslayton, desaikd, and popematt
Assets 2

v1.0.0 Release Candidate 1

19 Jul 21:59
@zslayton zslayton
v1.0.0-rc.1
42adf28
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.0 Release Candidate 1

This release is the first release candidate for ion-rust v1.0. It stabilizes the Element API for reading, writing, and manipulating Ion data. Please open issues for any API concerns that cannot be addressed in a backwards-compatible manner. We still intend to offer features like serde support and stabilize streaming readers and writers in future releases.

Breaking changes from v0.18

Experimental features

Portions of the API that are not ready to be stabilized have been moved into opt-in crate features.

Warning
Types requiring the experimental- features are still subject to breaking changes between minor versions.

This includes:

  • The streaming reader (experimental-reader)
  • The streaming writer (experimental-writer)
  • Ion hash (experimental-ion-hash).

Most inner modules are now private

Nearly all inner modules are now private. In almost all cases, types that were previously imported from paths like ion_rs::types::, ion_rs::element::, etc have been re-exported at the top level.

// Before
use crate::element::Element;
use crate::result::IonResult;
use crate::types::Int;

// After
use crate::{Element, Int, IonResult};

Int and UInt are now opaque structs, not enums

In order to minimize the number of third-party types exposed in the public API, the Int and UInt types are now opaque structs. This prevents users from encountering version mismatches between their own dependency on num-bigint and ion_rs's dependency on num-bigint.

// Before
let int = Int::from(5);

match int {
  Int::I64(i64_value) => {...},
  Int::BigInt(big_int_value) => {...},
};

// After
let int = Int::from(5);

let value: i64 = int.try_into()?;
// or
let value: BigInt = int.into()?;

Into/TryInto have been added for Int/UInt to and from most Rust integer types, including i128/u128.

Vec<Element> has been replaced by Sequence

In order to make implementation changes in the future, APIs that previously returned Vec<Element> now return Sequence, an opaque wrapper around Vec<Element>.

IonError is now non_exhaustive

We may need to add new error variants to IonError in the future, so it is now marked non_exhaustive.

IonError's variants are now opaque structs

In order to improve error handling and enable future implementation changes, each of the error variants is now an opaque struct.

match Element::read_one(...) {
  Ok(element) => {...},
  Err(Decoding(e)) => {...},
  Err(Encoding(e)) => {...},
  Err(Io(e)) => {...},
  Err(IllegalOperation(e)) = {...}
  // ...
}

Methods referring to 3rd party types have been removed

The streaming reader and writer (both now marked experimental) previously supported chrono DateTimes and BigDecimals, which were holdovers from before we implemented Timestamp and Decimal.

The TimestampBuilder API is now simpler

See #588; most method names have remained the same, but the types they return have changed, potentially leading to breakage.

Pull Requests

Full Changelog: v0.18.1...v1.0.0-rc.1

Contributors

zslayton, desaikd, and popematt
Assets 2

v0.18.1

09 Jun 21:20
@zslayton zslayton
v0.18.1
7bc5c74
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.18.1

What's Changed

  • Hotfix v0.18.1: Exposes raw bytes accessors in the SystemReader by @zslayton in #566

Full Changelog: v0.18.0...v0.18.1

Contributors

zslayton
Assets 2

v0.18.0

09 Jun 19:26
@zslayton zslayton
v0.18.0
53ae38c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.18.0

What's Changed

New Contributors

Full Changelog: v0.17.0...v0.18.0

Contributors

tommy, jpschorr, and 6 other contributors
Assets 2

v0.17.0

19 Apr 20:41
@zslayton zslayton
v0.17.0
049da35
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.17.0

What's Changed

  • Make Element implement Send by @zslayton in #497
  • Implements IntoIterator for container Elements by @zslayton in #499
  • Custom string type by @zslayton in #501
  • Makes List and SExp thin wrappers around Sequence by @zslayton in #502
  • Makes Value::List and Value::SExp wrap Sequence by @zslayton in #505
  • Introduces Bytes, Blob, and Clob wrapper types by @zslayton in #506
  • Handle incomplete errors when parsing escaped string sequences by @nirosys in #495
  • Add BlockingRawReader as a blocking wrapper around non-blocking text and binary readers by @nirosys in #493
  • Introduces an Annotations type by @zslayton in #508
  • Remove map_ APIs and replace them with read_ by @nirosys in #509
  • Adds conversion from byte literals to Bytes by @popematt in #510
  • Adds Copy/Clone derivation to StreamItem by @almann in #513
  • Version bump to v0.17.0 by @zslayton in #514

Full Changelog: v0.16.0...v0.17.0

Contributors

almann, zslayton, and 2 other contributors
Assets 2

v0.16.0

18 Mar 02:38
@zslayton zslayton
v0.16.0
ecdf439
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.16.0

What's Changed

  • Introduced builder APIs for List, SExp, and Struct
  • Introduced ion_list!, ion_sexp!, and ion_struct! macros for initializing container Elements.
  • Into<Element> implementations for Rust types that map intuitively to Ion types.
  • Modernized the ElementReader trait so all IonReader implementations can read the current value(s) as an Element.
  • Modernized the ElementWriter trait so all IonWriter implementations can write an Element.
  • Adds accessor methods for the various time unit fields in a Timestamp
  • ion-hash is now a feature of ion-rs instead of an independent crate, eliminating downstream dependency version mismatches
  • Removed the ion-c-sys crate
  • Removed the little-used value::borrowed module and the IonElement trait
  • Bugfixes for the text reader

PRs

Full Changelog: v0.15.0...v0.16.0

Contributors

zslayton, nirosys, and 3 other contributors
Assets 2

v0.15.0

29 Dec 23:14
@popematt popematt
v0.15.0
96cd7ce
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.15.0

What's Changed

New Contributors

Full Changelog: v0.14.0...v0.15.0

Contributors

zslayton, nirosys, and 3 other contributors
Assets 2

v0.14.0

13 Oct 20:51
@zslayton zslayton
v0.14.0
1ab131a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.14.0 Pre-release
Pre-release

Note: This release does not include a new version of ion-hash, which depends on ion-rust v0.13.0. A new version of ion-hash will be available with the next release of ion-rust. If you depend on ion-hash, please wait to upgrade.

What's Changed

New Contributors

Full Changelog: v0.13.0...v0.14.0

Contributors

zslayton, nirosys, and 4 other contributors
Assets 2