BaseX v0.7.0

Aside from the promised features, I've been reading Rust for Rustaceans lately, influencing some of the design decisions made in the internal code structure. I'm at 6 out of 13 chapters, can't wait for what's more in the store!

I highly recommend this book for any aspiring Rustacean 📖🦀

And last but not least, CI configuration has improved a lot. It contains multiple stages now, caches build results and dependencies, builds with multiple toolchains, does security audits, and brings static analysis with rustfmt and clippy 📎

The Book: Existential types

In the second chapter of the book, I've learned this trick with "Existential types." You can make a public trait with private implementation. And make methods that would return Type instead return impl Trait. That impl Trait thing is syntax sugar for generic return types.

That way you can change the implementation without introducing a breaking change.

fn info(&self) -> Result<impl Info> { // <- returns some implementation of the public trait
    // ...
    Ok(RawInfo::from_str(...).unwrap()) // <- returns an instance of private type implementing the trait
}

Now I'm wondering where is the stopping point? Should I go and turn all my public structs into traits and hide their implementation from the public API? Guess only the next release will tell!

Features

Let me introduce basex::serializer::Options and basex::compiler::Info. Both of these concepts belong to Query and are returned by Query::info and Query::options.

New: Query::info

fn info(&self) -> Result<impl Info>;

You can now collect parsed compiler info from the query—for instance, the optimized version of the query and its time to produce it.

let info = query.info()?;

// Read compilation info
println!("Optimized Query: {:?}", info.optimized_query());
println!("Total Time: {:?}", info.total_time());
println!("Hit(s): {:?}", info.hits());

The compilation info needs to be enabled before creating the query server-side. This influenced the design of Client::query, which now returns a builder object, asking you if you want to query with_info, unlocking Query::info for you, or without_info, reducing overhead but making Query::info inaccessible.

New: Query::options

fn options(&self) -> Result<Options>;

You can now read and update server-side query result serializer options. This influences the way the result looks and might for instance change encoding.

// Get options from the server for the query
let options = query.options()?;

// Read options
let encoding = options.get("encoding").unwrap();
let indent = options.get("indent").unwrap();

println!("Encoding: {}", encoding.as_str());
println!("Indent: {}", if indent.as_bool().unwrap() { "ON" } else { "OFF" });

// Set options
let encoding = options.set("encoding", "UTF-8");
let indent = options.set("indent", true);

println!("Set Encoding: {}", encoding.as_str());
println!("Set Indent: {}", if indent.as_bool().unwrap() { "ON" } else { "OFF" });

// Save options to session on server
options.save(client);

My decision was to make it like a key-value store. Now an exhaustive could be made using the BaseX documentation, but there is no guarantee that new options might be created in the next release of the BaseX server, making it incompatible with the client.

CI

The build pipeline now has a bunch of new jobs! 🚦

The code is now formatted by rustfmt and further analyzed by clippy.

There are 3 build jobs now using:

1. Nightly toolchain
2. Nightly toolchain with 6 months old Rust version
3. Nightly toolchain with minimal dependency versions

And the build is cached.

Packages are now checked for security issues via cargo-audit, which runs only on Cargo.toml changes and once per day.

Chores

• Set version range of all dependencies the widest possible.

Fixes

• Upgrade rust-embed minimal version from 0.5.2 to 0.6.3 due to RUSTSEC-2021-0126 critical security issue.

Upgrading from the previous version

Getting the code work is pretty straightforward, but writing it in a way that makes sense will require a bit more effort.

Before

let mut client = Client::connect("localhost", 1984, "admin", "admin")?;

let query = client.query("count(/None/*)")?;
let info = query.info()?;
let options = query.options()?;

After

let mut client = Client::connect("localhost", 1984, "admin", "admin")?;

let query = client.query("count(/None/*)")?.with_info()?;
let info = query.info()?.to_string();
let options = query.options()?.to_string();

Plans for next version

The feature set is pretty much complete as far as the required functionality goes. The library can be only polished further, introduce async or add features on top. Which is exactly what is planned next.

The next version temporarily stabilizes the API and then basex-rs is going async!

BaseX v0.6.0

The new version brings the AsResource, a trait providing into_read(): &mut impl Read, and it's now used everywhere instead of impl Read.

Another improvement is that Query::context now uses only AsResource as its argument, moving the responsibility of getting the reader into the AsResource implementation.

Features

New: AsResource trait now used instead of Read

This new trait is responsible for converting the owned value to another one that implements Read.

Separating this small bit of responsibility to a single place allows us to get rid of some duplicity in code:

Mainly the part of converting &str to &[u8] for Read compatibility is now implemented via AsResource, making the code as follows possible:

use basex::{Client, ClientError};
use std::io::Read;

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let info = client.create("lambada")?
        .with_input("<Root><Text/><Lala/><Papa/></Root>")?;
    assert!(info.starts_with("Database 'lambada' created"));

    let query = client.query("count(/Root/*)")?;

    let mut result = String::new();
    let mut response = query.execute()?;
    response.read_to_string(&mut result)?;
    assert_eq!(result, "3");

    let mut query = response.close()?;
    query.close()?;
    Ok(())
}

Previously we would have to do this:

let info = client.create("lambada")?
    .with_input(&mut "<Root><Text/><Lala/><Papa/></Root>".as_bytes())?;
// ...
let query = client.query(&mut "count(/Root/*)".as_bytes())?;

Another benefit is that you can implement AsResource for your types for direct support!

New: Query::context now accepts AsResource and does not require type

The method now just uses the AsResource value and always sets its type to document-node() as the same thing does the BaseX Java Client it seems to be the way to go.

Upgrading from the previous version

The AsResource change is backward-compatible. However, to make use of the improvement for &str arguments, you may change according to the following guide.

Also, the usage of Query::context now does not require to specify the type, and also uses AsResource.

Before

let info = client.create("lambada")?
    .with_input(&mut "<Root><Text/><Lala/><Papa/></Root>".as_bytes())?;
// ...
let query = client.query(&mut "count(/Root/*)".as_bytes())?;

let query = client.query(&mut "count(/Root/*)".as_bytes())?
    .context(Some("<test/>"), Some("document-node()"))?;

After

let info = client.create("lambada")?
    .with_input("<Root><Text/><Lala/><Papa/></Root>")?;
// ...
let query = client.query("count(/Root/*)")?;

let query = client.query("count(/Root/*)")?.context("<test/>")?;

Plans for next version

The next version will focus on Query::info() and Query::options() and considers the option of parsing the strings into structured data.

BaseX v0.5.0

The new version brings us typed Query::bind arguments!

Features

Canceled: Wildcards support for Client::execute

The previously planned feature, adding wildcards support to Client::execute, is canceled. The reasoning behind this decision is that putting wildcards is not the library's responsibility. There is more, however.

Upon further examination of the Server Protocol and Commands it has been noted that there is no call to Client::execute with a command that could contain XML resources or binary data. Every such command has its method counterpart (e.g. Client::create, Client::add and so on). So there is no need to escape the input or have wildcards for it.

New: Query::bind now accepts value of type T: ToQueryArgument

The new trait ToQueryArgument has the responsibility of providing the type and writing argument value both for XQuery. You can implement it for your own types for direct support!

/// Makes this type able to be interpreted as XQuery argument value.
pub trait ToQueryArgument<'a> {
    /// Writes this value using the given `writer` as an XQuery argument value.
    fn write_xquery<T: DatabaseStream>(&self, writer: &mut ArgumentWriter<T>) -> Result<()>;

    /// The type name of the XQuery representation.
    ///
    /// # Example
    /// ```
    /// use basex::ToQueryArgument;
    /// assert_eq!("xs:string", String::xquery_type());
    /// ```
    fn xquery_type() -> String;
}

There is no way to encode an array, a map, or any other container or structured type supported by the BaseX server protocol, so I assume it's impossible and the interface does not support it. However, If there would be a way to serialize these as external variables for BaseX, it would be just a matter of trait implementation!

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut query = client.query(
        &mut "declare variable $prdel as xs:string external; $prdel".as_bytes()
    )?;

    query.bind("prdel")?.with_value(1u8)?
        .bind("prdel")?.with_value(1i8)?
        .bind("prdel")?.with_value(2u16)?
        .bind("prdel")?.with_value(2i16)?
        .bind("prdel")?.with_value(3u32)?
        .bind("prdel")?.with_value(3i32)?
        .bind("prdel")?.with_value(3f32)?
        .bind("prdel")?.with_value(4u64)?
        .bind("prdel")?.with_value(4i64)?
        .bind("prdel")?.with_value(3f64)?
        .bind("prdel")?.with_value(true)?
        .bind("prdel")?.with_value("test")?
        .bind("prdel")?.without_value()?;

    Ok(())
}

Upgrading from the previous version

Upgrading is trivial as you need to only fix every call to Query::bind as such:

From	To
Query::bind("var", Some("test"), Some("xs:string"))?	Query::bind("var")?.with_value("test")?
Query::bind("var", None, None)?	Query::bind("var")?.without_value()?

The type, which was previously the third argument, is now provided from the ToQueryArgument trait.

Before

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut query = client.query(
        &mut "declare variable $prdel as xs:string external; $prdel".as_bytes()
    )?;
    query.bind("prdel", None, None)?;
    // or
    query.bind("prdel", Some("test"), Some("xs:string"))?;
    let mut response = query.execute()?;
    let mut actual_result = String::new();
    response.read_to_string(&mut actual_result)?;
    response.close()?.close()?;

    println!("{}", actual_result);
    Ok(())
}

After

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut query = client.query(
        &mut "declare variable $prdel as xs:string external; $prdel".as_bytes()
    )?;
    query.bind("prdel")?.without_value()?;
    // or
    query.bind("prdel")?.with_value("test")?;
    let mut response = query.execute()?;
    let mut actual_result = String::new();
    response.read_to_string(&mut actual_result)?;
    response.close()?.close()?;

    println!("{}", actual_result);
    Ok(())
}

Plans for next version

The next version focuses on Query::context and similarly polishes its interface like Query::bind.

BaseX v0.4.0

Brings a streamable Query response and error parsing.

Features

Ok variant of Query::execute result is now of type Response that implements Read

Now the result no longer has to be loaded all at once, which may have exhausted memory resources.

Example

In the following example, the query result gets written to file in a memory-efficient manner.

use basex::{Client, ClientError};
use std::fs::File;
use std::io;

fn main() -> Result<(), ClientError> {
    let client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut xquery = File::open("hornbach.xq")?;
    let mut output_file = File::create("bla.xml")?;

    let (client, _) = client.execute("OPEN lambada")?.close()?;
    let query = client.query(&mut xquery)?;

    let mut response = query.execute()?;
    io::copy(&mut response, &mut output_file)?;

    let query = response.close()?;
    query.close()?;
    Ok(())
}

Err variant of Query::execute result is now of type QueryFailed

This new type parses errors into several fields, useful for further debugging or automatic processing. Most interesting is QueryFailed::code, which contains XQuery Error.

Example

In the following example, the query contains undeclared variable $x, which results in the error of code XPST0008.

use basex::{Client, ClientError};

fn main() -> Result<(), ClientError> {
    let client = Client::connect("localhost", 1984, "admin", "admin")?;

    let query = client.query(&mut "$x".as_bytes())?;
    let actual_error = query.execute()?.close().err().unwrap();
    match &actual_error {
        ClientError::QueryFailed(err) => {
            assert_eq!("Stopped at ., 1/1:\n[XPST0008] Undeclared variable: $x.", err.raw());
            assert_eq!("Undeclared variable: $x.", err.message());
            assert_eq!(1, err.line());
            assert_eq!(1, err.position());
            assert_eq!(".", err.file());
            assert_eq!("XPST0008", err.code());
        },
        _ => assert!(false),
    };
    Ok(())
}

Upgrading from the previous version

This upgrade introduces an extra step to get the whole result read to string.

Before

let mut xquery = "count(/Root/*)".as_bytes();
let mut query = client.query(&mut xquery)?;

let result = query.execute()?;
assert_eq!(result, "3");

After

let mut xquery = "count(/Root/*)".as_bytes();
let query = client.query(&mut xquery)?;

let mut result = String::new();
let mut response = query.execute()?;
response.read_to_string(&mut result)?;
assert_eq!(result, "3");

Plans for next version

The next version will introduce wildcards to Client::execute.

BaseX v0.3.0

The main point of this version is a brand new feature - running database commands. Next, an important fix for escaping arguments with special bytes. And finally, an exciting new example called Home Depot.

Features

Run database commands with Client::execute

This new method lets you run any database command, aside from the ones directly supported by the server protocol such as Client::create, Client::store, etc.

Example

use basex::{Client, ClientError};
use std::io::Read;

fn main() -> Result<(), ClientError> {
    let client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut list = String::new();
    client.execute("LIST")?.read_to_string(&mut list)?;
    println!("{}", list);
    Ok(())
}

Fixes

Escape special bytes 0x00 and 0xFF.

The previous version has brought the ability to stream arbitrary binary streams as arguments. This is useful for Client::store, which is designed for binary files, as opposed to other methods using XML resources which are (or are convertible to) UTF-8 strings.

But it also introduced a bug when sending 0x00 (intended as value separator) or 0xFF (intended as escape byte) bytes. This could practically only affect codebases using Client::store, as there aren't many UTF-8 sequences containing such bytes.

An important reminder that even 100% test-covered codebase can have undiscovered bugs.

Example

This now works as expected, where Client::store would return an error previously.

use basex;
use basex::{Client, ClientError};
use std::io::Read;

fn main() -> Result<(), ClientError> {
    let client = Client::connect("localhost", 1984, "admin", "admin")?;

    let (client, info) = client.execute("OPEN lambada")?.close()?;
    assert!(info.starts_with("Database 'lambada' was opened"));

    let expected_result = [6u8, 1, 0xFF, 3, 4, 0, 6, 5];
    client.store("blob", &mut &expected_result[..])?;

    let mut actual_result: Vec<u8> = vec![];
    client.execute("RETRIEVE blob")?.read_to_end(&mut actual_result)?;
    assert_eq!(expected_result.to_vec(), actual_result);
    Ok(())
}

Examples

Introducing new Home Depot example, available at examples/home_depot. Check it out!

Refactor

• Change Query::updating return type Result<String> -> Result<bool>.
• Change Connection::read to just pass through to the underlying stream instead of tokenizing.

Tests

• Test escaping special bytes.
• Add integration test for storing and retrieving binary files.
• Add new integration tests:
  • Command without open database fails.
  • Command with open query succeeds.
  • Command with invalid argument fails.
  • Command with empty response finishes.
  • Query correctly recognizes if it contains updating statements.

Docs

• Remove some bloat from Client documentation examples.
• Add how to open an existing database to Readme.
• Add docs build status and download count badges.

Upgrading from the previous version

Upgrading is trivial since the only change is the return type of Query::updating. Aside from that, there are no changes needed.

Plans for next version

The next version will focus on Query and bring streaming responses and improve error parsing.

BaseX v0.2.0

This version mainly brings streamable XML resources, full test coverage, and improved documentation.

Features

Arguments for XML resources are now &mut of any type that implements Read instead of &str.

This lets you stream the XML resource, rather than having it all in memory at once. This allows for bigger XML resources and smaller memory requirements.

Also, you can provide XML resources from file or TCP stream for example.

Example

use basex::{Client, ClientError};
use std::fs::File;

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let mut xml = File::open("gargantuan_file.xml")?;

    let _ = client.create("lambada")?.with_input(&mut xml)?;
    Ok(())
}

Refactor

• Remove Option<...> in Client methods' arguments where input is not actually optional.
• Make Client::create return command builder that lets you provide, or choose not to, initial XML resource by calling either with_input or without_input. Reasons for this design are:
  • To keep singular methods - Alternatively, there would be Client::create_with_input and Client::create_without_input. I chose not to implement it in this way
  • To not force to specify the input type - If there would be Client::create with Option<&mut R> where R: Read, then you would be forced to specify a type even when calling with None.

Tests

• Add unit tests to Connection, Client, Query, ClientError.
• Improve test coverage to 100%.

Docs

• Revise Client documentation and provide examples.
• Improve readme.
• Add build, coverage, and version badge.

Upgrading from the previous version

1. Put string XML resources as a mutable reference to byte array
   • "<wojak pink_index=\"69\"></wojak>" => &mut "<wojak pink_index=\"69\"></wojak>".as_bytes()
2. Unwrap XML resources from Option<...> type:
   • Some("<wojak pink_index=\"69\"></wojak>") => &mut "<wojak pink_index=\"69\"></wojak>".as_bytes()
3. Follow calls to Client::create with call to either with_input or without_input.

Before

use basex::{Client, ClientError};

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let _ = client.create("boy_sminem", Some("<wojak pink_index=\"69\"></wojak>"))?;
    let _ = client.create("bogdanoff", None)?;
    Ok(())
}

After

use basex::{Client, ClientError};

fn main() -> Result<(), ClientError> {
    let mut client = Client::connect("localhost", 1984, "admin", "admin")?;
    let _ = client.create("boy_sminem")?.with_input(&mut "<wojak pink_index=\"69\"></wojak>".as_bytes())?;
    let _ = client.create("bogdanoff")?.without_input()?;
    Ok(())
}

Plans for next version

The next version brings a very important feature which is sending commands through the client. This will finally let you open an existing database or fetch a binary resource for instance.

BaseX v0.1.0

refactor: Fix all warnings

- common tests code not being separated by #[cfg(tests)]
- unused variables