[WEBSITE] Blog posts on multi-column sorting implementation #264
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on JIRA? Then could you also rename pull request title in the following format? See also: |
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
| │ "Bar" │ ───────────────▶│ 01 │ | ||
| └──────────┘ └─────┘ | ||
| ┌──────────┐ ┌─────┬─────┐ | ||
| │"Fabulous"│ ───────────────▶│ 01 │ 02 │ |
There was a problem hiding this comment.
If "Bar" is 01 and "Fabulous" is 01 02, how do you distinguish between both when you encounter a 01 byte?
There was a problem hiding this comment.
Since the rows are variable length, each also has a length.
Thus, in this case since the lengths are different (and the length is stored along with the row) "Bar" ([01]) is shorter and thus sorts before "Fabulous" [01 , 02])
Perhaps @tustvold can confirm
We should probably make it clearer in the text that the row format includes a length as well
Edit: I was incorrect
There was a problem hiding this comment.
We don't store lengths along with the rows, in the case of dictionary keys, they are stored null terminated. This is how we are able to distinguish
|
|
||
| One detail we have so far ignored over is how to support ascending and descending sorts (e.g. `ASC` or `DESC` in SQL). The Arrow Rust row format supports these options by simply inverting the bytes of the encoded representation, except the initial byte used for nullability encoding, on a per column basis. | ||
|
|
||
| Similarly, supporting SQL compatible sorting also requires a format that can specify the order of `NULL`s (before or after all non `NULL` values). The row format supports this option by optionally encoding nulls as `0xFF` instead of `0x00` on a per column basis. |
There was a problem hiding this comment.
So you have to escape 00 and FF bytes in the input to make sure they aren't confused with NULLs, right?
Also, do you try to handle floating-point NaNs in a specific way?
There was a problem hiding this comment.
Perhaps @tustvold can weigh in here
There was a problem hiding this comment.
So you have to escape 00 and FF bytes in the input to make sure they aren't confused with NULLs, right?
The encoding is designed in such a way that this isn't necessary, at no point is it ambiguous as to whether a byte is part of a sentinel (e.g. null) or value data
do you try to handle floating-point NaNs in a specific way?
Nans are ordered according to the IEEE 754 (2008 revision) total order predicate
There was a problem hiding this comment.
I added a sentence explaining this design in 7f89c31
Co-authored-by: Paddy Horan <5733408+paddyhoran@users.noreply.github.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
This Blog post describes the row format introduced in apache/arrow-rs#2593