Abstractions of text

[truchi]

Hello Sir,

I would like to talk about a use case I am thinking up. This is not really a feature request (as I understand it might be a huge patch), but an open question about building upon textwrap.

Reading the docs and a few bits of the sources I found out that it may not possible to work with abstractions of text.

Let's say I am writing a terminal rendering pipeline in a layer superimposing fashion (i.e. a simple DOM for terminal). I want my users to be able to wrap text inside their "div"s, but also to style text inline, HTML-style:

magic_macro!( // Totally fake, but describes the context
    <div>
        Hello, <red>World</red>! // Text with inline styles
    </div>
);

I might then work with text with something like this:

struct Cell {
    char: char,
    style: Style, // Meta alongside the char
}

type Text = Vec<Cell>; // or anything...

As you can see, I cannot get an &str from that kind of Text type. Yet I really need this styling meta with each char to do some sort of painter algorithm, due to how terminal CSI work (they're global, I guess you know what I mean).

What pieces from core would I be able to reuse, and to what extent, to build a wrap working with my Cells?

Thank you for your answer !

[mgeisler]

Hi @truchi, thanks for the question, it's a great one. Sorry for the slow reply, I wish I had more time for open source work :-)

With the changes I made for the 0.13 release, you should be able to wrap things that don't have a &str anywhere. A made-up example about building a house can be found here: https://docs.rs/textwrap/0.13.2/textwrap/core/fn.wrap_first_fit.html#examples

So in short, if you can give core::wrap_first_fit or core::wrap_optimal_fit a slice with "things which can be wrapped", then the functions will tell you where you should put your line breaks.

As you can see, I cannot get an &str from that kind of Text type. Yet I really need this styling meta with each char to do some sort of painter algorithm, due to how terminal CSI work (they're global, I guess you know what I mean).

Yeah, I think I know what you mean :-) I guess you want to turn on bold/cursive/... on a per-character basis?

For textwrap to work, you would only need to group your Cells into some kind of "words". These words would then need to implement the Fragment trait — this trait does not require you to produce a &str at any time! You could perhaps do something like this:

#[derive(Debug)]
struct Word {
    cells: Vec<Cell>,
    whitespace: usize,
}

impl Fragment for Word {
    fn width(&self) -> usize {
        self.cells.len()
    }

    fn whitespace_width(&self) -> usize {
        self.whitespace
    }

    fn penalty_width(&self) -> usize {
        0
    }
}

I wrote a full example here which shows how these words can be wrapped into one or more lines, depending on the line width: playground.

Please let me know if this helps, I would love to include an example with exactly this kind of functionality in the examples/ directory.

[truchi]

Hello, thanks for the reply!
That looks great!

0.13

Yep it changed a lot since my previous issue, kudos!

I guess you want to turn on bold/cursive/... on a per-character basis?

Yes, in individual layers. When printing the flattened canvas to the terminal we can simplify because we are (supposedly) the single owner of Stdout, hopefully making it faster.

I have to play with the Fragment trait to get a grasp of it. The "house building" example is still confusing to me since I didn't run it... I suggest including the output in the docs.

Looking at your playground.
We do not get unicode-width/hyphenation features through this? That would be a huge bummer (unicode-width esp.)!

I'm working on a different part of the project right now, I will be back at that sooner or later!

[mgeisler]

I have to play with the Fragment trait to get a grasp of it. The "house building" example is still confusing to me since I didn't run it... I suggest including the output in the docs.

Yeah, I agree that it's a rather abstract example! :-) I wanted to show that you could use the wrapping algorithm for something way outside the realm of text — if you really really want to. Notice that you see the "output" indirectly via the assert_eq! statements in the example. You can copy-paste the code to the playground and make it run if you add a fn main() { ... } around it: playground.

It is honestly not so interesting to use wrap_first_fit for this since the underlying algorithm is so simple — but wrap_optimal_fit is more interesting since it has the same API, but uses a non-trivial algorithm below the covers.

Looking at your playground.
We do not get unicode-width/hyphenation features through this? That would be a huge bummer (unicode-width esp.)!

You would be responsible for determining the width of each cell — use core::display_width for this.

The Fragment trait is just the building block which textwrap uses internally. Hyphenation is implemented by breaking each word down to its syllables (using core::split_words) but you would need your own splitting if you have your own struct which implements the Fragment trait.

[mgeisler]

Hey @truchi, I hope the above helped you a bit, otherwise please let me know. GitHub recently released a "discussion" feature and I'll try and convert this issue into the very first discussion in this repo 🎉

[truchi]

Hello again!

Yes you help me a lot, thank you for your time!
(Good, so we can talk without polluting your issues)

I created a toy repo to play with theses ideas. I am assuming:

• No unicode width (for now)
• No hyphenation (maybe never)
• No quality/performance concerns
• Simple (real life is more convoluted)

I did the very basics in that first try. The idea is somewhat different from what we talked about earlier. Have a look at StyledWord and example 1. We have HTML-like formatting (fn to_text) for testing convenience :)

So now I wonder how to use break_words since I still don't have Words but StyledWords. Also why isn't this streamlined in the wrap algorithm? Wouldn't it save iteration and allocation?
Would it be possible to wrap from an iterator of fragments to an iterator of fragments without allocating?

Besides that I wonder how to treat indentation and newlines in the input.

(Bonus point for bug at width 20: the exlam is on a newline...)

[mgeisler]

So now I wonder how to use break_words since I still don't have Words but StyledWords

Yeah, you cannot use it with anything custom, at least not right now. I think that people who want to wrap complex things such as text rendered in a PDF, or content on a webpage will need a custom implementation anyway.

Also why isn't this streamlined in the wrap algorithm? Wouldn't it save iteration and allocation?

It could save some allocation, yes. In terms of big-oh time complexity, it remains linear, even if we scan through the fragments twice. The underlying assumption is that it's okay to keep all fragments for a single paragraph in memory at a time.

If allocation is a concern, then you could generate your fragments of the right size up-front. Instead of finding words one by one and then later breaking them into smaller parts, you could generate the fragments of the right size in the first go. I didn't do this since it's conceptually easier to do things in multiple smaller steps as described here: https://docs.rs/textwrap/0.13.4/textwrap/core/. Also, this way the wrapping algorithms don't need to know anything about breaking long words.

You could also use a single Vec for all paragraphs: put your fragments into the Vec and wrap them. Then reuse the same Vec for the next paragraph:

let mut fragments: Vec::with_capacity(10);
for paragraph in paragraphs {
    find_styled_fragments(&mut fragments);
    for line in textwrap::core::wrap_optimal_fit(&fragments, |_| line_width) {
        render_line(line);
    }
    fragments.clear();  // Keep allocation, clear content.
}

[mgeisler]

(Bonus point for bug at width 20: the exlam is on a newline...)

Since you style the text as <red><bold>world</bold>!</red>, you end up with two StyledWords, one for "world" and one for "!". This means that there is a potential break point between the two words.

What I think you really want is to put "world!" into a single StyledWord and then you need multiple styles for this single word. You would then need something like a list of offsets for each style so you know how to convert the StyledWord into a list of Cells.

[mgeisler]

Hey @truchi! Wow, looks cool what you've put together. Adding a dbg! to the Screen::text function shows the structure of the Fragments generated:

[src/screen.rs:41] &text = [
    StyledWord {
        style: Style {
            foreground: Foreground(255, 0, 0),
            weight: Normal,
        },
        string: "Hello, ",
        white: 1,
    },
    StyledWord {
        style: Style {
            foreground: Foreground(255, 0, 0),
            weight: Normal,
        },
        string: "cruel ",
        white: 1,
    },
    StyledWord {
        style: Style {
            foreground: Foreground(255, 0, 0),
            weight: Bold,
        },
        string: "world",
        white: 0,
    },
    StyledWord {
        style: Style {
            foreground: Foreground(255, 0, 0),
            weight: Normal,
        },
        string: "!",
        white: 0,
    },
]

The first StyledWord contain "Hello, ",, including the trailing space. I see that the width of the Fragment is computed as

    fn width(&self) -> usize {
        // Lets say we live in a perfect world for now
        self.string.chars().count()
    }

So this means that the space is double-counted. I've opened truchi/styledtextwrap#1 to fix this in our repository 😄

[mgeisler]

Hi @truchi, did you manager to get your wrapping to work?

[mgeisler]

See also #312 where I made a little demo of how one can wrap text for display on a HTML canvas — using a custom CanvasWord class which can carry non-integer widths.