Tracking issue for IME / composition support

[murarth]

This issue tracks the implementation of IME composition events on each platform, an API initially proposed in #1293. PRs implementing this event should be made against the composition-event branch, which will be merged into master once all implementations are complete.

• Public API implemented (add composition event support #1404)
• API rename (rename CompostionEvent to IMEPreeditEvent #1622)
• Implemented on all platforms
  • Windows (Partially in Windows: implement Window.set_ime_position() with IMM API. #1790, more work required)
  • Mac OS (macOS: Composition Event #2147)
  • Linux X11 (add composition event support #1404)
  • Linux Wayland (Add IME event support for Wayland #2154)
  • iOS
  • Android
  • Web (Composition event on web #2173)
• Document IME
• Add an entry to CHANGELOG.md

[chrisduerr]

As a consumer of winit I'd recommend against naming these events CompositionEvent. Applications that deal with display servers will already know that term in connection with window compositing and it could easily be confused with something related to that.

It also does not fall in line with the existing winit naming of functions like set_ime_position.

I think it would be much better to have a more self descriptive name for this event that cannot be confused with anything else related to the winit project. I'd strongly suggest that IME should at least be part of the name.

[pickfire]

Oh, I thought CompositionEvent means some kind of compose key thing, I never used those and don't even know how to use compose key, I had only used IME to input characters not in the keyboard.

[garasubo]

@pickfire This event naming comes from web specification (Ref: https://developer.mozilla.org/en-US/docs/Web/API/Element/compositionstart_event). And IME will create those composition events.
But as @chrisduerr is pointed out, I agree that calling those events as "composition" is confusing.

[garasubo]

Proposed the rename these events to IMEPreeditEvent in #1622

[kaiwk]

For mac, maybe we can use this API? composedString

Seems setMarkedText does the trick:

• https://bugzilla.mozilla.org/show_bug.cgi?id=875674

[jakerr]

Just FYI here's a similar problem being solved in the glfw project that might provide some hints / inspiration: glfw/glfw#658

[kchibisov]

I've decided to take a look on the proposed events wrt compositing, since I've been working on IME handling in winit for Wayland recently.

Right now I can see that the current API that is being proposed looks more like

pub enum CompositionEvent {
    CompositionStart(String),
    CompositionUpdate(String, usize),
    CompositionEnd(String),
}

If I were modelling it from Wayland side it'll be more like.

pub enum CompositionEvent {
    /// IME got enabled on the surface you're on.
    CompositionEnabled
    /// We start a composition, a cursors are offsets into string, which could represent a 'highlight' string. `None` for both means hidden.
    CompositionPreedit(text: String, cursor_start: Option<usize>, cursor_end: Option<usize>)
    /// Deletes surrounding text before and after current cursor.
    CompositionDeleteSurroundingText(before_length: usize, after_length: usize)
    /// Composition is done, and that's the text that should be inserted into widget.
    CompositionCommit(text: String)
    /// Composition is done, user must process all events and apply them atomically
    CompositionDone
    /// IME focus got lost, no need to process any IME stuff anymore.
    CompositionDisabled
}

So those are just plain events that we have straight from a Wayland protocol. The thing is that it works in some kind of atomic commits way, if you think about it. So on CompositionDone you must perform the following:

1. Replace existing preedit string with the cursor.
2. Delete requested surrounding text.
3. Insert commit string with the cursor at its end.
4. Calculate surrounding text to send.
5. Insert new preedit text in cursor position.
6. Place cursor inside preedit text.

Now the question is how to model that with the proposed IME events.

So first of all I don't need CompostionDone event, since this is an event when I'd send things downstream, meaning that I don't need CompositionDeleteSurroundingText. And it seems like I don't need DeleteSurroundingText, since I can model it via CompositionPreedit. CompositionCommit(text: String) is indeed needed so the user will know when to insert text.

So if we now apply that to the proposed composition event it'll look more like.

pub enum CompositionEvent {
    CompositionEnabled,
    CompositionPreedit(String, Option<usize>, Option<usize>),
    CompositionCommit(String),
    CompositionDisabled,
}

I guess it should make it a bit clear, so we removed 1 event for users to carry about when editing takes place.

However what wasn't mentioned is that how user interacts with IME via requests. Right now
in winit we only have fn set_ime_position, which is just some basic stuff. However user may want to
opt for IME handling, via e.g. fn enable_ime, so they will react that way on CompositionEnabled(basically means that user got IME opened). And they could want to hide IME sometimes, when they interact with a keyboard, but it doesn't require IME things, like a game, where you have a chat with IME support, so they will need fn disable_ime.

Also, the user is likely want to set text around the cursor and a cursor position, so they will need set_surrounding_text, and also a content type of the current text being edited, with set_content_type, so IME can provide better suggestions.

In the end Window will get

fn set_ime_surrounding_text(text: String, cursor: usize);
fn enable_ime_handling();
fn disable_ime_handling();
fn set_contents_type(type: ContentType);

The changes shouldn't affect existing work on IME in winit much, and purely about ergonomics and additions.

@garasubo since you've started initial work on updating IME handling in winit, I'm curious whether proposed changes here make sense, and are possible on X11.

@kaiwk As the one who send composition event implementation. Are proposed changes sound reasonable to you and possible on macOS, or macOS could need a bit more things to work, nicely?

@chrisduerr As a winit consumer that deals with text input, and that should implement IME handling, does the proposed API sounds ergonomic to implement? One benefit is that you'll know precisely when you actually have IME, so you won't call to set_ime_position until you've got an event from a winit that IME was enabled for some window.

Also, since winit doesn't have IME handling on a lot of platforms to land IME handling APIs we can only implement it as a concept on platforms that support it, like X11/macOS, and soon™ Wayland. Platforms that don't support IME in particular could just send IME text via ReceivedChar event like IME is working right now.

[pickfire]

One benefit is that you'll know precisely when you actually have IME, so you won't call to set_ime_position until you've got an event from a winit that IME was enabled for some window.

Based on my experience with xinput on X11, I believe it is good to be able to call set_ime_position when the IME was enabled rather than updating it for every key press or some weird solution to make some timer to update it once it a while.

[garasubo]

However what wasn't mentioned is that how user interacts with IME via requests.

That's a great idea. But for me, adding those APIs sounds out of scope of this issue. My original motivation is to get IME related event from the window system to implement CompositionEvent API in the web browser engine. So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

[kaiwk]

Are proposed changes sound reasonable to you and possible on macOS, or macOS could need a bit more things to work, nicely?

Actually, i'm not very familiar about IME, the PR just fix my own problem, but i'll see what i can do for it. Is there any detailed doc about surrounding text and content type here? thanks.

[kchibisov]

So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

My motivation was to change enum to those 4 variants, so it's in scope of that issue, since I don't quit like the 3 variants we have right now. We can add window methods later however I'd really like to see enum reworked, so users stop calling IME functions without IME being enabled.

[pickfire]

Can it commit partially?

[garasubo]

So, I didn't add APIs to control IME from the application.
I think it would be better to have another issue to add those APIs to make this problem simple. I can support x11 implementation.

My motivation was to change enum to those 4 variants, so it's in scope of that issue, since I don't quit like the 3 variants we have right now. We can add window methods later however I'd really like to see enum reworked, so users stop calling IME functions without IME being enabled.

I see. I meant that adding API to control IME from window (e.g. enable_ime_handling) could be an independent issue, but either way is fine. I can help x11 implementation anyway.

I'm not sure what is the surrounding text concept. I think in x11 there is no such concept. Could you explain this more specifically? Maybe part of preedit string to convert, like I type "きょうもいいてんきですね" in IME and try to convert "きょう" to "今日" firstly, then, "きょう" is surrounding text and preedit string is "きょうもいいてんきですね"?
Also, when can the second variant of CompositPreedit (cursor_start) be None? In x11, a cursor pos is always defined.

[kchibisov]

I see. I meant that adding API to control IME from window (e.g. enable_ime_handling) could be an independent issue, but either way is fine. I can help x11 implementation anyway.

Yeah, I agree, we can do it separately.

I'm not sure what is the surrounding text concept. I think in x11 there is no such concept. Could you explain this more specifically? Maybe part of preedit string to convert, like I type "きょうもいいてんきですね" in IME and try to convert "きょう" to "今日" firstly, then, "きょう" is surrounding text and preedit string is "きょうもいいてんきですね"?

It's for when you already have some text around and start editing, so you tell IME that you have certain things around, and your cursor is at some position. You're not required to send this request on Wayland and if you don't know what is around your cursor you don't send it. So if before starting edition you had きょう and a cursor right behind that text, you send this string and a cursor position right after the last char, so IME will know that it already has きょう, so suggestions are not empty right away, but include your text.

Also, when can the second variant of CompositPreedit (cursor_start) be None? In x11, a cursor pos is always defined.

On Wayland it's None when you should hide a cursor, I'm not sure how to expose that, since Wayland passes negative numbers here, so for rust I've translated that part into None. If you have an idea how to translate that concept it'll be nice.

[kchibisov]

Can it commit partially?

The purpose of Commit on IMEs is that you insert text to a widget and move cursor to a new position. You can't commit partially, since it's basically 'apply change'. While you're editing you're getting preedits, which informs about the current string, but once user hit a key that commits its input like Enter(this is handled by IME) you get commit and start new editing.

[Isopod]

IMEs on Linux still don't work, as far as I can tell. I should say I'm not some kind of IME guru, I only use it for testing. I tested it with Anthy (Japanese IME) on X11. For the record, the IME works in most other programs, which includes Firefox, Chrome and all GTK and Qt applications.

I attempted to enable IME support like this:

window.set_ime_allowed(true);

This makes compose sequences (dead keys, e.g. ´+e = é) work, but not IMEs. In order to support IMEs on Linux, I think winit would need to implement IBus, since this is what most desktop environments (Gnome etc.) use these days.

[kchibisov]

Winit supports XIM and text_input_v3, dbus based implementation are out of scope.

GNOME works with both of them out of the box if you enable IME, same with kwin.

Be aware that once you enable ime support, you must handle the Ime events.

[Isopod]

At the moment I'm just dumping all events to the console. I'm not receiving any IME events. I just get ReceivedCharacter() events with latin letters.

[kchibisov]

if you use alacritty do you have IME input in it?

[kchibisov]

Also, make sure that what you're using is using XIM.

[Isopod]

I do happen to use alacritty and no, I don't have IME input there, either. When I manually set the input method to XIM by overriding environment variables (e.g. GTK_IM_MODULE=xim), then I don't have any IME input in those applications, either. But that's to be expected, isn't it? After all, I'm using IBus and not XIM. GTK and Qt have their own input modules for IBus, so that's not an issue.

Is IBus really supposed to be completely backward-compatible with XIM? Because that has never been the case for me , the compatibility seems to end at compose sequences/dead keys (*). I don't know if I'm doing anything wrong. I tried launching ibus with --xim, but it doesn't make any difference. I don't use any desktop environment, just i3.

Edit: (*) IBus might not have had any involvement in that at all.

[kchibisov]

Is IBus really supposed to be completely backward-compatible with XIM? Because that has never been the case for me, the compatibility seems to end at compose sequences/dead keys. I don't know if I'm doing anything wrong. I tried launching ibus with --xim, but it doesn't make any difference. I don't use any desktop environment, just i3.

I'd suggest to consult the ibus docs, I know that ibus the way setup on GNOME works without any issue with alacritty on X11 using XIM.

There're plenty IMEs supporting XIM (e.g. fcitx) and surely ibus can do that given that it works on stock fedora gnome(x11, wayland is also oob, but it has different story) for me.

[notgull]

dbus based implementation are out of scope.

Do you have a link to discussion on this topic? In my opinion, if IME is in scope then ibus should be in scope.

[kchibisov]

Do you have a link to discussion on this topic? In my opinion, if IME is in scope then ibus should be in scope.

Yet on Wayland you don't need it and having ibus where other IMEs exist is not an option (you have uim and so on as well, I never had ibus and use IME daily).

[Isopod]

Ok, I just tested it on a fresh Fedora 37 VM, can confirm that the IME works there on both Wayland and X11 (tested in alacritty). The UX isn't the best, but it appears to be usable. Not sure what Fedora is doing differently, it might just be an issue on my machine then.

[pickfire]

I used alacritty-git alacritty 0.13.0-dev (2df8f860) with IME on both x11 and wayland kde on arch linux and it works, and I have no idea what is XIM even though I heard about it, I don't think I touched that. But I remember on wayland some tweaks needs to be done, like set the virtual keyboard https://fcitx-im.org/wiki/FAQ#Non_Gtk.2FQt_Wayland_Application_.28Alacritty.2C_kitty.2C_etc.29.

[Isopod]

The issue I had might be specific to the Arch Linux ibus package. The PKGBUILD does not pass --enable-xim to configure. I recompiled the package with --enable-xim and it works now. Note that you also have to start ibus-daemon with -x or --xim.

[lucasmerlin]

I managed to add IME support to android and iOS:
Android input works really well, including autocomplete and suggestions:

Video

Screen_Recording_20230708_144153.mp4

iOS supports basic text entry, but no autocomplete /autocorrect yet:

Video

IMG_0149.MP4

The android support is based on @rib's work on android-activity here: rust-mobile/android-activity#24
I had to add a new Event to winit, basically just passing through the TextInputState to egui, handling the logic there.
Implementation in egui is pretty simple, basically

• send the initial TextInputState (text + cursor position/selection) to winit when focusing a text field + open the keyboard
• update the text in egui whenever a new TextInputState is received, handling the selection and compose region.

The TextInputState struct looks like this:

Expand me

/// This struct holds a span within a region of text from `start` (inclusive) to
/// `end` (exclusive).
///
/// An empty span or cursor position is specified with `Some(start) == Some(end)`.
///
/// An undefined span is specified with start = end = `None`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct TextSpan {
    /// The start of the span (inclusive)
    pub start: Option<usize>,

    /// The end of the span (exclusive)
    pub end: Option<usize>,
}

#[derive(Debug, Clone, PartialEq, Eq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct TextInputState {
    pub text: String,
    /// A selection defined on the text.
    pub selection: TextSpan,
    /// A composing region defined on the text.
    pub compose_region: TextSpan,
}

For iOS text support, I implemented the UIKeyInput api to receive basic key events but it doesn't support autocomplete / autocorrect / character composition yet. There also is UITextInput to support these features, I think it should be possible to write a implementation in winit that provides the same simple TextInputState api to applications, but I don't have the objective c skills to implement this.

If you want to try these in an egui app, you can add the following to your Cargo.toml to try my branches:

[patch.crates-io]
winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" }
egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}

Or try this example for android: https://github.com/lucasmerlin/rust-android-examples/tree/ime_support_showcase/agdk-eframe.

These are the relevant branches:
https://github.com/lucasmerlin/winit/tree/mobile_ime_support
https://github.com/lucasmerlin/egui/tree/mobile_ime_support
https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)

If there is interest to add this to winit I'm happy to open a draft PR.

[kchibisov]

Well, we'd need patches upstream to be merged first, you could open a draft PR if you want to though.

[rib]

Sorry for the delay following up here @lucasmerlin - very cool that you got something working here.

I'm hoping to get a chance to look at this soon.

It'll be good to compare the egui / winit changes with the ones I experimented with at the time:

https://github.com/rib/winit/tree/android-activity-ime-events
https://github.com/rib/egui/tree/android-winit-ime-support

It'll be good to test this on GameActivity 2.0.2 once I land this PR: rust-mobile/android-activity#88

[phnaharris]

So first of all I don't need CompostionDone event, since this is an event when I'd send things downstream, meaning that I don't need CompositionDeleteSurroundingText. And it seems like I don't need DeleteSurroundingText, since I can model it via CompositionPreedit. CompositionCommit(text: String) is indeed needed so the user will know when to insert text.

Hi @kchibisov, I'm wondering why you said that you don't need CompositionDeleteSurroundingText event and how we can model it via CompositionPreedit. Because, the surrounding text is the text around the cursor, even if user committed it to widget or not, as far as I know. I think what you mean is we can modify the preedit, but the problem became a thing if the IME want to delete the surrounding text via text input v3 after the preedit be committed to widget, I think we didn't have a mechanism for doing that yet.

Now I wanna support this feature about surrounding text. Based on the design of GTK for this feature, I need a way for downstream (I'm using egui) to send the surrounding text back to winit whenever it need, through Ime::RetrieveSurrounding event. It would be nice if you have any idea for doing that.

[kchibisov]

You should add delete surrounding text, just keep in mind that it doesn't exist without setting surrounding text. The issue is how to communicate that cross platform, but it's a technical detail.

Part of that was done in #2993 . IME will also be reworked for 0.31.

[phnaharris]

it doesn't exist without setting surrounding text

I've taken the work in this pull request and got the idea that how the surrounding text be updated in the event loop. I also implement set_ime_surrounding_text for Wayland and call it whenever it changed, especially when text input enabled (and commit it to compositor). But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

Thank you .

[kchibisov]

But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

it's IME dependent, you'll get it only when IME thinks that you should get it. Also, ensure to commit when setting surrounding text.

[phnaharris]

I understand. But if I reproduce the same situation in the same environment, I should expect the same result right? I've created a text box with gtk and egui (with winit), input the same content and when I go back to change some character, gtk worked fine with surrounding text while winit receive a new Preedit event.

Have you ever got a DeleteSurroundingText event in winit before? I just wanna know if the bug come from the flow that I'm implemented was wrong or something else?

My environment:

• Compositor: Hyprland.
• IME: Fcitx5.
• Toolkit: GTK and egui + winit.

[kchibisov]

The event is not handled by winit unless you've added it. Use WAYLAND_DEBUG=1 to compare patters and use that knowledge to troubleshoot, since usually it's all pretty deterministic.

[phnaharris]

Okay, I got the idea. Thank you for your help.