Add basic IME support for Android

[rib]

This adds basic support for Ime events to the Android backend.

Note that this will only work when running with the game-activity backend, which uses AGDK GameText to forward Android IME events:

https://developer.android.com/games/agdk/add-support-for-text-input

Normally on Android, input methods track three things:

• Surrounding text
• A compose region
• A selection

Since Winit doesn't track surrounding text and therefore also wouldn't be able to handle orthogonal compose + selection regions within some surrounding text, we can treat the whole text region that we edit as the "preedit" string, and we can then treat the compose region as the optional selection within the preedit string.

This is limited for now because there's no reliable event for committing the input via the "Done" key for a soft keyboard, but it's still usable without this for now.

I've tested this with Egui 0.33

Note: I've opened this as a draft against the v0.30.x branch since that's what I've tested with. I might get a chance to forward port to master but it looks like a lot has changed and I don't have anything convenient to test with currently.

• Tested on all platforms changed
• Added an entry to the changelog module if knowledge of this change could be valuable to users
• Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• [ ] Created or updated an example program if it would help users understand this functionality

[nicoburns]

I'm not sure if this counts as convenient, but this PR (DioxusLabs/blitz#324) is setup to run the Blitz Browser on Android using Winit 0.31-beta.2 and GameActivity.

Status on Winit 0.31-beta.2 is that tapping the URL bar causes the soft keyboard to pop up (and tapping away from the url bar causes it to hide), but at least on my Samsung S23 phone no text events are actually recieved when typing on the keyboard.

This PR looks very promising in terms of actually getting text input working. I'll probably give it a try myself soon!

[nicoburns]

I have this running against Winit master (forward port posted as #4459). Basic text input is somewhat working, although it's pretty janky. It's also not working well at all for the "enter"/"submit" key, which for some reason is coming through as two TextInput events, each of which add a \n to the text (I need to be able to hook to the enter key so that I can trigger a "submit form" action).

[nicoburns]

Looks like rust-mobile/android-activity#200 is one half of what I need for the enter key (that looks like it will allow it to be configured to trigger an "action" rather than just generating a newline). Then the other part would be the ability to hook into the onEditorAction event/callback.

[rib]

Just to mention, I took a second pass at this where I first exposed editor actions via android-activity so that Commit could be handled properly.

This initial draft also doesn't correctly take into account that Winit uses byte offsets for the cursor/selection offsets while Android uses character offsets.

I'll update this draft after opening the corresponding PRs for android-activity.

I also made a stab at implementing IME support for the winit master branch, though that's quite a bit more fiddly.

I think it will effectively make sense to implement twice, once for winit master/0.31 and once for winit 0.30 (I guess it won't be a trivial backport).

Android IMEs support:

• surrounding text
• a text selection span
• a compose region / pre-edit span

winit 0.30 only supports the pre-edit span and winit 0.31 adds support for surrounding text (neither supports a separate selection span).

Even though Android IMEs support surrounding text, the winit 0.31 IME API is harder to support on Android because it doesn't support changes to surrounding text (except deletion).

Things could get fiddly if winit has to fight with the IME by trying to block attempts to modify surrounding text.

[nicoburns]

Interested to try your second pass. The first pass seemed a little rough (although I may have broken in porting to 0.31)

---

That being said, in looking into IME APIs of mostly Android and iOS more, it looks to me like it might be difficult to create a good abstraction. And I'm wondering if it would be best just to expose each platform interface (InputConnection on Android, UITextInput on iOS) directly to users using a mechanism similar to https://docs.rs/winit-core/0.31.0-beta.2/winit_core/application/trait.ApplicationHandler.html#method.macos_handler

Libraries like Parley and Cosmic Text could then implement support for these traits (so users wouldn't have to implement the all details themselves. And it would allow for full fidelity text input on all platforms.

[rib]

Interested to try your second pass. The first pass seemed a little rough (although I may have broken in porting to 0.31)

I've pushed my second pass for reference.

This is now based on rust-mobile/android-activity#216 so it can support Ime::Commit properly.

If anyone wants to smoke test this then you can try running the agdk-egui example in this rust-android-example branch: https://github.com/rust-mobile/rust-android-examples/tree/rib/wip/ime-test

(note that for debugging it can be handy to disable the egui set_request_repaint_callback because egui will redraw continuously during text input which makes it awkward to view event logs)

I think the UX could also be notably improved with some Egui changes which I haven't looked at.

For instance Egui should re-call set_ime_allowed() if you tap a text input widget to allow for the possibility that you can dismiss an Android soft keyboard and so you want to be able to re-show the keyboard without a new focus event for the widget.

When Egui support winit 0.31 then it should be possible to avoid awkward situations where you can move the native widget cursor during text input and the soft keyboard won't know about that. In the mean time then maybe Egui should explicitly disable/enable IME if the user moves the cursor as a crude way of resetting the IME.

[nicoburns]

If anyone wants to smoke test this then you can try running the agdk-egui example in this rust-android-example branch: https://github.com/rust-mobile/rust-android-examples/tree/rib/wip/ime-test

I had a go with this. It's definitely better than it was in the first pass.

I'm still seeing the following bugs:

• Text is highlighted while you type (is this just egui rendering the preedit in a non-standard way?)
• The pre-edit never seems to commit (except when pressing the action button). Usually it will commit when space is pressed to start an new word.
• Action button doesn't differentiate between single and multi-line text inputs (should enter newline in multi-line input)
• In the code editor: tapping to move cursor while typing causes the preedit to be duplicated the next type any letters are typed
• Cursor control (e.g. by dragging the space bar on the keyboard) from the IME doesn't work
• (aforementioned) Issue around keyboard not re-opening when tapping a text field if it has been dismissed.

winit 0.30 only supports the pre-edit span and winit 0.31 adds support for surrounding text (neither supports a separate selection span).
Even though Android IMEs support surrounding text, the winit 0.31 IME API is harder to support on Android because it doesn't support changes to surrounding text (except deletion).
Things could get fiddly if winit has to fight with the IME by trying to block attempts to modify surrounding text.

Could we just change Winit's API? 0.31 is still beta so there should be scope for breaking changes.

[rib]

If anyone wants to smoke test this then you can try running the agdk-egui example in this rust-android-example branch: https://github.com/rust-mobile/rust-android-examples/tree/rib/wip/ime-test

I had a go with this. It's definitely better than it was in the first pass.

I'm still seeing the following bugs:

I think mostly everything here boils down to limitations in the winit 0.30 ime API combined with limitations in Egui's ime handling.

• Text is highlighted while you type (is this just egui rendering the preedit in a non-standard way?)

Winit's IME design is geared towards desktop input methods, with the general assumption being that the pre-edit string represents something like an incomplete kanji when typing in japanese or chinese. It wasn't really designed for general-purpose soft keyboards and this is one way in which that shows.

The text from the keyboard is being forwarded to Egui as preedit text and Egui is showing that as selected because with a desktop IME this would be something like a tentative kanji.

I wouldn't necessarily say this is non-standard on the part of Egui (it makes sense with desktop IMEs that you would highlight the pre-edit span) but it could also be nicer if Egui rendered this differently on Android.

• The pre-edit never seems to commit (except when pressing the action button). Usually it will commit when space is pressed to start an new word.

Yeah, winit's notion of commit doesn't really align with Android's notion of an action.

In the context of a desktop IME then pressing space could commit any kanji you were pre-editing since it becomes unambiguous at that point but the same thing doesn't apply here because we're having to cram Android's surround text + selection + compose region through a narrower IME abstraction that only supports a single pre-edit region.

So in the context of desktop IMEs it makes sense that you may get lots of intermediate commits before any logical finished / done event.

We could maybe experiment with trying to auto-commit whenever we get a Text update that contains no compose region but that also sounds a bit sketchy since it could imply also clearing the IME text which could lead to us fighting with keeping state in sync (there's a race between winit observing the lack of compose region and comitting + resetting the ime text because the user could start typing the next word). If we didn't reset the IME text (to avoid the race condition) then we would have to decide what to do if we ever see a Text update from the IME that modifies the prefix text that has already been comitted to the toolkit (because Winit doesn't provide any way for us to notify the toolkit of those changes).

Ideally winit's IME abstraction could also track committed text, outside the current pre-edit region and it would differentiate commit and some kind of done event.

• Action button doesn't differentiate between single and multi-line text inputs (should enter newline in multi-line input)

on Android this would be handled by configuring the action to be None (we need a separate API for setting imeOptions).

but also this can't currently be conveyed via the winit 0.30 IME api. Probably the best that could be done is to have some toolkit integration for setting imeOptions via android_activity and then maybe Winit would automatically synthesizing Enter key presses for Action::None.

• In the code editor: tapping to move cursor while typing causes the preedit to be duplicated the next type any letters are typed

Yep, I thought I mentioned this somewhere too but can't see that I did.

Winit doesn't provide a way to track a selection/cursor that's separate from the pre-edit span.

For the sake of keeping the state in sync at least then it would maybe make sense for Egui to disable/enable the IME if the user moves the cursor. As it is currently I think moving the cursor in Egui gets treated like a commit, followed by a cursor move and since the IME state hasn't been reset then the next update will re-insert all the text from the IME at the new cursor position. I think this can probably be considered to be an Egui issue. Even though winit 0.30 has no way to let our IME know about this cursor move Egui should at least try and reset the IME to avoid state getting out of sync.

• Cursor control (e.g. by dragging the space bar on the keyboard) from the IME doesn't work

Same as above - Winit doesn't track a separate selection/cursor from the pre-edit region.

• (aforementioned) Issue around keyboard not re-opening when tapping a text field if it has been dismissed.

winit 0.30 only supports the pre-edit span and winit 0.31 adds support for surrounding text (neither supports a separate selection span).
Even though Android IMEs support surrounding text, the winit 0.31 IME API is harder to support on Android because it doesn't support changes to surrounding text (except deletion).
Things could get fiddly if winit has to fight with the IME by trying to block attempts to modify surrounding text.

Could we just change Winit's API? 0.31 is still beta so there should be scope for breaking changes.

Potentially, yep.

I think the Android GameText model where IME updates are in the form of a text blob plus a selection span and a pre-edit span would be general enough for other platforms and also technically easy for toolkits to handle consistently.

My initial aim though has been to at least enable basic text input on Android via Winit's existing 0.30 API.