Add Android IME bridge via Fragment section to touch.md

- Fragment registration pattern from Passgo: app.ViewEvent → GioView → Context → Activity → FragmentManager
- IME Fragment architecture: hidden EditText in transparent Fragment overlay
- Two-way sync: Gio→EditText (cursor sync via setSelection), EditText→Gio (IME events via TextWatcher + JNI callback)
- Text window management: ±1KB around cursor, updated on cursor move and text changes
- IME feature support: autocorrect, swipe typing, voice, predictions, emoji, CJK composition
- Event merging: Main goroutine drains JNI callback channel alongside w.Event() calls
- Implementation notes: impl_android.go, jni_android.c, ime/ImeFragment.java

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
Greg Pomerantz 2026-05-08 14:39:11 -04:00
parent e4c5804170
commit fdb51810f4

View File

@ -165,10 +165,119 @@ The `TextField` element must be "focused" to receive keyboard events. Focus is a
| Ctrl+Y | `Press` | `Y` (with Ctrl) | Redo | | Ctrl+Y | `Press` | `Y` (with Ctrl) | Redo |
| Ctrl+F | `Press` | `F` (with Ctrl) | Show search bar | | Ctrl+F | `Press` | `F` (with Ctrl) | Show search bar |
### 5.4 IME and Composition ### 5.4 IME and Composition (Pure Gioui)
Android soft keyboards use IME composition for languages like Chinese, Japanese, or Korean. Gioui's `key.Event` (Type: `Edit`) handles the final committed text. Logic treats the committed text as a single insertion at the cursor. Android soft keyboards use IME composition for languages like Chinese, Japanese, or Korean. Gioui's `key.Event` (Type: `Edit`) handles the final committed text. Logic treats the committed text as a single insertion at the cursor.
**Limitation**: Gioui's `key.InputOp` on Android does not provide a full `InputConnection`. The IME has no context — it cannot read surrounding text for autocorrect, cannot anchor swipe gestures to our cursor, and cannot provide predictive suggestions. Only raw character commits are received.
## 5.5 Android IME Bridge via Fragment
To access the full suite of Android IME features (autocorrect, swipe typing, voice input, predictions), Pad uses a **hidden `EditText`** hosted in a native `Fragment`. This is the same pattern used by Passgo (see `cmd/passgo-gui/impl_android.go`, `PgpConnect.java`).
### 5.5.1 Fragment Registration
The Fragment is registered on app startup using `app.ViewEvent`:
1. **Go side**: `handleEvent` receives `app.ViewEvent` on app start. `e.View` is a `uintptr` pointing to the `GioView` (`android.view.View`).
2. **C side (JNI)**: `registerFragment()` receives the `jobject view`:
- Gets the `Context` from the View via `getContext()`
- Gets the `ClassLoader` from the Context via `getClassLoader()`
- Loads the Fragment class via `findClass("pad/ime/ImeFragment")`
- Creates an instance by calling the constructor with the View
3. **Java side (`ImeFragment.java`)**:
- `ImeFragment` extends `Fragment`
- Constructor receives the `View` (GioView), extracts `Context`
- In `onAttach()`, casts `Context``Activity`
- Uses `act.getFragmentManager().beginTransaction().add(inst, "ImeFragment").commitNow()`
- Inflates a layout containing a transparent, zero-size `EditText`
### 5.5.2 IME Fragment Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Android Activity │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ GioView (OpenGL rendering) │ │
│ │ - pointer.InputOp → pointer.Event │ │
│ │ - key.InputOp → key.Event │ │
│ └───────────────────────────────────────────────────────┘ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ ImeFragment (transparent overlay) │ │
│ │ ┌─────────────────────────────────────────────────┐ │ │
│ │ │ EditText (hidden, zero-size) │ │ │
│ │ │ - Holds text window around cursor │ │ │
│ │ │ - Selection synced to cursor position │ │ │
│ │ │ - IME receives all keyboard events │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
### 5.5.3 Two-Way Sync: Gio ↔ EditText
The IME bridge maintains a tight sync loop between the Gio editor state and the hidden `EditText`:
**Gio → EditText (Cursor Sync)**:
1. User taps to move cursor in Gio editor.
2. Logic updates `cursor_pos` in state.
3. Main goroutine calls a JNI function: `SetEditTextSelection(byteOffset)`.
4. C side calls `editText.setSelection(byteOffset)` on the `EditText`.
5. IME now knows the cursor position and can provide context-aware suggestions.
**EditText → Gio (IME Events)**:
1. User types/swipes/uses voice input on the soft keyboard.
2. IME commits text to the `EditText` via `EditText.onTextChanged()`.
3. A `TextWatcher` on the `EditText` captures the change.
4. The change is sent via JNI callback to Go: `ImeTextChanged(replacementText, start, before, count)`.
5. C side allocates memory for the string and calls a Go export function.
6. Go side creates a synthetic `key.Event` (Type: `Edit`) and sends it to the Main goroutine.
7. Main goroutine batches it into `[]InputEvent` alongside any pointer/keyboard events.
8. Logic processes the synthetic event as a text insertion at the cursor.
### 5.5.4 Text Window Management
The `EditText` does not hold the entire file — only a **text window** around the cursor:
- **Window size**: ±1KB of text around the cursor (configurable).
- **On cursor move**: The Main goroutine sends a JNI call to update the `EditText`'s text and selection.
- **On text insertion/deletion**: The Logic goroutine updates the in-memory buffer, then sends a JNI call to update the `EditText`'s text and selection.
This ensures the IME always has context for suggestions, even for multi-gigabyte files.
### 5.5.5 IME Feature Support
| Feature | Supported | How |
|---|---|---|
| Autocorrect | Yes | IME reads text window from `EditText`, suggests corrections |
| Swipe typing | Yes | IME anchors swipe to `EditText` selection (cursor position) |
| Voice input | Yes | IME commits voice text to `EditText` |
| Predictive bar | Yes | IME reads text window for context |
| Emoji keyboard | Yes | IME commits emoji to `EditText` |
| CJK composition | Yes | IME composition buffer → `EditText` → Gio |
### 5.5.6 Event Merging in the Main Loop
IME events from the Fragment are merged with Gioui events in the Main goroutine:
```
Main goroutine event cycle:
1. w.Event() → returns next event (FrameEvent, pointer.Event, key.Event)
2. Check for pending IME events (non-blocking channel read)
3. If IME event exists → create InputEvent, add to batch
4. If Gioui event exists → create InputEvent, add to batch
5. Send entire batch to Logic via inputChan
```
IME events are **not** returned by `w.Event()` — they arrive asynchronously via a JNI callback channel. The Main goroutine drains this channel in every cycle, ensuring IME events are batched alongside Gioui events and delivered to Logic atomically.
### 5.5.7 Implementation Notes
- The `ImeFragment` lives in `cmd/pad/impl_android.go` (Go), `jni_android.c` (C), and `ime/ImeFragment.java` (Java).
- The JNI callback uses a `BlockingQueue` or unbuffered channel to pass events from the Java main looper to the Main goroutine.
- The `EditText` is transparent (`android:background="@android:color/transparent"`) and zero-size (`android:layout_width="0dp" android:layout_height="0dp"`), so it does not affect the UI.
- The `EditText` is **not** focusable by touch — it only receives focus programmatically when the Gio editor needs the keyboard.
## 6. Scroll Momentum ## 6. Scroll Momentum
Pad implements a momentum model in the Logic goroutine to provide smooth scrolling. Pad implements a momentum model in the Logic goroutine to provide smooth scrolling.