diff --git a/doc/touch.md b/doc/touch.md index 2abf655..8040e4b 100644 --- a/doc/touch.md +++ b/doc/touch.md @@ -165,10 +165,119 @@ The `TextField` element must be "focused" to receive keyboard events. Focus is a | Ctrl+Y | `Press` | `Y` (with Ctrl) | Redo | | Ctrl+F | `Press` | `F` (with Ctrl) | Show search bar | -### 5.4 IME and Composition +### 5.4 IME and Composition (Pure Gioui) Android soft keyboards use IME composition for languages like Chinese, Japanese, or Korean. Gioui's `key.Event` (Type: `Edit`) handles the final committed text. Logic treats the committed text as a single insertion at the cursor. +**Limitation**: Gioui's `key.InputOp` on Android does not provide a full `InputConnection`. The IME has no context — it cannot read surrounding text for autocorrect, cannot anchor swipe gestures to our cursor, and cannot provide predictive suggestions. Only raw character commits are received. + +## 5.5 Android IME Bridge via Fragment + +To access the full suite of Android IME features (autocorrect, swipe typing, voice input, predictions), Pad uses a **hidden `EditText`** hosted in a native `Fragment`. This is the same pattern used by Passgo (see `cmd/passgo-gui/impl_android.go`, `PgpConnect.java`). + +### 5.5.1 Fragment Registration + +The Fragment is registered on app startup using `app.ViewEvent`: + +1. **Go side**: `handleEvent` receives `app.ViewEvent` on app start. `e.View` is a `uintptr` pointing to the `GioView` (`android.view.View`). +2. **C side (JNI)**: `registerFragment()` receives the `jobject view`: + - Gets the `Context` from the View via `getContext()` + - Gets the `ClassLoader` from the Context via `getClassLoader()` + - Loads the Fragment class via `findClass("pad/ime/ImeFragment")` + - Creates an instance by calling the constructor with the View +3. **Java side (`ImeFragment.java`)**: + - `ImeFragment` extends `Fragment` + - Constructor receives the `View` (GioView), extracts `Context` + - In `onAttach()`, casts `Context` → `Activity` + - Uses `act.getFragmentManager().beginTransaction().add(inst, "ImeFragment").commitNow()` + - Inflates a layout containing a transparent, zero-size `EditText` + +### 5.5.2 IME Fragment Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Android Activity │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ GioView (OpenGL rendering) │ │ +│ │ - pointer.InputOp → pointer.Event │ │ +│ │ - key.InputOp → key.Event │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ ImeFragment (transparent overlay) │ │ +│ │ ┌─────────────────────────────────────────────────┐ │ │ +│ │ │ EditText (hidden, zero-size) │ │ │ +│ │ │ - Holds text window around cursor │ │ │ +│ │ │ - Selection synced to cursor position │ │ │ +│ │ │ - IME receives all keyboard events │ │ │ +│ │ └─────────────────────────────────────────────────┘ │ │ +│ └───────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### 5.5.3 Two-Way Sync: Gio ↔ EditText + +The IME bridge maintains a tight sync loop between the Gio editor state and the hidden `EditText`: + +**Gio → EditText (Cursor Sync)**: +1. User taps to move cursor in Gio editor. +2. Logic updates `cursor_pos` in state. +3. Main goroutine calls a JNI function: `SetEditTextSelection(byteOffset)`. +4. C side calls `editText.setSelection(byteOffset)` on the `EditText`. +5. IME now knows the cursor position and can provide context-aware suggestions. + +**EditText → Gio (IME Events)**: +1. User types/swipes/uses voice input on the soft keyboard. +2. IME commits text to the `EditText` via `EditText.onTextChanged()`. +3. A `TextWatcher` on the `EditText` captures the change. +4. The change is sent via JNI callback to Go: `ImeTextChanged(replacementText, start, before, count)`. +5. C side allocates memory for the string and calls a Go export function. +6. Go side creates a synthetic `key.Event` (Type: `Edit`) and sends it to the Main goroutine. +7. Main goroutine batches it into `[]InputEvent` alongside any pointer/keyboard events. +8. Logic processes the synthetic event as a text insertion at the cursor. + +### 5.5.4 Text Window Management + +The `EditText` does not hold the entire file — only a **text window** around the cursor: + +- **Window size**: ±1KB of text around the cursor (configurable). +- **On cursor move**: The Main goroutine sends a JNI call to update the `EditText`'s text and selection. +- **On text insertion/deletion**: The Logic goroutine updates the in-memory buffer, then sends a JNI call to update the `EditText`'s text and selection. + +This ensures the IME always has context for suggestions, even for multi-gigabyte files. + +### 5.5.5 IME Feature Support + +| Feature | Supported | How | +|---|---|---| +| Autocorrect | Yes | IME reads text window from `EditText`, suggests corrections | +| Swipe typing | Yes | IME anchors swipe to `EditText` selection (cursor position) | +| Voice input | Yes | IME commits voice text to `EditText` | +| Predictive bar | Yes | IME reads text window for context | +| Emoji keyboard | Yes | IME commits emoji to `EditText` | +| CJK composition | Yes | IME composition buffer → `EditText` → Gio | + +### 5.5.6 Event Merging in the Main Loop + +IME events from the Fragment are merged with Gioui events in the Main goroutine: + +``` +Main goroutine event cycle: + 1. w.Event() → returns next event (FrameEvent, pointer.Event, key.Event) + 2. Check for pending IME events (non-blocking channel read) + 3. If IME event exists → create InputEvent, add to batch + 4. If Gioui event exists → create InputEvent, add to batch + 5. Send entire batch to Logic via inputChan +``` + +IME events are **not** returned by `w.Event()` — they arrive asynchronously via a JNI callback channel. The Main goroutine drains this channel in every cycle, ensuring IME events are batched alongside Gioui events and delivered to Logic atomically. + +### 5.5.7 Implementation Notes + +- The `ImeFragment` lives in `cmd/pad/impl_android.go` (Go), `jni_android.c` (C), and `ime/ImeFragment.java` (Java). +- The JNI callback uses a `BlockingQueue` or unbuffered channel to pass events from the Java main looper to the Main goroutine. +- The `EditText` is transparent (`android:background="@android:color/transparent"`) and zero-size (`android:layout_width="0dp" android:layout_height="0dp"`), so it does not affect the UI. +- The `EditText` is **not** focusable by touch — it only receives focus programmatically when the Gio editor needs the keyboard. + ## 6. Scroll Momentum Pad implements a momentum model in the Logic goroutine to provide smooth scrolling.