From d38a2ee05f3f582735056c84342757b6cccabb69 Mon Sep 17 00:00:00 2001 From: Greg Pomerantz Date: Sun, 13 Sep 2026 22:47:29 -0400 Subject: [PATCH] =?UTF-8?q?doc:=20ime.md=20=E2=80=94=20the=20IME=20contrac?= =?UTF-8?q?t,=20failure=20modes,=20and=20diagnostics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written up from the debugging sessions that fixed the IME commit corruption, the scroll slowdown, the mid-word caps, and the distant-commit clobbering: the three-model mental model, the invariants, the stock-gioui structural behaviors the design works around (post-commit selection dedup, restartInput shadowing, focus-gated selection commands, no acks), the log signatures, the test-layer coverage table, and the operational pitfalls hit along the way. --- doc/README.md | 3 +- doc/ime.md | 166 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 doc/ime.md diff --git a/doc/README.md b/doc/README.md index c2ca7ee..61c0b2c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,6 +1,6 @@ # Pad documentation -Four documents, kept at the level of *what, why, and invariants* — not +Five documents, kept at the level of *what, why, and invariants* — not line-by-line code — so they stay true as the implementation evolves. | Doc | What it is | @@ -9,6 +9,7 @@ line-by-line code — so they stay true as the implementation evolves. | [`architecture.md`](./architecture.md) | How it works: single-owner concurrency model, channel topology, Frame handoff contract, ownership rules, editor/browser/render internals. | | [`development_plan.md`](./development_plan.md) | The active plan: completed phases, remaining work, and the on-device observation loop. | | [`release.md`](./release.md) | The release process (`scripts/release.sh`): the gates, the install-to-all-devices policy, and the rule that on-device profiling/testing is diagnostic and needs explicit approval. | +| [`ime.md`](./ime.md) | The IME contract: the three text models, the invariants that keep commits landing where they should, stock-gioui structural behaviors (worked around, not patched), log signatures for diagnosing IME desync, and the test layers and what each can catch. | ## Package inventory diff --git a/doc/ime.md b/doc/ime.md new file mode 100644 index 0000000..5ac7a7c --- /dev/null +++ b/doc/ime.md @@ -0,0 +1,166 @@ +# The IME contract and its failure modes + +What the editor relies on when talking to Android's IME (Gboard), what +stock gioui does structurally, and the failure modes observed on device — +with their log signatures and the invariants that keep the system safe. +This is the map for the next time text lands in the wrong place. + +## The three models + +An IME session involves **three independent copies of the text** that must +be kept consistent: + +1. **The file** (logic goroutine, the truth). +2. **The pushed snippet** — the 32 KB window around the caret that we push + to the IME (`key.SnippetCmd`). All IME commit positions are in *this* + coordinate space (absolute file runes, `Range.Start` = window start). +3. **Gboard's local model** — Gboard's private copy of the snippet plus its + own caret/selection/composition state, which it updates from our pushes + *and* from its own bookkeeping of the commits it sends. + +Every bug in this area is a loss of consistency between model 3 and model +1. The commit that arrives names a range in model 3; applying it to model 1 +is only correct if the two still agree. + +## Invariants the code must maintain + +- **Commits map against the whole buffer, never a stale window.** + `HandleIMECommit` converts the commit's rune range to bytes with a + whole-buffer scan (`runeToByteWhole`), so it stays exact while the + visible window moves (a fling in flight). A renderer-side IME model was + tried and abandoned: it drifts whenever the render window moves + (commit `f54ca2f`). +- **The snippet window is hysteresis-gated and decoupled from the render + window** (`State.computeIMESnippetWindow`): 32 KB, re-anchored only when + the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a + `restartInput`, which starts a fresh input session that re-derives + auto-capitalization — a per-scroll or per-tap re-push is what caused the + random mid-word caps (commit `dd9493b`). +- **The post-commit caret is `Range.Start + len(text)`**, not + `Range.End + len(text)`. Identical for insertions; for a replacement + (autocorrect over a selection) the old formula lands `(End−Start)` runes + past the inserted text and desynchronizes the IME at the exact moment its + model is being updated (commit `51344b9`). The logic side + (`applyIMECommitBytes`) and the renderer's immediate push + (`ApplyIMECommitToModel`) must compute the *same* caret. +- **A legitimate commit is always local**: at the caret, inside the live + selection, or a correction a few runes left of the caret. The guard + (`HandleIMECommit`, `maxCommitDistance = 1024`) snaps anything else to + the caret and arms the resync. A 5-rune autocorrect 19,000 runes from + the caret is never legitimate; applying it verbatim clobbers distant + text (commit `a83cc1a`). +- **A desynchronized IME is healed only by a restart.** See the structural + fact below: selection pushes after a commit are deduplicated away, so + the *only* mechanism that makes Gboard re-read the truth is a snippet + change (`restartInput`). `IMEForceResync` arms on the **first** + anomaly, not a streak: the next frame ships the snippet trimmed by one + rune (one restart), the following frame ships the full window (one more), + then quiet. One-shot, so normal typing never pays for it + (commits `cb8ebc0`, `a83cc1a`). + +## What stock gioui does structurally (worked around, not patched) + +All of the following is stock v0.10.2 behavior. The fixes live entirely in +pad; the fork at `~/gioui` carries **diagnostic logging only** (the +`PADIME` lines) and can be dropped for stock gioui at any time without +behavioral change. + +1. **Post-commit selection pushes are deduplicated away.** The commit + callback (`callbacks.EditorReplace`) advances the window's stored + `imeState.Selection` *before* the frame's state comparison, so a + `SelectionCmd` pushed in the same frame as a commit is "no change" and + `imm.updateSelection` is never called. Consequence: after a commit, the + IME's caret can only be corrected by a snippet restart. This is why the + resync exists and why "just re-push the selection" is not a fix. +2. **`restartInput` shadows `updateSelection`.** In + `window.EditorStateChanged`, a snippet change takes an early-return + branch: the selection update in the same state change is dropped. + Gboard re-fetches the selection on restart, so this is harmless — but + it means a restart frame is the *only* frame where selection and text + are guaranteed to be re-read together. +3. **Selection commands are focus-gated.** `keyQueue.setSelection` + silently drops a `SelectionCmd` whose tag is not the current key-focus + (`req.Tag != state.focus`). A selection push can vanish with no log and + no feedback — the IME never acks selections. If taps stop moving + Gboard's caret, check focus first. +4. **There is no ack anywhere.** Snippet pushes, selection pushes, and + commits are fire-and-forget. The system can only *detect* desync + indirectly (a commit landing where no text expects it), which is what + the drift guard does. + +## Gboard behaviors observed + +- Gboard keeps a word selection on its own (a tap near a word selects it) + and commits autocorrect as a **replacement over its local selection** — + the commit range is then as wide as the word, not a point insertion. +- While composing, Gboard re-sends the whole word on every keystroke + (`"t"`, `"th"`, `"thi"`, … each replacing the previous), and on + backspace it re-sends the shrinking word down to `text=""`. Both are + normal; the drift guard must not treat them as anomalies (they are at + the caret). +- When its local model desynchronizes, Gboard can enter an endless + empty-fix-up loop (`text=""`, one commit per ~150 ms) trying to + reconcile text that is not in its model. The file is never damaged (the + guard snaps each one to the caret); the resync breaks the loop. +- The phone's AICore Gboard is stricter/more aggressive than the emulator's + Gboard. A flow that is clean on the emulator can still desync on device. + +## Log signatures → diagnosis + +Permanent, low-volume lines (see "Diagnostics"): + +| Signature in logcat | Meaning | +|---|---| +| `IME TAP … cursor=N winStart=M` | A tap moved the logic cursor (bytes). Compare with the next `IME COMMIT` ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, **compute it, don't assume a constant**). | +| `IME COMMIT range=[a,b) text=… -> [x,y)` | The logic applied the commit at bytes [x,y). The pre-`->` range is Gboard's coordinate space; the post-`->` range is where it actually landed. | +| `IME DRIFT-SNAP … snap to caret + resync` | Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping. | +| `IME PUSH snippet=[a,b) … (restart)` | We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling). | +| `IME PUSH sel=[s,e)` | We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit). | +| `PADIME EditorStateChanged sel A -> B` / `PADIME updateSelection sel=[…]` / `PADIME restartInput` | The gio→Gboard boundary: what actually crossed into Android. If `IME PUSH` shows a value the `PADIME` lines never show, the drop is inside gioui (focus gate / dedup); if `PADIME` shows it and Gboard still misbehaves, the IME ignored it. | + +**Capture the log at incident time.** Logcat rotates fast; the incident +that started this document was only reconstructable because the capture +happened minutes later. + +## Test layers and what each can catch + +| Layer | What it exercises | Catches | +|---|---|---| +| e2e harness (`internal/test/e2e`) | Logic + renderer headless; commits injected directly (`HandleIMECommit`) | Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. **Never** the IME contract: Gboard is not in the loop. | +| Emulator stress loops (`/tmp/loop4.sh`, `/tmp/stress.sh`) | Real app + adb touches, real Gboard | Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely). | +| Full-path scenario (`/tmp/s8.sh`) | Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs **exact file diff** | The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed. | + +Rule: **a bug in the IME boundary is invisible to every layer except the +last one.** When IME behavior is touched, run S8. + +## Operational pitfalls hit in these sessions + +- The launcher activity is `pad.pad/org.gioui.GioActivity`, not + `.MainActivity`. `am start` failures are silent if stderr is redirected — + check the output once, trust it after. +- `adb install`/build steps fail with "no adb device" when the emulator is + mid-reconnect; the APK may be built but not installed. Verify + `dumpsys package … lastUpdateTime` after installs. +- The "All files access" settings toggle can be a red herring: `appops + set MANAGE_EXTERNAL_STORAGE allow` is the reliable grant, and + `appops get` output has been observed to lag; `dumpsys appops` is + authoritative. +- `difflib.SequenceMatcher` on ~1 MB strings is minutes slow; use a linear + first-diff scan for test assertions on large files. +- Float-equality re-emit is a recurring trap (LastLineY, caret px): any + "re-emit when changed" comparison on a float sum needs an epsilon gate, + or it spins a loop that re-pushes IME state every frame and starves real + work. +- When a tap's pixel→text mapping is in question, the `IME TAP` line gives + both the local point and the resulting cursor — log both, or the mapping + bug is undiagnosable from the outside. + +## Diagnostics currently enabled (temporary) + +- Pad-side, permanent: `IME TAP`, `IME COMMIT`, `IME DRIFT-SNAP`, + `IME PUSH snippet/sel/commit-range` (bounded: only on change). +- Fork-side, **temporary** (in `~/gioui`, gated by the `replace` in + `go.mod`): `PADIME` boundary logs in `GioView.updateSelection`, + `GioView.restartInput`, and `window.EditorStateChanged`. Remove these, + drop the `replace`, `go mod tidy` — behavior is identical on stock + gioui. Keep until the informal on-device testing is settled.