gmp/Pad
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

doc: ime.md — the IME contract, failure modes, and diagnostics

Browse Source 
Written up from the debugging sessions that fixed the IME commit
corruption, the scroll slowdown, the mid-word caps, and the
distant-commit clobbering: the three-model mental model, the
invariants, the stock-gioui structural behaviors the design works
around (post-commit selection dedup, restartInput shadowing,
focus-gated selection commands, no acks), the log signatures, the
test-layer coverage table, and the operational pitfalls hit along
the way.
This commit is contained in:
Greg Pomerantz 2026-09-13 22:47:29 -04:00
parent a83cc1a72f
commit d38a2ee05f
2 changed files with 168 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
doc/README.md
View File

@ -1,6 +1,6 @@
# Pad documentation
Four documents, kept at the level of *what, why, and invariants* — not
Five documents, kept at the level of *what, why, and invariants* — not
line-by-line code — so they stay true as the implementation evolves.
| Doc | What it is |
@ -9,6 +9,7 @@ line-by-line code — so they stay true as the implementation evolves.
| [`architecture.md`](./architecture.md) | How it works: single-owner concurrency model, channel topology, Frame handoff contract, ownership rules, editor/browser/render internals. |
| [`development_plan.md`](./development_plan.md) | The active plan: completed phases, remaining work, and the on-device observation loop. |
| [`release.md`](./release.md) | The release process (`scripts/release.sh`): the gates, the install-to-all-devices policy, and the rule that on-device profiling/testing is diagnostic and needs explicit approval. |
| [`ime.md`](./ime.md) | The IME contract: the three text models, the invariants that keep commits landing where they should, stock-gioui structural behaviors (worked around, not patched), log signatures for diagnosing IME desync, and the test layers and what each can catch. |
## Package inventory

166
doc/ime.md Normal file
View File

@ -0,0 +1,166 @@
# The IME contract and its failure modes
What the editor relies on when talking to Android's IME (Gboard), what
stock gioui does structurally, and the failure modes observed on device —
with their log signatures and the invariants that keep the system safe.
This is the map for the next time text lands in the wrong place.
## The three models
An IME session involves **three independent copies of the text** that must
be kept consistent:
1. **The file** (logic goroutine, the truth).
2. **The pushed snippet** — the 32 KB window around the caret that we push
to the IME (`key.SnippetCmd`). All IME commit positions are in *this*
coordinate space (absolute file runes, `Range.Start` = window start).
3. **Gboard's local model** — Gboard's private copy of the snippet plus its
own caret/selection/composition state, which it updates from our pushes
*and* from its own bookkeeping of the commits it sends.
Every bug in this area is a loss of consistency between model 3 and model
1. The commit that arrives names a range in model 3; applying it to model 1
is only correct if the two still agree.
## Invariants the code must maintain
- **Commits map against the whole buffer, never a stale window.**
`HandleIMECommit` converts the commit's rune range to bytes with a
whole-buffer scan (`runeToByteWhole`), so it stays exact while the
visible window moves (a fling in flight). A renderer-side IME model was
tried and abandoned: it drifts whenever the render window moves
(commit `f54ca2f`).
- **The snippet window is hysteresis-gated and decoupled from the render
window** (`State.computeIMESnippetWindow`): 32 KB, re-anchored only when
the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a
`restartInput`, which starts a fresh input session that re-derives
auto-capitalization — a per-scroll or per-tap re-push is what caused the
random mid-word caps (commit `dd9493b`).
- **The post-commit caret is `Range.Start + len(text)`**, not
`Range.End + len(text)`. Identical for insertions; for a replacement
(autocorrect over a selection) the old formula lands `(End−Start)` runes
past the inserted text and desynchronizes the IME at the exact moment its
model is being updated (commit `51344b9`). The logic side
(`applyIMECommitBytes`) and the renderer's immediate push
(`ApplyIMECommitToModel`) must compute the *same* caret.
- **A legitimate commit is always local**: at the caret, inside the live
selection, or a correction a few runes left of the caret. The guard
(`HandleIMECommit`, `maxCommitDistance = 1024`) snaps anything else to
the caret and arms the resync. A 5-rune autocorrect 19,000 runes from
the caret is never legitimate; applying it verbatim clobbers distant
text (commit `a83cc1a`).
- **A desynchronized IME is healed only by a restart.** See the structural
fact below: selection pushes after a commit are deduplicated away, so
the *only* mechanism that makes Gboard re-read the truth is a snippet
change (`restartInput`). `IMEForceResync` arms on the **first**
anomaly, not a streak: the next frame ships the snippet trimmed by one
rune (one restart), the following frame ships the full window (one more),
then quiet. One-shot, so normal typing never pays for it
(commits `cb8ebc0`, `a83cc1a`).
## What stock gioui does structurally (worked around, not patched)
All of the following is stock v0.10.2 behavior. The fixes live entirely in
pad; the fork at `~/gioui` carries **diagnostic logging only** (the
`PADIME` lines) and can be dropped for stock gioui at any time without
behavioral change.
1. **Post-commit selection pushes are deduplicated away.** The commit
callback (`callbacks.EditorReplace`) advances the window's stored
`imeState.Selection` *before* the frame's state comparison, so a
`SelectionCmd` pushed in the same frame as a commit is "no change" and
`imm.updateSelection` is never called. Consequence: after a commit, the
IME's caret can only be corrected by a snippet restart. This is why the
resync exists and why "just re-push the selection" is not a fix.
2. **`restartInput` shadows `updateSelection`.** In
`window.EditorStateChanged`, a snippet change takes an early-return
branch: the selection update in the same state change is dropped.
Gboard re-fetches the selection on restart, so this is harmless — but
it means a restart frame is the *only* frame where selection and text
are guaranteed to be re-read together.
3. **Selection commands are focus-gated.** `keyQueue.setSelection`
silently drops a `SelectionCmd` whose tag is not the current key-focus
(`req.Tag != state.focus`). A selection push can vanish with no log and
no feedback — the IME never acks selections. If taps stop moving
Gboard's caret, check focus first.
4. **There is no ack anywhere.** Snippet pushes, selection pushes, and
commits are fire-and-forget. The system can only *detect* desync
indirectly (a commit landing where no text expects it), which is what
the drift guard does.
## Gboard behaviors observed
- Gboard keeps a word selection on its own (a tap near a word selects it)
and commits autocorrect as a **replacement over its local selection** —
the commit range is then as wide as the word, not a point insertion.
- While composing, Gboard re-sends the whole word on every keystroke
(`"t"`, `"th"`, `"thi"`, … each replacing the previous), and on
backspace it re-sends the shrinking word down to `text=""`. Both are
normal; the drift guard must not treat them as anomalies (they are at
the caret).
- When its local model desynchronizes, Gboard can enter an endless
empty-fix-up loop (`text=""`, one commit per ~150 ms) trying to
reconcile text that is not in its model. The file is never damaged (the
guard snaps each one to the caret); the resync breaks the loop.
- The phone's AICore Gboard is stricter/more aggressive than the emulator's
Gboard. A flow that is clean on the emulator can still desync on device.
## Log signatures → diagnosis
Permanent, low-volume lines (see "Diagnostics"):
| Signature in logcat | Meaning |
|---|---|
| `IME TAP … cursor=N winStart=M` | A tap moved the logic cursor (bytes). Compare with the next `IME COMMIT` ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, **compute it, don't assume a constant**). |
| `IME COMMIT range=[a,b) text=… -> [x,y)` | The logic applied the commit at bytes [x,y). The pre-`->` range is Gboard's coordinate space; the post-`->` range is where it actually landed. |
| `IME DRIFT-SNAP … snap to caret + resync` | Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping. |
| `IME PUSH snippet=[a,b) … (restart)` | We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling). |
| `IME PUSH sel=[s,e)` | We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit). |
| `PADIME EditorStateChanged sel A -> B` / `PADIME updateSelection sel=[…]` / `PADIME restartInput` | The gio→Gboard boundary: what actually crossed into Android. If `IME PUSH` shows a value the `PADIME` lines never show, the drop is inside gioui (focus gate / dedup); if `PADIME` shows it and Gboard still misbehaves, the IME ignored it. |
**Capture the log at incident time.** Logcat rotates fast; the incident
that started this document was only reconstructable because the capture
happened minutes later.
## Test layers and what each can catch
| Layer | What it exercises | Catches |
|---|---|---|
| e2e harness (`internal/test/e2e`) | Logic + renderer headless; commits injected directly (`HandleIMECommit`) | Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. **Never** the IME contract: Gboard is not in the loop. |
| Emulator stress loops (`/tmp/loop4.sh`, `/tmp/stress.sh`) | Real app + adb touches, real Gboard | Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely). |
| Full-path scenario (`/tmp/s8.sh`) | Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs **exact file diff** | The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed. |
Rule: **a bug in the IME boundary is invisible to every layer except the
last one.** When IME behavior is touched, run S8.
## Operational pitfalls hit in these sessions
- The launcher activity is `pad.pad/org.gioui.GioActivity`, not
`.MainActivity`. `am start` failures are silent if stderr is redirected —
check the output once, trust it after.
- `adb install`/build steps fail with "no adb device" when the emulator is
mid-reconnect; the APK may be built but not installed. Verify
`dumpsys package … lastUpdateTime` after installs.
- The "All files access" settings toggle can be a red herring: `appops
set <pkg> MANAGE_EXTERNAL_STORAGE allow` is the reliable grant, and
`appops get` output has been observed to lag; `dumpsys appops` is
authoritative.
- `difflib.SequenceMatcher` on ~1 MB strings is minutes slow; use a linear
first-diff scan for test assertions on large files.
- Float-equality re-emit is a recurring trap (LastLineY, caret px): any
"re-emit when changed" comparison on a float sum needs an epsilon gate,
or it spins a loop that re-pushes IME state every frame and starves real
work.
- When a tap's pixel→text mapping is in question, the `IME TAP` line gives
both the local point and the resulting cursor — log both, or the mapping
bug is undiagnosable from the outside.
## Diagnostics currently enabled (temporary)
- Pad-side, permanent: `IME TAP`, `IME COMMIT`, `IME DRIFT-SNAP`,
`IME PUSH snippet/sel/commit-range` (bounded: only on change).
- Fork-side, **temporary** (in `~/gioui`, gated by the `replace` in
`go.mod`): `PADIME` boundary logs in `GioView.updateSelection`,
`GioView.restartInput`, and `window.EditorStateChanged`. Remove these,
drop the `replace`, `go mod tidy` — behavior is identical on stock
gioui. Keep until the informal on-device testing is settled.