gmp/Pad
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9d5c02db03
Pad/doc/ime.md

12 KiB
Raw Blame History

The IME contract and its failure modes

What the editor relies on when talking to Android's IME (Gboard), what stock gioui does structurally, and the failure modes observed on device — with their log signatures and the invariants that keep the system safe. This is the map for the next time text lands in the wrong place.

The three models

An IME session involves three independent copies of the text that must be kept consistent:

  1. The file (logic goroutine, the truth).
  2. The pushed snippet — the 32 KB window around the caret that we push to the IME (key.SnippetCmd). All IME commit positions are in this coordinate space (absolute file runes, Range.Start = window start).
  3. Gboard's local model — Gboard's private copy of the snippet plus its own caret/selection/composition state, which it updates from our pushes and from its own bookkeeping of the commits it sends.

Every bug in this area is a loss of consistency between model 3 and model

  1. The commit that arrives names a range in model 3; applying it to model 1 is only correct if the two still agree.

Invariants the code must maintain

  • Commits map against the whole buffer, never a stale window. HandleIMECommit converts the commit's rune range to bytes with a whole-buffer scan (runeToByteWhole), so it stays exact while the visible window moves (a fling in flight). A renderer-side IME model was tried and abandoned: it drifts whenever the render window moves (commit f54ca2f).
  • The snippet window is hysteresis-gated and decoupled from the render window (State.computeIMESnippetWindow): 32 KB, re-anchored only when the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a restartInput, which starts a fresh input session that re-derives auto-capitalization — a per-scroll or per-tap re-push is what caused the random mid-word caps (commit dd9493b).
  • The post-commit caret is Range.Start + len(text), not Range.End + len(text). Identical for insertions; for a replacement (autocorrect over a selection) the old formula lands (End−Start) runes past the inserted text and desynchronizes the IME at the exact moment its model is being updated (commit 51344b9). The logic side (applyIMECommitBytes) and the renderer's immediate push (ApplyIMECommitToModel) must compute the same caret.
  • A legitimate commit is always local: at the caret, inside the live selection, or a correction a few runes left of the caret. The guard (HandleIMECommit, maxCommitDistance = 1024) snaps anything else to the caret and arms the resync. A 5-rune autocorrect 19,000 runes from the caret is never legitimate; applying it verbatim clobbers distant text (commit a83cc1a).
  • A desynchronized IME is healed only by a restart. See the structural fact below: selection pushes after a commit are deduplicated away, so the only mechanism that makes Gboard re-read the truth is a snippet change (restartInput). IMEForceResync arms on the first anomaly, not a streak: the next frame ships the snippet trimmed by one rune (one restart), the following frame ships the full window (one more), then quiet. One-shot, so normal typing never pays for it (commits cb8ebc0, a83cc1a).

What stock gioui does structurally (worked around, not patched)

All of the following is stock v0.10.2 behavior. The fixes live entirely in pad; the fork at ~/gioui carries diagnostic logging only (the PADIME lines) and can be dropped for stock gioui at any time without behavioral change.

  1. Post-commit selection pushes are deduplicated away. The commit callback (callbacks.EditorReplace) advances the window's stored imeState.Selection before the frame's state comparison, so a SelectionCmd pushed in the same frame as a commit is "no change" and imm.updateSelection is never called. Consequence: after a commit, the IME's caret can only be corrected by a snippet restart. This is why the resync exists and why "just re-push the selection" is not a fix.
  2. restartInput shadows updateSelection. In window.EditorStateChanged, a snippet change takes an early-return branch: the selection update in the same state change is dropped. Gboard re-fetches the selection on restart, so this is harmless — but it means a restart frame is the only frame where selection and text are guaranteed to be re-read together.
  3. Selection commands are focus-gated. keyQueue.setSelection silently drops a SelectionCmd whose tag is not the current key-focus (req.Tag != state.focus). A selection push can vanish with no log and no feedback — the IME never acks selections. If taps stop moving Gboard's caret, check focus first.
  4. There is no ack anywhere. Snippet pushes, selection pushes, and commits are fire-and-forget. The system can only detect desync indirectly (a commit landing where no text expects it), which is what the drift guard does.

Gboard behaviors observed

  • Gboard keeps a word selection on its own (a tap near a word selects it) and commits autocorrect as a replacement over its local selection — the commit range is then as wide as the word, not a point insertion.
  • While composing, Gboard re-sends the whole word on every keystroke ("t", "th", "thi", … each replacing the previous), and on backspace it re-sends the shrinking word down to text="". Both are normal; the drift guard must not treat them as anomalies (they are at the caret).
  • When its local model desynchronizes, Gboard can enter an endless empty-fix-up loop (text="", one commit per ~150 ms) trying to reconcile text that is not in its model. The file is never damaged (the guard snaps each one to the caret); the resync breaks the loop.
  • The phone's AICore Gboard is stricter/more aggressive than the emulator's Gboard. A flow that is clean on the emulator can still desync on device.

Log signatures → diagnosis

Permanent, low-volume lines (see "Diagnostics"):

Signature in logcat Meaning
IME TAP … cursor=N winStart=M A tap moved the logic cursor (bytes). Compare with the next IME COMMIT ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, compute it, don't assume a constant).
IME COMMIT range=[a,b) text=… -> [x,y) The logic applied the commit at bytes [x,y). The pre--> range is Gboard's coordinate space; the post--> range is where it actually landed.
IME DRIFT-SNAP … snap to caret + resync Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping.
IME PUSH snippet=[a,b) … (restart) We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling).
IME PUSH sel=[s,e) We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit).
PADIME EditorStateChanged sel A -> B / PADIME updateSelection sel=[…] / PADIME restartInput The gio→Gboard boundary: what actually crossed into Android. If IME PUSH shows a value the PADIME lines never show, the drop is inside gioui (focus gate / dedup); if PADIME shows it and Gboard still misbehaves, the IME ignored it.

Capture the log at incident time. Logcat rotates fast; the incident that started this document was only reconstructable because the capture happened minutes later.

Test layers and what each can catch

Layer What it exercises Catches
e2e harness (internal/test/e2e) Logic + renderer headless; commits injected directly (HandleIMECommit) Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. Never the IME contract: Gboard is not in the loop.
Emulator stress loops (/tmp/loop4.sh, /tmp/stress.sh) Real app + adb touches, real Gboard Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely).
Full-path scenario (/tmp/s8.sh) Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs exact file diff The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed.

Rule: a bug in the IME boundary is invisible to every layer except the last one. When IME behavior is touched, run S8.

Operational pitfalls hit in these sessions

  • The launcher activity is pad.pad/org.gioui.GioActivity, not .MainActivity. am start failures are silent if stderr is redirected — check the output once, trust it after.
  • adb install/build steps fail with "no adb device" when the emulator is mid-reconnect; the APK may be built but not installed. Verify dumpsys package … lastUpdateTime after installs.
  • The "All files access" settings toggle can be a red herring: appops set <pkg> MANAGE_EXTERNAL_STORAGE allow is the reliable grant, and appops get output has been observed to lag; dumpsys appops is authoritative.
  • difflib.SequenceMatcher on ~1 MB strings is minutes slow; use a linear first-diff scan for test assertions on large files.
  • Float-equality re-emit is a recurring trap (LastLineY, caret px): any "re-emit when changed" comparison on a float sum needs an epsilon gate, or it spins a loop that re-pushes IME state every frame and starves real work.
  • When a tap's pixel→text mapping is in question, the IME TAP line gives both the local point and the resulting cursor — log both, or the mapping bug is undiagnosable from the outside.

Diagnostics currently enabled (temporary)

  • Pad-side, permanent: IME TAP, IME COMMIT, IME DRIFT-SNAP, IME PUSH snippet/sel/commit-range (bounded: only on change).
  • Fork-side, temporary (in ~/gioui, gated by the replace in go.mod): PADIME boundary logs in GioView.updateSelection, GioView.restartInput, and window.EditorStateChanged.

Cleanup steps (when informal on-device testing is settled)

The fork is stock v0.10.2 plus the PADIME lines and nothing else (verify first: git -C ~/gioui diff --stat must show only app/GioView.java and app/os_android.go); behavior is identical on stock gioui.

# 1. Restore the fork to clean v0.10.2
git -C ~/gioui checkout -- .
git -C ~/gioui diff --stat            # must be empty

# 2. Drop the replace and re-tidy pad against stock gioui
cd ~/pad
go mod edit -dropreplace gioui.org
go mod tidy

# 3. Verify everything green on stock gioui
go build ./...
go test -race ./...
./scripts/check.sh

# 4. Rebuild both APKs and smoke-test
./scripts/build_emu.sh                # install to emulator
bash /tmp/s8.sh                       # full-path scenario must still PASS
./scripts/build_phone.sh              # install to phone, type + tap once

# 5. Commit and push the go.mod/go.sum change
git add -A && git commit -m "build: drop the temporary gioui diagnostic fork" && git push origin main

# 6. Remove this cleanup section (and the "temporary" fork-side bullet
#    above) from ime.md; commit + push.

If step 1 shows more than the two diagnostic files, do NOT checkout -- . blindly: review the diff and keep anything that is a real (non-log) change, porting it upstream or documenting it in this file first.