Pad/internal/ui/pinch_tracker.go
Greg Pomerantz 180fa966c8 Pinch-to-font-size (continuous, content-point pinned) + IME-open scroll fix
Two feature bodies accumulated in the working tree:

1. Pinch to change the app font size, continuously (no snapping):
   - internal/ui/pinch_tracker.go: logic-free touch state machine.
     Two-mover formation (the resting palm can land first or last;
     movement is the only signal valid for both), pair = the mover
     pair whose distance changed most, baseline = press distance
     (formDist), lazy pending releases, survivor-scroll forwarding
     after a pair break. Robust to ~1 fps frames: a whole pinch can
     land in one drain (formDist/brokeFactor/lazy releases).
   - render.go: pinch probe (raw pointer events) + grab lifecycle so
     the pair is exclusive (scroll sees nothing of the pair) and the
     survivor's finger keeps working as a scroll after the pinch.
   - state.go/logic.go/session.go/frame.go: app-local float font
     scale, content-point pin (buffer byte + offset from baseline,
     not a layout point, so rewrap keeps the same character under
     the center), restore/font pins, session persistence.
   - pinch_test.go, pinch_font_test.go, tag_identity_test.go,
     real_draw_probe_test.go: unit + real-Renderer/real-Router tests.

2. Soft keyboard must not shift content:
   - Root cause: gioui.org/app calls Router.RevealFocus on any frame
     the viewport shrinks (IME open under adjustResize) and
     synthesizes a pointer.Scroll nudge aimed at the focused field's
     stale pre-resize bounds; gesture.Scroll consumed it -> a 32 dp
     content jump.
   - Fix: main.go flags the shrink frame; render.go drains that one
     synthetic scroll for the gesture's tag before Update (scroll-
     range clamping cannot work: the router UNIONs ranges across
     frames). Finger scroll (pointer.Drag) and the flinger are
     untouched. reveal_focus_drain_test.go reproduces RevealFocus at
     the router level and verifies the drain + zero delta.

Also: tools/touchinject (platform-signed emulator multi-touch
injection harness + e2e script, adb has no two-finger input),
docs (spec 2.2 + development_plan 18-20), .gitignore, gofmt.
2026-08-23 09:00:51 -04:00

471 lines
16 KiB
Go

package ui
import (
"math"
"time"
"gioui.org/f32"
"gioui.org/io/pointer"
)
// pinchFreshWindow is how recently BOTH pair fingers must have pressed for a
// pinch to start. A finger that has been resting (palm edge, pinky parked on
// the screen) is not a pinch candidate; without this window a resting third
// finger could be paired with a new one and the font would follow a
// single-finger scroll.
const pinchFreshWindow = 300 * time.Millisecond
// pinchMoveEps is how far (window px, from the press position) a pending
// finger must travel to count as "moving" for pair formation. It filters
// touch jitter while still registering an intentional pinch start quickly.
const pinchMoveEps = 10.0
// Pair formation requires TWO MOVING fingers, and nothing else. A pinch is
// two fingers moving apart (or together); a scroll is one finger moving;
// a resting palm is a finger that never moves. No static rule about which
// finger is "the palm" (first? last? still?) survives both palm-first and
// palm-last hand lands — movement is the only signal that works for both.
// While 2-3 fresh fingers are down the tracker is "pending": it forms the
// pair the moment two of them have moved more than pinchMoveEps from where
// they pressed (the pair = those two). A lone mover is a scroll, never a
// pinch, and cannot form a pair; the pending state cancels on any release
// that leaves fewer than two fingers.
// pinchTracker is the pure state machine for the editor's two-finger pinch
// (font scaling). It is independent of the Gio input router so the whole
// gesture — including the failure modes found on device (pair switching,
// stale baselines, scroll-grab starvation) — is unit-testable with synthetic
// pointer-event sequences.
//
// Design: the pair is EXPLICIT and stable. When two fresh fingers are down
// inside the region, they are named as the pair and the adapter issues
// pointer.GrabCmds for both (exclusive event delivery: releases always
// arrive, even off-clip, and scroll/click are dropped with a Cancel). The
// per-frame factor is current pair distance / previous frame's pair
// distance. Neither the pair's composition nor its baseline is ever
// re-derived from whatever pointers happen to be present — that re-derivation
// (two lowest IDs + stale prevDist) is what made single-finger scrolls scale
// the font on the phone.
type pinchTracker struct {
// Pre-pinch ("pending"): fresh fingers down in the region, no pair yet.
// The pair forms when TWO of them have moved more than pinchMoveEps
// from where they pressed (see the comment above the constants).
observed map[pointer.ID]f32.Point
observedAt map[pointer.ID]time.Duration
// Position each pending finger had when it PRESSED: the formation
// baseline (formDist) is the press distance, so a pinch that has
// already spread by the time the pair starts owes that spread; and the
// displacement from here is the "mover" measurement.
pressPos map[pointer.ID]f32.Point
// Pending fingers that released in the CURRENT frame's drain, held lazy
// until factor() (which forms the pair — and may break it again — at
// the fingers' final positions). A normal active-pair release breaks
// immediately in step(); this map is only for releases that arrive
// while the pair is still pending.
released map[pointer.ID]bool
// Reform candidate: after a pair broke, the survivor is still grabbed
// and a NEW finger has landed. The pair re-forms (survivor + new) only
// when the new finger moves (pinchMoveEps) — the same two-mover rule;
// a palm landing on the survivor is not a pinch. reformPos is where
// the candidate PRESSED (the mover reference).
reformID pointer.ID
reformPos f32.Point
reformBase float32 // pair distance at the candidate's press
reformHave bool
// The active (grabbed) pair.
pair [2]pointer.ID
pos [2]f32.Point
on bool
prevDist float32
// The pair's distance at the moment of (re)formation: the distance of
// the two press positions. The Android driver replays
// historical samples, so drags of the forming pair can land in the same
// frame as the pair's formation; movement relative to the formation
// distance is real and owed.
formDist float32
// Set on the pair's formation frame (start or re-form): the baseline
// is being established, so factor() emits nothing unless the pair
// moved from its formation positions this frame.
fresh bool
// The pair broke (a finger lifted) during the current frame's drain.
// factor() runs AFTER the drain, so on a slow frame the pair's drags
// and the breaking release arrive together and the frame's factor
// would be lost (on==false by the time factor() runs). The factor is
// settled here instead.
brokeFactor float32
brokeMid f32.Point
// A pair finger was released while the other is still down (still
// grabbed by the probe; Gio v0.10 has no release-grab). The survivor's
// drags are forwarded as scroll so the finger is not dead.
survOn bool
survID pointer.ID
survPos f32.Point
// Accumulated survivor scroll for the current frame (window px,
// gesture.Scroll convention: positive = content scrolls up).
survScroll int
}
// pinchStep is the outcome of one pointer event.
type pinchStep struct {
// Grabs the adapter must issue (pointer.GrabCmd per ID). Only the
// survivor re-form path emits grabs per event; initial pair formation
// is decided in factor() (after the whole frame's events).
grabs []pointer.ID
}
// step feeds one pointer event into the tracker. The per-frame factor is not
// part of the step result: the adapter calls factor() once after draining
// the frame's events, because the Android driver replays historical samples
// (several drags per frame) and the font must see one factor per frame.
func (t *pinchTracker) step(pe pointer.Event) pinchStep {
var s pinchStep
id := pe.PointerID
switch pe.Kind {
case pointer.Press:
switch {
case t.on:
// Extra finger during an active pinch (palm rest, third
// finger): ignore — the pair is fixed.
case t.survOn && t.reformHave && id == t.reformID:
// The reform candidate lifted before moving: not a pinch.
t.reformHave = false
case t.survOn:
// A new finger lands while the survivor is down: CANDIDATE for
// a re-form; it must MOVE to become the pair (the same two-mover
// rule — a palm resting on the survivor is not a pinch). The
// formation baseline is the distance at this PRESS (any spread
// by re-form time is owed).
t.reformID = id
t.reformPos = pe.Position
t.reformBase = pairDist(t.survPos, pe.Position)
t.reformHave = true
default:
if t.observed == nil {
t.observed = make(map[pointer.ID]f32.Point)
t.observedAt = make(map[pointer.ID]time.Duration)
t.pressPos = make(map[pointer.ID]f32.Point)
}
t.observed[id] = pe.Position
t.observedAt[id] = pe.Time
t.pressPos[id] = pe.Position
// Prune stale fingers (resting palm/pinky); a pruned palm is
// out of the game entirely.
t.pruneStale(pe.Time)
// (Two or more fingers down is the PENDING state; the pair
// forms only when two of them are movers, decided in factor().)
}
case pointer.Drag:
switch {
case t.on:
if i := t.pairIndex(id); i >= 0 {
t.pos[i] = pe.Position
}
case t.survOn && id == t.survID:
t.survScroll += int(t.survPos.Y - pe.Position.Y)
t.survPos = pe.Position
case t.survOn && t.reformHave && id == t.reformID:
// The reform candidate moves: if it clears the epsilon from
// where it pressed it is a real second finger — re-form the
// pair (survivor + new).
dx, dy := pe.Position.X-t.reformPos.X, pe.Position.Y-t.reformPos.Y
if dx*dx+dy*dy > pinchMoveEps*pinchMoveEps {
t.pair = [2]pointer.ID{t.survID, id}
t.pos = [2]f32.Point{t.survPos, pe.Position}
t.survOn = false
t.reformHave = false
t.on = true
t.fresh = true
t.formDist = t.reformBase
s.grabs = []pointer.ID{id} // survivor already grabbed
}
default:
if _, ok := t.pressPos[id]; ok {
t.observed[id] = pe.Position
// Formation is NOT decided here: a per-event decision would
// lock in the first mover pair seen (e.g. (finger, drifting
// palm)) before the second pinch finger's drag lands in the
// same drain. factor() decides, after the full frame.
}
}
case pointer.Release, pointer.Cancel:
switch {
case t.on:
if i := t.pairIndex(id); i >= 0 {
t.breakPair(i)
}
case t.survOn && id == t.survID:
// The survivor lifts too: fully idle (the candidate, if any,
// was a lone finger — never a pair).
t.survOn = false
t.reformHave = false
default:
// Pending finger goes up: held LAZY until factor() — on a slow
// frame the whole pinch (drags AND releases) can land in one
// drain, and the pair must form at the fingers' final positions
// before the break settles the owed factor.
if _, ok := t.observed[id]; ok {
t.markReleased(id)
}
}
case pointer.Leave:
// Only meaningful pre-pinch: a finger that leaves the editor
// region is no longer a pinch candidate. (A grabbed pair finger
// keeps delivering through the grab; its Leave is ignored.)
if t.survOn && t.reformHave && id == t.reformID {
t.reformHave = false
} else if t.observed != nil {
if _, ok := t.observed[id]; ok {
t.markReleased(id)
}
}
}
return s
}
// breakPair settles the frame's owed factor (if any) and breaks the pair.
// The surviving finger stays grabbed by the probe (no release-grab in v0.10)
// and becomes the survivor — UNLESS it too released in this same drain
// (slow frame: a born-and-dead pair), in which case the tracker goes fully
// idle. Settling here (before factor() runs) is what keeps the factor on a
// frame where the drags and the breaking release arrive together.
func (t *pinchTracker) breakPair(i int) {
base := t.prevDist
if base == 0 && t.fresh {
base = t.formDist
}
if d := pairDist(t.pos[0], t.pos[1]); d > 0 && base > 0 && d != base {
if f := d / base; f >= 0.1 && f <= 10 {
t.brokeFactor = f
t.brokeMid = pairMid(t.pos[0], t.pos[1])
}
}
t.on = false
t.prevDist = 0
other := 1 - i
if !t.released[t.pair[other]] {
t.survOn = true
t.survID = t.pair[other]
t.survPos = t.pos[other]
}
}
// markReleased lazily records a pending finger's release (see the Release
// case). Cleared by factor() after it has formed (and possibly broken) the
// pair.
func (t *pinchTracker) markReleased(id pointer.ID) {
if t.released == nil {
t.released = make(map[pointer.ID]bool)
}
t.released[id] = true
}
// dropObserved removes a pending finger for good.
func (t *pinchTracker) dropObserved(id pointer.ID) {
if t.observed == nil {
return
}
delete(t.observed, id)
delete(t.observedAt, id)
delete(t.pressPos, id)
delete(t.released, id)
}
// movers returns the pending fingers that moved more than pinchMoveEps from
// where they pressed, most-displaced first.
func (t *pinchTracker) movers() []pointer.ID {
type dm struct {
id pointer.ID
d float64
}
var list []dm
for id, pp := range t.pressPos {
dx := float64(t.observed[id].X - pp.X)
dy := float64(t.observed[id].Y - pp.Y)
if d := math.Sqrt(dx*dx + dy*dy); d > pinchMoveEps {
list = append(list, dm{id, d})
}
}
for i := 0; i < len(list); i++ {
for j := i + 1; j < len(list); j++ {
if list[j].d > list[i].d {
list[i], list[j] = list[j], list[i]
}
}
}
out := make([]pointer.ID, 0, len(list))
for _, e := range list {
out = append(out, e.id)
}
return out
}
// tryFormPair forms the pair from the pending fingers if at least TWO are
// movers (two moving fingers = a pinch; one = a scroll, never a pair). Of
// the mover pairs, the one whose DISTANCE changed most wins: a pinch's pair
// distance changes, while a drifting palm's distance to a finger changes
// little. A mover pair whose distance does not change (fingers moving in
// unison) is a slide, not a pinch. Returns the IDs to grab, or nil when no
// pair forms.
func (t *pinchTracker) tryFormPairAny() []pointer.ID {
m := t.movers()
if len(m) < 2 {
return nil
}
bestA, bestB := m[0], m[1]
bestChange := -1.0
for i := 0; i < len(m); i++ {
for j := i + 1; j < len(m); j++ {
a, b := m[i], m[j]
now := float64(pairDist(t.observed[a], t.observed[b]))
press := float64(pairDist(t.pressPos[a], t.pressPos[b]))
if chg := math.Abs(now - press); chg > bestChange {
bestChange = chg
bestA, bestB = a, b
}
}
}
if bestChange <= pinchMoveEps {
return nil // unison movement: a slide, not a pinch
}
a, b := bestA, bestB
t.pair = [2]pointer.ID{a, b}
t.pos = [2]f32.Point{t.observed[a], t.observed[b]}
t.on = true
t.fresh = true
// Baseline = the PRESS distance of the pair: any spread that happened
// before the pair started is owed, not lost.
t.formDist = pairDist(t.pressPos[a], t.pressPos[b])
t.observed = nil
t.observedAt = nil
t.pressPos = nil
return []pointer.ID{a, b}
}
// pairIndex returns 0/1 if id is a member of the active pair, else -1.
func (t *pinchTracker) pairIndex(id pointer.ID) int {
if !t.on {
return -1
}
if t.pair[0] == id {
return 0
}
if t.pair[1] == id {
return 1
}
return -1
}
// pruneStale drops observed fingers that pressed more than
// pinchFreshWindow before now.
func (t *pinchTracker) pruneStale(now time.Duration) {
for id, at := range t.observedAt {
if now-at > pinchFreshWindow {
delete(t.observed, id)
delete(t.observedAt, id)
delete(t.pressPos, id)
}
}
}
// factor returns the current frame's relative pinch scale and the pair
// midpoint (window px), or ok=false when no factor applies (no active pair,
// first frame of a pinch, or an implausible jump). It must be called once
// per frame, after all of the frame's events have been stepped.
func (t *pinchTracker) factor() (f float32, mid f32.Point, ok bool, grabs []pointer.ID) {
// Pending pair formation is decided HERE, after the whole frame's
// events (not per event): a per-event decision would lock in the first
// mover pair seen — e.g. (finger, drifting palm) — before the second
// pinch finger's drag lands in the same drain.
if !t.on && !t.survOn && t.observed != nil {
if g := t.tryFormPairAny(); g != nil {
grabs = g
// Slow frame: a pair member may have released in this same
// drain (the release arrived while the pair was still pending):
// break it now at the fingers' final positions.
for i, id := range t.pair {
if t.released[id] {
t.breakPair(i)
break
}
}
}
}
// Pending fingers that released this frame are dropped for good.
if t.released != nil {
for id := range t.released {
t.dropObserved(id)
}
t.released = nil
}
if !t.on {
// The pair broke during this frame's drain: emit the settled
// factor (on a slow frame the drags and the breaking release can
// arrive together, and factor() runs only after the drain).
if t.brokeFactor > 0 {
f, mid, ok = t.brokeFactor, t.brokeMid, true
t.brokeFactor = 0
t.brokeMid = f32.Point{}
}
return f, mid, ok, grabs
}
d := pairDist(t.pos[0], t.pos[1])
if t.fresh {
// Formation frame: establish the baseline. Emit only if the pair
// already moved from its formation positions this frame (the
// Android driver replays historical samples, so drags can batch
// with the formation Press).
t.fresh = false
t.prevDist = d
if t.formDist > 0 && d != t.formDist {
f = d / t.formDist
if f < 0.1 || f > 10 {
return 0, f32.Point{}, false, grabs
}
return f, pairMid(t.pos[0], t.pos[1]), true, grabs
}
return 0, f32.Point{}, false, grabs
}
if t.prevDist > 0 && d > 0 && d != t.prevDist {
// No-movement guard: a stationary pair emits nothing (an f=1.0
// factor would churn the font pin's re-layout for no change).
f = d / t.prevDist
// Sanitize: a real pinch moves millimeters between frames; a
// factor this far off 1 is noise, not a finger.
if f < 0.1 || f > 10 {
t.prevDist = d
return 0, f32.Point{}, false, grabs
}
t.prevDist = d
return f, pairMid(t.pos[0], t.pos[1]), true, grabs
}
t.prevDist = d
return 0, f32.Point{}, false, grabs
}
// survivorScroll returns and resets the frame's accumulated survivor-finger
// scroll (window px, gesture.Scroll convention: positive = content scrolls
// up).
func (t *pinchTracker) survivorScroll() int {
d := t.survScroll
t.survScroll = 0
return d
}
// reset clears all state (the editor left the screen).
func (t *pinchTracker) reset() {
*t = pinchTracker{}
}
// pairDist is the distance between two pair points (window px).
func pairDist(a, b f32.Point) float32 {
dx, dy := a.X-b.X, a.Y-b.Y
return float32(math.Sqrt(float64(dx*dx + dy*dy)))
}
// pairMid is the midpoint of two pair points (window px).
func pairMid(a, b f32.Point) f32.Point {
return f32.Point{X: (a.X + b.X) / 2, Y: (a.Y + b.Y) / 2}
}