Pad/doc/browser_implementation_plan.md
Greg Pomerantz 3f1c881a06 browser: document lazy loading implementation plan
- Add detailed §15 lazy loading plan with 7 implementation steps
- Update Phase 2 checklist with specific tasks
- Add E2E test harness for browser files
- Refactor browser layout and handler structure
- Add .gitignore for common patterns

WIP: Actual lazy loading not yet implemented.
2026-05-31 18:43:46 +00:00

1142 lines
42 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Browser Implementation Plan
## 1. Overview
This plan specifies the implementation of the directory browser for Pad, covering lazy loading, virtualized rendering, alphabetical indexing, and search functionality.
### Current State (as of 2026-05-30)
| Feature | Status | Location |
|---|---|---|
| Browser page rendering | ✅ Implemented | `internal/browser/layout.go:BrowserLayout()` |
| Scroll with virtualization | ✅ Implemented | `internal/editor/state.go` + `ui/element.go:ListView` |
| Search (case-insensitive) | ✅ Implemented | `internal/browser/search.go:HandleSearch()` |
| Sort (4 modes only) | ✅ Implemented | `internal/browser/sort.go` (4 modes, not 6) |
| Worker pool | ✅ Implemented | `internal/io/pool/` |
| Filesystem-backed entries | ❌ Not started | Uses static `browserEntries` slice |
| Lazy loading / pagination | ⏳ Partial | Page model + eviction implemented; disk I/O not wired |
| Directory index / caching | ✅ Implemented | `internal/browser/index.go:buildIndex()` |
| Alphabetical index sidebar | ⏸ Deferred | See §14 |
### Design Decisions Summary
| Decision | Rationale |
|---|---|
| **Lazy loading required** | Directories with 100k+ entries must not block first paint or exhaust memory |
| **Page-based loading** | Matches the editor's chunked buffer pattern; predictable memory usage |
| **Pre-sorted index with position maps** | Index stores raw metadata; pre-computed position maps for each of 4 sort modes; O(1) lookup during frame rendering; no sorting on frame path |
| **Virtualized ListView** | Only render visible items; matches existing `ListView` element |
| **Worker-dispatched reads** | Directory I/O goes through high-priority worker channel; logic stays <16ms |
| **Editor-owned state** | All browser state is embedded in the editor's state to maintain single-owner pattern |
---
## 2. Package Structure
```
internal/browser/
browser.go # Main browser logic, state machine, page management
index.go # Directory index (lazy-loaded, cached, sorted)
layout.go # Browser layout computation (produces []Element)
handlers.go # Interaction handlers (tap, scroll, search, alpha index)
types.go # Browser-specific types (Entry, Page, etc.)
```
---
## 3. Core Types
### 3.1 Directory Entry
```go
// Entry represents a single file or directory in the browser.
type Entry struct {
Path string // Relative path from configured directory
Name string // Display name (basename)
Size int64 // File size in bytes (0 for directories)
ModTime time.Time // Last modification time
IsDir bool // True if this is a directory
IsFirst bool // True if this is the first entry for its letter section
}
```
### 3.2 Browser Page (Lazy Load Unit)
```go
// Page represents a chunk of directory entries loaded from disk.
type Page struct {
Index int // Page number (0-based)
Entries []Entry // Entries in this page
Loaded bool // True if page data is in memory
Dirty bool // True if page needs refresh (external change detected)
}
const (
PageSize = 100 // Entries per page (tunable; ~11KB per page in memory)
PrefetchDist = 2 // Pages to prefetch beyond visible region
)
```
### 3.3 Browser State
```go
// BrowserState holds all mutable browser state owned by the logic goroutine.
// This struct is embedded in the editor's State to maintain the single-owner pattern.
type BrowserState struct {
// Navigation
CurrentPath string // Currently browsed directory (relative to root)
ScrollIndex int // Index of first visible entry (not pixel offset)
VisibleCount int // Number of entries currently visible
// Lazy loading
Pages map[int]*Page // Loaded pages by page index
TotalEntries int // Total entry count (from cached index)
Loading bool // True if a page load is in flight
// Search
Query string // Current search query
SearchResults []int // Indices of matching entries (empty = no filter)
// Alphabetical index (DEFERRED - see §14.1)
// ActiveLetter string // Currently pressed letter (for highlighting)
// LetterOffsets map[string]int // First entry index for each letter
// Interaction
SelectedIndex int // Currently selected entry (-1 = none)
TapTimestamp time.Time // For double-tap detection
}
```
---
## 4. Lazy Loading Architecture
### 4.1 Page Lifecycle
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ VISIBLE │────▶│ PREFETCH │────▶│ EVICTED │
│ (in memory)│ │ (in memory)│ │ (on disk) │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
│ │ │
└────────────────────┴───────────────────┘
scroll triggers transitions
```
**Page states:**
1. **Visible**: Page is within the current viewport; entries are rendered
2. **Prefetched**: Page is within `PrefetchDist` of the viewport; entries are loaded but not rendered
3. **Evicted**: Page is outside prefetch range; entries are released from memory
### 4.2 Loading Flow
```
User scrolls browser
Logic goroutine: update ScrollIndex, compute visible page range
Logic goroutine: identify unloaded pages in [visible - prefetch, visible + prefetch]
Logic goroutine: pool.Dispatch(NewLoadPagesTask(dirPath, pageIndices, fs))
Worker: Task.Execute() → Result{TaskType: TypeLoadPages, Data: pages}
Worker: wp.post(result) — posts to encapsulated result channel
Logic goroutine: receives Result via pool.ResultChan(), applies to BrowserState.Pages
Logic goroutine: sendFrame() with updated ListView entries
```
### 4.3 Memory Management
- **Page eviction**: When memory pressure is detected (via the monitoring system in architecture.md §10), pages beyond the prefetch range are evicted
- **Max pages in memory**: `VisibleCount / PageSize + 2 * PrefetchDist + 2` (visible + prefetch buffer)
- **Per-page memory**: ~11KB for 100 entries (Entry struct is ~113 bytes: Path string ~50B, Name string ~30B, Size int64 8B, ModTime time.Time 24B, IsDir/IsFirst bool 2B)
- **Total browser memory budget**: ~1.1MB for typical viewport (10 pages)
- **Position maps**: 100k × 4 bytes × 4 sort modes = 1.6MB (shared across all pages)
### 4.4 Eviction Policy
```go
// Evict pages that are far from the current scroll position.
func (s *BrowserState) EvictPages() {
minPage := s.ScrollIndex/PageSize - PrefetchDist
maxPage := (s.ScrollIndex + s.VisibleCount)/PageSize + PrefetchDist
for idx, page := range s.Pages {
if idx < minPage || idx > maxPage {
page.Entries = nil // Release memory
page.Loaded = false
}
}
}
```
---
## 5. Directory Index
### 5.1 Sorted Index Architecture
The browser uses a **pre-sorted index with position maps** to deliver entries in the correct sort order without sorting during frame rendering. This ensures the 16ms frame budget is maintained even for directories with 100k+ entries.
**Core concept:** Instead of storing entries in sorted order (which requires sorting on every frame or mode change), we store entries in raw filesystem order and maintain pre-computed position maps that map sorted indices to raw indices.
```
┌─────────────────────────────────────────────────────────────┐
│ SORTED INDEX │
├─────────────────────────────────────────────────────────────┤
│ Entries[] (raw filesystem order, unsorted) │
│ ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐ │
│ │ 0 │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ ... │
│ └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘ │
│ │
│ Position Maps (one per sort mode, 4 total) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ NameAsc: [2, 5, 0, 7, 3, 1, 6, 4, ...] │ │
│ │ NameDesc: [4, 6, 1, 3, 7, 0, 5, 2, ...] │ │
│ │ DateAsc: [5, 0, 7, 2, 3, 1, 6, 4, ...] │ │
│ │ DateDesc: [4, 6, 1, 3, 7, 0, 5, 2, ...] │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
**Frame path (sub-16ms):**
```
BrowserLayout → computeVisibleEntries → getEntryByIndex(sortedPos, mode)
positionMap = SortOrders[mode]
rawPos = positionMap[sortedPos]
return Entries[rawPos]
```
### 5.2 Index File Format
The directory index is cached on disk to avoid re-reading and sorting on every browse:
```
.pad/indices/browser_<dir_hash>.json
```
```json
{
"path": "/user/documents",
"mtime": 1715000000,
"size": 4096,
"entry_count": 150000,
"entries": [
{"path": "a/file.txt", "name": "a", "size": 0, "mod_time": 1715000000, "is_dir": true},
...
],
"sort_orders": {
"name_asc": [2, 5, 0, 7, 3, 1, 6, 4, ...],
"name_desc": [4, 6, 1, 3, 7, 0, 5, 2, ...],
"date_asc": [5, 0, 7, 2, 3, 1, 6, 4, ...],
"date_desc": [4, 6, 1, 3, 7, 0, 5, 2, ...]
}
}
```
**Memory calculation for 100k entries:**
- Raw entries: 100k × 113 bytes = 11.3MB
- Position maps: 100k × 4 bytes × 4 modes = 1.6MB
- **Total: ~13MB** (acceptable on modern devices)
### 5.3 Index Build/Invalidation
| Event | Action |
|---|---|
| First browse of directory | Build index: read directory, compute position maps, write to `.pad/indices/` |
| Directory mtime changed | Rebuild index on next browse |
| External file created/deleted | Watcher triggers index rebuild for affected directory |
| Index file missing | Build from scratch |
### 5.4 Index Build Process
```go
// buildIndex reads the directory and produces a cached index with position maps.
func buildIndex(dirPath string) (*DirectoryIndex, error) {
// 1. Read directory entries (os.ReadDir)
entries, err := os.ReadDir(dirPath)
if err != nil {
return nil, err
}
// 2. Convert to Entry structs (unsorted)
var browserEntries []Entry
for _, e := range entries {
info, _ := e.Info()
browserEntries = append(browserEntries, Entry{
Path: filepath.Join(dirPath, e.Name()),
Name: e.Name(),
Size: info.Size(),
ModTime: info.ModTime(),
IsDir: info.IsDir(),
})
}
// 3. Build position maps for all 4 sort modes
sortOrders := make(map[SortMode][]int)
for mode := SortModeNameAsc; mode <= SortModeDateDesc; mode++ {
sortOrders[mode] = buildPositionMap(browserEntries, mode)
}
// 4. Write to cache
idx := &DirectoryIndex{
Path: dirPath,
Mtime: dirInfo.ModTime(),
EntryCount: len(browserEntries),
Entries: browserEntries,
SortOrders: sortOrders,
}
cacheIndex(idx)
return idx, nil
}
// buildPositionMap creates a sorted index → raw index mapping.
func buildPositionMap(entries []Entry, mode SortMode) []int {
n := len(entries)
indices := make([]int, n)
for i := range indices {
indices[i] = i
}
// Sort indices based on entry comparison
sort.SliceStable(indices, func(i, j int) bool {
a, b := entries[indices[i]], entries[indices[j]]
return comparator(mode)(a, b) < 0
})
return indices
}
```
---
## 6. Layout Computation
### 6.1 Browser Layout Function
```go
// BrowserLayout computes []Element for the directory browser page.
// Pure function: (screen dimensions, browser state) → []Element
// This function is in the browser package but called from the editor package.
func BrowserLayout(screenW, screenH ui.Dp, state *BrowserState) []ui.Element {
// Compute regions
headerH := ui.Dp(48)
searchBarH := ui.Dp(40)
bottomBarH := ui.Dp(24)
alphaIndexW := ui.Dp(24)
// Header
header := ui.NewLabel(
state.CurrentPath,
18,
ui.Region{X: ui.Dp(10), Y: ui.Dp(10), W: screenW - ui.Dp(20), H: headerH},
ui.AlignStart,
)
// Search bar (if query is non-empty or search is active)
var searchBar ui.Element
searchY := headerH + ui.Dp(10)
if state.Query != "" {
searchBar = ui.NewTextField(ui.Region{
X: ui.Dp(10), Y: searchY,
W: screenW - ui.Dp(34), H: searchBarH,
}, state.Query, "Search...")
searchY += searchBarH + ui.Dp(4)
}
// ListView region
listW := screenW - alphaIndexW - ui.Dp(20)
listH := screenH - searchY - bottomBarH - ui.Dp(10)
listRegion := ui.Region{
X: ui.Dp(10), Y: searchY,
W: listW, H: listH,
}
// Compute visible entries
var visibleEntries []ui.ListItem
startIndex := state.ScrollIndex
endIndex := startIndex + state.VisibleCount
// Use search results if filtering
var effectiveStart, effectiveEnd int
if len(state.SearchResults) > 0 {
effectiveStart = state.SearchResults[0]
effectiveEnd = state.SearchResults[len(state.SearchResults)-1]
} else {
effectiveStart = startIndex
effectiveEnd = endIndex
}
// Load pages as needed (sync if available, async if not)
for idx := effectiveStart; idx < effectiveEnd && idx < state.TotalEntries; idx++ {
pageIdx := idx / PageSize
page, loaded := state.Pages[pageIdx]
if !loaded || !page.Loaded {
// Show placeholder or skip; page load is in flight
continue
}
entry := page.Entries[idx%PageSize]
subtext := formatSize(entry.Size)
if entry.IsDir {
subtext = "Directory"
}
visibleEntries = append(visibleEntries, ui.ListItem{
Text: entry.Name,
Subtext: subtext,
Selected: idx == state.SelectedIndex,
})
}
listView := ui.NewListView(listRegion, visibleEntries, state.ScrollIndex, state.SelectedIndex)
// Alpha index (DEFERRED - see §14.1)
// alphaIndex := ui.NewAlphaIndex(...)
return []ui.Element{header, searchBar, listView} // alphaIndex deferred
}
```
---
## 7. Interaction Handlers
### 7.1 Tap Handler (File Selection)
```go
// HandleBrowserTap processes a tap on the ListView.
func HandleBrowserTap(data any) {
state := editor.TheState.Load()
state.Lock()
defer state.Unlock()
clickData := data.(gesture.ClickEvent)
// Convert click Y to entry index
entryIndex := state.Browser.ScrollIndex +
int(clickData.Location.Y/state.Browser.EntryHeight)
// Load entry if not in memory
pageIdx := entryIndex / PageSize
page, ok := state.Browser.Pages[pageIdx]
if !ok || !page.Loaded {
return // Page not loaded yet; will handle on next frame
}
entry := page.Entries[entryIndex%PageSize]
if entry.IsDir {
// Navigate into directory
state.Browser.CurrentPath = entry.Path
state.Browser.ScrollIndex = 0
state.Browser.Pages = make(map[int]*Page)
// Dispatch index build/load
dispatchLoadDirectory(entry.Path)
} else {
// Open file in editor
state.Editor.ActiveFile = entry.Path
state.Page = PageEditor
// Dispatch file open
dispatchOpenFile(entry.Path)
}
}
```
### 7.2 Scroll Handler
```go
// HandleBrowserScroll processes scroll events on the ListView.
func HandleBrowserScroll(data any) {
state := editor.TheState.Load()
state.Lock()
defer state.Unlock()
scrollData := data.(ScrollEvent)
delta := int(scrollData.Delta / state.Browser.EntryHeight)
oldScrollIndex := state.Browser.ScrollIndex
state.Browser.ScrollIndex += delta
// Clamp
maxScroll := state.Browser.TotalEntries - state.Browser.VisibleCount
if state.Browser.ScrollIndex < 0 {
state.Browser.ScrollIndex = 0
}
if state.Browser.ScrollIndex > maxScroll {
state.Browser.ScrollIndex = maxScroll
}
// Trigger prefetch for newly visible pages
if state.Browser.ScrollIndex != oldScrollIndex {
dispatchPrefetchPages(state.Browser)
state.Browser.EvictPages()
}
}
```
### 7.3 Alphabet Index Handler (DEFERRED)
**Status: DEFERRED** - The alphabetical index sidebar is not included in this implementation round (see §14.1). The handler would read `LetterOffsets` and set `ScrollIndex` directly, but this feature is not being implemented at this time.
### 7.4 Search Handler
```go
// HandleSearchInput filters entries by the current query.
func HandleSearchInput(data any) {
state := editor.TheState.Load()
state.Lock()
defer state.Unlock()
query := data.(string)
state.Browser.Query = query
if query == "" {
state.Browser.SearchResults = nil
return
}
// Filter entries (runs over loaded pages; for large directories,
// this may need to run over the cached index)
var results []int
queryLower := strings.ToLower(query)
for idx := 0; idx < state.Browser.TotalEntries; idx++ {
pageIdx := idx / PageSize
page, ok := state.Browser.Pages[pageIdx]
if !ok || !page.Loaded {
continue
}
entry := page.Entries[idx%PageSize]
if strings.Contains(strings.ToLower(entry.Name), queryLower) {
results = append(results, idx)
}
}
state.Browser.SearchResults = results
// Jump to first result
if len(results) > 0 {
state.Browser.ScrollIndex = results[0]
}
}
```
---
## 8. Worker Tasks
### 8.1 IO Delegation Model
The browser package follows the architecture's single-owner pattern:
1. **Logic goroutine** owns `BrowserState` the only goroutine that reads/writes browser state
2. **Logic goroutine** dispatches IO tasks via `pool.Dispatch()` or `pool.DispatchNonBlocking()`
3. **Workers** perform disk IO read directories, read/write index cache files
4. **Workers** call `Task.Execute()` which returns a generic `pool.Result`, then post via `wp.post(result)`
5. **Logic goroutine** receives results via `pool.ResultChan()` in its `select` loop and applies them to state
**Invariant**: Workers never access `BrowserState` directly. They only read from the filesystem and post structured results.
### 8.2 Result Routing
Browser tasks return generic `pool.Result` values with `TaskType` set to one of the browser task types (`TypeReadDir`, `TypeBuildIndex`, `TypeLoadIndex`, `TypeLoadPages`, `TypeStatDir`). The logic goroutine uses `result.IsBrowserResult()` to identify browser results and routes on `result.TaskType`:
```go
case res := <-wp.ResultChan():
if res.IsBrowserResult() {
switch res.TaskType {
case TypeBuildIndex:
applyDirectoryLoaded(res)
case TypeLoadPages:
applyPagesLoaded(res)
case TypeReadDir, TypeLoadIndex:
// Handle as needed
}
if res.IsError() {
applyBrowserError(res)
}
sendFrame()
}
```
**Note**: The plan originally described a custom `BrowserResult` type with fields like `Type`, `DirPath`, `Data`, `Err`. This was superseded by the generic `pool.Result` type during worker pool implementation. The generic type avoids per-domain type proliferation all the information a custom type would carry is already available via `Result.TaskType`, `Task.DirPath()`, `Result.Data`, and `Result.Error`.
### 8.3 Task Dispatch
Browser tasks are dispatched through the worker pool via `pool.Dispatch()` or `pool.DispatchNonBlocking()`. These methods encapsulate channel access internal channels are not exposed:
```go
// Dispatch a build index task (blocking — result is required)
pool.Dispatch(pool.NewBuildIndexTask(dirPath, fs))
// Dispatch a page load task (blocking — result is required)
pool.Dispatch(pool.NewLoadPagesTask(dirPath, pageIndices, fs))
```
**Do not use `DispatchNonBlocking` for browser tasks.** If a page load task is dropped, the logic goroutine will wait forever for a result that will never arrive, causing a livelock.
### 8.4 Task Implementations
Browser tasks implement the `pool.Task` interface defined in `internal/io/pool/task.go`:
```go
type Task interface {
Execute() Result // Performs the work, returns a Result
Priority() Priority // HighPriority or LowPriority
TaskID() string // Unique identifier for result matching
TaskType() TaskType // Routing key (e.g., TypeBuildIndex)
DirPath() string // Directory this task operates on
}
```
Workers call `Task.Execute()` which returns a `pool.Result`. The worker then calls `wp.post(result)` to send it to the logic goroutine. Tasks never write to channels directly channel access is encapsulated in the worker pool.
Existing task types in `internal/io/pool/task.go`: `ReadDirTask`, `BuildIndexTask`, `LoadIndexTask`, `LoadPagesTask`, `StatDirTask`. New browser-specific task types can be added there if needed.
### 8.5 Logic Goroutine Result Handling
The logic goroutine's `select` loop handles browser results as shown in §8.2. The key invariant is that every dispatched browser task produces exactly one result the worker pool guarantees this through blocking sends on both the work channel (via `Dispatch()`) and the result channel (via `wp.post()`).
### 8.6 Cache Persistence
The index cache (`.pad/indices/`) is written by workers during index build, not by the logic goroutine. This means:
- **Index build** is entirely async no blocking on the logic thread
- **Cache reads** are entirely async workers read from disk, post results via `wp.post()`
- **Logic never touches the filesystem** it only manages in-memory state
This is consistent with the architecture's partitioning: logic stays under 16ms, all IO is delegated.
---
## 9. State Persistence
### 9.1 Browser State in `state.json`
The browser scroll position is persisted in the existing `state.json`:
```json
{
"browser_scroll": 1200,
"browser_path": "/user/documents",
"browser_query": ""
}
```
### 9.2 Restore Flow
1. App launches read `state.json`
2. If `browser_path` exists and is valid load that directory
3. Restore `browser_scroll` position
4. If directory was deleted fall back to configured root directory
---
## 10. Implementation Phases
### Phase 1: Core Browser (MVP)
- [ ] `internal/browser/types.go` Define Entry, Page, BrowserState
- [ ] `internal/browser/index.go` Directory index build/cache
- [ ] `internal/browser/browser.go` State management, page loading
- [ ] `internal/browser/layout.go` BrowserLayout function
- [ ] Wire into main event loop (replace placeholder browser)
- [ ] Basic scroll and tap-to-open
### Phase 2: Lazy Loading (Detailed Plan)
See §15 for the complete lazy loading implementation plan with step-by-step details.
- [ ] Create `BrowserManager` orchestrator (Step 1)
- [ ] Wire initial directory load via `BuildIndexTask` (Step 2)
- [ ] Implement scroll-based prefetching (Step 3)
- [ ] Handle worker results in logic goroutine (Step 4)
- [ ] Implement page eviction (Step 5)
- [ ] Implement dirty page refresh (Step 6)
- [ ] Write E2E tests with mock filesystem (Step 7)
### Phase 3: Deferred — Alphabetical Index Sidebar
**Status: DEFERRED** The alphabetical index sidebar (tap-a-letter to jump) is not included in this implementation round. It can be added later as a standalone feature without changes to the core browser architecture.
Deferred items:
- Alphabetical index sidebar element
- Letter offset computation
- Tap-to-jump functionality
### Phase 4: Search
- [ ] Search bar UI
- [ ] Incremental filtering
- [ ] Search result navigation
### Phase 5: Polish
- [ ] Scroll momentum (reuse from touch.md §6)
- [ ] External change detection (watcher integration)
- [ ] State persistence and restore
- [ ] Performance testing with large directories
---
## 11. Performance Targets
| Operation | Target | Mechanism |
|---|---|---|
| First paint (empty directory) | < 100 ms | Show header + empty list immediately |
| First paint (100k entries) | < 200 ms | Load first page async, show loading skeleton |
| Scroll (60 fps) | 16 ms/frame | Virtualized ListView, only visible items rendered |
| Page load | < 50 ms | Cached index, sequential read |
| Search (100k entries) | < 100 ms | Index-level filtering, no disk I/O |
| Alpha index jump | < 10 ms | Direct index lookup, no iteration |
---
## 12. Edge Cases
| Scenario | Behavior |
|---|---|
| Directory with 0 files | Show empty state message |
| Directory with 1 file | Show single entry, no scroll |
| Directory with 1M+ entries | Lazy load, paginate, evict aggressively |
| Directory deleted externally | Show error, offer to go back or refresh |
| File renamed externally | Index rebuild on next browse |
| Rapid scroll | Debounce page loads; coalesce requests |
| Memory pressure | Evict non-visible pages immediately |
| Index file corrupted | Rebuild from scratch, log warning |
---
## 13. Testing Strategy — Test-First (TDD)
### 13.1 Methodology
**Rule: Tests are written before any implementation code. Every test must fail before its implementation is written.**
This ensures:
1. Tests are meaningful (they fail without code)
2. Implementation is minimal (only what's needed to pass)
3. No accidental pass (test passes without real logic)
**Process for each feature:**
1. Write the test file with the expected behavior
2. Run `go test ./internal/browser/...` confirm ALL tests fail
3. Implement the minimum code to make tests pass
4. Run tests again confirm ALL tests pass
5. Refactor if needed, keeping tests green
### 13.2 Test File Organization
```
internal/browser/
types_test.go # Tests for Entry, Page, BrowserState constructors
index_test.go # Tests for directory index build/cache/invalidation
browser_test.go # Tests for page loading, eviction, state management
layout_test.go # Tests for BrowserLayout pure function
scroll_test.go # Tests for scroll clamping and delta computation
search_test.go # Tests for query filtering and result navigation
fixtures/ # Synthetic directory structures for testing
empty/ # Empty directory
small/ # 10 entries
large/ # 100k+ entries (generated at test time)
```
### 13.3 Phase 1 Tests — Types (types_test.go)
**Write first. Must all fail.**
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestNewEntryFromFileInfo` | Entry constructed from `fs.DirEntry` | No `Entry` struct exists |
| `TestNewPage` | Page fields initialized correctly | `Page` struct missing |
| `TestPageSizeConstant` | `PageSize` is 100 | Constant not defined |
| `TestNewBrowserState` | Default state values (ScrollIndex=0, etc.) | `BrowserState` struct missing |
### 13.4 Phase 1 Tests — Index (index_test.go)
**Write after types tests pass. Must all fail.**
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestBuildIndex_EmptyDir` | Empty directory 0 entries, no error | `buildIndex` not implemented |
| `TestBuildIndex_SortedByName` | Entries sorted case-insensitively by name | Sort logic missing |
| `TestBuildIndex_DirsBeforeFiles` | Directories appear before files | Sort priority missing |
| `TestBuildIndex_CacheWritten` | Index written to `.pad/indices/` | Cache write not implemented |
| `TestBuildIndex_CacheInvalidatedOnMtimeChange` | Changed mtime triggers rebuild | Invalidation logic missing |
| `TestBuildIndex_CacheReusedWhenUnchanged` | Same mtime cached index reused | Cache read not implemented |
| `TestBuildIndex_LargeDirectory` | 10k entries built within time budget | Not tested at scale |
**Fixture setup:**
```go
func setupTestDir(t *testing.T, name string, entries []testEntry) string {
dir := t.TempDir()
// Create files and subdirectories
return dir
}
```
### 13.5 Phase 1 Tests — Browser State (browser_test.go)
**Write after index tests pass. Must all fail.**
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestLoadPage_FromIndex` | Page loaded from index at correct offset | Page loading not implemented |
| `TestLoadPage_LastPagePartial` | Last page has fewer than PageSize entries | Boundary handling missing |
| `TestLoadPage_OutOfBounds` | Requesting page beyond total returns error | Bounds check missing |
| `TestEvictPages_RemovesDistantPages` | Pages outside prefetch range are evicted | Eviction not implemented |
| `TestEvictPages_KeepsVisiblePages` | Visible pages are never evicted | Eviction logic incomplete |
| `TestEvictPages_KeepsPrefetchedPages` | Prefetched pages survive eviction | Prefetch boundary missing |
| `TestBrowserState_Reset` | Reset clears pages, scroll, selection | Reset method missing |
### 13.6 Phase 1 Tests — Layout (layout_test.go)
**Write after browser tests pass. Must all fail.**
These are pure function tests no mocks needed.
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestBrowserLayout_ElementCount` | Correct number of elements emitted | `BrowserLayout` not implemented |
| `TestBrowserLayout_HeaderRegion` | Header region matches expected position | Layout computation missing |
| `TestBrowserLayout_ListRegion` | ListView region matches expected position | Region calculation missing |
| `TestBrowserLayout_VisibleEntriesCount` | Only visible entries in ListView | Virtualization not implemented |
| `TestBrowserLayout_EmptyDirectory` | Empty state shown when no entries | Empty case not handled |
| `TestBrowserLayout_SingleEntry` | Single entry fills list correctly | Boundary case missing |
| `TestBrowserLayout_PartialPage` | Partial page at end of list | Partial page handling missing |
| `TestBrowserLayout_UnloadedPage` | Unloaded pages show placeholder | Placeholder logic missing |
### 13.7 Phase 2 Tests — Scroll (scroll_test.go)
**Write after Phase 1 tests pass. Must all fail.**
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestScrollClamp_Minimum` | ScrollIndex never goes below 0 | Clamping not implemented |
| `TestScrollClamp_Maximum` | ScrollIndex never exceeds max | Max calculation missing |
| `TestScrollDelta_Computation` | Delta computed correctly from pixel offset | Delta math missing |
| `TestScroll_PrefetchTriggered` | Scroll triggers prefetch for new pages | Prefetch dispatch missing |
| `TestScroll_EvictionTriggered` | Scroll triggers eviction of distant pages | Eviction on scroll missing |
### 13.8 Phase 2 Tests — Search (search_test.go)
**Write after scroll tests pass. Must all fail.**
| Test | What it verifies | Expected failure |
|---|---|---|
| `TestSearch_EmptyQuery` | Empty query returns all entries | Search not implemented |
| `TestSearch_CaseInsensitive` | "foo" matches "Foo.txt" | Case folding missing |
| `TestSearch_PartialMatch` | "abc" matches "abc.txt" and "xabc.txt" | Substring matching missing |
| `TestSearch_NoMatch` | No matches returns empty results | Empty result handling missing |
| `TestSearch_JumpToFirst` | Scroll jumps to first match | Jump logic missing |
| `TestSearch_LargeDirectory` | Search completes within time budget | Performance not verified |
### 13.9 Test Execution Order
Tests must be written and run in this order:
```
1. types_test.go → ALL FAIL → implement types → ALL PASS
2. index_test.go → ALL FAIL → implement index → ALL PASS
3. browser_test.go → ALL FAIL → implement browser → ALL PASS
4. layout_test.go → ALL FAIL → implement layout → ALL PASS
5. scroll_test.go → ALL FAIL → implement scroll → ALL PASS
6. search_test.go → ALL FAIL → implement search → ALL PASS
```
**Verification command at each step:**
```bash
# Before implementation:
go test ./internal/browser/... -run TestName -v
# Expected: "FAIL" or "panic: not implemented"
# After implementation:
go test ./internal/browser/... -run TestName -v
# Expected: "PASS"
```
### 13.10 Performance Tests (Post-Implementation)
These run after all functional tests pass. They verify targets but do not block development.
| Test | Target | Method |
|---|---|---|
| `BenchmarkBuildIndex_10k` | < 500 ms | `testing.B` with synthetic dir |
| `BenchmarkBuildIndex_100k` | < 5 s | Same, scaled up |
| `BenchmarkPageLoad` | < 50 ms | Time a single page load from cache |
| `BenchmarkBrowserLayout` | < 10 ms | Time layout computation |
| `BenchmarkSearch_100k` | < 100 ms | Search through 100k entries |
| `BenchmarkEviction` | < 5 ms | Time eviction of 100 pages |
---
## 15. Lazy Loading Implementation Plan (Detailed)
This section contains the detailed step-by-step plan for implementing lazy loading with the mock filesystem, enabling robust E2E testing with large simulated directories.
### 15.1 Current State Assessment
**What's Already Implemented:**
| Component | Status | Notes |
|-----------|--------|-------|
| Browser types | Complete | `BrowserState`, `Page`, `PageSize=100`, `PrefetchDist=2` |
| Page loading logic | Partial | `loadPageFromIndex()`, `LoadInitialPages()` exist but not wired |
| Worker pool | Complete | Priority dispatch, `LoadPagesTask` defined |
| Mock filesystem | Complete | Thread-safe, configurable delay |
| Test harness | Complete | Frame capture, input simulation |
**Gaps to Fill:**
1. **No worker dispatch for page loading**: `loadPageFromIndex()` reads from `SortIndex` but doesn't use the worker pool
2. **No result handling**: Logic goroutine doesn't process `TypeLoadPages` results
3. **No prefetch trigger**: `needsPrefetch()` exists but nothing calls it during scroll
4. **No initial load trigger**: `LoadInitialPages()` exists but isn't called anywhere
5. **No dirty page refresh**: `Page.Dirty` flag exists but no mechanism to refresh
---
### 15.2 Implementation Steps
#### Step 1: Add Browser Manager
Create a `BrowserManager` that orchestrates lazy loading operations.
**File:** `browser/manager.go` (new file)
```go
type BrowserManager struct {
state *BrowserState
workerPool *pool.WorkerPool
fs *mock.FileSystem
dirPath string
}
func NewBrowserManager(state *BrowserState, wp *pool.WorkerPool, fs *mock.FileSystem) *BrowserManager
func (bm *BrowserManager) NavigateTo(dirPath string)
func (bm *BrowserManager) OnScroll()
func (bm *BrowserManager) HandleResult(result pool.Result)
```
**Responsibilities:**
- Dispatch tasks to worker pool
- Handle results and update browser state
- Trigger prefetch on scroll
- Manage page lifecycle
---
#### Step 2: Wire Initial Directory Load
When navigating to a directory:
1. Dispatch `BuildIndexTask` to read from mock filesystem
2. On result: populate `SortIndex` and `TotalEntries`
3. Dispatch `LoadInitialPages()` to load visible pages
**Flow:**
```
NavigateTo(dirPath)
BuildIndexTask(dirPath)
Result: SortIndex populated
LoadInitialPages()
First frame rendered with visible entries
```
---
#### Step 3: Implement Scroll-Based Prefetching
In `HandleScroll()`:
1. Check which pages should be prefetched using `needsPrefetch()`
2. Dispatch `LoadPagesTask` for unloaded pages
3. Handle results by populating `Pages` map
**Key logic:**
```go
func (bm *BrowserManager) OnScroll() {
minPage, maxPage := computeVisiblePageRange(bm.state)
// Check for unloaded pages in prefetch range
for i := minPage - PrefetchDist; i <= maxPage + PrefetchDist; i++ {
if _, ok := bm.state.Pages[i]; !ok {
bm.dispatchLoadPage(i)
}
}
// Evict distant pages
bm.state.EvictPages()
}
```
---
#### Step 4: Handle Worker Results
In logic goroutine (editor/logic.go):
```go
case result := <-workerPool.ResultChan():
if result.IsBrowserResult() {
browserManager.HandleResult(result)
}
```
**Result handling:**
- `TypeBuildIndex`: Populate `SortIndex`, set `TotalEntries`, trigger initial page load
- `TypeLoadPages`: Update `Pages` map with loaded entries
- `TypeReadDir`: Handle directory read results
---
### 15.3 Memory Management
#### Step 5: Implement Page Eviction
In `HandleScroll()` or periodic timer:
1. Call `EvictPages()` to unload distant pages
2. Track memory usage (optional: add metrics)
**Eviction policy:**
- Keep pages within `PrefetchDist` of visible range
- Release `page.Entries` and set `Loaded = false`
- Keep `Page` struct for metadata
---
### 15.4 Dirty Page Handling
#### Step 6: Implement Dirty Page Refresh
When a page is marked dirty:
1. Reload from index
2. Merge into `Pages` map
**Dirty detection:**
- External file changes (via file watcher)
- Manual refresh trigger
- Periodic mtime check
---
### 15.5 E2E Tests
#### Step 7: Write Comprehensive Tests
**File:** `browser/lazy_loading_test.go` (new file)
```go
func TestLazyLoadingLargeDirectory(t *testing.T) {
// 1. Create mock filesystem with 10,000 entries
// 2. Navigate to directory
// 3. Verify only visible pages are loaded
// 4. Scroll down
// 5. Verify new pages are loaded
// 6. Verify old pages are evicted
}
func TestPrefetchOnScroll(t *testing.T) {
// 1. Load initial pages
// 2. Scroll slightly
// 3. Verify prefetch pages are loaded
}
func TestDirtyPageRefresh(t *testing.T) {
// 1. Load page
// 2. Mark as dirty
// 3. Verify page is reloaded
}
```
**Test scenarios:**
| Test | Purpose |
|------|--------|
| `TestLazyLoadingLargeDirectory` | Verify page-based loading works with 10k+ entries |
| `TestPrefetchOnScroll` | Verify prefetch distance is respected |
| `TestDirtyPageRefresh` | Verify dirty pages are reloaded |
| `TestEvictionUnderMemoryPressure` | Verify eviction works correctly |
| `TestConcurrentScrollAndLoad` | Verify no race conditions |
---
### 15.6 Architecture Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ Logic Goroutine │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────┐ │
│ │ BrowserState│◄──►│BrowserMgr │ │ Result Handler│ │
│ │ (Pages map) │ │ (orchest.) │ │ (HandleResult)│ │
│ └─────────────┘ └─────┬──────┘ └────────────────┘ │
│ │ │
└───────────────────────────┼────────────────────────────────┘
│ Dispatch
┌───────────────┐
│ Worker Pool │
│ (4 workers) │
└───────┬───────┘
│ Execute
┌───────────────┐
│ Mock Filesystem│
│ (ReadDir) │
└───────────────┘
```
---
### 15.7 File Changes Summary
| File | Change | Purpose |
|------|--------|---------|
| `browser/manager.go` | **New** | Browser lazy loading orchestrator |
| `browser/browser.go` | Modify | Add `LoadPagesTask` dispatch |
| `browser/handlers.go` | Modify | Add prefetch on scroll |
| `editor/logic.go` | Modify | Wire result handling |
| `browser/lazy_loading_test.go` | **New** | E2E tests for lazy loading |
---
### 15.8 Key Design Decisions
1. **Page-based loading**: Load 100 entries at a time (configurable)
2. **Prefetch distance**: Load 2 pages beyond visible region
3. **Priority**: Page loads are `HighPriority` (user-visible)
4. **Eviction**: Remove pages beyond prefetch distance
5. **Mock filesystem**: Use configurable delay to simulate slow I/O in tests
---
### 15.9 Verification Strategy
1. **Unit tests**: Test page loading logic in isolation
2. **Integration tests**: Test with mock filesystem and worker pool
3. **E2E tests**: Test full flow with frame capture
4. **Stress tests**: 10,000+ entries with concurrent operations
---
## 14. Deferred Features
### 14.1 Alphabetical Index Sidebar (DEFERRED)
Not included in this implementation round. The core browser architecture supports adding it later without changes:
- **Letter offsets** are computed during index build but not exposed
- **ListView element** does not need modification
- **New element** `AlphaIndex` would be added alongside ListView
- **Handler** would read `LetterOffsets` and set `ScrollIndex` directly
### 14.2 Future Considerations
| Feature | Status | Notes |
|---|---|---|
| Alphabetical index sidebar | Deferred | See §14.1 |
| Subdirectory navigation | Phase 3 | Tap directory to navigate in; breadcrumb for back |
| File type icons | Future | Small icon prefix in ListView items |
| Sort order toggle | Future | Name, date, size; persisted preference |
| Folder expansion | Future | Expand/collapse subdirectories inline |
| Thumbnail preview | Out of scope | Text editor only; images not in scope |