From eac61683f671ed6a6fa0b61bae91f17f1034228b Mon Sep 17 00:00:00 2001 From: Greg Pomerantz Date: Wed, 26 Aug 2026 21:41:07 -0400 Subject: [PATCH] RESEARCH.md: live progress notes + /tmp-cleanup infra lesson --- fundlab/RESEARCH.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/fundlab/RESEARCH.md b/fundlab/RESEARCH.md index 4e0df7c..b00a4ea 100644 --- a/fundlab/RESEARCH.md +++ b/fundlab/RESEARCH.md @@ -80,19 +80,26 @@ Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate. -### Attempts & progress -- [x] Survey: 10,372 unique class tickers in covers cache; screen - costs 0.3s/fund → local screen of ~5k families ≈ 25 min. - Bottlenecks: Yahoo verify (~20 min threaded) + goget - (~3–5 h for ~4k missing syms) — the overnight part. -- [ ] Yahoo verify all non-local tickers → `universe_cache/yahoo_meta.json` -- [ ] goget in resumable batches -- [ ] screen all → `search_all.json` (streamed, resumable) -- [ ] app reads search_all.json; cross-check: alpha funds whose names - DON'T match the alpha pattern (what the v1 filter missed) +### Notes while running (live - the log() below appends automatically) +- [x] survey: 10,372 class tickers in covers cache; 1,223 already local, + 9,149 to verify on Yahoo; screen costs 0.3s/fund. +- [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also + filed a base in window) - no need to include. +- [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min; + 112 dead/never-listed tickers dropped). Cached in + universe_cache/yahoo_meta.json. +- [x] select: **2,384 funds** (407 local, 1,977 external). Only 54 + match the alpha name pattern - the other 2,330 are exactly what + the v1 name-filter would have missed. Worklist cached in + universe_cache/selected.json. +- [ ] download: 1,960 missing symbols via goget, 200/batch, ~3.3s/sym + (ETA ~1:20 AM). Logs to fundlab/overnight.log + this file. +- [ ] screen: ~2,384 x 0.3s ≈ 25 min -> search_all.json. +- [ ] finalize: verdict counts + "candidates v1 would have missed". -### Notes while running -(empty — append as the run progresses) +Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the +project (fundlab/overnight.log, fundlab/universe_cache/), and use +`setsid nohup ... < /dev/null &` so a closed shell can't kill the job. - 2026-08-26 16:01 === stage verify === - 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo - 2026-08-26 16:01 verify: 500/9149