Greg Pomerantz
a94eac6545
fundlab/RESEARCH.md - running research log: sources that work/die (full-index = discovery workhorse; browse-edgar JS-dead; investment-company-tickers.json nonexistent; company_tickers.json useless for OTC; Yahoo crumb throttled but chart API fine), 13 hard-won learnings (OTC funds report exchange 'Nasdaq' -> use instrumentType; 497 SGML cover uses UNCLOSED line-based tags -> parse before tag-stripping; full-index columns drift -> regex the line; one quarter != universe -> 4-qtr union; accession paths relative to /Archives/ not /Archives/edgar/data/; family CIKs repeat -> dedupe by series name; portfolio is 50% MN so MN alpha funds are 'correlated', not diversifying). fundlab/overnight.py - resumable all-stage pipeline (kill/restart safe): verify (Yahoo chart per non-local ticker, 4-thread, 429 backoff, local tickers measured from CSV row counts) -> select (pure select_rows: MUTUALFUND, >=5y, one longest-history class per series name, alpha_name as TAG not filter) -> download (goget in 200-sym batches) -> screen (streamed, skip-already-done) -> finalize (verdict counts + candidates the v1 name-filter would have missed). Universe: 10,372 class tickers -> 10,260 verified -> 2,384 funds (407 local, 1,977 external; only 54 match the alpha name pattern - the v2 point is to screen the other 2,330). app: alpha table now dedupes by sym with search_all.json winning (comprehensive superset). tests: select_rows unit tests (ETF drop, short-history drop, class collapse, name tagging). 70/70 fundlab.
123 lines
6.5 KiB
Markdown
123 lines
6.5 KiB
Markdown
# Fund discovery research log
|
||
|
||
Running log of attempts, dead ends, and learnings for finding candidate
|
||
funds (alpha-driven, portfolio-complementing) with the `fundlab`
|
||
pipeline. Newest entries at the bottom of each section. Update as you
|
||
go — this file IS the knowledge base.
|
||
|
||
## Sources & what works
|
||
|
||
| Source | Status | Notes |
|
||
|---|---|---|
|
||
| Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern |
|
||
| SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers |
|
||
| SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `<SERIES-NAME>` (UNCLOSED tag) + `<CLASS-CONTRACT-TICKER-SYMBOL>` per class; often several series per filing |
|
||
| Yahoo chart API (`query1.../v8/finance/chart/<T>`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length |
|
||
| `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym |
|
||
| EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only |
|
||
| SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML |
|
||
| SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) |
|
||
| SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent |
|
||
| Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected |
|
||
| stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued |
|
||
|
||
## Learnings (hard-won)
|
||
|
||
1. **Famous multi-strategy/macro funds are private/offshore** — Millennium,
|
||
Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace,
|
||
Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497,
|
||
no Yahoo OTC ticker. Structural, not a search failure.
|
||
2. **Precision over recall for name→ticker resolution**: a guessed
|
||
ticker that "looks right" is worse than no answer. Chart-API name
|
||
gate + 2/3 token overlap rejected 23/24 memory-based guesses.
|
||
3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC
|
||
mutual funds report `fullExchangeName="Nasdaq"`. Use
|
||
`instrumentType` (MUTUALFUND vs ETF).
|
||
4. **497 SGML cover uses UNCLOSED tags, one per line** —
|
||
`<SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n`.
|
||
Closed-tag regexes find nothing; `to_text()` (tag stripping)
|
||
destroys the data. Parse line-based, BEFORE any tag stripping.
|
||
5. **Full-index columns drift** — don't trust fixed widths; the header
|
||
line and data rows don't align. Regex the whole line.
|
||
6. **One quarter ≠ the universe**: each fund's annual base-497
|
||
re-filing lands in a random quarter; union of 4 consecutive
|
||
quarters is the full active universe (1,209 in Q2 alone → 1,668
|
||
union).
|
||
7. **Accession paths in the index are relative to `/Archives/`**, not
|
||
`/Archives/edgar/data/` — doubling the prefix 404s.
|
||
8. **Amendments (497A/497VPU) may lack the series cover** — base 497 /
|
||
497K carry it. (497A inclusion pending — see below.)
|
||
9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs,
|
||
same series listed twice) — dedupe by series name, not CIK.
|
||
10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) —
|
||
the |t|>2 gate on added regressors is essential (decompose.py).
|
||
11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity
|
||
funds show the strongest alpha on screen (+12–17%/yr) but corr
|
||
0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
|
||
12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis
|
||
candidate sets (one rep per sleeve family) or the betas split
|
||
arbitrarily between ivv/vti/vt.
|
||
13. **Wrong-fund objectives are worse than none** — all EDGAR
|
||
resolution stages gate on name match before accepting.
|
||
|
||
## Pipeline stages (current)
|
||
|
||
```
|
||
full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
|
||
→ parse_cover (line-based SGML)
|
||
→ [name filter — REMOVED in v2, kept as a tag]
|
||
→ Yahoo chart verify (instrumentType, 20y length)
|
||
→ share-class dedupe (longest history)
|
||
→ goget missing (batched)
|
||
→ screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
|
||
corr vs portfolio & benchmark) [0.3s/fund]
|
||
→ search_*.json → app Fund Lab "Alpha search" table
|
||
```
|
||
|
||
## Overnight comprehensive screen (v2, started 2026-08-26)
|
||
|
||
Goal: screen EVERY OTC open-end fund with ≥5y history from the 497
|
||
universe — no name pre-filter. Name match becomes a tag/cross-check,
|
||
not a gate.
|
||
|
||
### Attempts & progress
|
||
- [x] Survey: 10,372 unique class tickers in covers cache; screen
|
||
costs 0.3s/fund → local screen of ~5k families ≈ 25 min.
|
||
Bottlenecks: Yahoo verify (~20 min threaded) + goget
|
||
(~3–5 h for ~4k missing syms) — the overnight part.
|
||
- [ ] Yahoo verify all non-local tickers → `universe_cache/yahoo_meta.json`
|
||
- [ ] goget in resumable batches
|
||
- [ ] screen all → `search_all.json` (streamed, resumable)
|
||
- [ ] app reads search_all.json; cross-check: alpha funds whose names
|
||
DON'T match the alpha pattern (what the v1 filter missed)
|
||
|
||
### Notes while running
|
||
(empty — append as the run progresses)
|
||
- 2026-08-26 16:01 === stage verify ===
|
||
- 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
|
||
- 2026-08-26 16:01 verify: 500/9149
|
||
- 2026-08-26 16:02 verify: 1000/9149
|
||
- 2026-08-26 16:03 verify: 1500/9149
|
||
- 2026-08-26 16:03 verify: 2000/9149
|
||
- 2026-08-26 16:04 verify: 2500/9149
|
||
- 2026-08-26 16:04 verify: 3000/9149
|
||
- 2026-08-26 16:05 verify: 3500/9149
|
||
- 2026-08-26 16:05 verify: 4000/9149
|
||
- 2026-08-26 16:06 verify: 4500/9149
|
||
- 2026-08-26 16:06 verify: 5000/9149
|
||
- 2026-08-26 16:07 verify: 5500/9149
|
||
- 2026-08-26 16:07 verify: 6000/9149
|
||
- 2026-08-26 16:08 verify: 6500/9149
|
||
- 2026-08-26 16:09 verify: 7000/9149
|
||
- 2026-08-26 16:09 verify: 7500/9149
|
||
- 2026-08-26 16:10 verify: 8000/9149
|
||
- 2026-08-26 16:10 verify: 8500/9149
|
||
- 2026-08-26 16:11 verify: 9000/9149
|
||
- 2026-08-26 16:11 verify done: 10260/10372 with data
|
||
- 2026-08-26 16:11 === overnight run finished in 0.2h ===
|
||
- 2026-08-26 21:31 === stage select ===
|
||
- 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
|
||
- 2026-08-26 21:31 === overnight run finished in 0.0h ===
|
||
- 2026-08-26 21:31 === stage download ===
|
||
- 2026-08-26 21:31 download: 1960 missing symbols
|