130 lines
7.0 KiB
Markdown
130 lines
7.0 KiB
Markdown
# Fund discovery research log
|
||
|
||
Running log of attempts, dead ends, and learnings for finding candidate
|
||
funds (alpha-driven, portfolio-complementing) with the `fundlab`
|
||
pipeline. Newest entries at the bottom of each section. Update as you
|
||
go — this file IS the knowledge base.
|
||
|
||
## Sources & what works
|
||
|
||
| Source | Status | Notes |
|
||
|---|---|---|
|
||
| Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern |
|
||
| SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers |
|
||
| SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `<SERIES-NAME>` (UNCLOSED tag) + `<CLASS-CONTRACT-TICKER-SYMBOL>` per class; often several series per filing |
|
||
| Yahoo chart API (`query1.../v8/finance/chart/<T>`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length |
|
||
| `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym |
|
||
| EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only |
|
||
| SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML |
|
||
| SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) |
|
||
| SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent |
|
||
| Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected |
|
||
| stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued |
|
||
|
||
## Learnings (hard-won)
|
||
|
||
1. **Famous multi-strategy/macro funds are private/offshore** — Millennium,
|
||
Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace,
|
||
Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497,
|
||
no Yahoo OTC ticker. Structural, not a search failure.
|
||
2. **Precision over recall for name→ticker resolution**: a guessed
|
||
ticker that "looks right" is worse than no answer. Chart-API name
|
||
gate + 2/3 token overlap rejected 23/24 memory-based guesses.
|
||
3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC
|
||
mutual funds report `fullExchangeName="Nasdaq"`. Use
|
||
`instrumentType` (MUTUALFUND vs ETF).
|
||
4. **497 SGML cover uses UNCLOSED tags, one per line** —
|
||
`<SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n`.
|
||
Closed-tag regexes find nothing; `to_text()` (tag stripping)
|
||
destroys the data. Parse line-based, BEFORE any tag stripping.
|
||
5. **Full-index columns drift** — don't trust fixed widths; the header
|
||
line and data rows don't align. Regex the whole line.
|
||
6. **One quarter ≠ the universe**: each fund's annual base-497
|
||
re-filing lands in a random quarter; union of 4 consecutive
|
||
quarters is the full active universe (1,209 in Q2 alone → 1,668
|
||
union).
|
||
7. **Accession paths in the index are relative to `/Archives/`**, not
|
||
`/Archives/edgar/data/` — doubling the prefix 404s.
|
||
8. **Amendments (497A/497VPU) may lack the series cover** — base 497 /
|
||
497K carry it. (497A inclusion pending — see below.)
|
||
9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs,
|
||
same series listed twice) — dedupe by series name, not CIK.
|
||
10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) —
|
||
the |t|>2 gate on added regressors is essential (decompose.py).
|
||
11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity
|
||
funds show the strongest alpha on screen (+12–17%/yr) but corr
|
||
0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
|
||
12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis
|
||
candidate sets (one rep per sleeve family) or the betas split
|
||
arbitrarily between ivv/vti/vt.
|
||
13. **Wrong-fund objectives are worse than none** — all EDGAR
|
||
resolution stages gate on name match before accepting.
|
||
|
||
## Pipeline stages (current)
|
||
|
||
```
|
||
full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
|
||
→ parse_cover (line-based SGML)
|
||
→ [name filter — REMOVED in v2, kept as a tag]
|
||
→ Yahoo chart verify (instrumentType, 20y length)
|
||
→ share-class dedupe (longest history)
|
||
→ goget missing (batched)
|
||
→ screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
|
||
corr vs portfolio & benchmark) [0.3s/fund]
|
||
→ search_*.json → app Fund Lab "Alpha search" table
|
||
```
|
||
|
||
## Overnight comprehensive screen (v2, started 2026-08-26)
|
||
|
||
Goal: screen EVERY OTC open-end fund with ≥5y history from the 497
|
||
universe — no name pre-filter. Name match becomes a tag/cross-check,
|
||
not a gate.
|
||
|
||
### Notes while running (live - the log() below appends automatically)
|
||
- [x] survey: 10,372 class tickers in covers cache; 1,223 already local,
|
||
9,149 to verify on Yahoo; screen costs 0.3s/fund.
|
||
- [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also
|
||
filed a base in window) - no need to include.
|
||
- [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min;
|
||
112 dead/never-listed tickers dropped). Cached in
|
||
universe_cache/yahoo_meta.json.
|
||
- [x] select: **2,384 funds** (407 local, 1,977 external). Only 54
|
||
match the alpha name pattern - the other 2,330 are exactly what
|
||
the v1 name-filter would have missed. Worklist cached in
|
||
universe_cache/selected.json.
|
||
- [ ] download: 1,960 missing symbols via goget, 200/batch, ~3.3s/sym
|
||
(ETA ~1:20 AM). Logs to fundlab/overnight.log + this file.
|
||
- [ ] screen: ~2,384 x 0.3s ≈ 25 min -> search_all.json.
|
||
- [ ] finalize: verdict counts + "candidates v1 would have missed".
|
||
|
||
Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the
|
||
project (fundlab/overnight.log, fundlab/universe_cache/), and use
|
||
`setsid nohup ... < /dev/null &` so a closed shell can't kill the job.
|
||
- 2026-08-26 16:01 === stage verify ===
|
||
- 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
|
||
- 2026-08-26 16:01 verify: 500/9149
|
||
- 2026-08-26 16:02 verify: 1000/9149
|
||
- 2026-08-26 16:03 verify: 1500/9149
|
||
- 2026-08-26 16:03 verify: 2000/9149
|
||
- 2026-08-26 16:04 verify: 2500/9149
|
||
- 2026-08-26 16:04 verify: 3000/9149
|
||
- 2026-08-26 16:05 verify: 3500/9149
|
||
- 2026-08-26 16:05 verify: 4000/9149
|
||
- 2026-08-26 16:06 verify: 4500/9149
|
||
- 2026-08-26 16:06 verify: 5000/9149
|
||
- 2026-08-26 16:07 verify: 5500/9149
|
||
- 2026-08-26 16:07 verify: 6000/9149
|
||
- 2026-08-26 16:08 verify: 6500/9149
|
||
- 2026-08-26 16:09 verify: 7000/9149
|
||
- 2026-08-26 16:09 verify: 7500/9149
|
||
- 2026-08-26 16:10 verify: 8000/9149
|
||
- 2026-08-26 16:10 verify: 8500/9149
|
||
- 2026-08-26 16:11 verify: 9000/9149
|
||
- 2026-08-26 16:11 verify done: 10260/10372 with data
|
||
- 2026-08-26 16:11 === overnight run finished in 0.2h ===
|
||
- 2026-08-26 21:31 === stage select ===
|
||
- 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
|
||
- 2026-08-26 21:31 === overnight run finished in 0.0h ===
|
||
- 2026-08-26 21:31 === stage download ===
|
||
- 2026-08-26 21:31 download: 1960 missing symbols
|