gmp/f
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
a94eac6545
f/fundlab/RESEARCH.md
Greg Pomerantz a94eac6545 Overnight comprehensive screen: drop the name pre-filter, screen all 2,384 funds 
fundlab/RESEARCH.md - running research log: sources that work/die
  (full-index = discovery workhorse; browse-edgar JS-dead;
  investment-company-tickers.json nonexistent; company_tickers.json
  useless for OTC; Yahoo crumb throttled but chart API fine), 13
  hard-won learnings (OTC funds report exchange 'Nasdaq' -> use
  instrumentType; 497 SGML cover uses UNCLOSED line-based tags ->
  parse before tag-stripping; full-index columns drift -> regex the
  line; one quarter != universe -> 4-qtr union; accession paths
  relative to /Archives/ not /Archives/edgar/data/; family CIKs
  repeat -> dedupe by series name; portfolio is 50% MN so MN alpha
  funds are 'correlated', not diversifying).

fundlab/overnight.py - resumable all-stage pipeline (kill/restart safe):
  verify (Yahoo chart per non-local ticker, 4-thread, 429 backoff,
  local tickers measured from CSV row counts) -> select (pure
  select_rows: MUTUALFUND, >=5y, one longest-history class per series
  name, alpha_name as TAG not filter) -> download (goget in 200-sym
  batches) -> screen (streamed, skip-already-done) -> finalize
  (verdict counts + candidates the v1 name-filter would have missed).

Universe: 10,372 class tickers -> 10,260 verified -> 2,384 funds
(407 local, 1,977 external; only 54 match the alpha name pattern -
the v2 point is to screen the other 2,330).

app: alpha table now dedupes by sym with search_all.json winning
(comprehensive superset).
tests: select_rows unit tests (ETF drop, short-history drop, class
collapse, name tagging). 70/70 fundlab.
2026-08-26 21:33:11 -04:00

6.5 KiB
Raw Blame History

Fund discovery research log

Running log of attempts, dead ends, and learnings for finding candidate funds (alpha-driven, portfolio-complementing) with the fundlab pipeline. Newest entries at the bottom of each section. Update as you go — this file IS the knowledge base.

Sources & what works

Source Status Notes
Local stocks DB (~/prog/fin/stocks/, 8,250 syms) ✅ primary ~100 open-end alt families already present; dbmine.py mines by name pattern
SEC full-index Archives/edgar/full-index/YYYY/QTRn/company.gz ✅ discovery workhorse lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers
SEC full-submission .txt (per accession) ✅ ~10–50KB; line-based SGML cover with <SERIES-NAME> (UNCLOSED tag) + <CLASS-CONTRACT-TICKER-SYMBOL> per class; often several series per filing
Yahoo chart API (query1.../v8/finance/chart/<T>) ✅ no crumb needed; meta has instrumentType/exchange/longName; range=20y gives history length
goget (~/go/bin/goget) ✅ batch downloader, idempotent, ~2–5s/sym
EDGAR FTS (efts.sec.gov/LATEST/search-index) ⚠️ fragile phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only
SEC browse-edgar company listing ❌ dead now JS-rendered, no data in HTML
SEC /files/investment-company-tickers.json ❌ doesn't exist 404 (misremembered)
SEC company_tickers.json ❌ for our purpose only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent
Yahoo search/crumb API ❌ throttled IP-level "Too Many Requests" on fc.yahoo.com/getcrumb; chart API unaffected
stockanalysis.com/funds/ ❌ 404 path guessed wrong, not pursued

Learnings (hard-won)

  1. Famous multi-strategy/macro funds are private/offshore — Millennium, Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace, Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497, no Yahoo OTC ticker. Structural, not a search failure.
  2. Precision over recall for name→ticker resolution: a guessed ticker that "looks right" is worse than no answer. Chart-API name gate + 2/3 token overlap rejected 23/24 memory-based guesses.
  3. Yahoo exchange name is a USELESS fund/ETF discriminator: OTC mutual funds report fullExchangeName="Nasdaq". Use instrumentType (MUTUALFUND vs ETF).
  4. 497 SGML cover uses UNCLOSED tags, one per line — <SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n. Closed-tag regexes find nothing; to_text() (tag stripping) destroys the data. Parse line-based, BEFORE any tag stripping.
  5. Full-index columns drift — don't trust fixed widths; the header line and data rows don't align. Regex the whole line.
  6. One quarter ≠ the universe: each fund's annual base-497 re-filing lands in a random quarter; union of 4 consecutive quarters is the full active universe (1,209 in Q2 alone → 1,668 union).
  7. Accession paths in the index are relative to /Archives/, not /Archives/edgar/data/ — doubling the prefix 404s.
  8. Amendments (497A/497VPU) may lack the series cover — base 497 / 497K carry it. (497A inclusion pending — see below.)
  9. Family CIKs repeat across the index (e.g. AB under 2 CIKs, same series listed twice) — dedupe by series name, not CIK.
  10. Large-n BIC is knife-edge (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) — the |t|>2 gate on added regressors is essential (decompose.py).
  11. The portfolio is 50% market-neutral (qspnx) — MN/L-S-equity funds show the strongest alpha on screen (+12–17%/yr) but corr 0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
  12. Near-duplicate sleeves make OLS knife-edge — distinct-axis candidate sets (one rep per sleeve family) or the betas split arbitrarily between ivv/vti/vt.
  13. Wrong-fund objectives are worse than none — all EDGAR resolution stages gate on name match before accepting.

Pipeline stages (current)

full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
  → parse_cover (line-based SGML)
  → [name filter — REMOVED in v2, kept as a tag]
  → Yahoo chart verify (instrumentType, 20y length)
  → share-class dedupe (longest history)
  → goget missing (batched)
  → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
     corr vs portfolio & benchmark)  [0.3s/fund]
  → search_*.json → app Fund Lab "Alpha search" table

Overnight comprehensive screen (v2, started 2026-08-26)

Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate.

Attempts & progress

  • Survey: 10,372 unique class tickers in covers cache; screen costs 0.3s/fund → local screen of ~5k families ≈ 25 min. Bottlenecks: Yahoo verify (~20 min threaded) + goget (~3–5 h for ~4k missing syms) — the overnight part.
  • Yahoo verify all non-local tickers → universe_cache/yahoo_meta.json
  • goget in resumable batches
  • screen all → search_all.json (streamed, resumable)
  • app reads search_all.json; cross-check: alpha funds whose names DON'T match the alpha pattern (what the v1 filter missed)

Notes while running

(empty — append as the run progresses)

  • 2026-08-26 16:01 === stage verify ===
  • 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
  • 2026-08-26 16:01 verify: 500/9149
  • 2026-08-26 16:02 verify: 1000/9149
  • 2026-08-26 16:03 verify: 1500/9149
  • 2026-08-26 16:03 verify: 2000/9149
  • 2026-08-26 16:04 verify: 2500/9149
  • 2026-08-26 16:04 verify: 3000/9149
  • 2026-08-26 16:05 verify: 3500/9149
  • 2026-08-26 16:05 verify: 4000/9149
  • 2026-08-26 16:06 verify: 4500/9149
  • 2026-08-26 16:06 verify: 5000/9149
  • 2026-08-26 16:07 verify: 5500/9149
  • 2026-08-26 16:07 verify: 6000/9149
  • 2026-08-26 16:08 verify: 6500/9149
  • 2026-08-26 16:09 verify: 7000/9149
  • 2026-08-26 16:09 verify: 7500/9149
  • 2026-08-26 16:10 verify: 8000/9149
  • 2026-08-26 16:10 verify: 8500/9149
  • 2026-08-26 16:11 verify: 9000/9149
  • 2026-08-26 16:11 verify done: 10260/10372 with data
  • 2026-08-26 16:11 === overnight run finished in 0.2h ===
  • 2026-08-26 21:31 === stage select ===
  • 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
  • 2026-08-26 21:31 === overnight run finished in 0.0h ===
  • 2026-08-26 21:31 === stage download ===
  • 2026-08-26 21:31 download: 1960 missing symbols