fundlab/RESEARCH.md - running research log: sources that work/die
(full-index = discovery workhorse; browse-edgar JS-dead;
investment-company-tickers.json nonexistent; company_tickers.json
useless for OTC; Yahoo crumb throttled but chart API fine), 13
hard-won learnings (OTC funds report exchange 'Nasdaq' -> use
instrumentType; 497 SGML cover uses UNCLOSED line-based tags ->
parse before tag-stripping; full-index columns drift -> regex the
line; one quarter != universe -> 4-qtr union; accession paths
relative to /Archives/ not /Archives/edgar/data/; family CIKs
repeat -> dedupe by series name; portfolio is 50% MN so MN alpha
funds are 'correlated', not diversifying).
fundlab/overnight.py - resumable all-stage pipeline (kill/restart safe):
verify (Yahoo chart per non-local ticker, 4-thread, 429 backoff,
local tickers measured from CSV row counts) -> select (pure
select_rows: MUTUALFUND, >=5y, one longest-history class per series
name, alpha_name as TAG not filter) -> download (goget in 200-sym
batches) -> screen (streamed, skip-already-done) -> finalize
(verdict counts + candidates the v1 name-filter would have missed).
Universe: 10,372 class tickers -> 10,260 verified -> 2,384 funds
(407 local, 1,977 external; only 54 match the alpha name pattern -
the v2 point is to screen the other 2,330).
app: alpha table now dedupes by sym with search_all.json winning
(comprehensive superset).
tests: select_rows unit tests (ETF drop, short-history drop, class
collapse, name tagging). 70/70 fundlab.
Answer to 'find other alpha-driven funds that complement the portfolio':
- fundlab/search.py: complementarity screen - each fund's daily total
returns vs the same 21 broad sleeve axes (BIC forward selection,
|t|>2), full + 5y; alpha (intercept t), R2, rolling 6m alpha
persistence, correlation vs the current qspnx/pmaix portfolio and the
spy/agg/tlt benchmark mix. Verdict tiers: CANDIDATE (alpha +
persistent + portfolio-corr<0.3) / semi-alpha / alpha-but-correlated /
sleeve mix / weak.
- fundlab/dbmine.py: the actual search universe - the local stocks DB
already holds ~100 US open-end alternatives (AQR, PIMCO, JPM,
Principal, Calamos, GMO, Franklin K2, ...). Name-pattern miner with
share-class family dedupe (keeps the longest-history class).
- fundlab/tickers.py + searchlist.py: external longlist resolution
(chart-API name gate + EDGAR 497 cover tickers). Finding: the famous
multi-strategy/macro names (Millennium, Balyasny, Two Sigma, Winton,
Marshall Wace, Brevan Howard, AQR Event-Driven) are private/offshore
or terminated - not US open-end accessible. Fidelity Multi-Asset
Income (FMSDX) resolved and screens as weak alpha.
- app Fund Lab: 'Alpha search - all screened funds, ranked' table
(80 funds: 13 shortlist + 59 mined + 1 external).
- results (ranked candidates, 5y alpha / t / portfolio-corr):
wmnix Westwood Alt Income +3.8% t6.5 c0.09 | pyaix Payden ARB +3.0%
t4.8 c0.13 | srdax Stone Ridge Div Alts +7.7% t4.2 c0.10 | padqx PGIM
ARB +2.3% t2.4 c0.27 | bxmdx Blackstone Alt MS +3.5% t2.4 c0.30 |
aqmix AQR Mngd Futures +8.0% t2.2 c0.21 | cmnix/gioix semi-alpha.
Key insight: AQR MN / L/S-equity / Vanguard MN show strong alpha but
corr 0.35-0.76 with the portfolio - it is already 50% market-neutral
(qspnx), so more MN is not diversifying.
- tests: 59/59 fundlab (resolver gates, query ladder, family dedupe,
ticker regex), 32/32 app, 14/14 data
- fundlab/nport.py: parse the fund's own category/percentage lines,
as-of date, net assets and dollar-valued positions from N-PORT
schedules of investments (handles per-fund and combined multi-fund
family filings; conservative keyword bucketing of positions)
- fundlab/nport_cache/<sym>.json: parsed snapshots for 15 of 16 funds
(raw SOI HTML kept locally, gitignored; source URLs + filing dates in
nport_manifest.json, md5-verified against EDGAR)
- atesx: no current SOI found (Anchor's recent filings cover the Income
fund) - listed with an honest note
- app.py: new 'Fund Lab' tab - pick a fund, see objective, reported
composition (bar + table), rough keyword buckets, top positions, and
the prospectus strategy excerpt
- tests: parser unit tests (section finding, category regex, buckets)
- extract_strategy(): finds the 'Principal Investment Strategies' /
'main investment strategies' section, scores candidates (strategy prose
+3, Q&A heading +2, TOC -5, risk subheading -5, stop-heading -2),
truncates at the next section heading; falls back to the prose after
the objective sentence when no heading exists
- fundinfo --strategy [--refresh]: populates the strategy field of
funds.json from each fund's EDGAR document
- funds.json now carries objective + strategy for 20 funds (the 9
curated index funds have no strategy: their objective is the strategy)
- ticker -> CIK scoped FTS (ciks filter) over the fund's own registrant,
word queries shed Yahoo abbreviations shortest-first
- fuzzy cover gate (SequenceMatcher, sliding window for wrapped names)
+ abbreviation table (Mgd/Glbl/Macr/Abs/Ret/Advtg/...) applied to both
sides of the match
- family-section matching inside multi-fund filings; similarity tiers:
gate (0.85) returns immediately, floor (0.70) is a CIK-scoped fallback
ranked against sibling funds' documents (best sim, then newest)
- objective phrasings: 'seeks to ...', 'seeks investment results ...',
'The Fund's investment objective is ...', '(the Fund) investment
objective is ...' (boilerplate 'is not fundamental' rejected); TOC
headings skipped by trying all heads
- sec_get retries on mid-stream connection drops
- edgar.py: SEC FTS + submissions API; strict cover-gate extraction
(name in title position or (TICKER) on the cover; underlying-reference
names like leveraged wrappers rejected); 4-pass fetch (ticker->CIK
filings, name search, annual reports, ticker search); keyword category
classifier. Returns None rather than a wrong fund's objective.
- fundinfo.py CLI: curated -> cached -> EDGAR resolution into funds.json
- funds_curated.json: human-verified objectives for 13 benchmark-pool
funds (iShares/Vanguard family-trust classes the scraper can't reach)
- tests: 26 checks incl. live EDGAR fetch of VTSAX
