Greg Pomerantz
a94eac6545
fundlab/RESEARCH.md - running research log: sources that work/die (full-index = discovery workhorse; browse-edgar JS-dead; investment-company-tickers.json nonexistent; company_tickers.json useless for OTC; Yahoo crumb throttled but chart API fine), 13 hard-won learnings (OTC funds report exchange 'Nasdaq' -> use instrumentType; 497 SGML cover uses UNCLOSED line-based tags -> parse before tag-stripping; full-index columns drift -> regex the line; one quarter != universe -> 4-qtr union; accession paths relative to /Archives/ not /Archives/edgar/data/; family CIKs repeat -> dedupe by series name; portfolio is 50% MN so MN alpha funds are 'correlated', not diversifying). fundlab/overnight.py - resumable all-stage pipeline (kill/restart safe): verify (Yahoo chart per non-local ticker, 4-thread, 429 backoff, local tickers measured from CSV row counts) -> select (pure select_rows: MUTUALFUND, >=5y, one longest-history class per series name, alpha_name as TAG not filter) -> download (goget in 200-sym batches) -> screen (streamed, skip-already-done) -> finalize (verdict counts + candidates the v1 name-filter would have missed). Universe: 10,372 class tickers -> 10,260 verified -> 2,384 funds (407 local, 1,977 external; only 54 match the alpha name pattern - the v2 point is to screen the other 2,330). app: alpha table now dedupes by sym with search_all.json winning (comprehensive superset). tests: select_rows unit tests (ETF drop, short-history drop, class collapse, name tagging). 70/70 fundlab.
3 lines
72 B
Plaintext
3 lines
72 B
Plaintext
21:31:57 === stage download ===
|
|
21:31:57 download: 1960 missing symbols
|