CEF pass: universe (SEC report, 973 -> 295 listed) + stage 1 screen + stage 2a character
fundlab/cef_universe.py: SEC 'Closed-End Fund Information' report (973 active CEFs) + company_tickers CIK join -> 295 listed common classes (preferreds and 6(c)-converted companies flagged). Supersedes the N-2 full-index approach (N-2/A annual updates + 404ing index paths). Prices via goget. fundlab/cef_screen.py: 290 screened - t5/t12, vol5, maxdd5, the 5 crash episodes, 12m payout proxy. Energy/midstream CEFs (EMO +305%, SRV +234%, NML +232%) top return AND crash resilience; Voya Dividend-Premium series (IGD maxDD -16%); EM CEFs volatile + 14-24% dist; long-dur munis -26..-42%. fundlab/cef_character.py: 50-fund shortlist, 35-sleeve character + crude tax_arb = character x (upside + 0.4 x vol). Tests: 6 new cef checks (105 total). RESEARCH.md: CEF form facts (N-2ASR, N-PX, N-23C-3A, BDC caveats) + remaining stage 2b work.
This commit is contained in:
parent
f8409fff7a
commit
d1e85026bf
2
adx-split.csv
Normal file
2
adx-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2000-10-20,3:2
|
||||
|
2
aef-split.csv
Normal file
2
aef-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
1995-07-17,2:1
|
||||
|
2
asa-split.csv
Normal file
2
asa-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2010-05-04,3:1
|
||||
|
2
brw-split.csv
Normal file
2
brw-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2022-05-20,1:2
|
||||
|
3
bto-split.csv
Normal file
3
bto-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
1997-12-02,4:1
|
||||
2008-12-30,1:4
|
||||
|
3
clm-split.csv
Normal file
3
clm-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
2008-12-23,1:4
|
||||
2014-12-29,1:4
|
||||
|
3
crf-split.csv
Normal file
3
crf-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
2008-12-23,1:2
|
||||
2014-12-29,1:4
|
||||
|
2
evg-split.csv
Normal file
2
evg-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2019-12-27,999649:1000000
|
||||
|
|
|
@ -732,3 +732,61 @@ swing, so volatility is the right selection variable. Refinements:
|
|||
identical (different holdings/NAV) - the standard harvest-and-swap
|
||||
is safe; the 250-candidate universe makes a like-for-like
|
||||
substitute usually available.
|
||||
|
||||
### CEF pass (2026-08-27)
|
||||
**Universe** (fundlab/cef_universe.py): the SEC's official
|
||||
"Closed-End Fund Information" report (ALL active 1940-Act CEFs; 2026
|
||||
file = 973; sec.gov/files/investment/data/other/closed-end-fund-
|
||||
information/closed-end-investment-company-<YYYY>.csv - hyphenated
|
||||
filenames 2024+, underscored before). CIK join with
|
||||
company_tickers.json -> 295 listed common-class tickers (hyphenated
|
||||
preferred series dropped; 668 small/dark/OTC funds have no listed
|
||||
ticker; several 6(c)-converted "CEFs" are now operating companies -
|
||||
Powerlaw, RoboStrategy, C1 Fund, Foxby - excluded). The N-2
|
||||
full-index approach was abandoned: annual updates file as N-2/A, and
|
||||
full-index paths 404 for newer filings; the SEC report is the whole
|
||||
active population and is one download.
|
||||
|
||||
CEF form facts learned:
|
||||
- CEFs file NPORT-P (same format as open-end - xcheck.py reuses)
|
||||
- shareholder reports: N-CSR/N-CSRS AND/OR N-2ASR (rocdetect must try both)
|
||||
- N-PX proxy = CEF signal (open-end funds have no boards)
|
||||
- N-23C-3A/-2 = Rule 23c-3 repurchase-offer (tender) notifications - common
|
||||
- BDCs also file N-2; their names don't contain "Business Development"
|
||||
(Ares Capital Corp etc) - detect via form history (10-K vs N-2ASR)
|
||||
- Yahoo instrumentType for CEFs is usually EQUITY - do NOT apply the
|
||||
open-end MUTUALFUND filter
|
||||
- CEF price = MARKET price (premium/discount on top of NAV); Adj Close
|
||||
includes reinvested distributions, so the payout proxy works
|
||||
|
||||
**Stage 1** (fundlab/cef_screen.py): 290/295 screened (goget prices;
|
||||
phd/unid/uniu 404). Stats: t5/t12, vol5, maxdd5, the 5 crash
|
||||
episodes via drawdown.fund_windows, 12m payout proxy, n_pos_scen.
|
||||
Findings:
|
||||
- Energy/midstream infra CEFs dominate 5y return AND crash
|
||||
resilience: EMO +305%, SRV +234%, NML +232%, KYN +206%, PEO +192%,
|
||||
TYG +170% - all positive in 2022 (+10..25%) and 2026 Q1 (+10..17%),
|
||||
dist 10-13%/yr.
|
||||
- Voya "Dividend & Premium" series (IGD/IHD/IAE/EOD): +77..86%,
|
||||
maxDD only -16..-30%, 11-13% dist - premium management keeps the
|
||||
market price near NAV.
|
||||
- EM/China CEFs (TWN +348%, KF +139%, EMF +98%, AEF, MXF): high vol,
|
||||
-34..-47% in 2022 (harvestable), 14-24% dist.
|
||||
- Long-duration muni CEFs (PCQ/PML/PNI/TDF) -26..-42% over 5y: the
|
||||
2022-23 rate spike, duration not alpha.
|
||||
- LTCFX (+1101%, 2023 +799%) and DXYZ (maxDD -90%) are
|
||||
derivatives/meme vehicles - flag, don't rank.
|
||||
|
||||
**Stage 2a** (fundlab/cef_character.py): 50-fund shortlist (top 40
|
||||
return + >=3 positive scenarios + top 15 payout) -> 35-sleeve
|
||||
factor_screen -> taxplan.sleeve_score character -> crude
|
||||
tax_arb = character x (upside + 0.4 x vol). Top: KF (char 0.63),
|
||||
LENDX (0.69, maxDD -12%, all scenarios ~flat/positive), EMF/AEF/HQL/
|
||||
HQH/IGD equity-character 0.55-0.75; EMO/NML/SRV/KYN/TYG char
|
||||
0.29-0.39 (income-heavy -> IRA-tilted, ROC share pending the 1099).
|
||||
|
||||
**Remaining (stage 2b+):** per-fund N-2ASR/N-CSR distribution
|
||||
character (the income/gains/ROC split is decisive for the EMO
|
||||
family), NPORT NAV-per-share -> quarterly discount series, leverage
|
||||
from NPORT financials, BDC flag via form history, tender-offer
|
||||
status; app CEF tab.
|
||||
|
|
|
|||
1270
fundlab/cef_character.json
Normal file
1270
fundlab/cef_character.json
Normal file
File diff suppressed because it is too large
Load Diff
116
fundlab/cef_character.py
Normal file
116
fundlab/cef_character.py
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
"""CEF stage 2a: tax-character proxy for the shortlist.
|
||||
|
||||
Takes the cef_screen.json shortlist (top 5y return + crash-resilient +
|
||||
high-payout CEFs), runs the 35-sleeve factor screen on each, and scores
|
||||
the sleeve mix with taxplan.sleeve_score (same proxy the open-end
|
||||
candidates use). Then ranks by a crude "tax-arb per dollar" heuristic
|
||||
for a constrained taxable account:
|
||||
|
||||
tax_arb ~= character x (upside + 0.4 x vol)
|
||||
|
||||
character share of the return mix that is tax-favorable (0..1)
|
||||
upside max(t5,0)/5 - the LTCG-deferral term
|
||||
0.4 x vol5 ~ expected |annual loss| (normal approx) - the
|
||||
harvest term (0.4 = 1/sqrt(2*pi))
|
||||
|
||||
Both terms are per dollar of balance; the heuristic is a SCREENER,
|
||||
the EDGAR deep dive (N-2ASR/N-CSR distribution character + NPORT NAV
|
||||
for the discount) is what decides.
|
||||
|
||||
Output: fundlab/cef_character.json
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from fundlab import factors, taxplan
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
SCREEN = HERE / "cef_screen.json"
|
||||
OUT = HERE / "cef_character.json"
|
||||
|
||||
# 6(c)-converted / non-fund tickers that show up in the SEC CEF report
|
||||
# (organized as CEFs, later converted to operating companies)
|
||||
NON_FUNDS = {"cfnd", "pwrl", "bot", "fxby"}
|
||||
SPECULATIVE = re.compile(r"corp\.?$|tech100", re.I)
|
||||
|
||||
|
||||
def select(res: dict, n_ret: int = 40, n_pay: int = 15) -> list[str]:
|
||||
rows = [v for v in res.values()
|
||||
if v.get("t5") is not None and v["sym"] not in NON_FUNDS]
|
||||
by_ret = sorted(rows, key=lambda x: -x["t5"])[:n_ret]
|
||||
by_res = [v for v in rows if v.get("n_pos_scen", 0) >= 3]
|
||||
by_pay = sorted(rows, key=lambda x: -(x.get("payout_12m") or 0))[:n_pay]
|
||||
out: dict[str, dict] = {}
|
||||
for v in by_ret + by_res + by_pay:
|
||||
out.setdefault(v["sym"], v)
|
||||
return list(out)
|
||||
|
||||
|
||||
def _betas(sym: str) -> dict:
|
||||
fs = factors.factor_screen(sym)
|
||||
if not fs:
|
||||
return {}
|
||||
return (fs.get("full") or fs.get("rec5") or {}).get("betas") or {}
|
||||
|
||||
|
||||
def run() -> dict:
|
||||
res = json.loads(SCREEN.read_text())
|
||||
syms = select(res)
|
||||
print(f"{len(syms)} shortlisted CEFs", flush=True)
|
||||
out: dict = {}
|
||||
for i, s in enumerate(syms, 1):
|
||||
v = res[s.upper()]
|
||||
comps = _betas(s.lower())
|
||||
char = taxplan.sleeve_score(comps) if comps else None
|
||||
t5 = v.get("t5") or 0.0
|
||||
vol = v.get("vol5") or 0.0
|
||||
upside = max(t5, 0.0) / 5.0
|
||||
harvest = 0.4 * vol
|
||||
arb = (char if char is not None else 0.3) * (upside + harvest)
|
||||
out[s] = {
|
||||
**{k: v.get(k) for k in ("name", "bdc", "days", "last")},
|
||||
"sym": s, "t5": t5, "t12": v.get("t12"), "vol5": vol,
|
||||
"maxdd5": v.get("maxdd5"),
|
||||
"2022": v.get("2022 bear mkt"), "2023": v.get("2023 rate shock"),
|
||||
"2025t": v.get("2025 tariff crash"),
|
||||
"2026": v.get("2026 Q1 drawdown"),
|
||||
"n_pos_scen": v.get("n_pos_scen"),
|
||||
"payout_12m": v.get("payout_12m"),
|
||||
"character": round(char, 2) if char is not None else None,
|
||||
"sleeves": {k: round(b, 2) for k, b in comps.items()
|
||||
if abs(b) > 0.15},
|
||||
"tax_arb": round(arb, 4),
|
||||
}
|
||||
print(f"{i:2}/{len(syms)} {s:7} char={out[s]['character']} "
|
||||
f"arb={out[s]['tax_arb']}", flush=True)
|
||||
OUT.write_text(json.dumps(out, indent=1, default=str))
|
||||
print(f"wrote {OUT}")
|
||||
return out
|
||||
|
||||
|
||||
def _pc(v: float | None) -> str:
|
||||
return "" if v is None else f"{v:+.1%}"
|
||||
|
||||
|
||||
def _print(out: dict, top: int = 35) -> None:
|
||||
rows = sorted(out.values(), key=lambda x: -x["tax_arb"])
|
||||
print(f"{'fund':7} {'name':40} {'char':>5} {'5yTR':>8} {'vol':>6}"
|
||||
f" {'maxDD':>8} {'2022':>8} {'2026':>8} {'dist12':>8} {'tax_arb':>8}")
|
||||
for v in rows[:top]:
|
||||
c = v["character"]
|
||||
print(f"{v['sym']:7} {v.get('name','')[:40]:40} "
|
||||
f"{('' if c is None else f'{c:.2f}'):>5} "
|
||||
f"{v['t5']:+8.1%} {v['vol5']:6.2f} "
|
||||
f"{_pc(v.get('maxdd5')):>8} {_pc(v.get('2022')):>8} "
|
||||
f"{_pc(v.get('2026')):>8} {_pc(v.get('payout_12m')):>8} "
|
||||
f"{v['tax_arb']:8.3f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
o = run()
|
||||
_print(o)
|
||||
6977
fundlab/cef_screen.json
Normal file
6977
fundlab/cef_screen.json
Normal file
File diff suppressed because it is too large
Load Diff
137
fundlab/cef_screen.py
Normal file
137
fundlab/cef_screen.py
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
"""CEF screen: stage 1 of the CEF pass.
|
||||
|
||||
Reads cef_universe.json (SEC active-CEF report + ticker map) and local
|
||||
price files (goget-downloaded). Per ticker:
|
||||
t5 / t12 total return (Adj Close)
|
||||
vol5 annualized daily-return vol, 5y
|
||||
maxdd5 deepest peak->trough drawdown, 5y
|
||||
scen_* return over each of the 5 market-crash episodes
|
||||
(drawdown.py episodes, fund_windows)
|
||||
payout_12m distribution proxy = 12m adj return - 12m price
|
||||
return (Close = market price, Adj Close = total
|
||||
return incl. reinvested distributions)
|
||||
p5y_share 5y version of the same
|
||||
|
||||
CEF-specific context for interpreting the output:
|
||||
- price = MARKET price; premium/discount dynamics are on top of NAV
|
||||
(stage 2 adds the NPORT NAV-per-share series)
|
||||
- distributions are NOT the fund's discretion to skip - a CEF usually
|
||||
keeps paying even in drawdowns (that's what the leverage is for),
|
||||
so the payout proxy is more stable than for open-end funds
|
||||
- BDCs in the set pay ordinary income + occasional ROC (stage 2
|
||||
flags them from the EDGAR form history)
|
||||
|
||||
Output: fundlab/cef_screen.json
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from fundlab import drawdown
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
UNIVERSE = HERE / "cef_universe.json"
|
||||
EPS = json.loads((HERE / "drawdown_results.json").read_text())["episodes"]
|
||||
OUT = HERE / "cef_screen.json"
|
||||
SCEN_KEYS = ["2022 bear mkt", "2023 rate shock", "2024 vol spike",
|
||||
"2025 tariff crash", "2026 Q1 drawdown"]
|
||||
|
||||
|
||||
def _load(sym: str) -> pd.DataFrame | None:
|
||||
f = Path.home() / "prog/fin/stocks" / f"{sym.lower()}-history.csv"
|
||||
if not f.exists():
|
||||
return None
|
||||
try:
|
||||
d = pd.read_csv(f, parse_dates=["Date"])
|
||||
except Exception:
|
||||
return None
|
||||
d = d.set_index("Date").sort_index()
|
||||
d = d[~d.index.duplicated(keep="last")]
|
||||
return d
|
||||
|
||||
|
||||
def _perf(d: pd.DataFrame) -> dict:
|
||||
adj = d["Adj Close"].dropna()
|
||||
px = d["Close"].dropna()
|
||||
out: dict = {}
|
||||
for tag, n in (("5", 5 * 252), ("12", 252)):
|
||||
a = adj.tail(n)
|
||||
p = px.tail(n)
|
||||
out[f"t{tag}"] = float(a.iloc[-1] / a.iloc[0] - 1) if len(a) > 20 else None
|
||||
out[f"p{tag}"] = float(p.iloc[-1] / p.iloc[0] - 1) if len(p) > 20 else None
|
||||
a5 = adj.tail(5 * 252)
|
||||
if len(a5) > 20:
|
||||
r = a5.pct_change().dropna()
|
||||
out["vol5"] = float(r.std() * np.sqrt(252))
|
||||
peak = np.maximum.accumulate(a5.to_numpy())
|
||||
out["maxdd5"] = float(((a5.to_numpy() / peak) - 1).min())
|
||||
else:
|
||||
out["vol5"] = out["maxdd5"] = None
|
||||
if out.get("t12") is not None and out.get("p12") is not None:
|
||||
out["payout_12m"] = out["t12"] - out["p12"]
|
||||
if out.get("t5") is not None and out.get("p5") is not None:
|
||||
out["payout_5y_share"] = (out["t5"] - out["p5"]) / out["t5"] \
|
||||
if abs(out["t5"]) > 1e-4 else None
|
||||
return out
|
||||
|
||||
|
||||
def screen_one(sym: str) -> dict:
|
||||
s = sym.lower() # price files are lowercase
|
||||
d = _load(s)
|
||||
if d is None or len(d) < 30:
|
||||
return {"error": "no data"}
|
||||
out = _perf(d)
|
||||
out.update(drawdown.fund_windows(s, EPS))
|
||||
return out
|
||||
|
||||
|
||||
def run() -> dict:
|
||||
uni = json.loads(UNIVERSE.read_text())
|
||||
res: dict = {}
|
||||
for t in sorted(uni):
|
||||
info = dict(uni[t])
|
||||
info["sym"] = t.lower()
|
||||
s = screen_one(t)
|
||||
info.update(s)
|
||||
info["n_pos_scen"] = sum(
|
||||
1 for k in SCEN_KEYS if info.get(k) is not None and info[k] > 0)
|
||||
res[t] = info
|
||||
n_ok = sum(1 for v in res.values() if "t5" in v)
|
||||
print(f"{n_ok}/{len(res)} CEFs screened", flush=True)
|
||||
OUT.write_text(json.dumps(res, indent=1, default=str))
|
||||
print(f"wrote {OUT}")
|
||||
return res
|
||||
|
||||
|
||||
def _fmt(x, pct: bool = True, dec: int = 2) -> str:
|
||||
if x is None:
|
||||
return ""
|
||||
if pct:
|
||||
return f"{x:+.{dec}%}"
|
||||
return f"{x:.{dec}f}"
|
||||
|
||||
|
||||
def _print(res: dict, top: int = 30, sort: str = "t5") -> None:
|
||||
rows = [v for v in res.values() if v.get("t5") is not None]
|
||||
print(f"{'fund':6} {'name':42} {'5yTR':>8} {'vol':>6} {'maxDD':>8}"
|
||||
f" {'2022':>8} {'2023':>8} {'2025t':>8} {'2026':>8}"
|
||||
f" {'dist12':>8} {'n+':>3}")
|
||||
for v in sorted(rows, key=lambda x: -x.get(sort, -9))[:top]:
|
||||
print(f"{v['sym']:6} {v.get('name', '')[:42]:42}"
|
||||
f" {_fmt(v.get('t5')):>8} {_fmt(v.get('vol5'), False):>6}"
|
||||
f" {_fmt(v.get('maxdd5')):>8}"
|
||||
f" {_fmt(v.get('2022 bear mkt')):>8}"
|
||||
f" {_fmt(v.get('2023 rate shock')):>8}"
|
||||
f" {_fmt(v.get('2025 tariff crash')):>8}"
|
||||
f" {_fmt(v.get('2026 Q1 drawdown')):>8}"
|
||||
f" {_fmt(v.get('payout_12m')):>8}"
|
||||
f" {v.get('n_pos_scen', 0):>3}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
r = run()
|
||||
_print(r)
|
||||
2657
fundlab/cef_universe.json
Normal file
2657
fundlab/cef_universe.json
Normal file
File diff suppressed because it is too large
Load Diff
159
fundlab/cef_universe.py
Normal file
159
fundlab/cef_universe.py
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
"""CEF universe: the SEC's official "Closed-End Fund Information" report.
|
||||
|
||||
Primary source: https://www.sec.gov/files/investment/data/other/
|
||||
closed-end-fund-information/closed-end-investment-company-<YYYY>.csv
|
||||
(all ACTIVE 1940-Act closed-end companies: CIK, name, addresses,
|
||||
filing status - 973 for 2026). This supersedes the N-2 full-index
|
||||
approach (N-2/N-2-A filers are the CEFs that updated their
|
||||
registration within the window - a biased subset; the report is the
|
||||
whole active population).
|
||||
|
||||
Tickers: CIK join with the SEC company_tickers.json exchange map
|
||||
(305 of 973 match - the rest are small/dark/OTC funds). Hyphenated
|
||||
tickers (GAM-PB etc.) are PREFERRED series - dropped (their price
|
||||
history is the preferred's near-flat NAV, useless for screening);
|
||||
the common class is usually the fund's other listed security.
|
||||
|
||||
CEF notes for the rest of the pipeline:
|
||||
- CEFs file NPORT-P (same format as open-end - xcheck.py works)
|
||||
- shareholder reports are N-CSR/N-CSRS AND/OR N-2ASR (rocdetect must
|
||||
try both)
|
||||
- CEFs file N-PX proxies (open-end funds don't - a CEF signal)
|
||||
- price = MARKET price (premium/discount to NAV is a free variable)
|
||||
- Yahoo instrumentType is usually EQUITY - do NOT apply the open-end
|
||||
MUTUALFUND filter
|
||||
- BDCs are in the report too (name-flagged)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import urllib.request
|
||||
from datetime import date, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
CACHE = HERE / "universe_cache"
|
||||
CEF_CSV = CACHE / "cef_sec_2026.csv"
|
||||
UNIVERSE = HERE / "cef_universe.json"
|
||||
DATA = Path.home() / "prog/fin/stocks"
|
||||
UA = {"User-Agent": "research test@example.com"}
|
||||
MIN_DAYS = 1250 # >=5y of trading history (same bar as the 497 pass)
|
||||
STALE_DAYS = 35 # last quote older than this = delisted
|
||||
|
||||
|
||||
def _get(url: str) -> bytes:
|
||||
req = urllib.request.Request(url, headers=UA)
|
||||
return urllib.request.urlopen(req, timeout=120).read()
|
||||
|
||||
|
||||
def fetch_cef_report(year: int = 2026) -> list[dict]:
|
||||
"""The SEC active-CEF report (CIK, name, ...). Cached locally."""
|
||||
if CEF_CSV.exists():
|
||||
raw = CEF_CSV.read_bytes()
|
||||
else:
|
||||
CACHE.mkdir(exist_ok=True)
|
||||
# 2024+ uses hyphenated filenames, earlier underscored
|
||||
for pat in (f"closed-end-investment-company-{year}.csv",
|
||||
f"closed-end_investment_company_{year}.csv",
|
||||
"closed-end_investment_company.csv"):
|
||||
url = ("https://www.sec.gov/files/investment/data/other/"
|
||||
f"closed-end-fund-information/{pat}")
|
||||
try:
|
||||
raw = _get(url)
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
else:
|
||||
raise RuntimeError("could not fetch the SEC CEF report")
|
||||
CEF_CSV.write_bytes(raw)
|
||||
rows = list(csv.DictReader(io.StringIO(
|
||||
raw.decode("utf-8-sig"))))
|
||||
return [{"cik": r["CIK"].lstrip("0") or "0",
|
||||
"name": r["Registrant_Name"].strip()} for r in rows]
|
||||
|
||||
|
||||
def _company_tickers() -> dict:
|
||||
"""cik(int) -> (ticker, title) from the SEC exchange-listed map."""
|
||||
d = json.loads(_get("https://www.sec.gov/files/company_tickers.json"))
|
||||
out: dict[int, tuple] = {}
|
||||
for v in d.values():
|
||||
cik = v.get("cik_str", v.get("cik"))
|
||||
t = v.get("ticker", v.get("symbol"))
|
||||
if cik is None or not t:
|
||||
continue
|
||||
out[int(cik)] = (t.upper(), v.get("title", ""))
|
||||
return out
|
||||
|
||||
|
||||
def _history(sym: str) -> tuple[int, str]:
|
||||
"""(trading days, last date) from the local price file."""
|
||||
p = DATA / f"{sym}-history.csv"
|
||||
if not p.exists():
|
||||
return 0, ""
|
||||
last = ""
|
||||
n = 0
|
||||
with p.open() as f:
|
||||
next(f, None) # header
|
||||
for line in f:
|
||||
n += 1
|
||||
last = line.split(",")[0]
|
||||
return n, last
|
||||
|
||||
|
||||
def run() -> dict:
|
||||
cefs = fetch_cef_report()
|
||||
print(f"{len(cefs)} active CEFs in the SEC report", flush=True)
|
||||
tick = _company_tickers()
|
||||
|
||||
# NOTE: do NOT exclude tickers present in the local DB - goget
|
||||
# writes <sym>.json there, so after the first download every CEF
|
||||
# would look "known". The SEC report already defines the set.
|
||||
known = set()
|
||||
for j in (Path(__file__).parent.parent / "funds.json",
|
||||
HERE / "search_all.json", HERE / "search_external.json",
|
||||
HERE / "xcheck_report.json"):
|
||||
if j.exists():
|
||||
d = json.loads(j.read_text())
|
||||
if isinstance(d, dict):
|
||||
known |= {k.upper() for k in d}
|
||||
|
||||
stale = (date.today() - timedelta(days=STALE_DAYS)).toordinal()
|
||||
out: dict[str, dict] = {}
|
||||
n_pref = n_unmatched = 0
|
||||
for c in cefs:
|
||||
hit = tick.get(int(c["cik"]))
|
||||
if not hit:
|
||||
n_unmatched += 1
|
||||
continue
|
||||
t, _yt = hit
|
||||
if "-" in t: # preferred/series class
|
||||
n_pref += 1
|
||||
continue
|
||||
if t in known:
|
||||
continue
|
||||
days, last = _history(t.lower())
|
||||
live = False
|
||||
try:
|
||||
live = date.fromisoformat(last).toordinal() >= stale
|
||||
except ValueError:
|
||||
pass
|
||||
out[t] = {"cik": c["cik"], "name": c["name"],
|
||||
"bdc": bool(re.search(r"business development",
|
||||
c["name"], re.I)),
|
||||
"listed": days >= MIN_DAYS and live,
|
||||
"days": days, "last": last, "known": False}
|
||||
n_listed = sum(1 for v in out.values() if v["listed"])
|
||||
n_bdc = sum(1 for v in out.values() if v["bdc"])
|
||||
print(f"{len(out)} CEF tickers ({n_listed} live with >=5y history, "
|
||||
f"{n_bdc} BDCs, {n_pref} preferred-series dropped, "
|
||||
f"{n_unmatched} CIKs with no listed ticker)", flush=True)
|
||||
UNIVERSE.write_text(json.dumps(out, indent=1))
|
||||
print(f"wrote {UNIVERSE}")
|
||||
return out
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
137
fundlab/goget_cef.log
Normal file
137
fundlab/goget_cef.log
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
goget
|
||||
|
||||
Symbol = txemf
|
||||
txemf: Bad format (timestamp)
|
||||
Symbol = phd
|
||||
Error fetching phd. Retries: 5: aborting
|
||||
Get error (phd): 404 Not Found
|
||||
Symbol = glq
|
||||
Symbol = fmy
|
||||
Symbol = pgp
|
||||
Symbol = etv
|
||||
Symbol = etw
|
||||
Symbol = iga
|
||||
Symbol = spxx
|
||||
Symbol = ety
|
||||
Symbol = bta
|
||||
Symbol = glo
|
||||
Symbol = agd
|
||||
Symbol = caf
|
||||
Symbol = fof
|
||||
Symbol = exg
|
||||
Symbol = aod
|
||||
Symbol = nie
|
||||
Symbol = jce
|
||||
Symbol = iae
|
||||
Symbol = eod
|
||||
Symbol = edd
|
||||
Symbol = awp
|
||||
Symbol = etj
|
||||
Symbol = chw
|
||||
Symbol = srv
|
||||
Symbol = ide
|
||||
Symbol = nuw
|
||||
Symbol = eot
|
||||
Symbol = igi
|
||||
Symbol = rcg
|
||||
Symbol = jls
|
||||
Symbol = gdo
|
||||
Symbol = nbb
|
||||
Symbol = bsl
|
||||
Symbol = nhs
|
||||
Symbol = bbn
|
||||
Symbol = oxlcg
|
||||
Symbol = gbab
|
||||
Symbol = ihd
|
||||
Symbol = ltcfx
|
||||
Symbol = heq
|
||||
Symbol = hyi
|
||||
Symbol = psf
|
||||
Symbol = edf
|
||||
Symbol = bgx
|
||||
Symbol = bwg
|
||||
Symbol = nxg
|
||||
Symbol = dpg
|
||||
Symbol = ardc
|
||||
Symbol = kio
|
||||
Symbol = emo
|
||||
Symbol = mmd
|
||||
Symbol = ccid
|
||||
Symbol = bgh
|
||||
Symbol = dma
|
||||
Symbol = vgi
|
||||
Symbol = isd
|
||||
Symbol = jri
|
||||
Symbol = pdi
|
||||
Symbol = dxr
|
||||
Symbol = bgb
|
||||
Symbol = ldp
|
||||
Symbol = bpre
|
||||
Symbol = ghy
|
||||
Symbol = pgz
|
||||
Symbol = nml
|
||||
Symbol = etx
|
||||
Symbol = dmb
|
||||
Symbol = fpf
|
||||
Symbol = banx
|
||||
Symbol = thq
|
||||
Symbol = nms
|
||||
Symbol = qqqx
|
||||
Symbol = eccv
|
||||
Symbol = jgh
|
||||
Symbol = bst
|
||||
Symbol = thw
|
||||
Symbol = acv
|
||||
Symbol = rsf
|
||||
Symbol = lendx
|
||||
Symbol = nrsax
|
||||
Symbol = ra
|
||||
Symbol = xflt
|
||||
Symbol = ascix
|
||||
Symbol = cpz
|
||||
Symbol = occim
|
||||
Symbol = pmfax
|
||||
Symbol = nichx
|
||||
Symbol = fins
|
||||
Symbol = rmi
|
||||
Symbol = owscx
|
||||
Symbol = larax
|
||||
Symbol = eica
|
||||
Symbol = pdx
|
||||
Symbol = pdskx
|
||||
Symbol = rmm
|
||||
Symbol = aio
|
||||
Symbol = rfm
|
||||
Symbol = ndmo
|
||||
Symbol = asgi
|
||||
Symbol = pdo
|
||||
Symbol = fthy
|
||||
Symbol = vcrdx
|
||||
Symbol = sdhy
|
||||
Symbol = rfmz
|
||||
Symbol = wdi
|
||||
Symbol = npct
|
||||
Symbol = nbxg
|
||||
Symbol = megi
|
||||
Symbol = nmai
|
||||
Symbol = ecat
|
||||
Symbol = gug
|
||||
Symbol = npfd
|
||||
Symbol = rmmz
|
||||
Symbol = rlty
|
||||
Symbol = paxs
|
||||
Symbol = eiia
|
||||
Symbol = dxyz
|
||||
Symbol = spma
|
||||
Symbol = pdpa
|
||||
Symbol = cfnd
|
||||
Symbol = unid
|
||||
Error fetching unid. Retries: 5: aborting
|
||||
Get error (unid): 404 Not Found
|
||||
Symbol = uniu
|
||||
Error fetching uniu. Retries: 5: aborting
|
||||
Get error (uniu): 404 Not Found
|
||||
Symbol = bot
|
||||
Symbol = rvi
|
||||
Symbol = pwrl
|
||||
Failed downloads: [phd unid uniu]
|
||||
2
fxby-split.csv
Normal file
2
fxby-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2021-05-07,1:5
|
||||
|
2
grf-split.csv
Normal file
2
grf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2005-05-11,115:100
|
||||
|
3
herz-split.csv
Normal file
3
herz-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
2026-02-09,1:10
|
||||
2021-11-18,1034:1000
|
||||
|
2
iaf-split.csv
Normal file
2
iaf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2025-10-23,1:3
|
||||
|
3
kf-split.csv
Normal file
3
kf-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
1988-10-12,3:1
|
||||
2008-12-22,1:10
|
||||
|
3
mci-split.csv
Normal file
3
mci-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Date,Stock Splits
|
||||
1998-01-21,2:1
|
||||
2011-02-22,2:1
|
||||
|
2
mxf-split.csv
Normal file
2
mxf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
1993-09-03,10547:10000
|
||||
|
2
nro-split.csv
Normal file
2
nro-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2025-01-23,1048:1000
|
||||
|
2
peo-split.csv
Normal file
2
peo-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2000-10-20,3:2
|
||||
|
2
rvt-split.csv
Normal file
2
rvt-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2013-10-18,1084:1000
|
||||
|
2
saba-split.csv
Normal file
2
saba-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2024-06-24,1:2
|
||||
|
2
swz-split.csv
Normal file
2
swz-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
1998-10-16,2:1
|
||||
|
|
|
@ -576,6 +576,40 @@ def test_drawdown() -> None:
|
|||
check("fund_windows callable", callable(w), "")
|
||||
|
||||
|
||||
def test_cef() -> None:
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from fundlab import cef_screen, cef_character
|
||||
print("\ncef")
|
||||
# _perf: flat market price, total-return line +30% over 3y -> the
|
||||
# whole return is payout, maxDD ~ 0 (monotone up)
|
||||
n = 3 * 252
|
||||
idx = pd.bdate_range("2023-01-02", periods=n)
|
||||
adj = 100.0 * (1 + np.arange(n) / 252 * 0.10)
|
||||
p = cef_screen._perf(pd.DataFrame({"Close": 100.0,
|
||||
"Adj Close": adj}, index=idx))
|
||||
check("_perf t5 approx +30% (3y window)",
|
||||
p["t5"] is not None and 0.29 < p["t5"] < 0.31, f"{p['t5']}")
|
||||
check("_perf payout_12m == t12 - p12",
|
||||
abs(p["payout_12m"] - (p["t12"] - p["p12"])) < 1e-12,
|
||||
f"{p['payout_12m']}")
|
||||
check("_perf vol5 positive", p["vol5"] is not None and p["vol5"] > 0)
|
||||
check("_perf maxdd5 <= 0 (monotone up -> ~0)",
|
||||
p["maxdd5"] is not None and p["maxdd5"] <= 1e-9,
|
||||
f"{p['maxdd5']}")
|
||||
# select(): top return + resilient + payout; NON_FUNDS excluded
|
||||
scr = {
|
||||
"AAA": {"sym": "aaa", "t5": 0.5, "n_pos_scen": 0, "payout_12m": 0.0},
|
||||
"BBB": {"sym": "bbb", "t5": 0.4, "n_pos_scen": 4, "payout_12m": 0.0},
|
||||
"CCC": {"sym": "ccc", "t5": -0.5, "n_pos_scen": 0, "payout_12m": 0.2},
|
||||
"CFND": {"sym": "cfnd", "t5": 0.9, "n_pos_scen": 0, "payout_12m": 0.0},
|
||||
}
|
||||
sel = cef_character.select(scr)
|
||||
check("select excludes NON_FUNDS", "cfnd" not in sel, str(sel))
|
||||
check("select includes top return + resilient + payout",
|
||||
{"aaa", "bbb", "ccc"} <= set(sel), str(sel))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
test_pool()
|
||||
test_text_and_objective()
|
||||
|
|
@ -590,6 +624,7 @@ def main() -> int:
|
|||
test_xcheck()
|
||||
test_taxplan()
|
||||
test_drawdown()
|
||||
test_cef()
|
||||
test_edgar_live()
|
||||
print(f"\n{PASS} passed, {FAIL} failed")
|
||||
return 1 if FAIL else 0
|
||||
|
|
|
|||
2
tyg-split.csv
Normal file
2
tyg-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2020-05-01,1:4
|
||||
|
2
utf-split.csv
Normal file
2
utf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2025-09-22,1017:1000
|
||||
|
2
vlt-split.csv
Normal file
2
vlt-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2009-05-26,1:5
|
||||
|
2
ztr-split.csv
Normal file
2
ztr-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Date,Stock Splits
|
||||
2012-06-27,1:4
|
||||
|
Loading…
Reference in New Issue
Block a user