f/pubdx-dividend.csv at bdb887dee8126844e24b26d300f1f9da84f59d74

gmp/f

Greg Pomerantz 3c5bee96ce Clean multi-vintage 'ctime' files: 1,065 files, 2.2M stale rows

Some dump files carry a trailing 'ctime' column: an older pipeline
appended a FULL re-download of the symbol per download (up to ~28
vintages per date, 2019-2022). The consumer keeps the last row per date,
so it silently used stale (~2021) vintages where Yahoo later revised
values. scripts/clean_multivintage.py keeps, per date, the rows from the
newest ctime (event files keep distinct same-day amounts; history/split
keep the single newest row), drops the ctime column, sorts by date.
Applied to the data root and overrides/frozen (which carried the same
artifact): 1,065 files, 2,202,712 stale rows dropped.

Also: stripped 3 UTF-8 BOMs (teg/lo/krft), refreshed the event backup
with the cleaned files, and re-ran the double-listing scan: category C
(83 symbols, 18,636 repeated within-file rows) is fully explained by the
multi-vintage artifact and is now gone; A (6,589 pairs, corrected) and
B (12,366 pairs, open review list) are unchanged. Whole-set structural
QC after cleaning: 0 duplicate dates, 3 syms/15 rows of OHLC invariant
violations, 15 syms/2,897 rows of non-positive prices (mostly
long-delisted tickers), 8 unsorted files (reader sorts).

2026-08-31 22:25:02 -04:00

2.1 KiB

Raw Blame History

1	Date	Dividends
2	2008-08-29	0.021
3	2008-09-30	0.014
4	2008-10-31	0.008
5	2008-11-28	0.012
6	2008-12-10	0.004
7	2008-12-31	0.01
8	2009-01-30	0.011
9	2009-02-27	0.018
10	2009-03-31	0.03
11	2009-04-30	0.037
12	2009-05-29	0.039
13	2009-06-30	0.034
14	2009-07-31	0.028
15	2009-08-31	0.025
16	2009-09-30	0.018
17	2009-10-30	0.029
18	2009-11-30	0.02
19	2009-12-09	0.253
20	2009-12-31	0.031
21	2010-01-29	0.026
22	2010-02-26	0.021
23	2010-03-31	0.022
24	2010-04-30	0.054
25	2010-05-28	0.022
26	2010-06-30	0.016
27	2010-07-30	0.016
28	2010-08-31	0.02
29	2010-09-30	0.019
30	2010-10-29	0.02
31	2010-11-30	0.02
32	2010-12-08	0.024
33	2010-12-31	0.018
34	2011-01-31	0.017
35	2011-02-28	0.013
36	2011-03-31	0.017
37	2011-04-29	0.016
38	2011-05-31	0.02
39	2011-06-30	0.016
40	2011-07-29	0.012
41	2011-08-31	0.009
42	2011-09-30	0.009
43	2011-10-31	0.01
44	2011-11-30	0.012
45	2011-12-28	0.082
46	2011-12-30	0.006
47	2012-01-31	0.004
48	2012-02-29	0.012
49	2012-03-30	0.02
50	2012-04-30	0.02
51	2012-05-31	0.02
52	2012-06-29	0.015
53	2012-07-31	0.008
54	2012-08-31	0.006
55	2012-09-28	0.006
56	2012-10-31	0.008
57	2012-11-30	0.009
58	2012-12-12	0.014
59	2012-12-27	0.197
60	2012-12-31	0.005
61	2013-01-31	0.003
62	2013-02-28	0.006
63	2013-03-28	0.005
64	2013-04-30	0.007
65	2013-05-31	0.007
66	2013-06-28	0.003
67	2013-07-31	0.003
68	2013-08-30	0.003
69	2013-09-30	0.002
70	2013-10-31	0.001
71	2013-11-29	0.005
72	2013-12-11	0.043
73	2013-12-31	0.005
74	2014-01-31	0.005
75	2014-02-28	0.005
76	2014-03-31	0.007
77	2014-04-30	0.008
78	2014-05-30	0.01
79	2014-06-30	0.01
80	2014-07-31	0.014
81	2014-08-29	0.016
82	2014-09-30	0.015
83	2014-10-31	0.021
84	2014-11-28	0.02
85	2014-12-29	0.021
86	2014-12-31	0.022
87	2015-01-30	0.01
88	2015-02-27	0.005
89	2015-03-31	0.006
90	2015-04-30	0.018
91	2015-05-29	0.024
92	2015-06-30	0.024
93	2015-07-31	0.039
94	2015-08-31	0.028
95	2015-09-30	0.025
96	2015-10-30	0.025
97	2015-11-30	0.025
98	2015-12-29	0.33
99	2015-12-31	0.023
100	2016-01-29	0.015
101	2016-02-29	0.015
102	2016-03-31	0.018
103	2016-04-29	0.019
104	2016-05-31	0.019
105	2016-06-30	0.017
106	2016-07-29	0.016
107	2016-08-31	0.017
108	2016-09-30	0.014
109	2016-10-31	0.018
110	2016-11-30	0.02
111	2016-12-30	0.017
112	2017-01-31	0.008
113	2017-02-28	0.012
114	2017-03-31	0.017
115	2017-04-28	0.014
116	2017-05-31	0.014
117	2017-06-30	0.016
118	2017-07-31	0.015
119	2017-08-31	0.017
120	2017-09-29	0.017
121	2017-10-31	0.018
122	2017-11-30	0.019
123	2017-12-27	0.277
124	2017-12-29	0.02
125	2018-01-31	0.019
126	2018-02-28	0.02

2.1 KiB Raw Blame History

2.1 KiB

Raw Blame History