gmp/rssd
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Walk all YouTube sources fully; channel order is not newest-first

Browse Source 
The Iron Culture channel lists recent episodes mid-list (Ep 392/393 at
~position 360 of 396), so the 'channels enumerate newest-first' boundary
assumption is wrong: with it, rssd would break at the first pre-start-date
video and miss recent episodes, while re-resolving a boundary video every
poll. Walk channels exactly like playlists: always record out-of-range
videos as skipped so discovery stays incremental.

Also ignore media files (manual yt-dlp downloads have landed in the
working tree before).
This commit is contained in:
Greg Pomerantz 2026-10-21 08:17:21 -04:00
parent c562ecc411
commit 4b5e48a26d
2 changed files with 11 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.gitignore vendored Normal file
View File

@ -0,0 +1,6 @@
# Podcast download byproducts
*.webm
*.mp4
*.opus
*.mp3

19
main.go
View File

@ -175,11 +175,6 @@ func pollYouTube(feed config.Feed, sm *state.Manager, jobs chan<- downloader.Dow
})
start := feed.PublishedAt()
// Channels enumerate newest-first, so the first video older than the start
// date is a stable boundary: everything after it is older too. Playlists
// have no guaranteed order, so they must be walked fully and their
// out-of-range videos recorded as skipped to avoid re-resolving them.
newestFirst := !strings.Contains(strings.ToLower(feed.URL), "list=")
newCount, enqueued, skipped := 0, 0, 0
// Per-video resolve timeout. A single pathological video (JS-challenge
@ -200,16 +195,12 @@ func pollYouTube(feed config.Feed, sm *state.Manager, jobs chan<- downloader.Dow
continue
}
// Out of the start_date window.
// Out of the start_date window. Record as skipped so it isn't
// re-resolved next poll. (Channel video order is not reliably
// newest-first — e.g. the Iron Culture channel lists recent episodes
// mid-list — so we can't stop at the first out-of-range video and
// must walk the whole source.)
if !start.IsZero() && !info.UploadDate.IsZero() && info.UploadDate.Before(start) {
if newestFirst {
// Stop at the first out-of-range video without recording it. It
// stays "new" each poll, so it remains a stable boundary and we
// never walk backward through older videos.
fmt.Fprintf(os.Stderr, "rssd: %s reached pre-start-date boundary, stopping\n", feed.URL)
break
}
// Playlist: record as skipped so it isn't re-resolved next poll.
dp := computeYTDestPath(feed.OutputDir, info.UploadDate, info.Title, v.ID)
if sm.AddJobWithStatus(feed.URL, watchURL(v.ID), dp, state.StatusSkipped, info.Duration) {
skipped++