--- title: "Latent Space Podcast Downloader" description: "Latent Space Podcast Downloader This project is a Python script that downloads podcast episodes from latent.space, with automatic YouTube fallback for episodes not present in the RSS feed. The challen…" doc_version: 1 last_updated: 2026-10-17 canonical_url: "http://parc.yolo.scapegoat.dev/note/projects/2026/05/02/proj-latent-space-podcast-downloader" --- # Latent Space Podcast Downloader Latent Space Podcast Downloader This project is a Python script that downloads podcast episodes from latent.space, with automatic YouTube fallback for episodes not present in the RSS feed. The challen… ## Tags - `project` - `python` - `podcast` - `rss` - `web-scraping` - `youtube` - `surf-cli` - `docmgr` ## Content # Latent Space Podcast Downloader This project is a Python script that downloads podcast episodes from latent.space, with automatic YouTube fallback for episodes not present in the RSS feed. The challenge that made it interesting was not the download itself—it is a solved problem—but the discovery process: figuring out that the RSS feed has only 20 items, that older podcast episodes get pushed out by AINews posts, that the audio detection requires reading MIME types rather than file extensions, and that the surf CLI has non-obvious flag conventions that require exploration before use. > [!summary] > This project has three intertwined identities: > 1. a **podcast downloader** that parses the latent.space RSS feed and falls back to YouTube > 2. a **surf CLI field guide** documenting what works, what doesn't, and why for this particular toolchain > 3. a **web discovery journal** showing how to navigate from a search result to a downloadable file when the naive approach fails ## Why this project exists latent.space is one of the most technically substantive AI engineering podcasts available. The episode about OpenAI's internal agent orchestration system "Symphony" — featuring Ryan Lopopolo, who led the experiment of building a million-line codebase with zero human-written code — deserved to be downloaded and kept. The podcast is hosted on Substack, which means the audio lives behind a feed, but the feed is not paginated and the episode page itself does not expose a direct download link in the browser. The goal was to produce a durable, reusable script that could download any latent.space episode by slug or title, with intelligent fallback when the feed does not have the episode. ## Current project status The project is complete and working. What exists: - `scripts/01-download-latent-space.py` — a Python download script with feed parsing and YouTube fallback - `podcasts/latent-space-harness-eng-ryans-lopopolo-openai-symphony.mp3` — the downloaded episode (106 MB, 78 min, 189 kbps) - A docmgr ticket documenting the investigation: `ttmp/2026/05/02/latent-space-openai-symphony--download-latent-space-podcast-episode-openai-symphony-harness-engineering/` - A detailed implementation diary in the ticket's `reference/02-diary.md` What remains: - A full `--help` docstring and command-line interface could be polished - The script currently writes to `podcasts/` in the working directory; could be made configurable via environment variable - Archive page support for latent.space (see "Open questions" below) ## Project shape ``` claw-stuff/ podcasts/ latent-space-harness-eng-ryans-lopopolo-openai-symphony.mp3 # downloaded episode scripts/ (global scripts, if any) ttmp/ 2026/ 05/ 02/ latent-space-openai-symphony--download-latent-space-podcast-episode-openai-symphony-harness-engineering/ index.md changelog.md tasks.md reference/ 02-diary.md # implementation journal scripts/ 01-download-latent-space.py # the download script ``` ## Architecture The download system has three layers, tried in sequence: ```mermaid flowchart LR A[User query -- slug or title] --> B[Parse RSS feed latent.space/feed] B --> C{Episode found with audio URL?} C -- Yes --> D[Download from Substack CDN] C -- No --> E[Search YouTube yt-dlp] E --> F[Extract audio as MP3] D --> G[Save to podcasts/] F --> G ``` The feed lookup is the primary path. YouTube is the fallback. The script never hardcodes a URL—it always discovers the target through the feed or a search. ## Surf CLI: A Field Guide Before any download could begin, the surf CLI had to be understood. Surf is a Go-based browser automation CLI that communicates with a local Chromium host over a Unix socket. It has the structure of a multi-level command tree (`surf `) and uses a NDJSON protocol over a socket for request/response. The documentation in the tool itself is sparse and the help output references `surf-go` while the binary is named `surf`. These are the notes from the field. ### Commands that work The following commands were verified working and used in this project: ```bash # Search via Kagi surf kagi search --query "latent.space podcast openai symphony" # Open a URL in a new tab (note: --args-json, not --url) surf tab new --args-json '{"url":"https://www.latent.space/p/harness-eng"}' # Read the page content (interactive snapshot) surf page read # Read plain text only surf page text # Execute JavaScript in the page context surf js 'document.title' # Clear and list captured network requests surf network clear surf network list # Screenshot surf screenshot ``` ### Commands that fail and why The most common failure mode with surf is positional arguments. Unlike most CLIs where `surf kagi search "query with spaces"` would work, surf interprets anything after `kagi search` as a structured flag and throws "Too many arguments" when it encounters a bare string. The correct form always uses a named flag: ```bash # WRONG — "Too many arguments" surf kagi search "latent.space podcast" # CORRECT — uses --query flag surf kagi search --query "latent.space podcast" ``` The same pattern applies to `navigate` and other commands that expect structured input: ```bash # WRONG — unknown flag or too many arguments surf navigate --url "https://..." surf tab new --url "https://..." # CORRECT for tab creation — JSON wrapper surf tab new --args-json '{"url":"https://..."}' ``` The `tab new` command does not accept `--url` directly. The URL must be wrapped as a JSON object and passed via `--args-json`. This is the most non-obvious convention in the tool. ### Commands that do not exist Some expected commands are simply absent: - `surf page snapshot` — there is no `snapshot` subcommand under `page`. The correct subcommands are `read`, `text`, `search`, and `state`. - `surf wait --time N` — `wait` exists as a group but has no time-based sleep subcommand. Use bash `sleep N` instead. - `surf network list --filter "pattern"` — no filter flag exists. Filter results post-hoc with `--jq` or pipe to `grep`. ### Why surf exists in this context Surf is the right tool for navigating authenticated pages or pages with dynamic JavaScript rendering. For the latent.space episode page, the audio player was rendered by JavaScript and the page itself noted "Audio playback is not supported on your browser"—surf could have been used to inspect the DOM for the audio element. In practice, for this project, surf was used primarily for the Kagi search and page reading. The download itself used Python's `urllib` for direct HTTP and `yt-dlp` for YouTube extraction. ## RSS Feed Investigation The first approach was to parse the RSS feed directly and find the audio enclosure URL for the target episode. ### The 20-item problem The feed at `https://www.latent.space/feed` contains exactly 20 items. This is a hard limitation—Substack does not paginate the default RSS feed, and latent.space publishes an AINews post nearly every day. Each AINews post pushes one of the podcast episodes out of the 20-item window. The harness-eng episode was published on April 7, 2026. By May 2, it was no longer in the feed. The 20 most recent items at that time were mostly AINews posts from late April and early May. ``` RSS feed contents (May 2, 2026): 5 items with audio enclosures (actual podcast episodes) 15 items with image enclosures (AINews posts) Total items: 20 ``` This is the core constraint of the feed-based approach. It works for the 5 most recent podcast episodes but fails for anything older. ### Detecting audio enclosures correctly The initial feed parsing checked for common audio file extensions in the enclosure URL (`.mp3`, `.m4a`, etc.). This approach failed silently because the AINews posts have image enclosures (PNG, JPEG) whose URLs have image-like paths, and some enclosures had no extension at all. The correct approach is to check the `type` attribute of the `` element: ```xml ``` The Python implementation filters by MIME type prefix: ```python def is_audio(enclosure) -> bool: """Check if enclosure is an audio file based on MIME type.""" if enclosure is None: return False typ = enclosure.get("type", "") return typ.startswith("audio/") ``` This distinction is clean and correct. A podcast episode enclosure has `type="audio/mpeg"` or `type="audio/mp4"`. An AINews post enclosure has `type="image/png"` or `type="image/jpeg"`. ### What the feed reveals Once the audio detection is corrected, the feed shows exactly 5 podcast episodes among the 20 items. The feed does not expose a pagination parameter. The only way to reach older episodes via the feed would be to find an archive page or an alternate feed URL. The five episodes in the feed (as of May 2, 2026): ``` Physical AI that Moves the World — Qasar Younis & Peter Ludwig url: https://api.substack.com/feed/podcast/195677117/75596dbd...mp3 AIE Europe Debrief + Agent Labs Thesis: Unsupervised Learning url: https://api.substack.com/feed/podcast/195264855/02d7a076...mp3 Shopify's AI Phase Transition: 2026 Usage Explosion url: https://api.substack.com/feed/podcast/195067855/a8388eaf...mp3 🔬 Training Transformers to solve 95% failure rate of Cancer Trials url: https://api.substack.com/feed/podcast/194810752/a0db6b56...mp3 Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs url: https://api.substack.com/feed/podcast/194195821/4c29b9e9...mp3 ``` All five have `type="audio/mpeg"` enclosures and can be downloaded directly via curl or Python's urllib. ## YouTube Fallback When an episode is not found in the feed, the script searches YouTube via `yt-dlp`'s search capability: ```python def search_youtube(query: str) -> str | None: """Search YouTube for a video and return the first URL.""" search_query = f"ytsearch3:latent space {query}" result = subprocess.run( ["yt-dlp", "--no-playlist", "--get-id", search_query], capture_output=True, text=True, timeout=30, ) first_id = result.stdout.strip().split("\n")[0].strip() if first_id and not first_id.startswith("["): return f"https://www.youtube.com/watch?v={first_id}" return None ``` For the harness-eng episode, the search for "latent space harness engineering token billionaires lopopolo openai symphony" returned `CeOXx-XTYek` as the first result. The yt-dlp download command: ```bash yt-dlp -x --audio-format mp3 --audio-quality 0 \ --newline --quiet \ -o "podcasts/latent-space-harness-eng.%(ext)s" \ "https://www.youtube.com/watch?v=CeOXx-XTYek" ``` The flags mean: - `-x` — extract audio (remove video) - `--audio-format mp3` — convert to MP3 - `--audio-quality 0` — best quality - `--newline --quiet` — suppress progress bars (per user request for cleaner output) - `%(ext)s` — placeholder for the actual extension (yt-dlp converts to the target format) The downloaded file is 106 MB with a duration of 4673 seconds (78 minutes) and a bitrate of 189 kbps. The YouTube source is a mirror of the podcast, uploaded by a third party (latent.space does not appear to have an official YouTube channel, so this is likely user-submitted). ## Implementation Details ### Download script structure The script has four main functions: 1. `parse_feed()` — fetches and parses the RSS feed, returns a list of episode dicts with title, link, audio_url, and pub_date 2. `find_episode(episodes, query)` — matches by slug or title substring (case-insensitive) 3. `download_from_url(url, output_path)` — downloads a direct URL using Python urllib with a progress bar 4. `download_via_ytdlp(url, output_path, silent)` — downloads via yt-dlp for YouTube and other supported sites 5. `search_youtube(query)` — searches YouTube via yt-dlp and returns the first video URL The main logic: ```python # Feed lookup first ep = find_episode(episodes, search_query) if ep and ep["audio_url"]: ok = download_from_url(ep["audio_url"], output_path) elif ep and not ep["audio_url"]: # Episode exists in feed but no audio URL — fall back to YouTube yt_url = search_youtube(ep["title"]) ok = download_via_ytdlp(yt_url, output_path, silent=args.silent) else: # Episode not in feed — search YouTube directly yt_url = search_youtube(search_query) ok = download_via_ytdlp(yt_url, output_path, silent=args.silent) ``` The fallback chain is deliberate. An episode can be in the feed (meaning it exists as a page on latent.space) but without an audio enclosure (meaning it is an AINews post, not a podcast episode). Or it can be completely absent from the feed (the harness-eng scenario). The script handles both cases without separate user-facing flags. ### File naming The output filename is derived from the episode slug: ```python slug = ep["link"].rstrip("/").split("/")[-1] ext = Path(urllib.parse.urlparse(ep["audio_url"] or "").path).suffix or ".mp3" output_path = output_dir / f"{slug}{ext}" ``` For the harness-eng episode, this produces `latent-space-harness-eng-ryans-lopopolo-openai-symphony.mp3`. The full title is included because the slug alone (`harness-eng`) is not descriptive enough for a file that might live in `podcasts/` alongside many others. ### Command-line interface ```bash # List available episodes python3 scripts/01-download-latent-space.py # Download by slug (from feed) python3 scripts/01-download-latent-space.py harness-eng # Download by title (from feed with auto YouTube fallback) python3 scripts/01-download-latent-space.py "Ryan Lopopolo" # Force YouTube search (skip feed lookup) python3 scripts/01-download-latent-space.py --youtube "harness engineering token billionaires" # Silent download (no yt-dlp progress bars) python3 scripts/01-download-latent-space.py harness-eng --silent # Auto-confirm overwrite python3 scripts/01-download-latent-space.py harness-eng --yes # Custom output directory python3 scripts/01-download-latent-space.py harness-eng -o /path/to/podcasts ``` ## What was tricky ### The MIME type vs extension trap The first implementation checked for audio by looking at the file extension in the enclosure URL. This failed because AINews posts have image enclosures whose URLs sometimes contain encoded paths that look like they could be audio files, and because some enclosure URLs have no extension at all. The fix—checking `type="audio/mpeg"`—is simple in hindsight but required reading the raw XML to understand what the feed was actually returning. ### The feed 20-item ceiling Substack's RSS feed is not paginated. There is no `?page=2` or `