evo-web-essay-scraper
Scrapes and extracts clean text from Paul Graham essays on paulgraham.com. Handles legacy table-based HTML, character encoding, and text chunking for TTS APIs.
SKILL.mdtasks-evolved/pg-essay-to-audiobook/environment/skills/evo-web-essay-scraper/SKILL.md
GitHubevo-web-essay-scraper
Scrapes Paul Graham essays and prepares text for TTS consumption.
Key Functions
scrape_pg_essay(url)- Fetches and extracts clean text from a PG essay URLclean_text(text)- Cleans text artifacts, encoding issues, nav boilerplatechunk_text(text, max_chars=4000)- Splits text into API-safe chunks by sentence boundaries
Essay URLs
- "Do Things that Don't Scale":
http://www.paulgraham.com/ds.html - "Founder Mode":
http://www.paulgraham.com/foundermode.html
Usage
import sys
sys.path.insert(0, '/app/environment/skills/evo-web-essay-scraper/scripts')
from utils import scrape_pg_essay, clean_text, chunk_text
text = scrape_pg_essay('http://www.paulgraham.com/ds.html')
chunks = chunk_text(text, max_chars=4000)
Technical Notes
- PG site uses table-based HTML layout, text in
<font>tags inside<table width="435">or similar - Encoding: use
response.apparent_encodingto handle smart quotes/em-dashes - Remove nav boilerplate like "Want to start a startup?"
- Chunk at sentence boundaries, max 4000 chars for OpenAI TTS (limit is 4096)