download-paper

Download the PDF of an academic paper (by DOI, publisher/doi.org URL, arXiv id, Anna's Archive md5, or title) with `paper-fetch`, the single paper-acquisition tool, including its browser fallback for

SKILL.md

GitHub

Download Paper

Core rule

paper-fetch is the only implementation for obtaining a paper PDF. Do not write a new downloader, call Anna's Archive or LibGen directly, or reach for annas-mcp or Playwright. Every route that works from a shell is already inside the tool, in the right order, with identity verification; the skill's job is to run it, and to drive the one step it cannot do itself: a real browser session for hosts that only answer a real browser. Shared logic: ~/My Drive/dotfiles/lib/python/paper_fetch.py.

Procedure

  1. Run the tool with a staging directory outside Drive. Use the citekey as --name when the paper is going into the bibliography.

    paper-fetch get "IDENT" --out "STAGING-DIR" [--name CITEKEY] [--author SURNAME]
    

    IDENT may be a DOI, a doi.org or publisher URL, an arXiv id, a 32-hex Anna's Archive md5, or a title (add --author to disambiguate a title). The tool resolves metadata through Crossref/OpenAlex, then tries open access (Unpaywall, OpenAlex, arXiv, publisher page), then LibGen -> Anna's Archive member fast download, then Anna's SciDB. It reads the Anna's Archive key itself; never pass or print a key or a signed download URL.

    Anna's Archive's only API is that member fast-download endpoint, which takes an md5; it has no search API, and the key does not unlock search. Going from a title or ISBN to an Anna's md5 therefore needs either LibGen (inside the tool) or Anna's /search in the browser. For books, that browser step is paper-fetch book-candidates's annas_browser_search handoff (docs/book-acquisition.md, "Anna's Archive search").

  2. Read status and act on it:

    status meaning next step
    ok (exit 0) file staged and its text matched the requested title or DOI use file; hand to add-bib-entry for attachment when wanted
    needs-browser (2) the only remaining copies sit behind a bot challenge (PhilPapers, PhilArchive, Anna's HTML) run the browser job below, then paper-fetch collect JOB
    not-member (3) Anna's Archive answered "Not a member": the paid membership lapsed tell Pablo; only he can renew. The md5 is in the message, so rerun with --md5 after renewal
    unavailable (4) every route failed read attempts; a failed route is not proof the work is unavailable everywhere. Try --author, a DOI instead of a title, or a version (preprint vs published)
    error (1) configuration problem (invalid key, quota, missing curl_cffi) fix the cause named in message

    A verification.verdict of unverified-no-text means a scanned PDF without a text layer: inspect the first page yourself before trusting it. mismatch files are deleted by the tool, never staged.

  3. Browser job (needs-browser). The tool wrote browser_job.path (JSON) and one JavaScript snippet per origin. Load chrome-use first, then use the session's Chrome control (Claude: Claude in Chrome; Codex CLI: the Playwright MCP browser_* tools; Codex desktop app: control-chrome), in a tab you create, and follow this order for each origin — it matters:

    1. Navigate the tab to the first item URL of that origin (the PDF itself). The challenge page ("Just a moment…", "DDoS-Guard") clears on its own within about ten seconds; wait until the tab title changes. Opening the origin's home page is not enough: PhilPapers challenges /archive/ separately.
    2. Navigate to the origin root (an HTML page, e.g. https://philpapers.org/). The snippet must run on an HTML document; Chrome's PDF viewer swallows anchor downloads.
    3. Read the snippet file and evaluate it in that tab (wrap it as await (async () => { ... })() if the tool needs a returned promise). It fetches each item with the cleared cookies and saves it through a normal Chrome download under the job's file name. It returns one small record per item: saved + bytes, or an HTTP status, or md5_groups for a SciDB page (eight-character hex groups of the record's md5; join them and rerun paper-fetch get IDENT --md5 MD5 to download through the API instead).
    4. paper-fetch collect "JOB.json" --out STAGING-DIR [--name CITEKEY]. It finds the job's files in ~/Downloads, verifies identity, installs the first verified file, and leaves anything else in place.

    If the snippet reports saved but collect says no-downloads, Chrome blocked the download: it allows one automatic download per site and then requires the "Automatic downloads" permission. Ask Pablo to allow it once for that origin (the blocked-download icon in the address bar, or chrome://settings/content/automaticDownloads), then rerun steps 3–4. This is a per-site, one-time browser permission; nothing else can substitute for it — bytes cannot be passed back through the tool result (encoded blobs are redacted) or posted to localhost (Private Network Access blocks it).

    If the snippet reports 403 for an item, the challenge was not cleared or has expired (clearance lasts roughly half an hour): repeat step 1 for that item.

    Anna's Archive SciDB items (kind: scidb or a DOI the API could not serve): open https://<annas-host>/scidb/DOI/ in the tab and, once the page has loaded, click its Download link ([...document.querySelectorAll('a')].find(a => a.textContent.trim() === 'Download').click()). That is a plain navigation download from the partner server and works whenever the viewer would show the PDF. Two failure shapes are known and are not worth retrying: the page redirects to /search (no SciDB record), and the partner answers 404 for DOIs whose suffix contains / or : (10.1093/mind/fzv208, 10.1023/a:…) because the partner path double-encodes the separator. For those, use the page's Sci-Hub link instead: on sci-hub.ru/DOI the PDF is the object/embed/iframe source; fetch it in-page and save through an anchor when it is same-origin, and when it lives on another Sci-Hub host (sci-hub.red, sci-net.xyz) navigate the tab to it and fetch from there. Sci-Hub shows an "Are you a robot? → No" gate once per session; that click is Pablo's, not the agent's. The extension refuses site-level permission for Sci-Hub hosts, so budget one approval per paper and do the whole fetch-and-save in a single JavaScript action per DOI.

    Chrome's automatic-download gate applies per site: the first download from sci-hub.ru, sci-hub.red, sci-net.xyz or an Anna's host lands, later ones are dropped silently until that site is allowed. Confirm each save in ~/Downloads before moving on rather than trusting the snippet's saved.

  4. Hand-off. A staged PDF is a file, not a bibliography attachment. When the paper belongs in the bibliography, continue with add-bib-entry, which imports the metadata and attaches the staged file through Ebib with the same operation.

Batches

Run one identifier per paper-fetch get, in order; the tool is serial by design so that Anna's Archive quota and challenge state stay legible. Group browser items by origin: one cleared tab serves every item of that origin. Record per-paper outcomes (status, route, file) so a rerun can skip successes.

Failure reporting

Report the tool's status and attempts verbatim rather than a paraphrase. "Every route failed" is the accurate summary of unavailable; "the paper is not available" is not. Never present a mismatch or unverified file as the requested work, and never substitute a related paper from a SciDB results page.

Verification before finishing

  • status: ok and verification.verdict: verified (or a manually inspected unverified-no-text scan), and the staged path exists and is non-empty.
  • The first page's text names the requested title or DOI.
  • Browser tabs you created are closed; nothing left in ~/Downloads except files collect deliberately kept.