fc
Perform web search then firecrawl scrape each result and combine.
SKILL.mdweb-workflow/fc/SKILL.md
GitHubDescription: Perform a web search for a given query, then scrape each result URL using Firecrawl, and combine the extracted content into a single response.
Trigger: Slash command /fc <query> where <query> is the search term.
Prerequisites:
- Firecrawl API key in
~/.hermes/.envasFIRECRAWL_API_KEY. - Firecrawl SDK installed in the venv:
uv pip install firecrawl-py --python venv/bin/python.
Steps:\n1. Parse Input: Check if input is a URL (starts with http) or a search query.\n2. URL Mode (if input is a URL like /fc https://x.com/...):\n - Try 1: npx -y firecrawl@latest scrape "<url>" (CLI approach)\n - Try 2: Python SDK FirecrawlApp(api_key).scrape(url, formats=['markdown'])\n - Fallback (for X/Twitter or blocked sites): Browser automation workflow:\n a. mcp_chrome_devtools_new_page(url=<url>)\n b. mcp_chrome_devtools_take_screenshot(fullPage=true, filePath=/tmp/page.png)\n c. vision_analyze(image_url=/tmp/page.png, question="Extract all text content")\n d. Alternative: mcp_chrome_devtools_evaluate_script to extract DOM text\n3. Search Mode (if input is a query):\n - Call web_search with the query, requesting up to 5 results.\n - Extract the url field from each result.\n4. Firecrawl Scrape (for each URL in Search Mode):\n - Use the Firecrawl SDK to scrape the content.
- Use the Firecrawl SDK to scrape the content.
- Python Implementation Pattern:
from firecrawl import FirecrawlApp import os # Load API key from .env api_key = None with open(os.path.expanduser('~/.hermes/.env')) as f: for line in f: if line.startswith('FIRECRAWL_API_KEY='): api_key = line.split('=', 1)[1].strip() break app = FirecrawlApp(api_key=api_key) doc = app.scrape(url=target_url, formats=['markdown']) content = doc.markdown - Fallback: If the SDK returns
InternalServerError(common on X.com/Twitter), use the CLI:npx firecrawl-cli@latest scrape <url>
- Combine Results:
- Concatenate all scraped markdown sections, separating each with a heading indicating the source URL (e.g.,
## Source: <url>). - If any URL fails to scrape, note the error and continue.
- Concatenate all scraped markdown sections, separating each with a heading indicating the source URL (e.g.,
- Output:
- Return the combined markdown to the user.
- Include the original web search URLs as references (per user preference).
Pitfalls:
- SDK v2 API: Remember that
app.scrape_urlis deprecated; useapp.scrape(url=..., formats=['markdown']). - Pydantic Objects: The SDK returns a
Documentobject; useresult.markdown, notresult['markdown']. - Auth: The SDK does NOT auto-read
FIRECRAWL_API_KEYfrom env; it must be passed explicitly. - Anti-Bot Protection: Some sites block the SDK's playwright backend. The
npx firecrawl-cliis the recommended workaround for these cases. - X/Twitter: Often blocks Firecrawl entirely (returns
InternalServerError). Use browser automation fallback:- Open URL via
mcp_chrome_devtools_new_page - Take full-page screenshot with
mcp_chrome_devtools_take_screenshot(fullPage=true) - Extract text using
vision_analyzeon the screenshot - Alternatively, use
mcp_chrome_devtools_evaluate_scriptto extract DOM text directly
- Open URL via
- Government / Weather Sites (e.g., CWA/中央氣象署 for 台灣天氣):
- Web search for real-time weather may return only climate averages or outdated summary pages instead of current forecasts.
- Target government sites (cwa.gov.tw, etc.) often block both
web_extractand Firecrawl SDK. - Workaround: use browser automation (
mcp_chrome_devtools_take_snapshot) on the official forecast page for structured, parseable text — prefertake_snapshotovervision_analyzewhen the page has well-structured tables/lists.
Verification:
- After running
/fc test query, verify that:- The response contains sections for each successfully scraped URL.
- Each section starts with
## Source: <url>. - The combined content is in markdown format.
Example:
/fc latest AI advancements 2024
Will search the web, scrape the top result pages via Firecrawl, and return a consolidated markdown summary.