dk-wikibot
Fetch, search, and extract information from Wikipedia, Wikidata, or any MediaWiki-based wiki using pywikibot. Use when the user asks to scrape Wikipedia articles or infoboxes, iterate over Wikipedia c
SKILL.mdskills/dk-wikibot/SKILL.md
GitHubPywikibot Skill
Pywikibot wraps the MediaWiki API. For read-only tasks (the common case), no credentials are
needed — set PYWIKIBOT_NO_USER_CONFIG=1.
Always write a standalone Python script and execute it via bash_tool. Do not import pywikibot
interactively — it makes blocking network calls and must run as a script.
Setup
uv add pywikibot mwparserfromhell
export PYWIKIBOT_NO_USER_CONFIG=1
Every script must start with:
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
import pywikibot
Step-by-step instructions
1. Fetch a Wikipedia article
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
import pywikibot
site = pywikibot.Site('en', 'wikipedia')
page = pywikibot.Page(site, 'Grammy Award for Best Rock Song')
print(page.text[:3000]) # raw wikitext — inspect this first before parsing
Always inspect page.text before writing any parser. Wikitext structure varies per article.
2. Parse wikitext
import mwparserfromhell
wikicode = mwparserfromhell.parse(page.text)
templates = wikicode.filter_templates() # e.g. {{won}}, {{nom}}, {{infobox}}
Alternatively, use regex directly on page.text:
import re
year_sections = re.split(r'\n==\s*(\d{4})\s*==\n', page.text)
links = re.findall(r'\[\[([^\|\]]+)(?:\|[^\]]+)?\]\]', block)
3. Iterate a Wikipedia category
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
import pywikibot
site = pywikibot.Site('en', 'wikipedia')
cat = pywikibot.Category(site, 'Category:Grammy Award for Best Rock Song')
for page in cat.articles(total=50):
print(page.title())
Use cat.subcategories() for subcats, cat.members() for everything.
4. Query Wikidata via SPARQL
Best for clean structured data (winners, dates, relationships). Find the QID for any entity at https://www.wikidata.org/wiki/Special:Search first, then use it in the query.
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
from pywikibot.data import sparql
QUERY = """
SELECT ?year ?songLabel ?artistLabel WHERE {
?award wdt:P31 wd:Q1377714 . # instance of Grammy Award for Best Rock Song
?award wdt:P585 ?date .
BIND(YEAR(?date) AS ?year)
FILTER(?year >= 2010)
SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
ORDER BY ?year
"""
results = sparql.SparqlQuery().select(QUERY)
for row in results:
print(row)
5. Fetch a Wikidata item directly
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
import pywikibot
site = pywikibot.Site('en', 'wikipedia')
page = pywikibot.Page(site, 'Douglas Adams')
item = pywikibot.ItemPage.fromPage(page)
item.get()
print(item.labels.get('en'))
print(item.claims) # dict of property_id -> list of Claims
Decision guide
| Task | Approach |
|---|---|
| Award winner lists, formatted tables | Wikipedia page.text + wikitext parsing |
| Clean structured facts (dates, IDs, relationships) | Wikidata SPARQL or ItemPage |
| Category membership | Category.articles() |
| Multiple entities at once | Wikidata SPARQL (most efficient) |
Examples
Find all Grammy Best Rock Song winners since 2010
import os; os.environ['PYWIKIBOT_NO_USER_CONFIG'] = '1'
import re, pywikibot
site = pywikibot.Site('en', 'wikipedia')
page = pywikibot.Page(site, 'Grammy Award for Best Rock Song')
text = page.text
# Inspect first — then adapt the pattern to what you see
print(text[:2000])
# Split by year section headers
sections = re.split(r'\n==\s*(\d{4})\s*==\n', text)
# sections alternates: [preamble, year, content, year, content, ...]
results = []
for i in range(1, len(sections) - 1, 2):
year = int(sections[i])
if year < 2010:
continue
content = sections[i + 1]
won_blocks = re.findall(r'\{\{won\}\}.*?\n\|-', content, re.DOTALL)
for block in won_blocks:
links = re.findall(r'\[\[([^\|\]]+)(?:\|[^\]]+)?\]\]', block)
if links:
results.append({'year': year, 'song': links[0], 'artist': links[1] if len(links) > 1 else ''})
for r in results:
print(f"{r['year']}: {r['song']} — {r['artist']}")
Common edge cases
NoPageError: page doesn't exist — check spelling or follow a redirect.IsRedirectPageError: follow withpage.getRedirectTarget().- Template name varies: the winner template may be
{{won}},{{winner}},{{yes}}, etc. Always checkpage.textfirst. - SPARQL timeouts: add
LIMIT 100or tighten filters. - Rate limits: pywikibot has built-in throttling; don't add manual
sleep().
LLM-based extraction (PydanticAI + LMStudio)
For pages with messy, inconsistent, or deeply nested wikitext, skip regex/template parsing entirely
and pass page.text directly to a local LLM. The LLM handles wikitext syntax natively and returns
a validated Pydantic model. This is the most agent-native approach.
See the free-models-pydantic-ai skill for full setup and usage patterns. The pattern here is:
- Fetch
page.textvia pywikibot as usual - Truncate to fit context window (8000 chars is a safe default for most local models)
- Pass to a PydanticAI agent with a structured
output_typematching your desired schema - Get back a validated Pydantic model — no regex, no template parsing
When to use this over regex/mwparserfromhell:
- Page structure is inconsistent across years
- Templates are non-standard or undocumented
- You need to extract prose context alongside structured fields
- Rapid prototyping — skip the wikitext archaeology
Caveats:
- Local models may hallucinate values — validate against known data if accuracy is critical
- Slower and heavier than SPARQL or regex for well-structured pages; use those first
See references/api.md for a full API reference of commonly used classes and methods.