nltk
Use NLTK for classical natural-language processing in Python: install the package and data, tokenize and tag text, stem or lemmatize, build grammars and parse trees, train classical models, score metr
SKILL.mdskills/repositories/repo-skills/nltk/SKILL.md
GitHubNLTK
Use this skill when a task names NLTK or asks for classical Python NLP workflows involving tokenization, corpora, POS tagging, stemming, parsing, grammar, text classification, sentiment, language models, translation metrics, or NLTK data packages.
NLTK 3.10.x is a broad toolkit rather than one end-to-end model. Start from the user's artifact and task: raw text, tokens, tagged sentences, a corpus/resource, a grammar, features, an aligned parallel corpus, or an evaluation score.
Install and import check
NLTK supports Python 3.10 through 3.14. The base install provides defusedxml, click, joblib, regex, and tqdm.
python -m pip install nltk
python - <<'PY'
import nltk
print(nltk.__version__)
PY
Optional extras are deliberately separate: machine_learning (numpy, python-crfsuite, scikit-learn, scipy), plot (matplotlib), tgrep (pyparsing), twitter (twython), and corenlp (requests). Install only the extra or external tool the task requires.
NLTK code and NLTK data are separate. Do not treat import nltk as proof that punkt_tab, tagger models, corpora, WordNet, or VADER data are installed.
Route by task
- Install, locate, download, or repair corpora/models/data: read
sub-skills/data-and-downloader/SKILL.md. It coversNLTK_DATA,nltk.data.path,nltk.data.find/load, targeted downloader commands, corpus readers, and safe recovery. - Tokenize, detokenize, tag, stem, lemmatize, or run sentiment preprocessing: read
sub-skills/tokenize-tag-stem/SKILL.md. It coversword_tokenize,sent_tokenize, thenltk tokenizeCLI,pos_tag, tagger backoff, stemmers, WordNet lemmatization, and VADER. - Write grammars, chunk POS-tagged text, parse, manipulate trees, evaluate dependencies, or interpret logical semantics: read
sub-skills/grammar-parse-semantics/SKILL.md. - Train classical classifiers, cluster vectors, fit n-gram language models, use probability distributions, score metrics, or build translation alignments: read
sub-skills/ml-metrics-and-translation/SKILL.md.
For a pipeline that spans routes, use them in order: data/resource checks → tokenization/tagging → grammar or feature construction → model/metric evaluation. Keep each stage's input shape explicit.
Shared operating rules
- Prefer small, deterministic, no-download checks before downloading data or installing optional dependencies.
- When a
LookupErrornames an attempted resource, use that exact path to choose the smallest package; do not default toallorpopular. - Preserve tokenization and tagset choices in outputs. A parser, classifier, or metric is not comparable if preprocessing changes silently.
- Treat third-party Java/binary wrappers, GUI apps, network downloads, credentials, and large corpora as explicit optional prerequisites.
- Never manually unzip untrusted NLTK packages or bypass
nltk.datapath-security checks.
Shared references and diagnostics
- Read
references/troubleshooting.mdfor cross-cutting installation, data, optional-dependency, API-shape, CLI, and security failures. - Read
references/native-verification.mdwhen selecting a safe smoke or native test; it records what was verified and what remains data- or tool-dependent. - Run
scripts/nltk_doctor.pywith--helpfirst, then--jsonfor a read-only import/CLI/resource diagnostic. It never downloads data unless--downloadis explicitly supplied. - Read
references/repo-provenance.mdbefore deciding whether this skill matches a changed NLTK checkout or should be refreshed.
Boundaries
This is a runtime package skill, not a release or contributor manual. It does not bundle NLTK corpora, Java tools, Stanford/CoreNLP/Senna/Malt/MEGAM/TADM binaries, GUI assets, credentials, or the original repository's tests/docs. Those artifacts informed the routes and verification plan; runtime helpers and distilled references live inside this skill tree.