nlp

Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when clas

SKILL.md

GitHub

nlp — tokenize the text, pick the model type, pick the metric

You own the language-modeling discipline: how raw text becomes tokens, which transformer architecture fits a task, and which metric actually tells you whether it worked. When the question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3× the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the RAG loop, prompt wording, and the training step itself — those route out (below).

Route out first (loud — do not duplicate these)

Decision: model type per task (get this right before anything else)

Pick the architecture from the task's shape, not from what is trendy. A decoder LLM can technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more accurate on a fixed-label task.

Task shape Architecture Why Example families*
Understand / label a whole input (classification, NER, extractive QA, similarity) Encoder (bidirectional) Attends to the full sentence both directions; cheap to fine-tune and to serve BERT, RoBERTa, DistilBERT, ModernBERT
Free-form generation, chat, few-shot Decoder (autoregressive) Attends only to prior tokens; predicts the next token GPT-style, Llama, Gemma, Qwen
Transform input → new text (summarize, translate, generative QA) Encoder-decoder / seq2seq Encoder reads all of the source, decoder writes conditioned on it T5 / FLAN-T5, BART, mT5

* Architecture families are stable; specific checkpoints and their licenses are not — check the HF model card before you commit (licenses change; see open-weights). ModernBERT (2024) is a current long-context encoder; verify the latest at author time.

The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an encoder does it for a fraction of the cost), and forcing an encoder to generate (it cannot — it has no decoder).

1. Tokenization

Every downstream number depends on this step, and its failures are silent. The single load-bearing rule:

Load the tokenizer that shipped with the checkpoint, and use the same one at train and inference. AutoTokenizer.from_pretrained(same_checkpoint). A train/inference tokenizer mismatch — different vocab, different special tokens, different casing/normalization — maps text to token ids the model never saw and corrupts everything downstream with no error.

from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)

tok = AutoTokenizer.from_pretrained("bert-base-cased")
enc = tok("Tokenizers matter.", return_offsets_mapping=True)
tok.convert_ids_to_tokens(enc["input_ids"])
# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials

The four algorithms (full mechanics in references/tokenization.md):

Algorithm Builds vocab by… Applies by… Used by
BPE merging the most frequent adjacent pair, repeatedly split to chars, replay learned merges GPT-2 (byte-level), many
WordPiece merging pairs that maximize a likelihood score longest-match subword from the front (## continuations) BERT family
Unigram (SentencePiece) start large, remove tokens that least hurt corpus likelihood most-probable segmentation T5, ALBERT, mT5
Byte-level BPE BPE over the 256 raw bytes, not Unicode chars same as BPE on bytes GPT-2, RoBERTa
  • Byte-level BPE has no [UNK]. Base vocab is exactly 256 (all byte values), so every emoji, accent, and script maps to some byte sequence — nothing falls out as unknown (verified: HF NLP course ch.6). WordPiece/word-level tokenizers do have [UNK] and lose OOV content.
  • SentencePiece is reversible — it treats space as a normal symbol (the ▁ meta-symbol), so decode(encode(x)) == x without language-specific detokenization rules. That is why it dominates multilingual models.

Special tokens are not decoration. [CLS]/<s> carries the pooled sentence representation for classification; [SEP]/</s> marks segment/end; [PAD] fills a batch (and must be masked out via attention_mask); [MASK] is the MLM target; [UNK] is the fallback. Names differ by model ([CLS] in BERT vs <s> in RoBERTa) — another reason to never hand-roll the tokenizer.

Why it matters — three concrete costs:

  • $ cost & context. Token count is the bill and the context budget. Fewer tokens per sentence = cheaper calls and more room in the window.
  • OOV / information loss. A tokenizer that emits [UNK] throws away content it can't represent; byte-level/SentencePiece degrade gracefully instead.
  • Fairness. Vocab trained mostly on English fragments other scripts far harder — the same meaning costs more tokens, more money, and more latency (section 5).

2. Tasks

Text classification (encoder + classification head)

from transformers import pipeline
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
clf("The service was slow but the food was incredible.")
# [{'label': 'POSITIVE', 'score': 0.99...}]

Metric: accuracy on balanced data; macro-F1 the moment classes are imbalanced (accuracy lies when 95% of rows are one class).

Token classification / NER (encoder, per-token labels)

Labels are B-/I-/O spans aligned to subword tokens: the first subword of a word gets the label, continuation subwords and special tokens get -100 (ignored by the loss). Evaluate with seqeval at the entity level, never per-token accuracy (per-token accuracy is inflated by the flood of O tokens).

from transformers import pipeline
ner = pipeline("token-classification", aggregation_strategy="simple")
ner("Ada Lovelace worked in London.")
# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'

Seq2seq — summarization / translation (encoder-decoder)

summ = pipeline("summarization", model="facebook/bart-large-cnn")
summ(long_article, max_length=130, min_length=30)

Metric: ROUGE for summarization, BLEU/chrF for translation — with the heavy caveat in section 4.

Sentence embeddings (SBERT — the task, not retrieval)

from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining

Producing/judging embeddings for retrieval (model choice, chunking, recall@k, rerankers) is embeddings-search, not here.

3. Evaluation — pick the metric that matches the task

Task Primary metric Catches Trap
Classification accuracy + macro-F1 wrong labels accuracy hides minority-class failure
NER / token entity-level F1 (seqeval) missed/partial spans per-token accuracy is inflated by O
Translation BLEU / chrF n-gram overlap w/ reference weak on meaning; chrF better for morphology
Summarization ROUGE (1/2/L) recall of reference n-grams rewards copying; blind to faithfulness
Generation (LM) perplexity how well the model predicts held-out text tokenizer-dependent — not comparable across tokenizers
Open-ended / chat LLM-as-judge + human quality overlap metrics miss judge bias (position, verbosity, self-preference)

The caveat that governs this whole section: BLEU, ROUGE, and chrF are n-gram/character overlap metrics and correlate weakly with human judgment on open-ended and creative generation — they reward matching the reference's exact phrasing, so a correct paraphrase scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and MT-evaluation literature). Use them for regression tracking on a fixed reference set, never as the final verdict on quality. For open-ended output, use an LLM-as-judge rubric plus a human spot-check — and know the judge has its own biases (position, verbosity, self-preference), so pin the rubric and randomize order.

Perplexity = exp(mean token NLL): lower means the model predicts held-out text better. It is tokenizer-dependent, so two models with different tokenizers have non-comparable perplexities — only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE, perplexity, and an LLM-judge harness are in references/evaluation.md.

4. Multilingual pitfalls

The English-centric trap: a tokenizer whose vocab was learned mostly on English over-fragments other scripts. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take 2–15× more tokens than its English equivalent (Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023). That "fertility" (tokens per word) inflation is a triple tax:

  • Money — more tokens per identical meaning = a proportionally larger bill for the same work.
  • Context — over-fragmented text eats the window faster, so fewer few-shot examples fit and long documents truncate sooner.
  • Quality — sequences fragmented into byte-shards are harder to model, degrading accuracy for exactly the users the tool already serves worst.

Mitigations: prefer a multilingual tokenizer/model (mT5, XLM-R, a SentencePiece-based model) whose vocab actually covers your languages; measure fertility on your own corpus (tokens per word, per language) before you commit; and don't benchmark cost or latency only on English.

Guardrails / gotchas

  • Tokenizer must match the checkpoint, at train and inference. Mismatch corrupts silently, no error. The most expensive bug in this skill.
  • Encoders can't generate; don't classify with a giant decoder by default. Match architecture to task shape.
  • BLEU/ROUGE/chrF ≠ quality on open-ended text. Overlap metrics; weak human correlation. Track regressions with them; judge quality with an LLM-judge + human.
  • Entity-F1 (seqeval), not token accuracy, for NER. O tokens inflate accuracy toward 1.0.
  • macro-F1, not accuracy, on imbalanced classes.
  • Perplexity is tokenizer-relative — never compare it across different tokenizers.
  • Don't benchmark tokenization/cost only in English — fertility varies 2–15× across scripts.
  • Never assert a model's license from memory — check the current model card; license classes shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).

Related skills

  • embeddings-search — retrieval embeddings, chunking, recall@k, reranking. NLP owns making/judging sentence embeddings as a task; using them to search is theirs.
  • rag — the full retrieve→generate answer loop and groundedness.
  • prompt-engineering — the wording of the prompt.
  • finetuning + deep-learning — actually training/adapting the network (this skill picks the type and metric; those move the weights).
  • training-data — building the labeled corpus you train on.

Checklist

  • Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
  • Tokenizer loaded from the same checkpoint, used identically at train and inference.
  • Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
  • Special tokens and attention_mask handled (padding masked, -100 on ignored labels).
  • Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF only as a regression signal, LLM-judge + human for open-ended.
  • Perplexity compared only within one tokenizer.
  • Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.

References

  • references/tokenization.md — BPE/WordPiece/Unigram/byte-level training mechanics, special tokens per family, offset mapping, and a fertility-measuring snippet.
  • references/evaluation.md — runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an LLM-as-judge rubric, with when each lies.