nlp-deberta-classification

Fine-tunes DeBERTa-v3 for text classification tasks. Use when building text classifiers, sentiment analysis, or multi-label classification on domain-specific data.

SKILL.md

GitHub

DeBERTa Classification

Overview

DeBERTa-v3 (Decoding-enhanced BERT with disentangled attention) is a strong baseline for text classification. It outperforms BERT/RoBERTa on most NLU benchmarks while maintaining similar inference cost.

Quick Start

from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer

model_name = "microsoft/deberta-v3-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

Workflow

  1. Prepare dataset in HuggingFace Dataset format (text + label columns)
  2. Tokenize with max_length=512, dynamic padding
  3. Fine-tune with TrainingArguments(lr=2e-5, epochs=3, warmup_ratio=0.1)
  4. Evaluate with accuracy, F1, and confusion matrix
  5. Export model with model.save_pretrained()

Key Decisions

  • Base vs Large: Use deberta-v3-base (86M params) first. Only upgrade to large (304M) if base plateaus.
  • Max length: 512 covers most tasks. Truncate long docs or use sliding window.
  • Learning rate: 2e-5 is a safe default. Search [1e-5, 3e-5] if needed.

References