nlp-lda-topic-modeling
Latent Dirichlet Allocation on CountVectorizer bag-of-words to discover latent topics with per-document topic distributions for feature engineering or EDA
SKILL.mdskills/nlp/lda-topic-modeling/SKILL.md
GitHubLDA Topic Modeling
Overview
Latent Dirichlet Allocation discovers latent topics in a text corpus. Each document is modeled as a mixture of topics, each topic as a distribution over words. Fit LDA on a CountVectorizer bag-of-words matrix to extract topic distributions per document — usable as features for downstream models or for understanding corpus structure.
Quick Start
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
cvec = CountVectorizer(min_df=4, max_features=50000,
ngram_range=(1, 2), stop_words='english')
bow = cvec.fit_transform(texts)
lda = LatentDirichletAllocation(
n_components=20, learning_method='online',
max_iter=20, random_state=42)
topic_dist = lda.fit_transform(bow) # shape: (n_docs, n_topics)
vocab = cvec.get_feature_names_out()
for i, topic in enumerate(lda.components_):
top_words = [vocab[j] for j in topic.argsort()[:-11:-1]]
print(f"Topic {i}: {', '.join(top_words)}")
Workflow
- Build bag-of-words with
CountVectorizer(not TF-IDF — LDA expects raw counts) - Fit LDA with
learning_method='online'for large corpora fit_transformreturns per-document topic distributions (features)- Inspect
lda.components_to label topics by their top words - Append topic distributions as features to the original dataset
Key Decisions
- n_components: 10-30 for EDA, tune via coherence score for production
- CountVectorizer not TfidfVectorizer: LDA's generative model assumes word counts
- online vs batch: online is faster for large datasets, batch is more accurate
- min_df: 3-5 removes rare terms that create noise topics