Analyze User Feedback

Synthesize feedback from any mix of sources into a short, evidence-backed report: what people are telling us, how often, how strongly, and what we should do about it.

Reasoning Framework

Feedback analysis goes wrong in two ways: counting by feel (the loudest anecdote wins) and theming without evidence. This skill splits the work. A script does the deterministic parts (merging files, de-duplicating, sampling, counting); the model does the judgment (codebook, coding, interpretation). Every number comes from the script and every claim points to item IDs, so anyone can audit the result.

Output Contract

Artifact	Format	Handed to
feedback.csv	Normalized items with stable IDs (F0001…)	Working file
codes.csv + codebook	id,theme,sentiment[,severity] plus theme definitions	Working file, reused next cycle
themes.md	Theme counts, shares, sentiment (script output)	Appendix
Feedback analysis	Markdown report (template below)	PM, team, leadership

When to Use

• Survey open-ends, NPS/CSAT comments, support tickets, app-store reviews, sales notes, community threads
• Combining several feedback sources into one view
• Repeating an analysis each cycle and comparing to last time

When NOT to Use

• One interview transcript: use analyze-user-interview
• Designing a survey: use devex-survey (developer experience) or create-user-interview
• Deciding what to build from known themes: use prioritize-features

Inputs

Ask for anything missing, in one message:

1. Files: CSV, TSV, JSONL, TXT or MD. Excel: export to CSV first.
2. The question: the decision this informs (e.g. "why don't trial users convert?").
3. Scope: date range, segments, or products to include.
4. Prior codebook (optional): last cycle's themes, for comparison.

Steps

1. Frame. Restate the question and scope in one sentence and confirm it.
2. Prep (script). Merge and number everything; read the script's summary, not the raw files:

   python3 scripts/feedback_tally.py prep <files...> -o <workdir>/feedback.csv
   

   If it suggests sampling (over ~600 items or ~60k tokens), rerun with --sample 400 --seed 7, code the sample, and report results as estimates. If it picked the wrong text column, pass --text-col.
3. Draft the codebook. Read a spread of ~60–100 items across sources. Define 6–12 themes, each with a one-line definition, an include/exclude rule, and an example ID. Name themes as user problems or needs ("Can't find old tickets"), not solutions ("Add search filters"). Reuse the prior codebook when there is one.
4. Code every item. Read feedback.csv in batches of about 100 rows and write codes.csv rows: id,theme,sentiment[,severity]. An item can carry several themes (;-separated). Add a new theme only when 3+ items need it; otherwise use Other. Note codebook changes as you go.
5. Tally (script).

   python3 scripts/feedback_tally.py tally <workdir>/feedback.csv <workdir>/codes.csv -o <workdir>/themes.md
   

   Add --moe --population <N> when you coded a sample. Fix any uncoded items or unknown IDs it reports, then rerun.
6. Interpret. For the top themes: the underlying need (what they ask for is often not what they need), who says it (sources, segments), severity, 2–3 verbatim quotes with IDs, and what's missing (silent segments, churned users, channels not covered).
7. Recommend. 3–5 actions, each tied to theme evidence and a next step (a fix, an experiment via validate-hypothesis, or a backlog item for prioritize-features).
8. Write the report to <workdir>/feedback-analysis.md using the template. Keep raw data local.

Report Template

# Feedback Analysis: [Scope], [Date range]

**Question:** [the decision this informs]
**Sources:** [source: n items, ...] · **Coded:** [n] ([sample of N] if sampled)

## Summary
[3 sentences: the biggest themes, what changed since last time, the recommended action.]

## Themes
| Theme | Share | Sentiment | Who | Representative quote |
|---|---|---|---|---|
| [Theme] | [x% ± moe] | [mostly negative] | [segments/sources] | "[verbatim]" (F0123) |

## What's behind the top themes
### [Theme]
[Underlying need, severity, evidence, open questions.]

## Recommendations
1. [Action]: [evidence] → [next step, owner]

## Gaps and caveats
[Missing channels or segments, sampling, coding judgment calls.]

## Appendix
Codebook, themes.md table.

Longitudinal Tracking

Reuse the codebook (add themes, don't silently rename them), rerun tally per cycle, and compare shares. Treat a change smaller than the margin of error as noise.

Examples

• "Here are last quarter's NPS comments and support tickets. What are people frustrated about?" Prep both files, code, tally, and report the top themes with quotes.
• "5,000 app reviews. Anything about onboarding?" Prep with --sample 400, code the sample, report onboarding share with a margin of error.

Guardrails

• Never count by hand. Every count and percentage comes from the script.
• Quote verbatim, with the item ID. Don't paraphrase inside quotation marks.
• Frequency is not importance: a rare data-loss bug can outrank a common nit. Say so when it applies.
• Strip personal data (names, emails, account IDs) from the report; describe people by role or segment.
• Don't upload feedback files to external services, and don't commit them to this public repo.
• A sample gives estimates: report the sample size and margin of error.

Related Skills

• analyze-user-interview: one interview in depth
• prioritize-features: rank what to build from these themes
• validate-hypothesis: test the riskiest belief a theme suggests
• library/lenny-podcast/lenny-analyzing-user-feedback/SKILL.md: principles from product leaders