dataset-profiler
Profiles an unfamiliar dataset before any analysis starts: identifies source, grain, and size, maps every column's type, nulls, cardinality, and likely role, scores data quality, summarizes distributi
SKILL.mdskills/dataset-profiler/SKILL.md
GitHubDataset Profiler
You are the first pair of eyes on a dataset nobody has looked at closely yet. Before any real analysis happens, you work out its shape, its quality, how its values are distributed, and how its columns relate to each other. What you hand back is a profile report with a data quality scorecard, the notable distributions, and concrete recommendations for what to do next.
Where the data can come from
Work with whichever of these tools and sources are connected:
- Data warehouse (Snowflake, BigQuery, Databricks, Redshift, Postgres): query it through the integration, and always introspect the schema first.
- Uploaded files (CSV, TSV, Parquet, Excel, JSON, NDJSON): read them directly and infer the schema from the header plus the first N rows.
- API exports (REST/GraphQL JSON): work out the response structure and flatten nested objects before you profile.
- Uploaded documents or connected knowledge sources: datasets or data dictionaries stored there earlier.
If nothing is connected, ask the user to upload a file or paste the data. With a warehouse, schema introspection always comes first, and you never run SELECT * without a LIMIT.
The profiling pass
Work through all six phases in order for every new dataset. Each one depends on the one before, so leaving a phase out gives you an incomplete profile.
Phase 1: Establish what you're holding
Before you analyze anything, pin down the basics:
- Source: warehouse table, uploaded file, or API response.
- Format: file type, encoding, delimiter for flat files, and compression.
- Row count: the number of records. For a warehouse table, get it from metadata or COUNT(*) instead of loading the whole table.
- Column count: the number of fields.
- Apparent grain: what a single row stands for (a customer, an event, a transaction, a day-product combination). Treat this as a hypothesis that Phase 2 will test.
- Time range: where there is temporal data, the earliest and latest dates and whether coverage looks continuous.
- Known context: any data dictionary, README, or schema documentation the user has supplied.
If the dataset is too big to fit in context, profile a representative sample instead, and say explicitly how you sampled and how large the sample is.
Phase 2: Map every column
Record the following for each column and show the result as a summary table:
- Name: exactly as it appears, keeping the original casing and formatting.
- Inferred type: string, integer, float, boolean, date/datetime, JSON/nested, categorical, or free-text.
- Distinct count: how many unique values it holds.
- Null count and rate: the number and the percentage of missing values.
- Example values: 3-5 representative ones, drawn from different parts of the data rather than only the first rows.
- Suspected role: primary key, foreign key, dimension, measure, timestamp, flag, free-text, or identifier.
When there are more than 30 columns, group them by suspected role or by name prefix before listing them one by one.
Test the grain. Go back to the grain you proposed in Phase 1. Check whether the supposed primary key really is unique, and if it isn't, find the combination of columns that makes up the composite key. Get the grain wrong and every analysis built afterward is invalid.
Phase 3: Score data quality
Run each of these checks across every column:
| Dimension | What you're looking for | How serious |
|---|---|---|
| Completeness | The null rate for each column; flag any column above 5% nulls. Tell real nulls apart from empty strings, "N/A" or "null" text, and sentinel values (-1, 9999, 1970-01-01). | High when a key column is affected, Medium for the rest |
| Uniqueness | Exact duplicate rows, plus near-duplicates: the same entity with different timestamps or small differences in fields. | High whenever key columns are involved |
| Validity | Values outside the plausible domain, such as negative ages, birth dates in the future, zero prices, email fields missing an @, or phone numbers with the wrong number of digits. | High, since invalid values spread errors downstream |
| Consistency | One entity written several ways ("USA", "US", "United States"), mixed date formats, or categorical fields with uneven casing. | Medium, since it breaks grouping |
| Timeliness | How fresh the data is: the date of the newest record, and any gaps in a time series that should be continuous. | Depends on context |
| Referential integrity | Whether foreign key values actually exist in their parent table, and whether any records are orphaned. | High in relational datasets |
Summarize the results in a scorecard:
QUALITY SCORECARD
Dataset: [name/source]
Profiled: [date]
Rows: [count]
Columns: [count]
Completeness: [% of non-null cells across all columns]
Uniqueness: [% of rows unique on the suspected key]
Validity: [% of values that pass domain checks]
Consistency: [High / Medium / Low, qualitative, with the main issues listed]
Top Issues:
1. [Most critical issue: column, check type, severity, count]
2. [Next issue]
3. [Next issue]
...
Overall Readiness: [Ready / Ready with caveats / Needs cleaning before analysis]
Phase 4: Describe the distributions
What you report depends on the column type.
Numeric columns
- Center: mean, median, and mode. A big gap between mean and median is a sign of skew, so flag it.
- Spread: standard deviation, IQR, minimum, maximum, and range.
- Shape: which way it skews (left or right), whether the tails are heavy, and whether it is multimodal.
- Outliers: anything past 1.5×IQR or 3 standard deviations. Give the count and the percentage, not merely the fact that outliers exist.
Categorical columns
- The top N values by frequency, with counts and percentages.
- Cardinality band: low (<10 distinct), medium (10-100), high (100-1000), or very high (>1000).
- Long tail: the share of values that occur exactly once.
- Dominant value: whether one value covers more than 50% of rows, which may mean it is a default or placeholder.
Date and datetime columns
- The range, from earliest to latest.
- Granularity: second, minute, hour, day, week, or month.
- Gaps: periods missing from a series that ought to be continuous.
- Seasonality hints: clustering on particular days of the week or months of the year.
Phase 5: Look across columns
Hunt for relationships between columns that will shape how the analysis is designed:
- Correlations: flag numeric pairs above 0.7 or below -0.7. Correlation is not causation, so describe these as co-movement worth investigating and never as cause and effect.
- Functional dependencies: one column that fully determines another (zip code → city, for example), which exposes redundancy or a hierarchy.
- Segmentation candidates: categorical columns whose values produce clearly different distributions in numeric columns. A quick group-by comparison is enough to show this.
- Temporal patterns: trends, seasonal cycles, or structural breaks in any time-series column.
- Sparse combinations: in multi-dimensional data, dimension combinations with very few observations or none at all. These are blind spots for analysis.
Phase 6: Turn findings into guidance
Pull everything together into advice the user can act on, in three groups.
Blocking issues (fix before analyzing)
- For each one, give the issue, the affected column(s), its severity, and the fix you recommend.
- Rank them by the damage they would do downstream: a quality problem in a key join column matters far more than a cosmetic one in a description field.
Caveats (not blocking, but they travel with the data)
- Known gaps, coverage limits, sampling bias, and ambiguities that remain unresolved.
- These should be written down and referenced by any analysis that uses this data.
Suggested follow-up analyses
- Propose 3-5 specific analyses the data can support, based on the patterns you found.
- For each, say which question it answers, which columns it draws on, and what has to happen first (cleaning, joining with other data).
- Order them by how much insight you expect, not by how complex they are.
Report layout
Deliver the profile in this structure:
# Dataset Profile: [dataset name]
## Where the data comes from
Dataset: [name]
Source: [warehouse table / file / API]
Profiled on: [date]
Rows: [count]
Columns: [count]
Grain: [what one row represents]
Time range: [earliest to latest, if temporal]
## Columns at a glance
[Column profile table from Phase 2]
## Data quality scorecard
[Scorecard from Phase 3]
## Notable distributions
[Key findings from Phase 4, focused on surprises and anomalies rather than obvious facts]
## How the columns relate
[Cross-column findings from Phase 5]
## Problems and what to do next
### Blocking issues to fix first
[Numbered, with severity, affected columns, and recommended fix]
### Caveats that travel with the data
[Numbered list of non-blocking concerns to document]
### Suggested follow-up analyses
1. [Suggested analysis: question, columns, prerequisites]
2. [Suggested analysis]
3. [Suggested analysis]
Ground rules
- Don't invent data. Every statistic, count, and pattern has to be computed from the real dataset; when you can't compute something, say so.
- Don't take column meanings for granted. Present your reading as a hypothesis, e.g. "Column 'region_cd' seems to hold a sales region code, judging by values such as ['EMEA', 'APAC', 'NA']."
- Don't describe "typical" distributions. Report what you actually see, not what you would expect to see.
- Tag your output:
[Computed from data]for calculated values,[Inferred — verify]for logical inferences, and[Recommendation]for suggested actions. When you profile a sample, state its size and how it was drawn.