polars
Use when working with dataframes, data wrangling, ETL pipelines, or replacing pandas with Polars; Use when the user asks about fast, memory-efficient tabular data processing in Python
SKILL.mdskills/polars/SKILL.md
GitHubYou are an expert in Polars, the blazing-fast DataFrame library for Python written in Rust, designed as a modern replacement for pandas.
Use this skill when
- Loading, transforming, and analyzing tabular data with Polars
- Migrating code from pandas to Polars
- Building ETL or data pipelines
- Working with lazy evaluation and query optimization
- Reading/writing CSV, Parquet, JSON, Arrow, Excel files
- Performing aggregations, joins, pivots, and window functions
Do not use this skill when
- The user explicitly requires pandas
- The task involves ML model training (use pandas/numpy there if needed)
Instructions
- Always prefer the lazy API (
pl.scan_*,.lazy(),.collect()) for large datasets. - Use expression-based transforms rather than row-wise operations.
- Avoid
.to_pandas()unless absolutely required for interop. - Always type-annotate DataFrames where possible.
Core API
Creating DataFrames
import polars as pl
df = pl.DataFrame({
"name": ["Alice", "Bob", "Charlie"],
"age": [30, 25, 35],
"score": [88.5, 92.1, 78.0],
})
# From list of dicts
df = pl.from_dicts([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
# Schema-enforced
df = pl.DataFrame({"a": [1, 2, 3]}, schema={"a": pl.Int32})
Reading & Writing Files
# CSV
df = pl.read_csv("data.csv")
lf = pl.scan_csv("data.csv") # lazy
df.write_csv("out.csv")
# Parquet (preferred for large data)
df = pl.read_parquet("data.parquet")
lf = pl.scan_parquet("data.parquet") # lazy
df.write_parquet("out.parquet")
# JSON / NDJSON
df = pl.read_json("data.json")
df = pl.read_ndjson("data.ndjson")
# Excel
df = pl.read_excel("data.xlsx", sheet_name="Sheet1")
Selecting & Filtering
# Select columns
df.select(["name", "age"])
df.select(pl.col("age"), pl.col("score"))
df.select(pl.all().exclude("name"))
# Filter rows
df.filter(pl.col("age") > 28)
df.filter((pl.col("age") > 25) & (pl.col("score") >= 90))
# Slice
df.head(10)
df.tail(5)
df.slice(10, 20) # offset, length
Transformations
# Add / rename / drop columns
df.with_columns(
(pl.col("score") * 1.1).alias("adjusted_score"),
pl.col("name").str.to_uppercase().alias("NAME"),
)
df.rename({"name": "full_name"})
df.drop(["score"])
# Type casting
df.with_columns(pl.col("age").cast(pl.Float64))
# String operations
df.with_columns(
pl.col("name").str.contains("Ali").alias("is_ali"),
pl.col("name").str.split(" ").alias("parts"),
pl.col("name").str.replace("Charlie", "Chuck"),
)
# Date/time
df.with_columns(
pl.col("date").str.to_date("%Y-%m-%d"),
pl.col("dt").dt.year(),
pl.col("dt").dt.strftime("%b %Y"),
)
Aggregations & GroupBy
df.group_by("department").agg(
pl.col("salary").mean().alias("avg_salary"),
pl.col("salary").max().alias("max_salary"),
pl.len().alias("count"),
)
# Multiple group keys
df.group_by(["dept", "year"]).agg(pl.col("revenue").sum())
# Pivot
df.pivot(index="year", on="category", values="amount", aggregate_function="sum")
Joins
df1.join(df2, on="id", how="inner") # inner
df1.join(df2, on="id", how="left") # left
df1.join(df2, left_on="user_id", right_on="id", how="full") # full outer
df1.join_asof(df2, on="timestamp", strategy="backward") # asof/range join
Lazy API (recommended for large data)
result = (
pl.scan_csv("large.csv")
.filter(pl.col("value") > 100)
.group_by("category")
.agg(pl.col("value").sum())
.sort("value", descending=True)
.collect() # execute the query plan
)
# Explain the query plan
lf.explain()
lf.explain(optimized=True)
Window Functions
df.with_columns(
pl.col("score").rank().over("department").alias("rank_in_dept"),
pl.col("value").sum().over("group").alias("group_total"),
pl.col("ts").sort().over("user_id"), # sort within group
)
Expressions & Utilities
# Conditional / when-then-otherwise
df.with_columns(
pl.when(pl.col("score") >= 90)
.then(pl.lit("A"))
.when(pl.col("score") >= 80)
.then(pl.lit("B"))
.otherwise(pl.lit("C"))
.alias("grade")
)
# Null handling
df.drop_nulls()
df.fill_null(0)
df.fill_null(strategy="forward")
df.with_columns(pl.col("x").fill_nan(None))
# Sorting
df.sort("age", descending=True)
df.sort(["dept", "salary"], descending=[False, True])
Schema & Metadata
df.schema # dict of column -> dtype
df.dtypes # list of dtypes
df.columns # list of column names
df.shape # (rows, cols)
df.describe() # summary statistics
pandas → Polars Migration Cheatsheet
| pandas | Polars |
|---|---|
df["col"] |
df["col"] or df.get_column("col") |
df[["a","b"]] |
df.select(["a","b"]) |
df[df["x"]>1] |
df.filter(pl.col("x")>1) |
df.assign(c=...) |
df.with_columns(...) |
df.groupby().agg() |
df.group_by().agg() |
df.merge() |
df.join() |
df.apply(fn) |
use expressions; last resort: df.map_rows(fn) |
pd.read_csv() |
pl.read_csv() / pl.scan_csv() |
df.to_parquet() |
df.write_parquet() |
Best Practices
- Prefer lazy over eager for anything beyond toy-sized data.
- Use expressions instead of Python-level loops or
.map_rows(). - Avoid
.to_pandas()— interop cost is high; stay in Polars as long as possible. - Use Parquet as your storage format: columnar, compressed, fast to scan.
- Specify dtypes explicitly when reading CSVs to avoid inference overhead.
- Use
.sink_parquet()/.sink_csv()for streaming writes with the lazy API. - Partition large datasets with
hive_partitioning=Trueon Parquet reads/writes.
Common Patterns
Top-N per group
df.sort("score", descending=True).group_by("dept").head(3)
Unnesting structs / lists
df.unnest("address") # struct column → individual columns
df.explode("tags") # list column → multiple rows
Concatenation
pl.concat([df1, df2]) # vertical (same schema)
pl.concat([df1, df2], how="diagonal") # vertical, fill missing with null
pl.concat([df1, df2], how="horizontal") # side-by-side