Pandas runs on a single core, copies data often, and materializes every intermediate step in memory. Polars is a DataFrame library written in Rust on the Apache Arrow columnar format. It parallelizes work across cores, plans queries before running them, and can process datasets larger than RAM. This guide maps each Pandas habit to its Polars equivalent and shows where the two models differ.
Quick Takeaways
- Think in expressions, not mutations. Polars builds transformations with
pl.col()and applies them throughselect,with_columns,filter, andgroup_by().agg(). - No index. Polars has no row index, so
loc,iloc,reset_index, andMultiIndexdo not exist. Usefilter,join, and window functions instead. - Use lazy mode for pipelines.
scan_*plus.collect()enables predicate pushdown, projection pushdown, and parallel execution. - Keep Pandas at the edges. Convert with
.to_pandas()for Scikit-Learn, Statsmodels, or plotting libraries that expect it.
| Aspect | Pandas | Polars |
|---|---|---|
| Execution | Eager, single-threaded for most operations | Eager or lazy, multi-threaded |
| Memory model | NumPy blocks (Arrow optional) | Apache Arrow columnar |
| Row index | Yes | No |
| Missing values | NaN, None, pd.NA mixed |
null for all types, NaN only for floats |
| Query optimization | None | Query planner in lazy mode |
| Larger-than-RAM data | Chunking by hand | Streaming engine |
| Ecosystem maturity | Very large | Growing fast |
Install and Import
pip install polars pyarrow # pyarrow is required for .to_pandas() conversions
import polars as pl
import pandas as pd # keep for interop during migration
The Mental Shift: Expressions
A Pandas workflow mutates a DataFrame step by step. A Polars workflow describes expressions, which are reusable recipes for computing a column. The engine runs independent expressions in parallel.
# Pandas: mutate and assign
df["revenue"] = df["price"] * df["quantity"]
df = df[df["revenue"] > 100]
# Polars: describe the result
df = (
df.with_columns(revenue=pl.col("price") * pl.col("quantity"))
.filter(pl.col("revenue") > 100)
)
Three rules cover most of the migration:
- DataFrames are immutable. Every method returns a new DataFrame, so
df["col"] = ...raises an error. - Expressions inside one
with_columnscall run in parallel. An expression cannot reference a column created in the same call. Chain a secondwith_columnsinstead. - Column selection is explicit. Use
pl.col("a"),pl.col("a", "b"),pl.col(pl.Float64), orpl.all().
Pandas to Polars Cheat Sheet
| Task | Pandas | Polars |
|---|---|---|
| Read CSV | pd.read_csv(p) |
pl.read_csv(p) / pl.scan_csv(p) |
| Read Parquet | pd.read_parquet(p) |
pl.read_parquet(p) / pl.scan_parquet(p) |
| Select columns | df[["a", "b"]] |
df.select("a", "b") |
| Filter rows | df[df.a > 1] |
df.filter(pl.col("a") > 1) |
| New column | df.assign(c=df.a + df.b) |
df.with_columns(c=pl.col("a") + pl.col("b")) |
| Rename | df.rename(columns={"a": "x"}) |
df.rename({"a": "x"}) |
| Sort | df.sort_values("a", ascending=False) |
df.sort("a", descending=True) |
| Group and aggregate | df.groupby("k").agg(...) |
df.group_by("k").agg(...) |
| Merge | df.merge(o, on="k", how="left") |
df.join(o, on="k", how="left") |
| Fill missing | df.fillna(0) |
df.fill_null(0) |
| Drop missing | df.dropna() |
df.drop_nulls() |
| Cast type | df.a.astype("float64") |
pl.col("a").cast(pl.Float64) |
| Wide to long | df.melt(...) |
df.unpivot(...) |
| Group transform | df.groupby("k")["v"].transform("sum") |
pl.col("v").sum().over("k") |
| Conditional | np.where(c, a, b) |
pl.when(c).then(a).otherwise(b) |
| Unique count | df.a.value_counts() |
df["a"].value_counts() |
Step 1: Load and Inspect Data
df = pl.read_csv("sales.csv", try_parse_dates=True)
print(df.shape) # (rows, columns)
print(df.schema) # column names and dtypes
print(df.head(5))
print(df.describe()) # summary statistics, similar to pandas describe()
print(df.null_count()) # nulls per column in one call
Polars infers dtypes from a sample of rows. For large or messy files, pass an explicit schema_overrides mapping to avoid inference errors.
Step 2: Core Transformations
Select, filter, and derive columns
result = (
df
.filter(
(pl.col("status") == "completed") & pl.col("order_date").is_not_null()
)
.with_columns(
revenue=pl.col("price") * pl.col("quantity"),
order_month=pl.col("order_date").dt.truncate("1mo"), # month bucket
)
.select("order_id", "region", "order_month", "revenue")
)
Use &, |, and ~ for boolean logic. Wrap each condition in parentheses.
Conditional logic
df = df.with_columns(
tier=pl.when(pl.col("revenue") >= 1000).then(pl.lit("enterprise"))
.when(pl.col("revenue") >= 200).then(pl.lit("mid"))
.otherwise(pl.lit("small"))
)
Group-by aggregations
summary = (
df.group_by("region", "order_month")
.agg(
pl.col("revenue").sum().alias("total_revenue"),
pl.col("revenue").mean().alias("avg_order_value"),
pl.col("order_id").n_unique().alias("orders"),
)
.sort("region", "order_month")
)
group_by does not guarantee row order by default. Add maintain_order=True if order matters, or sort afterward.
Window functions replace transform
Pandas needs groupby().transform() to broadcast a group statistic back to each row. Polars uses .over().
df = df.with_columns(
region_share=pl.col("revenue") / pl.col("revenue").sum().over("region"),
rank_in_region=pl.col("revenue").rank("dense", descending=True).over("region"),
)
Joins
customers = pl.read_parquet("customers.parquet")
enriched = df.join(customers, on="customer_id", how="left")
Polars supports inner, left, right, full, semi, anti, and cross joins. Semi and anti joins replace the isin filtering patterns Pandas users often write.
Step 3: Handle Nulls and NaN Correctly
Pandas treats NaN as the missing marker for floats. Polars separates the two concepts. A null means missing data. A NaN is a floating-point value that results from an invalid computation.
| Operation | Pandas | Polars |
|---|---|---|
| Detect missing | isna() |
is_null() |
| Detect NaN | isna() (same call) |
is_nan() |
| Replace missing | fillna(x) |
fill_null(x) |
| Replace NaN | fillna(x) |
fill_nan(x) |
| Forward fill | ffill() |
fill_null(strategy="forward") |
df = df.with_columns(
pl.col("price").fill_nan(None) # convert NaN to null first
.fill_null(strategy="forward") # then forward fill
.over("sku") # respect product boundaries
)
Convert NaN to null early. Aggregations like mean() skip nulls but propagate NaN, which causes silent surprises.
Step 4: Use Lazy Evaluation
Lazy mode is the largest single performance gain. A LazyFrame records operations and optimizes the full plan when you call .collect().
lazy_result = (
pl.scan_parquet("events/*.parquet") # nothing is read yet
.filter(pl.col("event_date") >= pl.date(2026, 1, 1)) # pushed down to the scan
.select("user_id", "event_type", "amount") # only these columns are read
.group_by("user_id")
.agg(
pl.col("amount").sum().alias("total_spend"),
pl.len().alias("event_count"),
)
)
print(lazy_result.explain()) # inspect the optimized query plan
df_out = lazy_result.collect()
The optimizer applies these techniques automatically:
| Optimization | What it does |
|---|---|
| Predicate pushdown | Applies filters at the file scan so fewer rows load |
| Projection pushdown | Reads only the columns the query uses |
| Common subplan elimination | Reuses repeated computations |
| Parallel execution | Runs independent branches on separate threads |
For datasets that exceed memory, run the query with the streaming engine: lazy_result.collect(engine="streaming"). Check the Polars documentation for your installed version, because streaming support has evolved across releases.
Step 5: Time Series Operations
# Hourly aggregation (replaces df.resample("1h").mean())
hourly = (
sensor_df
.sort("timestamp")
.group_by_dynamic("timestamp", every="1h")
.agg(pl.col("temperature").mean().alias("avg_temp"))
)
# Rolling statistics (replaces df["v"].rolling(5).mean())
sensor_df = sensor_df.with_columns(
rolling_avg=pl.col("temperature").rolling_mean(window_size=5),
rolling_std=pl.col("temperature").rolling_std(window_size=5),
)
group_by_dynamic requires a sorted time column. Sort first or the results will be wrong.
Common Migration Pitfalls
| Pitfall | Pandas habit | Polars fix |
|---|---|---|
Row-wise apply |
df.apply(func, axis=1) |
Rewrite with expressions; use map_elements only as a last resort because it runs Python per row |
| Index lookups | df.loc[idx] |
df.filter(...) or a join |
| Chained assignment | df["a"][mask] = 0 |
with_columns(pl.when(mask).then(0).otherwise(pl.col("a"))) |
| In-place edits | inplace=True |
Reassign the returned DataFrame |
| Dtype looseness | Auto-upcasting | Cast explicitly; mixed-type operations raise errors |
| String ops | df.a.str.lower() |
pl.col("a").str.to_lowercase() |
Real-World Use Case: Customer Churn Features
A subscription business stores raw transactions and needs per-customer features for a churn model. Polars builds the feature table in one lazy query, then hands it to Scikit-Learn.
import polars as pl
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
snapshot = pl.date(2026, 10, 31)
features = (
pl.scan_parquet("transactions/*.parquet")
.group_by("customer_id")
.agg(
pl.col("amount").sum().alias("lifetime_value"),
pl.col("amount").mean().alias("avg_order"),
pl.len().alias("n_orders"),
(snapshot - pl.col("order_date").max()).dt.total_days().alias("days_since_last"),
pl.col("order_date").n_unique().alias("active_days"),
)
.join(
pl.scan_parquet("customers.parquet").select("customer_id", "churned"),
on="customer_id",
how="inner",
)
.collect()
)
# Convert at the modeling boundary
X = features.drop("customer_id", "churned").to_pandas()
y = features["churned"].to_numpy()
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42
)
model = GradientBoostingClassifier(random_state=42).fit(X_train, y_train)
print("AUC-ROC:", roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))
The heavy aggregation across millions of transaction rows stays in Polars. Only the final, compact feature matrix crosses into the Pandas and Scikit-Learn world.
Should You Migrate Everything?
| Scenario | Recommendation |
|---|---|
| ETL and feature pipelines on files over 1 GB | Migrate to Polars |
| Repeated group-bys and joins on wide tables | Migrate to Polars |
| Small exploratory notebooks (under 100k rows) | Either; Pandas is fine |
Heavy reliance on index-based logic or MultiIndex |
Redesign first, or stay on Pandas |
| Libraries that require Pandas input | Convert at the boundary with .to_pandas() |
Migrate incrementally. Replace the slowest pipeline stage first, convert with pl.from_pandas(df) and .to_pandas() at the seams, and validate outputs against the old implementation with assert_frame_equal from polars.testing.
FAQ
Is Polars faster than Pandas?
For most large-scale operations such as group-bys, joins, and file scans, Polars is faster because it uses all CPU cores, stores data in Arrow format, and optimizes lazy queries. The gap depends on workload and data size. On small datasets the difference is often negligible.
Can I use Polars and Pandas together?
Yes. Convert with pl.from_pandas(pdf) and pldf.to_pandas(). Both directions need pyarrow installed. Many teams use Polars for heavy data preparation and Pandas for modeling and plotting libraries that require it.
Does Polars have an index like Pandas?
No. Polars deliberately omits the row index. Use filter() for row selection, join() for alignment, and .over() for group-wise calculations. Add a positional column with df.with_row_index() if you need row numbers.
When should I use LazyFrame instead of DataFrame?
Use a LazyFrame for multi-step pipelines, large files, and anything read from Parquet or CSV. It enables predicate and projection pushdown. Use an eager DataFrame for interactive exploration, where you want to see each result immediately.



