Polars for Pandas Users: A Practical Migration Guide

Pandas runs on a single core, copies data often, and materializes every intermediate step in memory. Polars is a DataFrame library written in Rust on the Apache Arrow columnar format. It parallelizes work across cores, plans queries before running them, and can process datasets larger than RAM. This guide maps each Pandas habit to its Polars equivalent and shows where the two models differ.

Quick Takeaways

  • Think in expressions, not mutations. Polars builds transformations with pl.col() and applies them through select, with_columns, filter, and group_by().agg().
  • No index. Polars has no row index, so loc, iloc, reset_index, and MultiIndex do not exist. Use filter, join, and window functions instead.
  • Use lazy mode for pipelines. scan_* plus .collect() enables predicate pushdown, projection pushdown, and parallel execution.
  • Keep Pandas at the edges. Convert with .to_pandas() for Scikit-Learn, Statsmodels, or plotting libraries that expect it.
Aspect Pandas Polars
Execution Eager, single-threaded for most operations Eager or lazy, multi-threaded
Memory model NumPy blocks (Arrow optional) Apache Arrow columnar
Row index Yes No
Missing values NaN, None, pd.NA mixed null for all types, NaN only for floats
Query optimization None Query planner in lazy mode
Larger-than-RAM data Chunking by hand Streaming engine
Ecosystem maturity Very large Growing fast

Install and Import

pip install polars pyarrow  # pyarrow is required for .to_pandas() conversions
import polars as pl
import pandas as pd  # keep for interop during migration

The Mental Shift: Expressions

A Pandas workflow mutates a DataFrame step by step. A Polars workflow describes expressions, which are reusable recipes for computing a column. The engine runs independent expressions in parallel.

# Pandas: mutate and assign
df["revenue"] = df["price"] * df["quantity"]
df = df[df["revenue"] > 100]

# Polars: describe the result
df = (
    df.with_columns(revenue=pl.col("price") * pl.col("quantity"))
      .filter(pl.col("revenue") > 100)
)

Three rules cover most of the migration:

  1. DataFrames are immutable. Every method returns a new DataFrame, so df["col"] = ... raises an error.
  2. Expressions inside one with_columns call run in parallel. An expression cannot reference a column created in the same call. Chain a second with_columns instead.
  3. Column selection is explicit. Use pl.col("a"), pl.col("a", "b"), pl.col(pl.Float64), or pl.all().

Pandas to Polars Cheat Sheet

Task Pandas Polars
Read CSV pd.read_csv(p) pl.read_csv(p) / pl.scan_csv(p)
Read Parquet pd.read_parquet(p) pl.read_parquet(p) / pl.scan_parquet(p)
Select columns df[["a", "b"]] df.select("a", "b")
Filter rows df[df.a > 1] df.filter(pl.col("a") > 1)
New column df.assign(c=df.a + df.b) df.with_columns(c=pl.col("a") + pl.col("b"))
Rename df.rename(columns={"a": "x"}) df.rename({"a": "x"})
Sort df.sort_values("a", ascending=False) df.sort("a", descending=True)
Group and aggregate df.groupby("k").agg(...) df.group_by("k").agg(...)
Merge df.merge(o, on="k", how="left") df.join(o, on="k", how="left")
Fill missing df.fillna(0) df.fill_null(0)
Drop missing df.dropna() df.drop_nulls()
Cast type df.a.astype("float64") pl.col("a").cast(pl.Float64)
Wide to long df.melt(...) df.unpivot(...)
Group transform df.groupby("k")["v"].transform("sum") pl.col("v").sum().over("k")
Conditional np.where(c, a, b) pl.when(c).then(a).otherwise(b)
Unique count df.a.value_counts() df["a"].value_counts()

Step 1: Load and Inspect Data

df = pl.read_csv("sales.csv", try_parse_dates=True)

print(df.shape)        # (rows, columns)
print(df.schema)       # column names and dtypes
print(df.head(5))
print(df.describe())   # summary statistics, similar to pandas describe()
print(df.null_count()) # nulls per column in one call

Polars infers dtypes from a sample of rows. For large or messy files, pass an explicit schema_overrides mapping to avoid inference errors.

Step 2: Core Transformations

Select, filter, and derive columns

result = (
    df
    .filter(
        (pl.col("status") == "completed") & pl.col("order_date").is_not_null()
    )
    .with_columns(
        revenue=pl.col("price") * pl.col("quantity"),
        order_month=pl.col("order_date").dt.truncate("1mo"),  # month bucket
    )
    .select("order_id", "region", "order_month", "revenue")
)

Use &, |, and ~ for boolean logic. Wrap each condition in parentheses.

Conditional logic

df = df.with_columns(
    tier=pl.when(pl.col("revenue") >= 1000).then(pl.lit("enterprise"))
           .when(pl.col("revenue") >= 200).then(pl.lit("mid"))
           .otherwise(pl.lit("small"))
)

Group-by aggregations

summary = (
    df.group_by("region", "order_month")
      .agg(
          pl.col("revenue").sum().alias("total_revenue"),
          pl.col("revenue").mean().alias("avg_order_value"),
          pl.col("order_id").n_unique().alias("orders"),
      )
      .sort("region", "order_month")
)

group_by does not guarantee row order by default. Add maintain_order=True if order matters, or sort afterward.

Window functions replace transform

Pandas needs groupby().transform() to broadcast a group statistic back to each row. Polars uses .over().

df = df.with_columns(
    region_share=pl.col("revenue") / pl.col("revenue").sum().over("region"),
    rank_in_region=pl.col("revenue").rank("dense", descending=True).over("region"),
)

Joins

customers = pl.read_parquet("customers.parquet")

enriched = df.join(customers, on="customer_id", how="left")

Polars supports inner, left, right, full, semi, anti, and cross joins. Semi and anti joins replace the isin filtering patterns Pandas users often write.

Step 3: Handle Nulls and NaN Correctly

Pandas treats NaN as the missing marker for floats. Polars separates the two concepts. A null means missing data. A NaN is a floating-point value that results from an invalid computation.

Operation Pandas Polars
Detect missing isna() is_null()
Detect NaN isna() (same call) is_nan()
Replace missing fillna(x) fill_null(x)
Replace NaN fillna(x) fill_nan(x)
Forward fill ffill() fill_null(strategy="forward")
df = df.with_columns(
    pl.col("price").fill_nan(None)           # convert NaN to null first
      .fill_null(strategy="forward")         # then forward fill
      .over("sku")                           # respect product boundaries
)

Convert NaN to null early. Aggregations like mean() skip nulls but propagate NaN, which causes silent surprises.

Step 4: Use Lazy Evaluation

Lazy mode is the largest single performance gain. A LazyFrame records operations and optimizes the full plan when you call .collect().

lazy_result = (
    pl.scan_parquet("events/*.parquet")                 # nothing is read yet
      .filter(pl.col("event_date") >= pl.date(2026, 1, 1))  # pushed down to the scan
      .select("user_id", "event_type", "amount")         # only these columns are read
      .group_by("user_id")
      .agg(
          pl.col("amount").sum().alias("total_spend"),
          pl.len().alias("event_count"),
      )
)

print(lazy_result.explain())  # inspect the optimized query plan
df_out = lazy_result.collect()

The optimizer applies these techniques automatically:

Optimization What it does
Predicate pushdown Applies filters at the file scan so fewer rows load
Projection pushdown Reads only the columns the query uses
Common subplan elimination Reuses repeated computations
Parallel execution Runs independent branches on separate threads

For datasets that exceed memory, run the query with the streaming engine: lazy_result.collect(engine="streaming"). Check the Polars documentation for your installed version, because streaming support has evolved across releases.

Step 5: Time Series Operations

# Hourly aggregation (replaces df.resample("1h").mean())
hourly = (
    sensor_df
    .sort("timestamp")
    .group_by_dynamic("timestamp", every="1h")
    .agg(pl.col("temperature").mean().alias("avg_temp"))
)

# Rolling statistics (replaces df["v"].rolling(5).mean())
sensor_df = sensor_df.with_columns(
    rolling_avg=pl.col("temperature").rolling_mean(window_size=5),
    rolling_std=pl.col("temperature").rolling_std(window_size=5),
)

group_by_dynamic requires a sorted time column. Sort first or the results will be wrong.

Common Migration Pitfalls

Pitfall Pandas habit Polars fix
Row-wise apply df.apply(func, axis=1) Rewrite with expressions; use map_elements only as a last resort because it runs Python per row
Index lookups df.loc[idx] df.filter(...) or a join
Chained assignment df["a"][mask] = 0 with_columns(pl.when(mask).then(0).otherwise(pl.col("a")))
In-place edits inplace=True Reassign the returned DataFrame
Dtype looseness Auto-upcasting Cast explicitly; mixed-type operations raise errors
String ops df.a.str.lower() pl.col("a").str.to_lowercase()

Real-World Use Case: Customer Churn Features

A subscription business stores raw transactions and needs per-customer features for a churn model. Polars builds the feature table in one lazy query, then hands it to Scikit-Learn.

import polars as pl
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

snapshot = pl.date(2026, 10, 31)

features = (
    pl.scan_parquet("transactions/*.parquet")
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("lifetime_value"),
        pl.col("amount").mean().alias("avg_order"),
        pl.len().alias("n_orders"),
        (snapshot - pl.col("order_date").max()).dt.total_days().alias("days_since_last"),
        pl.col("order_date").n_unique().alias("active_days"),
    )
    .join(
        pl.scan_parquet("customers.parquet").select("customer_id", "churned"),
        on="customer_id",
        how="inner",
    )
    .collect()
)

# Convert at the modeling boundary
X = features.drop("customer_id", "churned").to_pandas()
y = features["churned"].to_numpy()

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = GradientBoostingClassifier(random_state=42).fit(X_train, y_train)
print("AUC-ROC:", roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

The heavy aggregation across millions of transaction rows stays in Polars. Only the final, compact feature matrix crosses into the Pandas and Scikit-Learn world.

Should You Migrate Everything?

Scenario Recommendation
ETL and feature pipelines on files over 1 GB Migrate to Polars
Repeated group-bys and joins on wide tables Migrate to Polars
Small exploratory notebooks (under 100k rows) Either; Pandas is fine
Heavy reliance on index-based logic or MultiIndex Redesign first, or stay on Pandas
Libraries that require Pandas input Convert at the boundary with .to_pandas()

Migrate incrementally. Replace the slowest pipeline stage first, convert with pl.from_pandas(df) and .to_pandas() at the seams, and validate outputs against the old implementation with assert_frame_equal from polars.testing.

FAQ

Is Polars faster than Pandas?

For most large-scale operations such as group-bys, joins, and file scans, Polars is faster because it uses all CPU cores, stores data in Arrow format, and optimizes lazy queries. The gap depends on workload and data size. On small datasets the difference is often negligible.

Can I use Polars and Pandas together?

Yes. Convert with pl.from_pandas(pdf) and pldf.to_pandas(). Both directions need pyarrow installed. Many teams use Polars for heavy data preparation and Pandas for modeling and plotting libraries that require it.

Does Polars have an index like Pandas?

No. Polars deliberately omits the row index. Use filter() for row selection, join() for alignment, and .over() for group-wise calculations. Add a positional column with df.with_row_index() if you need row numbers.

When should I use LazyFrame instead of DataFrame?

Use a LazyFrame for multi-step pipelines, large files, and anything read from Parquet or CSV. It enables predicate and projection pushdown. Use an eager DataFrame for interactive exploration, where you want to see each result immediately.

Hot this week

SQL for Time Series: Date Functions, Rolling Averages, and Gaps

Master SQL time series analysis: date functions, rolling averages, and gap filling with copy-ready PostgreSQL code. Start querying smarter today.

Build a RAG App From Scratch: Documents, Embeddings, and Answers

Build a RAG app from scratch in Python: chunk documents, create embeddings, retrieve context, and generate cited answers. Copy the full code now.

SHAP vs LIME: Explain Any Model’s Predictions

Compare SHAP vs LIME for explaining ML predictions. See code, speed tradeoffs, and a decision guide. Pick the right explainer for your model today.

Predict House Prices With Linear Regression: A Complete Project

Build a house price predictor with linear regression in Python. Clean data, train, and evaluate with RMSE and R². Start the full project now.

AI and Copyright: The Lawsuits, the Rulings, and What They Mean for Creators

Track every major AI copyright lawsuit in 2026: Anthropic's $1.5B deal, NYT v. OpenAI, Ross, Midjourney. See what it means for your work. Read now.

Topics

SQL for Time Series: Date Functions, Rolling Averages, and Gaps

Master SQL time series analysis: date functions, rolling averages, and gap filling with copy-ready PostgreSQL code. Start querying smarter today.

Build a RAG App From Scratch: Documents, Embeddings, and Answers

Build a RAG app from scratch in Python: chunk documents, create embeddings, retrieve context, and generate cited answers. Copy the full code now.

SHAP vs LIME: Explain Any Model’s Predictions

Compare SHAP vs LIME for explaining ML predictions. See code, speed tradeoffs, and a decision guide. Pick the right explainer for your model today.

Predict House Prices With Linear Regression: A Complete Project

Build a house price predictor with linear regression in Python. Clean data, train, and evaluate with RMSE and R². Start the full project now.

AI and Copyright: The Lawsuits, the Rulings, and What They Mean for Creators

Track every major AI copyright lawsuit in 2026: Anthropic's $1.5B deal, NYT v. OpenAI, Ross, Midjourney. See what it means for your work. Read now.

Build Your First Spam Email Classifier

Build a spam email classifier in Python with TF-IDF and Naive Bayes. Copy the code, hit 98%+ accuracy, and ship your first ML model today.

Best AI Music and Voice Generators in 2026

Compare Suno v6, ElevenLabs Music, Udio, Eleven v4, Cartesia and Fish Audio on price, rights and latency. Pick your tool and start creating today.

Train/Test Splits Done Right: Best Practices and Common Mistakes

Avoid data leakage with proper train/test splits. Compare random, stratified, group and time-series methods with Python code. Fix your pipeline today.

Related Articles

Popular Categories