Pandas vs Polars vs DuckDB in 2026: Which Should You Use?

Pandas, Polars, and DuckDB now overlap on the same job: turning tabular data into answers on a single machine. They differ in execution model, memory behavior, and the language you write in. Pandas runs eagerly and has the biggest ecosystem. Polars runs a lazy, multi-threaded, Arrow-native query engine behind a DataFrame API. DuckDB is an in-process analytical SQL database that reads files and DataFrames directly. Pick by workload shape, not by hype.

Quick Takeaways

• Pandas wins on ecosystem depth (scikit-learn, statsmodels, plotting) and for exploratory work on data that fits comfortably in RAM.
• Polars wins on expression-based transformation pipelines, parallelism, and lazy evaluation with query optimization.
• DuckDB wins on SQL-first analytics, joins and aggregations over Parquet/CSV, and out-of-core processing of data larger than memory.
• The tools interoperate through Apache Arrow. Mixing them in one pipeline is normal, not a compromise.

How Each Engine Executes Your Code

Execution model explains most performance differences, so start here.

Pandas: Eager and Index-Centric

Every pandas operation runs immediately and materializes its result. That makes debugging simple, because you inspect each intermediate step. It also means pandas cannot reorder or prune work across steps. Chained operations allocate intermediate frames.

Pandas 3.x modernized several long-standing pain points. Copy-on-Write semantics are now the default, which removes the ambiguity behind SettingWithCopyWarning and reduces defensive copying. A dedicated string dtype replaces slow object-dtype strings for text columns. The dtype_backend="pyarrow" option gives Arrow-backed columns for better memory use and null handling.

Polars: Lazy Plans and Expressions

Polars separates describing a computation from running it. With scan_* functions and .lazy(), you build a logical plan. The optimizer then applies projection pushdown (read only needed columns), predicate pushdown (filter at the file scan), and common-subplan elimination. Execution is parallel across cores by default. A streaming engine processes data in batches, so many queries run on datasets larger than RAM.

DuckDB: A Database Engine Without the Server

DuckDB is a columnar, vectorized, in-process OLAP database. You query Parquet, CSV, JSON, Arrow tables, pandas DataFrames, and Polars DataFrames directly with SQL. Its cost-based optimizer handles join ordering and filter pushdown. Operators spill to disk when memory runs short, so a 40 GB join on a 16 GB laptop is realistic.

Setting Up a Fair Comparison

Run the same task in each tool: filter events, aggregate by user and day, and join to a dimension table. Install the libraries first:

pip install -U pandas polars duckdb pyarrow

Generate a reproducible test dataset:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)  # fixed seed for reproducibility
n = 5_000_000

events = pd.DataFrame({
    "user_id": rng.integers(1, 200_000, n),
    "event_ts": pd.Timestamp("2026-01-01")
                + pd.to_timedelta(rng.integers(0, 86_400 * 90, n), unit="s"),
    "amount": rng.gamma(2.0, 20.0, n).round(2),
    "country": rng.choice(["US", "DE", "BR", "IN", "JP"], n),
})
users = pd.DataFrame({
    "user_id": np.arange(1, 200_000),
    "segment": rng.choice(["free", "pro", "enterprise"], 199_999),
})

# Parquet is the common interchange format for all three engines
events.to_parquet("events.parquet", index=False)
users.to_parquet("users.parquet", index=False)

The Same Pipeline in Three Tools

Pandas Implementation

import pandas as pd

# Read only the columns you need; Arrow-backed dtypes reduce memory
events = pd.read_parquet(
    "events.parquet",
    columns=["user_id", "event_ts", "amount", "country"],
    dtype_backend="pyarrow",
)
users = pd.read_parquet("users.parquet", dtype_backend="pyarrow")

result = (
    events
    .loc[events["amount"] > 10]                       # filter rows
    .assign(day=lambda d: d["event_ts"].dt.floor("D"))  # derive a day column
    .merge(users, on="user_id", how="inner")          # join dimension table
    .groupby(["segment", "day"], as_index=False)
    .agg(total_amount=("amount", "sum"),
         active_users=("user_id", "nunique"))
)

Polars Implementation

import polars as pl

# scan_parquet is lazy: nothing is read until collect()
events = pl.scan_parquet("events.parquet")
users = pl.scan_parquet("users.parquet")

result = (
    events
    .filter(pl.col("amount") > 10)                    # pushed down to the scan
    .with_columns(pl.col("event_ts").dt.truncate("1d").alias("day"))
    .join(users, on="user_id", how="inner")
    .group_by(["segment", "day"])
    .agg(
        pl.col("amount").sum().alias("total_amount"),
        pl.col("user_id").n_unique().alias("active_users"),
    )
    .collect(engine="streaming")                      # run with the streaming engine
)

Call .explain() on the lazy frame before .collect() to inspect the optimized plan. It shows which filters and column selections moved into the Parquet scan.

DuckDB Implementation

import duckdb

# DuckDB queries Parquet files in place; no load step
result = duckdb.sql("""
    SELECT
        u.segment,
        date_trunc('day', e.event_ts) AS day,
        SUM(e.amount)                 AS total_amount,
        COUNT(DISTINCT e.user_id)     AS active_users
    FROM 'events.parquet' AS e
    JOIN 'users.parquet'  AS u USING (user_id)
    WHERE e.amount > 10
    GROUP BY u.segment, day
""").pl()   # return a Polars DataFrame; use .df() for pandas

Performance and Memory: What to Expect

Benchmark your own data, because file layout, column cardinality, and hardware change results. These patterns hold up across independent benchmarks:

Use this harness for a fair timing comparison:

import time
import tracemalloc

def benchmark(fn, runs=5):
    """Return median wall time in seconds; discard the first warm-up run."""
    fn()  # warm-up: populates OS file cache, imports, JIT-like effects
    times = []
    for _ in range(runs):
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)
    return sorted(times)[len(times) // 2]

print("polars :", benchmark(run_polars))
print("duckdb :", benchmark(run_duckdb))
print("pandas :", benchmark(run_pandas))

Wrap each pipeline in a run_* function. For peak memory, use memory_profiler or watch process RSS externally. tracemalloc misses native allocations from Arrow and Rust, so it understates all three engines.

Feature Trade-Off Matrix

Practical Examples and Real-World Use Cases

Predicting Customer Churn with Python

Feature engineering for churn involves rolling aggregates per customer (sessions in the last 7, 30, and 90 days) over hundreds of millions of event rows. Use DuckDB or Polars for feature computation, then hand a compact feature table to scikit-learn.

import duckdb
from sklearn.ensemble import GradientBoostingClassifier

features = duckdb.sql("""
    SELECT
        user_id,
        COUNT(*) FILTER (WHERE event_ts >= DATE '2026-03-24') AS sessions_7d,
        COUNT(*) FILTER (WHERE event_ts >= DATE '2026-03-01') AS sessions_30d,
        AVG(amount)                                           AS avg_amount,
        MAX(event_ts)                                         AS last_seen
    FROM 'events.parquet'
    GROUP BY user_id
""").df()  # pandas hand-off for scikit-learn

# model = GradientBoostingClassifier().fit(features[[...]], labels)

The heavy aggregation runs in a multi-threaded engine. Pandas only touches the final, small matrix, which is where its ecosystem advantage matters.

Handling Missing Data in Real-Time Sensor Streams

Sensor feeds contain gaps, duplicate timestamps, and drift. Polars expresses forward-fill and interpolation compactly and executes in parallel across devices:

import polars as pl

clean = (
    pl.scan_parquet("sensors/*.parquet")
    .sort(["device_id", "ts"])
    .with_columns(
        pl.col("temperature")
          .fill_null(strategy="forward")
          .over("device_id")                          # per-device forward fill
          .alias("temp_filled")
    )
    .unique(subset=["device_id", "ts"], keep="last")  # drop duplicate timestamps
    .collect(engine="streaming")
)

Ad-Hoc Analysis of a Folder of Parquet Files

An analyst receives 300 GB of partitioned Parquet from a data lake. DuckDB’s glob reads ('lake/year=2026/*/*.parquet') and Hive-partition pruning let a laptop answer questions without ingestion or cluster provisioning.

Statistical Modeling and Visualization

When you need statsmodels regression diagnostics, seaborn plots, or index-aware time-series work, convert the filtered result with .to_pandas() or .df(). Pandas stays the lingua franca of the Python data science stack.

Decision Framework: Which Tool Should You Use?

Interoperability: Mix the Engines

Arrow makes conversion cheap, often zero-copy:

import duckdb
import polars as pl

pl_df = pl.read_parquet("events.parquet")

# DuckDB queries the Polars DataFrame by variable name (replacement scan)
agg = duckdb.sql("""
    SELECT country, SUM(amount) AS revenue
    FROM pl_df
    GROUP BY country
""").pl()

pdf = agg.to_pandas()   # hand off to pandas for plotting or modeling

Migration Tips

1. Start at the I/O boundary. Replace pd.read_csv with Polars scan_csv or a DuckDB read_csv query. Convert to Parquet once and reuse it.
2. Rewrite apply calls. Row-wise df.apply(func, axis=1) is the biggest performance sink. Polars expressions and SQL functions are vectorized replacements.
3. Drop the index mindset. Polars and DuckDB have no row index. Use explicit key columns and joins instead of label alignment.
4. Keep pandas where it earns its place. Indexing-heavy time-series work and ML library handoffs are valid reasons to stay.

FAQ

Is Polars faster than pandas?

For most multi-column transformations, aggregations, and joins on medium to large data, yes. Polars runs multi-threaded with a query optimizer, while pandas executes eagerly on mostly a single thread. On small datasets the gap narrows or disappears.

Is DuckDB faster than Polars?

Neither wins everywhere. DuckDB often leads on large joins and complex SQL aggregations thanks to its cost-based optimizer and disk spilling. Polars often matches or beats it on expression-heavy pipelines inside Python. Benchmark your actual query and file layout.

Can DuckDB replace pandas?

For analytical queries, aggregation, and ETL, largely yes. For indexed time-series manipulation, ML preprocessing hooks, and plotting integrations, pandas remains more convenient. Many teams use DuckDB for heavy lifting and pandas for the last mile.

Should beginners learn pandas, Polars, or DuckDB first?

Learn pandas first for its documentation depth and its role in tutorials and courses. Learn SQL alongside it, which transfers directly to DuckDB. Add Polars when you hit performance limits or want stricter, more explicit pipelines.