Narwhals: Building Dataframe-Agnostic Libraries with Zero Dependencies

After the publication of our article about Ibis, Dr André Schemaitat pointed us to a similar tool with growing popularity – Narwhals. Narwhals describes itself as an "extremely lightweight and extensible compatibility layer between dataframe libraries". At its core, it uses a subset of the Polars API and has zero dependencies; it only uses what the user passes in, so the library can stay as lightweight as possible.

While both Narwhals and Ibis enable dataframe-agnostic code, they serve fundamentally different audiences. Ibis is designed as a complete analytical framework for end users, data scientists and analysts performing their daily work. Narwhals, by contrast, is built for library maintainers and application developers who want to accept multiple dataframe types as inputs without requiring all of them as dependencies. This distinction shapes every aspect of Narwhals' design, from its minimal API surface to its strict backwards compatibility guarantees.

What Does Narwhals Do?

Writing dataframe-agnostic code is hard because the same expression can produce different results depending on the library. A unified, simple and predictable API can help developers focus on behavior rather than subtle implementation differences. The installation of the library itself is as easy as expected and no dependencies are needed, apart from the dataframe library or libraries we actually want to use. Another interesting aspect is Narwhals' handling of both pandas and Polars frequently deprecating behavior. Narwhals tests against nightly builds of both libraries and handles backwards compatibility internally, so the user does not have to worry about it.

Narwhals is primarily aimed at library maintainers rather than end users. As such, it takes stability and backward compatibility very seriously. Public functions in the stable releases v1 and v2 will never be removed or changed. If backwards-incompatible changes have to be made, they will only be pushed into the main narwhals namespace and eventually into narwhals.stable.v3, but v1 and v2 will always stay unaffected and will be maintained indefinitely. This means different packages can depend on different Narwhals stable APIs, and end users can use all of them in the same project without conflict.

Because Narwhals implements a subset of the Polars API, and Polars' syntax is subject to change, this stability guarantee is particularly valuable. Users may encounter deprecated functions (like we did in our first benchmark) or even breaking changes in upstream libraries. Narwhals shields users from this: code written with the stable namespace will keep working, even in newer versions of Polars where functions have been rewritten. This ambitious promise has its limits, which Narwhals acknowledges. Unambiguous bugs will be fixed without that counting as a breaking change, type hints may be refined, and anything labeled "unstable" can change. The developers also state that if Polars were to remove expressions entirely or pandas dropped support for categorical data, Narwhals itself would have to be reconsidered, but they consider such radical changes unlikely. Old Python versions will be dropped around the time of their end of life.

At this point, it might seem a bit overwhelming, as there seem to be lots of different versions. The documentation acknowledges this and provides help. In general, narwhals should be used for prototyping, so users can iterate quickly. If the product becomes something users wish to release as production-ready and stable, they should switch to narwhals.stable.v2. For an entirely new project, users should either use narwhals or narwhals.stable.v2. In case of a project already using narwhals.stable.v1 without the need for newer features, there is no reason to switch to v2. If users wish to use v2, they should require at least narwhals>=2.0. The v1 version is older and has noticeable differences, which can be seen here. There are no missing features in v2 and everything that is stable from the main namespace is available in that version.

Narwhals can generate SQL via the narwhals.sql module. Currently, this module requires DuckDB to be installed. By setting the pretty parameter of the to_sql function to True, the SQL gets formatted to a more readable format, but this in turn requires sqlparse to be installed. As the SQL module relies on DuckDB, the generated SQL code follows DuckDB's dialect. To translate it to other dialects, users can use SQLGlot directly, or alternatively use Ibis or SQLFrame, which both leverage SQLGlot internally, as we discussed in our previous article about Ibis. This dependency requirement is likely to change to align with Narwhals' zero-dependency approach, but it was not as high a priority as for Ibis given their different target audiences.

Narwhals vs Ibis

Our initial instinct was to compare Ibis to Narwhals, as they both appear to do the same thing. The Narwhals documentation acknowledges this and interestingly enough states that they consider them to be very different and not in competition. Narwhals even supports Ibis tables, meaning that dataframe-agnostic code written using Narwhals' lazy API also supports Ibis.

The documentation outlines several key differences between the two tools. Most fundamentally, Narwhals is designed for library maintainers building tools that need to accept multiple dataframe types, while Ibis targets end users, data scientists and analysts performing their analytical work. This difference in audience carries through to every design choice.

Narwhals allows users to write functions that take a DataFrame and return one in the exact same format, preserving the input type. Ibis can materialize to pandas, Polars, and PyArrow, but has no built-in way to return the exact input type. On the data type side, Narwhals supports Categorical and Enum types across its backends, while Ibis does not. Their execution models also differ: Ibis focuses on lazy execution with SQL generation, whereas Narwhals separates between lazy and eager APIs, with the eager API providing very fine control over dataframe operations.

From a dependency perspective, Ibis requires pandas and PyArrow for all backends by default, while Narwhals has zero required dependencies, it only uses what the user passes in (unless converting to SQL, which requires DuckDB). Ibis currently supports more backends (20+ execution engines), but Narwhals still supports pandas and Dask, for which Ibis has deprecated support. Perhaps the most relevant difference regarding daily usage is the API itself: Narwhals uses a subset of the Polars API, while Ibis uses its own pandas/dplyr-inspired API.

In practice, these tools can be complementary rather than competing. A library built with Narwhals can accept Ibis tables as inputs, and users working with Ibis can leverage Narwhals-powered libraries seamlessly.

There are already some libraries and tools that use Narwhals for their dataframe interoperability needs, like Bokeh for interactive data visualization in the browser, Marimo for a reactive notebook, or the interactive graphing library Plotly, all with around 20,000 stars on GitHub.

Practice

To write a dataframe-agnostic function, we first need to initialize a Narwhals DataFrame or LazyFrame by passing our dataframe to nw.from_native. All the calculations stay lazy if we start with a LazyFrame, and Narwhals will never automatically trigger computation without asking. After that, we can express our logic using the subset of the Polars API supported by Narwhals. Finally, we can return a dataframe in its original library using nw.to_native. Since steps 1 and 3 are so common, Narwhals provides a utility @nw.narwhalify decorator, so we only have to explicitly write step 2. Let's consider the following example with a simple group-by and mean operation.

1import narwhals as nw
2from narwhals.typing import FrameT
3
4
5@nw.narwhalify
6def func(df: FrameT) -> FrameT:
7    return df.group_by("a").agg(nw.col("b").mean()).sort("a")

We can then simply use it with whatever engine we like:

1import pandas as pd
2
3df = pd.DataFrame({"a": [1, 1, 2], "b": [4, 5, 6]})
4print(func(df))

If we want to change from pandas to Polars, we do not have to rewrite the logic itself, only the import (and reference).

1import polars as pl
2
3df = pl.LazyFrame({"a": [1, 1, 2], "b": [4, 5, 6]})
4print(func(df).collect())

When dealing with different libraries, especially their more complex operations, it is possible to encounter functions that may not yet be implemented in Narwhals. In such cases, Narwhals can still be useful as a thin DataFrame ingestion layer. If a library developed with Narwhals wants to accept dataframes in any format but operates on pandas internally, Narwhals can handle the conversion. This is significantly more lightweight than including all dataframe libraries as dependencies, and the implementation is straightforward:

1def df_to_pandas(df: IntoDataFrame) -> pd.DataFrame:
2    return nw.from_native(df).to_pandas()

This overview does not nearly cover all capabilities of Narwhals, and the documentation offers great resources to get started.

Performance Overhead

The documentation claims that the overhead of running pandas via Narwhals compared to running pure pandas is negligible and sometimes "even negative". The developers were careful to avoid unnecessary copies and index resets. For lazy backends, Narwhals respects the backends' laziness and never evaluates a full query unless explicitly asked via .collect(). In some places, such as joins and selects, Narwhals does need to inspect a dataframe's schema to mimic Polars' behavior, but this is typically cheap since it can be done from metadata alone without reading the full dataset into memory. To minimize this overhead, Narwhals caches schema and column name evaluations.

How Narwhals Works Under the Hood

At the core of Narwhals is one rule: "An expression is a function from a DataFrame to a sequence of Series." A Series is a one-dimensional labeled array capable of holding any data type. In its simplest form, nw.col('a') returns the Series a from the DataFrame. Expressions can also return multiple Series (e.g., nw.col('a', 'b')), but all columns must have been derived from the same dataframe. By itself, an expression doesn't produce a value, it only produces one once given to a DataFrame context. What happens depends on which context is used: .select creates a DataFrame with only the result of the given expression, .with_columns produces a DataFrame like the current one plus the expression's result, and .filter keeps only rows where the expression evaluates to True.

Each implementation in Narwhals defines their own Narwhals-compliant objects in subfolders such as narwhals._pandas_like, narwhals._arrow, or narwhals._polars. Meanwhile the top-level modules such as narwhals.dataframe and narwhals.series coordinate how to dispatch the Narwhals API to each backend. So in the end, there are a couple of layers: the nw.DataFrame is backed by a Narwhals-compliant DataFrame, such as narwhals._pandas_like.dataframe.PandasLikeDataFrame or narwhals._arrow.dataframe.ArrowDataFrame. These Narwhals-compliant DataFrames are then again backed by a native DataFrame, in our case a pandas DataFrame and a PyArrow Table.

When a user executes code, some top-level Narwhals API is being called. The API then forwards the call to a Narwhals-compliant dataframe wrapper, like PandasLikeDataFrame or PolarsDataFrame. The DataFrame wrapper then forwards the call to the underlying library, like the pandas or Polars DataFrame.

Each operation in Narwhals is a node, which can be accessed via ._nodes. Additionally, Narwhals provides metadata of its expressions. Here we can see how and whether the expression expands to multiple outputs, how many order-dependent operations it contains, and whether the output of the expression is always length-1, and more.

One noteworthy trick (of many) of Narwhals is the elementwise push-down. SQL is picky about over operations, whereas Polars isn't. In SQL, abs(sum(a)) over (partition by b) is not valid. In Polars, pl.col('a').sum().abs().over('b') is valid. Narwhals rewrites expressions to keep Polars' level of flexibility when translating to SQL engines. Specifically, it pushes down over nodes past elementwise ones. In our Polars example, Narwhals automatically inserts the over operation before the abs one. The idea is that elementwise operations operate row-by-row and don't depend on the rows around them. An over node partitions or orders a computation. Therefore, an elementwise operation followed by an over operation is the same as doing the over operation followed by the same elementwise operation. But it is important to keep in mind that query optimization is out-of-scope for Narwhals. This expression rewrite is considered acceptable by the developers because it is simple and allows users to evaluate operations that would otherwise not be allowed for certain backends.

Conclusion

Narwhals and Ibis both address dataframe portability, but for different audiences. Ibis is a complete analytical framework for end users who want to write logic once and run it across 20+ backends. Narwhals is built for library maintainers who want to accept multiple dataframe types without requiring them all as dependencies.

The key insight: these tools are complementary, not competing. Narwhals-powered libraries like Plotly, Bokeh, and Marimo can accept Ibis tables as inputs, so end users can work with their preferred framework while library maintainers avoid the complexity of supporting every dataframe format directly.

Narwhals is the right choice when building a library or tool that needs to accept dataframes as inputs, when supporting multiple dataframe libraries without maintaining separate codebases, or when backwards compatibility guarantees matter for production deployments. Ibis is the better fit for end users performing analytical work across multiple database backends, for workflows that need comprehensive SQL backend support across 20+ engines, or for teams that want to develop locally on DuckDB and deploy to BigQuery or Snowflake without code changes.

Together, these tools reduce friction and eliminate lock-in. Ibis enables portability of analytical intent across execution engines; Narwhals enables portability of dataframe inputs across tools and libraries. As DuckDB, Polars, and other high-performance engines gain adoption, tools like Narwhals ensure that competition and innovation in dataframe libraries don't fragment the broader ecosystem.

If you are interested in exploring how these tools fit into modern data analytics workflows, check out our related articles on DuckDB vs Polars vs Pandas benchmarking, Ibis for backend-agnostic analytics, and our DuckDB and Polars stress test exploring extreme-scale workloads.