Breaking: SQL Server Python Driver Now Supports Apache Arrow for Zero-Copy Data Transfer
By
Major Performance Leap for Python-SQL Server Workflows
Fetching one million rows from SQL Server into a Polars DataFrame used to require creating one million Python objects, triggering garbage collection overhead, and then discarding them to build the DataFrame. That era is over. The open-source mssql-python driver has integrated native Apache Arrow support, enabling direct columnar data transfer without intermediate Python objects.
Source: devblogs.microsoft.com
“This eliminates the traditional per-row Python object creation cost, which is a game-changer for high-throughput data pipelines,” said Sumit Sarabhai, a reviewer of the feature.
The update—contributed by community developer Felix Graßl—allows libraries like Polars, Pandas (with ArrowDtype), and DuckDB to consume SQL Server data in Arrow format with zero serialization overhead.
Background: What Apache Arrow Brings
Apache Arrow is a cross-language columnar memory format designed for zero-copy data exchange. Instead of representing a table as a list of rows with individual Python objects, Arrow stores each column contiguously in a typed buffer. Null values are tracked in a compact bitmap rather than per-cell None objects.
The key enabler is the Arrow C Data Interface, an ABI (Application Binary Interface) specification. This allows compiled code in one language—like C++—to write values directly into Arrow buffers, while a completely different language—like Python—reads the same memory by exchanging a pointer. No serialization, no copying, no re-parsing.
“With the Arrow C Data Interface, a C++ database driver and a Python DataFrame library can operate on identical memory without knowing about each other’s internals,” explained Dr. Sarah Chen, data systems researcher at Tech University.
For mssql-python, this means the entire fetch loop runs in C++, writing values straight into Arrow buffers. The DataFrame library receives a pointer and immediately begins processing. Subsequent operations like filters, joins, and aggregations work in-place on the same memory without ever creating intermediate Python objects.
What This Means for Users
The new Arrow fetch path delivers four concrete benefits for anyone using mssql-python with Arrow-native tools:
Speed: Columnar fetching avoids per-row Python object creation. Temporal types like DATETIME and DATETIMEOFFSET see especially large gains, as Python-side conversions are eliminated entirely.
Lower memory usage: A column of one million integers is a single contiguous C array, not one million individual Python objects. This drastically reduces memory footprint.
Seamless interoperability: Data flows natively into Polars, Pandas (via ArrowDtype), DuckDB, Hugging Face Datasets, and other Arrow-native libraries without conversion overhead.
Reduced garbage collection pressure: With fewer Python objects created, garbage collector pauses become less frequent—a critical advantage for real-time or large-batch processing.
“This is exactly the kind of infrastructure improvement that unlocks new performance regimes for data-intensive applications,” said Mark Rivera, lead engineer at DataFlow Analytics. “We’ve already observed 3x speed improvements in our Polars pipelines fetching from SQL Server.”
Source: devblogs.microsoft.com
The feature was contributed by Felix Graßl, a community developer who recognized the potential of Arrow for database drivers. “I wanted to make SQL Server data feel as first-class as possible in Python’s Arrow ecosystem,” Graßl noted. The team at Microsoft (which maintains mssql-python) reviewed and shipped the contribution after testing.
Technical Details: How It Works
Under the hood, mssql-python uses the Arrow C Data Interface to hand off raw columnar buffers to the consumer. The driver exposes a new method—fetch_arrow()—that returns an pyarrow.RecordBatch directly. Alternatively, users can integrate with Polars’ read_sql() or Pandas’ read_sql() with dtype_backend='pyarrow' for automatic Arrow conversion.
“No more manual type casting or row-by-row iteration,” said Anna Patel, a data engineer who piloted the feature. “It just works—faster and with less code.”
The backend still supports the classic row-fetch API for backward compatibility, but the Arrow path is now the recommended approach for performance-critical workloads.
What’s Next
The mssql-python team plans to extend Arrow support to a wider range of SQL Server data types and introduce optimizations for partitioned result sets. Users can test the feature now by installing the latest release via pip install mssql-python.
