Skip to content

pyodbc -> mssql-python, setting chunksize does not stream results #390

@aa-matthias

Description

@aa-matthias

Describe the bug

Migrating from pyodbc and trying to fetch a huge table via pandas.read_sql_query by setting chunksize
It looks like the whole result set is loaded into memory which definitely was not the case with pyodbc.
Is that supported already?

To reproduce

        import mssql_python
        import pandas as pd
        import logging

        connection_string = f"SERVER=tcp:{server},1433;DATABASE={database};UID={username};PWD={password};Encrypt=no;"
        conn_MSSQL = mssql_python.connect(connection_string)

        for chunk in pd.read_sql_query(query, conn_MSSQL, chunksize=50000):
            logging.info(f"Fetched {chunk.shape[0]} Results from MSSQL")

Expected behavior

results should be streamed

Further technical details

Python version: 3.12.0
SQL Server version: SQL Server 2019
Operating system: Windows Server 2019

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions