Deptho

Guides

pandas read_csv MemoryError on a large file: why it happens and how to fix it

read_csv holds parser buffers and the finished DataFrame at once, so peak memory is well above the final frame's size.

Why it happens

read_csv holds parser buffers and the finished DataFrame at the same time. In our benchmark, parsing one 1M-row daily Binance trades file produced a 47 MB DataFrame with a peak of 87 MB of heap memory, about 1.9x the final frame (methodology). Text columns cost more again, because each value becomes a Python object.

Fix it with plain pandas

import pandas as pd

# 1. Read only the columns you need, with compact dtypes.
df = pd.read_csv(
    "BTCUSDT-aggTrades-2026-09-30.csv",
    usecols=["price", "quantity", "transact_time"],
    dtype={"price": "float32", "quantity": "float32", "transact_time": "int64"},
)

# 2. Or stream it in chunks and reduce each chunk before keeping it.
bars = []
for chunk in pd.read_csv("big.csv", chunksize=2_000_000, usecols=["price", "transact_time"]):
    chunk.index = pd.to_datetime(chunk.pop("transact_time"), unit="ms")
    bars.append(chunk["price"].resample("1min").ohlc())
bars = pd.concat(bars).groupby(level=0).agg(
    {"open": "first", "high": "max", "low": "min", "close": "last"})

Or with Deptho

dp.load samples the file first and tells you the peak RAM and parse time before pandas starts. If it won't fit, you get a clear error instead of a dead kernel.

import deptho as dp

df = dp.load("BTCUSDT-aggTrades-2026-09-30.csv")   # warns or raises before you run out of memory

Working with files bigger than RAM? Deptho Pro memory-maps them.

Related