How do you efficiently read a large CSV file (multiple...

A few tactics compound here. Loading only the columns you need with usecols is the easiest win — a 50-column file where you need 5 columns means an instant 90% memory reduction. Downcasting dtypes explicitly (float64 to float32, low-cardinality strings to category) cuts memory further, often by half or more. When the file genuinely doesn’t fit in memory at all, chunksize in read_csv processes it in pieces instead of loading everything at once.

But the single highest-ROI move, for any dataset you’ll read more than once, is converting CSV to Parquet: a one-time conversion that yields roughly 10x faster reads and 60-80% less disk space afterward, because Parquet is a columnar format that stores each column’s data together and compresses it accordingly. For 10 million rows and 20 columns, a CSV at float64 might run 1.6GB while the Parquet equivalent runs around 200MB — and every subsequent load is fast from then on. For anything beyond a one-off script, that conversion should happen on first ingest, not be repeated per run.