Pandas Should Go Extinct: The Case for DuckDB and Polars
Pandas forces users to adopt distributed systems like Spark or Snowflake long before their data justifies the complexity. Amazon Redshift fleet data shows 94.68% of tables hold under 100GB and 86.9% of queries process 80GB or less — squarely Medium Data. Benchmarks on a 1-billion-row CSV reveal Polars and DuckDB finish in about 5 seconds versus Pandas' 4m28s, using 2x and 19x less memory respectively.
You likely do not have Big Data, and probably never will. You have Medium Data problems, and need Medium Data solutions.
- minimaxir
It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant.
I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.
- rmunn
Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepared to see if the author had a serious argument to present.
Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.
- sjtrny
> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
- mulmboy
People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning.
Don't skip your profiling
- latent-person
In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
- crazysim
Am I crazy or did the OP swap the contents of the posts around accidentally?
https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/
- __mharrison__
I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)
Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).
If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.
- evolve-maz
Only in the last few years did I start using SQL properly. Before that my pipelines would live in python. Now I offload as much to the db as possible, and keep my python simple glue. I'm very happy with this compared to other methods in pandas or polars.
If I still need to do db-like things in python I think duckdb is better.