Pydantic v2's Rust core hides a costly secret at the Python boundary
Libraries Run Rust Inside Python (With PyO3)
Pydantic v2's speed comes from pydantic-core, a Rust extension built with PyO3. This post walks through a JSON parser in Rust exposed to Python via PyO3 and maturin. The surprising bottleneck: converting the parsed Rust tree into Python objects can cost more than the parsing itself. For large documents, that materialization loop dominates end-to-end time. The advice: profile the boundary, not just the algorithm, and consider lazy views instead of eagerly building the whole tree.
A document with 100,000 values means on the order of 100,000 Python objects being created at the boundary, all after parsing is completely done.
- simonw
I was a bit nervous about this trend when it started picking up because I care about Pyodide (Python running via WebAssembly) and libraries that use PyO3 might not work in Pyodide.
Thankfully that's now been mostly solved - you can compile and publish WASM builds of Rust or C extensions on PyPI now and a Pyodide can then use them.
Here's the WASM build of the Rust-including Pydantic-core package for example: https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0...
- the__alchemist
This is a nice party trick! I use it to provide easy installation of software that is written in rust, but is primarily used by Python users. (e.g.: Biology tools) They can do `pip install <name>`, since some people prefer this over downloading executables/installers. Bonus: Maturin/PyO3 can build "manylinux" binaries automatically, which helps with the Linux ABI diaspora.
- skeledrew
But does it work everywhere Python works? My main issue with this Rust move has always been compatibility. Python can be embedded and ran in a heck of a lot of places. What's the story when libraries that I may want to depend on are actually implemented in Rust and my target doesn't/can't handle the toolchain and there's no build target?
- roywiggins
> reach for, a Rust extension does the work
ai; dr, sorry
- startup_zombie_
How much of the existing PyPI ecosystem do you think could realistically work this way without package authors doing anything specifically for WASM?