Three ways to smuggle SQLite into Nix
Nix's builtins.fromJSON eagerly parses entire JSON files, making large indexes costly. The author explores three workarounds to use SQLite for efficient queries: builtins.exec (running sqlite3 and parsing output), builtins.importNative (loading a C++ shared object), and a giant Nix file (leveraging laziness). A fourth option, builtins.wasm, is also discussed, with a custom read_file_range patch enabling SQLite's wasm build to read databases.
The whole point: the database handle outlives a single query, so the b-tree pages we touch stay warm for the rest of the evaluation.
- Naru41
Just convert it to a binary data structure suitable for this purpose and use the old-fashioned bsearch. It's so simple that you wouldn't even need SQLite.
- kevincox
It seems that you could just compile the data into the WASM blob. Then use a more optimized query engine than sqlite. This should be very fast to compile (most of the WASM is just a byte buffer, the code is just a few binary searches and some result encoding). The downside is that you need to recompile to update the repo, but I don't think that should be particularly expensive.
- setheron
(author)
If you have questions, happy to try and answer some.