SpacetimeDB 2.0's benchmarks are dishonest, reviewer says
SpacetimeDB: A Short Technical Review
A technical review of SpacetimeDB 2.0 finds its published benchmarks are not honest, comparing the database unfairly against competitors. The reviewer notes the product is an all-in-one database and application server with in-memory storage, a single global mutex, and an asynchronous write-ahead log. SpacetimeDB's own benchmarks show poor performance against competitors in an alternate test.
You don't need 'INSANE BENCHMARKS' to win at this.
- cloutiertyler
I'm a SpacetimeDB developer and this post has many inaccuracies. I have actually asked Vicent to fix them and he's agreed to address them.
Please read our reply to this and other criticism here: https://spacetimedb.com/blog/benchmarking
- nemothekid
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
- Escapado
I appreciate the article from a technical perspective. Is there anyone here who has used it in production regardless of the criticism and has some perspective to share on where it actually falls flat or situations where performance degrades massively? Them having built an MMO around it makes it look like the technical limitations around this pattern might(!) only matter for very specific workloads or for poorly implemented reducers - and my gut feeling is that you can also reach for postgres or sqlite and write very inefficient queries that quickly bog your system down as well. Just curious how it stacked up in the real world.
- LarsDu88
I remember a friend showing me some stuff about this when they were still trying to pursue the MMORPG route and I am surprised this company is still alive given the amount of unnecessary overengineering that went into that effort.
The model of putting ECS like logic as code and putting all that code as stored procs solves a handful of db transaction issues that none of the major mmorpgs really had issues with 20 years ago when mmorpgs were all the rage and when computers were 100x-1000x slower than today. A lot of engineering is going to solving and cutting out these database round trips that were not even an issue 20 years ago and even less of an issue today.
On the other hand the actual hard parts about building a multiplayer game, such as roll back, physics, collisions, all of that sort of stuff, well this actually doesn't really help with any of that. If anything it makes all of those things even more difficult. I imagine that the developers of this game probably had to come up with some pretty interesting hacks that probably didn't benefit from their DB to get anything resembling physics to work.
Layered on top of that are the known anti-patterns around using stored procs within databases, which spacetimedb does't address at all. You can't really run your tests just on your client because the application logic is split between client and db-server leading to spaghetti code and poor iteration cycles for game design. The database itself also doesn't horizontally scale beyo […]
- Groxx
Broadly enjoyed my read, though I do have a nitpick here:
>The absence of side effects or stalls cannot be enforced by the type system ...
Side effects is probably correct, but stalls would imply wasm code that calls out to a long-time blocking function - that's generally quite easy to type-system-ify, and wasm stuff often does so with promise-like constructs. So there would be a need for spacetimedb-side markers for "this func might do HTTP", but that kind of marker for WASM-contact-able code is very much a normal expectation.
If they don't have that kind of marker, and do allow blocking calls in their beta API, then yeah - huge problem with that kind of internal structure (shared global lock), completely agreed.