Why your local LLM feels dumber than it is
This technical deep-dive from Level1Techs reveals why your local LLM may underperform despite using the same weights as the reference model. Through controlled experiments on Qwen3.6-27B, the author demonstrates that implementation details—such as attention backend, KV cache quantization, and weight quantization—cause measurable divergence in token probabilities, leading to subtle errors and even tool-calling failures. The post emphasizes that benchmarks must reflect real workloads and that methodology is crucial when interpreting KLD claims.
The methodology matters as much as the number and plenty of people get it wrong.
- big-chungus4
I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch
- jonplackett
I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
- heywoods
So to what extent does this apply to cloud hosted LLM’s? Are there benchmarks that score models across cloud providers? My experience using LLM’s during day time vs evening sessions has felt “night and day” and I’ve chalked that mostly up to it must be my imagination or just the general indeterministic nature of LLM’s. Sessions resumed after a day away also feel “dumb” sometimes so I can see an aggressive kv cache eviction policy playing a role if it’s reasonable to extrapolate what the article is saying about local inference.
Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?
- a11r
Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.
- utopiah
Comments are mostly showing off M5s and 5090s without addressing the article.
- walrus01
Much of this is why I stick to the rule of:
a) Don't quantize your KV cache
b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
- InvertedRhodium
I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
- nullpoint420
At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering