Ask HN: What is one simple thing LLMs are insanely bad at?
I am looking for ideas on what to train a specialized model for. What is one simple thing you repeatedly ask ChatGPT, Claude, or another model to do that it still somehow messes up?
Spatial reasoning and 3d rigging and animation. Oh, you said simple. Speaking like a human.
- jampa
Serious answer: no model ever gets close to writing an architectural floor plan that makes sense.
They understand all the rules and best practices, they can (sometimes) spot a bad idea in a floor plan, they can describe a good floor plan.
But ask them to make one, even if you give it every detail (even a "node graph" of rooms), they will still output nonsense. Same for text and image models.
Floor plans should be the new Pelican Benchmark.
- ghostpepper
They don't generate keyword search queries very well. They can overcome this by brute force but if you watch what they search you will cringe.
nhl toronto scores
nhl hockey toronto scores
"nhl hockey" toronto score today
nhl "hockey score toronto"
"hockey" who won toronto
etc.
Somehow being good at semantic search makes them bad at keyword search, for whatever reason.
- mojuba
One unexpected discovery that I have made while building an AI-based system: the LLM's are bad at designing prompts.
We tend to think that the AI has some sort of self-knowledge and should be good at designing prompts for itself but it's really not.
Been struggling with a task that heavily depended on prompts, ended up rewriting all my prompts from scratch in my own words, and it finally worked. Then every time I ask Claude to fix something in the prompts, it invariably makes it worse.
A very strange phenomenon that can probably be explained by the quality of prompt design advice that made it to the training dataset. Bottomline, all the prompt design advice that you can find on the internet is really not great.
- tartoran
LLMs are bad at not inventing stuff (hallucinating facts, sources etc), they're also bad at not over explaining, remembering details reliably, asking the right question and avoiding repetition.
- sandcat_
Video game tips. Constant mistakes and hallucinations, in my experience. Seen this across a lot of different games. Even in really well documented games, such as OSRS (which has multiple fantastic wikis).
Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.