I'm a seeing-eye dog for a computer
As a robot programmer in 2026, the author often relies on LLM coding assistants, but finds them hopeless at visual debugging. Despite having image encoders and MCP servers, these models lack intuition about how robots actually move and struggle to navigate GUI tools, turning simple tasks into half-hour ordeals. The author ends up doing the debugging manually, acting as a 'seeing-eye dog' for the computer, and concludes that the most effective workflow is to do the fun part themselves.
I used to argue with people on the internet. After about six replies, you realize that you’re speaking to someone incapable of thought.
- dwedge
> I used to argue with people on the internet, after about six replies, you realize that you’re speaking to someone incapable of thought
He realises the current woe of things but doesn't realise he's been arguing with bot farms and teams of people hired just for this reason - to sow doom, arguments and engagement.
Around 10 years ago I noticed this happening on trending topics of Twitter - it wasn't that the opponents were stupid because they disagreed, it was that they were simultaneously intelligent and stupid in the way they spoke in a way that I realised I'd never seen in genuine people, making me realise it was probably different people or bots under one account.
If I realised it 10 years ago it was probably happening for at least 15. He wasn't better than these people he was falling into their trap
- onion2k
One of the first things I tell the junior/mid-level developers I mentor is "You can't debug something just by reading the code." We all have a mental model of how our code works, and it's usually a bit wrong. Bugs are the real world manifestations of those mistakes. When you read the code it's all filtered through your model, and that makes you blind to seeing why something unexpected happened. In order to debug something you have to be able to put the system in the state where the bug happens to see why it occurred.
LLMs generally only debug systems by reading the code with whatever information you give them in a prompt. The image in the article is meta-prompt - the prompt is whatever comes from the vision model the AI happens to use to 'understand' the red circle annotation. That won't work. To successfully debug what's going on it will need much better state information. Has the 'shelf' been explained to is? Is the contrast and lack of shadows in the image messing up the vision model? Why isn't the 'lid' in the image? And so on.
LLMs are clever but they're not magical. Treat them like a naive junior dev. Give them enough data about the state of something to understand it properly.
- hspeiser
I completely understand this. I’ve worked on robot hands and 5.6/Fable 5 were practically useless at helping me debug anything visually.
What I have found super useful actually is having models make a interactive 3d viewer in which I use move / highlight / paint (soft body painting directly onto the geometry for issues and different colors mean different failures). This gives a much better way to communicate the physical relationships and positions that are hard to get across in a labeled screenshot.
Its for sure still a lot of manual work so the "seeing-eye dog" description definitely holds. But I have found that after a couple of examples with the extra context the model gets much better at handling the problem and becomes useful.
- beklein
A bit off topic, but I absolutely love the little robot on the author's main project's landing page (https://rerun.io/).
I normally condemn mouse hijacking, but this implementation will be allowed.
- Fr0styMatt88
I've found that LLMs are specifically bad at a certain kind of debugging, though I can't quite put my finger on what that is.
"Spot the bug in this code" when the code can be looked at and pattern-matched against bugginess is something they seem really good at.
Some parts of debugging, like "Here is this logfile, what do you think is going on?" are also surprisingly good.
It's that thing kind of in the middle -- I know it when I see it honestly is the best way I can put it into words. An example from recently, I'm receiving some bad data on a network message parser. Immediately I don't know whether it's a my-side or their-side thing, but I know if I try and just vaguely describe the behaviour to the LLM it will start churning tokens.
My current approach to problems like this is -- I need to tell the LLM what it needs to do to give itself the data it needs to solve the problem. My first reaction now isn't "It's not working, there's a bug, it's not doing X". It's "Okay, this isn't quite working properly; I need you to add some debug logging around X, Y and Z so we can figure this out". That tends to avoid spirals and get me out of the situation much more quickly.
The seeing eye dog analogy is pretty apt actually. I would love to see some transcripts from the author if they are able.
Edit to add: I think the 'thing' I'm alluding to might be -- if I have trouble expressing the buggy behaviour clearly in words, then I know it's probably going to be a fair few back-and-forths […]