How We Found and Fixed a 5-Year-Old Go Runtime Bug on 32-Bit ARM
Hunting Down a Go Runtime Bug on 32-Bit Embedded Systems
A Go application on a 32-bit ARM embedded Linux system crashed sporadically with a fatal netpoll error. The root cause was a pointer aliasing bug in the Go runtime: on 32-bit little-endian systems, the raw pointer to the event fd aliased with the fdseq tag in the ev.Data field, causing socket fds to be mistaken for the event fd after millions of pollDesc recycles. The bug, introduced in Go 1.14, went unnoticed until 2025. The authors created a reproducer and proposed a fix that stores a tagged nil pollDesc instead, which was merged into Go.
We can only speculate, but this suggests that Google itself no longer runs any 32-bit Go programs. Otherwise they would have hit the bug themselves long before we did.
- ReactiveJelly
So a stronger type system in the boundary between Go and Linux would have caught this?
- dlopes7
Very interesting read, these bugs that only happen after a lot of time in production are the worse to troubleshoot
- BlackRabbit1
Not directly related to your embedded Go bug, but maybe helpful:
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.