Fuzzing the Gleam Compiler: Finding Bugs by Generating Random Programs
Gleam is a functional language that compiles to both Erlang and JavaScript, making it an ideal candidate for structure-aware fuzzing. The author built a fuzzer that generates random Gleam programs to uncover compiler bugs, particularly those causing differences between the two backends. The project uses a generation-based approach, leveraging the compiler's public API to test the parser and type checker. It found a regression in the nightly build involving a pipeline operator in a const expression. The author plans to extend the fuzzer to generate type-safe programs for deeper testing.
One is definitely more than zero.
- nwellnhof
> We can compare the output of the same program for both targets and flag any differences.
This is called differential fuzzing and is one of the most powerful methods to find bugs in all kinds of software.
- goranmoomin
I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
- stephenlf
Fantastic article. Thanks for sharing. I love the honest take on LLM-based fuzzing. It’s exactly something I would do with a similar problem.