I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
I love good programming languages, if that counts for anything, and I think there's a huge amount of room and value in "using an LLM to translate one programming language to another".
I also think your instincts are on a useful path. Perhaps using a very small and stupid LLM to generate plausible-sounding-but-probably-wrong programs might be a path to generating interesting test cases?
Any fuzzer needs to compare its results to AFL (American Fuzzy Lop), "a free software fuzzer that employs genetic algorithms in order to efficiently increase code coverage of the test cases." https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software)
At one point it was considered state-of-the-art. As a project it's since been superseded by AFL++ - https://aflplus.plus/ .
I'm sorry, but I can't understand how AFL++ could be applied to the Gleam case.
The main presentation of the tool lacks any description of its coverage, but from deep inside the documentation[^1] I gather that AFL++ is only relevant for GCC/LLVM languages. Gleam is not one of those.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
[0]: https://github.com/dtcxzyw/llvm-hackme
reply