Finding bugs by fuzzing your code

If you’ve ever worked on a large scale project, you know that finding and tracking bugs can be very difficult and lengthy.

Do you know if this can be automated? There are several ways to achieve this, starting with unit and integration tests to ensure regression detection is run regularly, end-to-end tests, and much more.

However, writing those tests is also a lengthy process, and you miss some hard to find bugs. We’re going to focus on fuzzing, an automated crash detection process.


What is Fuzzing?

According to Wikipedia, Fuzzing is an automated software testing technique that involves providing invalid, unexpected or random data as input to a computer program (https://en.wikipedia.org/wiki/Fuzzing).

Given a set of inputs your program can work with, a fuzzer will generate as much diverging data as possible and feed it into your program, recording each crash.

fuzzing in practice

If you don’t know me yet, I’m the developer of ArcScript, an easy-to-embedded scripting language, and I’ve worked on dozens of new functionalities over the past year. However, it can (and does have) introduce bugs, sometimes quite hard to find.

Fuzzing comes to the rescue here! I didn’t want (nor had the time) to write thousands of tests by hand, so I just wrote basic tests, checking that every good input leads to the expected output. What was missing was “bad input leads to bad output” type tests.

Introducing AFL++

AFL++ is an improved fork of AFL (American Fuzzy Loop), a fuzzer originally developed by Google.

Here’s how it works:

It’s very easy to use, you just have to recompile your project using afl-cc and/or afl-c++, give it an input corpus, and let it work for you until you’re satisfied. Give.

generate an input corpus

Since I wanted to fuzz a programming language, it was easy to put together my input corpus: code samples, parts of the standard library, some test files.

The process is as follows:

  1. Given a corpus, we want to generate a unique corpus to remove input from that corpus that does not generate a new path/coverage in the target
  2. Shrinking the Corpus: The smaller the input files traverse the same path within the target, the better the fuzzing will be.
# step 1)
afl-cmin -i fuzzing/corpus -o fuzzing/unique -- ${buildFolder}/arkscript @@ -L ./lib

mkdir -p fuzzing/input
cd fuzzing/unique

# step 2)
for i in *; do
  afl-tmin -i "$i" -o "../input/$i" -- ../../${buildFolder}/arkscript @@ -L ../../lib
done
enter fullscreen mode

exit fullscreen mode

you must have noticed -- ${buildFolder}/arkscript @@ -L ./lib bit: This is the command to run the input, with @@ Being the filename of the input generated by AFL++.

Then we can run Fuzzer as follows, and get the crash:

afl-fuzz -i fuzzing/input -o fuzzing/output -s 0 -- ${buildFolder}/arkscript @@ -L ./lib
enter fullscreen mode

exit fullscreen mode

-s 0 is here to set the rng seed to 0, to be able to reproduce rng based crashes more easily; will be stored under every accident fuzzing/output/,

And here we are, Fuzzer is running and finding bugs for us.
AFL++ is underway

crash analysis

Now comes the hard part, truncating the input to find the smallest input sample that still produces the bug. Often, this has to be done by hand, and it’s a tedious process, but finding those little things by hand takes longer, so it’s still a win-win!

AFL++ has tools to reduce crashes, to help you find bugs:

afl-tmin -i fuzzing/output/main/crashes/id... -o fuzzing/minimized_result -- ./build/arkscript @@ -L ./lib
enter fullscreen mode

exit fullscreen mode

Once you’ve got your smallest input possible for a given crash, and you’ve fixed it, it’s a good idea to put it somewhere to be able to run the next version of your program. Whether it is still fixed or not. It has personally helped me to start collecting bad inputs, to check in my tests whether they are handled correctly or not.


A few things to note about fuzzing:

  1. A lot of crashes can be very similar, when AFL++ Fuzzer finds a bug it will use it and get others to find it
  2. Because of 1) you may want to run multiple Fuzzers at the same time, this will get more bugs, plus it was designed to work this way (one master instance and multiple variants)
  3. You don’t need to limit the memory allocated to each fuzzer, but if you don’t you may end up exhausting all your RAM.
  4. A Fuzzer can run for a very long time and find nothing, it doesn’t mean that your program is bug free!

Because fuzzing can require a lot of time and resources, you might want to run those tests periodically, for example for each new release rather than on each commit or test added.

Leave a Comment