Monday, June 11, 2012

Practical considerations implementing fuzzing acceleration via checkpointing

Hi all,
After giving my presentation in HITB I was fortunate to talk to some of the people that attended it and also some that contacted me via email afterwards. One of the most frequent questions I got was "how practical is it to implement a fuzzing acceleration framework?". Following up on that I decided to write a short post on the practical considerations of implementing an accelerated fuzzing framework by using DBI.
Some of the material in this post is based on one such Q&A discussion that I had with Peter Van Eeckhoutte (@corelanc0d3r) and published with his gracious permission.

Introduction to fuzzing acceleration via checkpointing

This section is aiming to provide a quick introduction to the subject. If you're familiar you can jump to the next section - practical implementation considerations.
If you're unfamiliar the concept of fuzzing acceleration via checkpointing and restoration the following slide from my HITB Amsterdam presentation provides an overview:

You can also find a very simple example I wrote here. This example does not restore the memory but only the CPU context and is meant to be run on this specific program. However, it still shows the very basics of creating a checkpoint and restoring it using PIN. Pay attention to the use of PIN_ExecuteAt and PIN_SaveContext for the creation and then restoration of the checkpoint. Note that these two APIs only save the CPU context and not any other part of the program context.

Practical implementation considerations

Well, that's it for the intro. Time to get down to business. The following notes are meant to highlight potential performance savings and pitfalls. The scenario discussed here is a program that reads some input file and we want to fuzz this file input.

What memory to save / restore?
Normally, you don't know exactly what memory areas will require saving and restoring. If you do - lucky you. Implementing a save and restore for a known memory area or data structure is not something I need to explain. If not you still have several choices:
  1. Collect the information on the relevant memory areas via memory access tracing. You can find an example tool that does memory tracing included in the PIN kit. After you run several representative workloads you should have a pretty good understanding of what memory areas you need to save and restore.
  2. Use the memory tracing approach in real-time (slower but more accurate). i.e. save all the addresses modified between the save point and the restore point and just restore those to their original values.
  3. Just save and restore all writable program memory - this is not recommended for programs using more than a few MB of memory.

Please note that there will usually be some memory areas that you can ignore restoring because those will be overwritten by the next file you load.

A final note on saving and restoring - due to performance considerations you should strive to save a copy in RAM rather than saving anything to disk.

Target file
A common pitfall many tend to use is to actually rename a fuzz file on disk to the name that the program was instructed to load. That is not advised. Disk access is far more costly in performance compared to RAM access. Therefore I recommend changing the file name string held in memory by the program you're fuzzing. There are two main options:
  1. Modifying the string in place - the problem with this approach is that due to memory constraints you'll have to stay with file names of the same length or shorter
  2. Modifying the pointer to the string - changing that pointer to point to a string allocated from your pintool.

System Resources and the choice of Restore Point
The consumption of system resources is one of the biggest problems you'll face when fuzzing like this. That depends greatly on your choice of where to locate your restore point. For example say you are fuzzing MS-Word. A restore point chosen after you close the file should not require any freeing of resources to be done by you (that is because MS-Word is supposed to handle this automatically). On the other hand, if you choose to restore immediately after completing the load of the file because the actual unload is not of interest for your fuzzing project then the responsibility to free the system resources falls on you.

Some types resources you might want to monitor and free before going back to a checkpoint:
  1. memory allocations
  2. file handles
  3. any other handle / system resource claimed
It is fairly easy to just hook the allocation functions for those resources and save the details then free those once before we restore to our checkpoint.

I highly recommend considering your restore point carefully and considering the option that you missed freeing some resource when debugging the first couple of crashes you get from this.

Random notes
  1. Bug chaining is a high possibility using this technique so verify several test cases back if you get a result which doesn’t repeat without PIN. 
  2. It might be that in some cases you need to save less than all the context (my example saves all context). It is possible to do so to save overhead and then just modify IP using the proper PIN API.
  3. The first test cases will appear slow, that is to be expected and because of JITing. If you stop the fuzzing and restart it you have to pay the JITing overhead again.
  4. If you see major slowness post the early test cases consider modifying the PIN code cache size parameters (look in the command line)
  5. Don’t forget to log things like the current fuzz file name to somewhere outside the process you're fuzzing. If you use a log file make sure to flush it. While it seems trivial it is very easy to miss a test case like this.

Well, that's it for today. If you have practical questions or comments feel free to leave those.

I hope you enjoyed this post,


  1. Hey, what's the difference between your fuzzer and an in-memory fuzzer?

  2. good question :-)

    It is true you could classify this as a "in-memory fuzzer" (depending on your definition). I intentionally avoided the term "in-memory fuzzing" because I find the way it is used ambiguous. there are two main questions about the way this is used:

    * the question of what you're fuzzing: a single function, a small sequence of functions or a whole functional piece of the program

    * the question of what you're restoring: CPU state, memory contents or even system resources

    The in-memory fuzzers I saw so far usually belong to the simplest answers in these categories. there is one more distinction you can add, whether your fuzzer "honors" the program filters or not.

    That being said, regardless of how you call it, my main purpose here is to provide notes on the pitfalls you'll face in implementing such a fuzzer.


    p.s - BTW, if you look at the file name in my example it is indeed called InMemoryFuzz. I chose the name because it works on single function and restores only the CPU state so there is no question this is a classic case of an in-memory fuzzer

  3. With regards to the "what memory to save/restore" question, is there any value to making use of Copy-on-Write OS capabilities when feasible? Or would forking cost more than its worth?