Practical considerations implementing fuzzing acceleration via checkpointing

Hi all,
After giving my presentation in HITB I was fortunate to talk to some of the people that attended it and also some that contacted me via email afterwards. One of the most frequent questions I got was "how practical is it to implement a fuzzing acceleration framework?". Following up on that I decided to write a short post on the practical considerations of implementing an accelerated fuzzing framework by using DBI.
Some of the material in this post is based on one such Q&A discussion that I had with Peter Van Eeckhoutte (@corelanc0d3r) and published with his gracious permission.

Introduction to fuzzing acceleration via checkpointing

This section is aiming to provide a quick introduction to the subject. If you're familiar you can jump to the next section - practical implementation considerations.
If you're unfamiliar the concept of fuzzing acceleration via checkpointing and restoration the following slide from my HITB Amsterdam presentation provides an overview:

Post Vegas Post - BlackHat / DEF CON Workshop Materials, Notes and Some Mess

Hi All,
I've finally solved some internet access problems and can upload the materials I wrote for my BH and DEF CON workshops. For those in a hurry the materials are here: [presentation] [code examples]. If you want to catch me giving this workshop (in a somewhat modified version) come hear me in hack.lu.
BH & DC this year were great - great parties (except MS that initially didn't want to let me in until after the party started and I made other plans :P), some good talks and (re)meeting great people.
Thanks again to everyone that came to listen to me.

Some notes regarding these materials:

• Please note that the presentation was not written with offline reading in mind so it might make a hard read in some places. Questions are welcome in the comments.
• The Defcon presentation is mostly a variant of the BlackHat one (or vice versa) which is why I did not include both.
• The examples might be included in the official Pin distribution some day. For now I'm going to start a page on this website to preserve these materials. If you have any submissions of updates, fixes or additional examples you want included send those to me. notifications of updates will be published via  twitter.
• Since I was lazy and intended to control the length of my presentations by deciding what to include in the demos on the fly the details of the demos / examples mostly do not appear in the presentation itself. That is why I've included notes about the demos and examples below. Please note the examples were written with educational purposes in mind and not
• The end of the presentation contains a ton of references to all sorts of DBI usages for security or whitepapers on the subject - highly recommended read for those interested in the field.

Examples and demos:

• Exploiter1 - An example of automated exploit development for classic stack buffer overflows and assuming no DEP and ASLR. If you had it 5-7 years ago you could write a worm that would actually "multiply" and learn new attack vectors as it goes along (muhahaha :D). I'm intentionally releasing this degenerate version so as not to provide people that couldn't write this code alone with the ability. If you're interested in full capabilities along these lines - contact me privately, I'm considering it. I might write a longer blog post on this subject (if I ever get time to write).
• InMemFuzz - An example of in memory fuzzing and of using checkpointing to accelerate fuzzing. please note the checkpointing part only covers registers. One of the exercises in the Defcon workshop was to complete it. Again, a blog post on this will be added to the fuzzing series if / when I have time.
• RetAddrProtect - An implementation of "shadow stack" method to defend against return address overwrites in your program. I originally invented this method for some job back in early 2004 unknowing that the term "shadow stack" existed I called it "protection stack" and only very recently discovered the proper name for it which is why, since I'm lazy, the tool still calls it "protection stack". See the work from Caro'11 by MS guys on the matter (link in the presentation).
• Taint 1-3 - Examples of basic taint analyzers. These are simple examples that only handle the propagation of taint through the MOV class of instructions. The first outputs some of the dataflow, the second improves the log to show only taint flow and the third adds tracking of the taint source.
• Taint_vis_2D - is a script that is meant to show how a very simple (practically dumb) visualization of taint is still very useful to us compared to reading log files. Read the source to understand how it works.
• Program data-flow visualization - Two examples of memory visualizations are included in the kit. Those are meant for working with a recording of all memory accesses generated by the "pinatrace" (memory access trace) example from the official Pin kit. When executed on the log file generated by pinatrace these tools generate a 3D view of the program memory accesses that allows us to analyze the program visually. For example, it is fairly easy to identify loops and extrapolate potential usages of those loops. To understand the information generated: one axis represents the address, the second represents the PC location at the time of the memory access and the third axis represents the order of the operations. Blue means a read, red means a write. You can navigate through the visualizations using the mouse (zoom in/out and rotate). Again, if you're interested in advanced taint / data-flow visualization capabilities I'm open to discussion privately. See example images below.
• Anti Debug - the first anti debug example is meant to show that Pin will not trip this anti-debug protection while using a debugger would. The second is meant to show that the transparent debug feature in Pin allows you to debug ignoring Anti-Debug techniques. Please note that the first example will trip the debugger under the transparent debugger - if you want transparent debugging to work on it you need to write a pintool (which was meant as education / exercise).
• Double Free - An example of how to detect potential double free cases. Some might say that it is less relevant due to the fact that is a fairly easy task using static analysis and modern libc protections however, sometimes you don't have the luxury of having access to the source code or you work on legacy systems where it becomes relevant. In addition, vulns like the OpenSSH double-free race condition vulnerability would not normally be caught by a static analyzer looking for double free.
• Malloc Fault injection - this simple PinTool does exactly what the name suggests. It injects malloc failures into the program at a random interval. For our case I set the frequency to be very high (every ~100 calls to malloc) it is interesting to note that the Linux loader and libc initializations usually fail intelligbly with a reasonable message. 
• VulnEx# - these are vulnerable code examples to run the tools on - each example purpose is documented.

Pretty Pictures:
Here are some pictures of visualizations with a small explanation on how to read those. these are based on the very early work of the OS loader in Linux while starting to execute "ls". If you're looking for why I choose this specifically the answer is "because" or in detail - because it fit me at that specific moment (yeah, I'm lazy).

Notice the series of memory read-writes a little up and to the right from the center - this is a loop, almost certainly with a counter. since no other memory writes are visible most probably this is a search loop or a loop related to register calculation

Same loop as before, now a little to the left from the center and depicted using separate lines for reads and writes. Note that it is now almost certain that the top red line represents some sort of counter.

On the center and a little below and to the right notice another loop - note the symmetry in the reads and writes relative positions. figuring out what it probably is is left as an exercise to the reader :P (you can also see this same loop from the top in the first picture to the left of the center)

~1000 memory accesses, note that blocks with relations are easily identifiable. Easy to see that a filtering capability is very valuable. finding more interesting patterns will require some more staring.

BTW, the ZIP files on google docs are downloadable - through "file" -> save original

-- The End (for today)
Please leave a comment and share if you liked it.

Posts or possibly talks or whitepapers in planning / phases of writing, though I don't know when I'll get to each particular item: "how I would go about blowing up batteries if I was 0xcharlie" (pending legal review - meaning it is too close to my work research and besides if batteries start blowing up there could be trouble...), "Fuzzing series part 2 - monitoring OR why I'm tired of hearing about !exploitable", "fuzzing series part 3 - fuzzing protocols from the RFCs using semi-context aware grammars", "how to attack HW using SW" (pending legal review - again close to work stuff), "Enough with the code coverage! - feedback driven fuzzing based on data coverage and taint analysis", "old/new heap attacks", "increasing exploit reliability". Every post or potential talk depends popular demand, free time and me not getting sued.

-- Gal

Binary Instrumentation for Hackers

I've been planning to write something about the "Binary Instrumentation for Hackers" presentation I gave at the Israeli Defcon user group (DC9723) on 26 April right after the presentation but I kinda got sidetracked (which I will try to avoid from now on). At first I waited for the video to be online but right now I don't know if and when it will be online and later I got too busy with some work . Anyways, my slides are available HERE and if the video will be uploaded it will be HERE (the slides are English but the presentation was given in Hebrew). I'd like to thank DC9723 organizers for having me, I hope to come back again someday.

Also, the slides from my lightning talk at Hackito Ergo Sum 2011 are up with the other conference slides HERE.

I have (today) sent submissions of workshops and talks on the subject of binary instrumentation to BlackHat USA & Defcon 19 (check out the Defcon site, it just opened), hopefully I'll get a chance to talk about some awesome stuff with the great people in attending these cons.

Oh, and if anyone happens to be in Amsterdam next week (15-21) for HITB, I'm there on business. I won't attend HITB but would love to meet anybody that actually reads my blog for a beer in the evening.

While we're at it, If you are in HITB don't miss Itzik's & Zuk's presentations - I know they have some kwel stuff planned. I also recommend going to see whatever Travis Goodspeed & Claudio Crisocone present - I haven't talked with them about the details of their presentations but they usually do great stuff. 

(yeah, in case it was unclear I'm disappointed to be in  Amsterdam at the same time HITB happens and not attend HITB).

-- Gal

Hackito Ergo Sum 2011 - Day 3 (and last)

Day 3 was another great day: we had a couple of cybercrime talks, Ruby on rails, DNIe , USB autorun for Linux and the solution to the conference crackme. I want to take a chance to thank the organizers, I really enjoyed this conference, the people were great and the talks interesting and I hope to be back next year, maybe even to present a talk.

As usual I welcome comments, either here or directly to me - feel free to let me know anything constructive.

Raoul Chiesa / Keynote

Raoul's keynote focused on cybercrime and UNICRI and their hacker profiling project (HPP). Cool quote - "you got information, you got power". The talk was cool and included parts focusing on who's behind cybercrime that I won't recount here out of respect to the fact he asked not to record anything during these parts of the talk. He shared his views on why cybercrime works:

• New user every day = new fools every day
• Making money  (fits to the economical crisis)
• Technical know how is easy to get
• Back in the day we worked hard to get something
• Easy to recruit idiots ("mules")
• Psychological - "they will never find me/bust me"
• Psychological - Lack of violent actions

And how HPP classifies hackers:

• Amateur
• Wanna be lamer (9-16)
• Script kiddie (10-18)
• Hobbyist
• Cracker (17-30)
• Ethical hacker (15-50)
• Quiet, paranoid, skilled hacker (16-40)
• Pros
• Cyber warrior (18-50)
• Industrial spy (22-45)
• Gov agent (25-45)
• Military hacker (25-45)

He had a lot of interesting stuff to say and the talk actually went overtime. I can't possibly begin to recount even the public parts of the talk in a short summary but I highly advise you to listen to him if you get a chance.

Joernchen of Phenolit / Ruby on Rails form a code auditor's perspective

This talk focused on how to audit Ruby on Rails (RoR) code. Key points:

• 3 layers called MVC - Models, Views, Controllers - review them
• Start from the controllers
• Look at the database
• Look at the filters
• RoR has some fancy magic that can go wrong - for example: mass assignments (CCC.de had a vuln because of using this)

Nice talk and definitely someone you should talk to if you audit RoR, personally this isn't in my focus areas.

Gabriel Gonzalez Garcia / Man in the Remote: PKCS 11 for fun and profit

This talk focused on ways to defeat the DNIe which is the Spanish government system of digital IDs used for authentication and non-repudiation certificates. Basically he implemented an attack against the untrusted terminal problem that the PC is posing between the DNIe device and what it authenticates to. There are two ways to use DNIe for web auth: Java applet or SSL + client certificate. The device itself has an EAL 4+ certification and defends against most hardware tampering attacks. What he found were two practical attacks on the terminal:

1. Write a fake interface library that acts as an RPC server connecting his machine with a remote DNIe
2. Write a fake java applet that acts as the proxy and runs on the client machine (easier)

He proposed a solution that I personally disagree with to use "distance bounding" == measure response time and disallow slow responses.

Jon Larimer / USB autorun attacks against Linux

This talk focused on how to implement autorun attacks against Linux and showed one successful attack. To be exact it is successful against GNOME. So first of all the specs of freedesktop.org forbid systems from running code without asking the user. That being said a lot of processing is done when a new storage device is connected: file system drivers execute, file browsers read the contents and thumbnailers create thumbnails. These attacks are basically physical access which usually means "game over" because you can use DMA attacks using 1394 vulns and other attacks like cold boot attacks. However, assuming the latter is complicated for you and you are on a system where protections for against 1394 DMA attacks were implemented you need another way. From here on Jon described his processes of research which I will skip directly to the end - he decided to focus on Thumbnailers used by Nautilus (GNOME file manager). He found a vulnerability in the Evince thumbnailer for DVI files (among others). Luckily (or unfortunately, depending on you view) Evince uses AppArmor and compiles as PIE, in addition the kernel has ASLR enabled (but 32bit system). He worked on overcoming this:

• ASLR and PIE can be defeated by brute forcing -
• On a 32bit system there are only about 3000 addresses that the Linux kernel can load libc to.
• Using this we can just generate 3000 files - 1 for each address
• Loading will be slow (about 10 min), but success is almost sure
• Interesting result he found were that in his statistics around 10% of the addresses were used significantly more than others
• So you can create only about 300 files and have high chance of success
• He says he didn't research why is this. This is something that someone should really pay attention to.
• AppArmor can be worked around
• Doesn't protect against X11 library calls

He showed a demo of killing the lockscreen using a USB stick. Jon Oberhide added an interesting comment that there is a rather new layer for partition parsing in Linux that had a lot of bugs recently.

Yuval Vadim Polevoy / Money is in the eye of the beholder: new and exciting ways to steal you money

This talk again focused on Cybercrime, Yuval is from RSA research lab and really interesting to talk to. He focused on how cyber-criminals make money and the underground economy. I will recount some of the most interesting points, I'm not sure I can do justice to the details but that is the cost of summarizing...

• A cybercrime operation requires:
• Bots
• Campaign
• Drop point
• Bot plugins
• Hiring & managing mules
• Establishing covert channels
• Maintain fast-flux (optional)
• The skill set required for that is:
• Low level programmer
• Oday researcher
• Spammer
• Hosting owner
• JS programmer
• HR recruiter
• E-commerce export
• IT specialist

These are too many skills for most people, so the common solution nowadays is outsourcing parts of this effort to others.

For the end, he focused a bit on the future techniques that malware authors will use, in particular screen grabbing. Yuval showed a demo of two techniques that can be used for screen grabbing in Windows, thus defeating on screen keyboard and allowing other stuff like cheating in online poker…

Eloi Vanderbéken / Hackito Ergo Sum crackme

For the end Eloi talked about the crackme. I won't go into all the details but here are the highlights:

• Verification alg - based on modified RC4
• Obfuscation
• Inst. Mutation
• Control flow graph obfuscation
• Encryption layers
• Direct native API call
• Using sysenter (also uses a lot of random invalid syscalls) which is why the crackme only worked on 32bit systems
• Anti-X
• Anti debugger
• Anti-instrumentation
• This was especially interesting to me, in particular because Eloi used Pin as the example for instrumentation engine
• Detect hook on KiUserExceptionDispatcher nad KiUserCallbackDispatcher
• Detect stack reuse by instrumentation code: place a constant in esp-4, execute syscall, check constant
• Anti data-tainting

This is definitely a respectable set of protections. Only 3 solutions were submitted. I might try to write a PinTool that will be able to handle all these protections and post it here if I can.

Well, that's it for Hackito this year - I hope you will read my future posts.

Hackito Ergo Sum 2011 - Second half of the first day

Short summaries of the talks in the second half of the first day:


Itzik Kotler / Let Me stuxnet you
Although the name of the talk has Stuxnet in it it does not another one of these boring Stuxnet talks but focuses on Permanent DOS (PDOS) of your hardware using all sorts of methods. This is a research that is still WIP.
Several key concepts:

• SW can harm HW by making it perform harmful operations
• SW can harm FW and cause it to do harmful operations
• SW can take advantage of one piece of HW to harm another HW

PDOS can be achieved in many manners:

• Phlashing - overwriting FW on Flash to brick parts
• over-clocking
• over-volting
• over-using - esp. in mechanical parts
• power cycling - uses temperature flection

Can attack CPUs, GPUs, RAM, HDDs, SSDs, Flash, NICs, CRTs, Floppys and more...
Interesting focus on examples how to implement these attacks so the user doesn't realize it's done until his system dies.

Marc Heuse / Recent advanced in IPv6 insecurities
This is Van Hauser from THC. First part of the talk focused on IPv6 basics. He then covered some attacks against IPv6 he released since 2005 and then went on to talk about new attacks - I'll try to cover everything real quick. In particular I liked the quote: "if you start fuzzing things will blow up".
First it is important to mention the THC IPv6 attack tool he is developing, many parts of which are not yet released BUT if you join the dev team he is willing to share (and he is looking for help)... He also released a new version of THC Hydra during his talk.
So, a real quick overview of IPv6 attacks he mentioned (only the headlines or I won't finish this post):

• ARP Spoofing => ND spoofing
• Duplicate address detection DOS
• Many ways to MITM using redirects
• Using Autoconfig (like DHCP but built in IPv6) for pretending to be a router
• Replacing the default router using lifetime=0
• Most Linux distros (and in general) don't filter most IPv6, especially weird headers
• Can announce a remote network as local to spoof it
• RA flooding - can DOS cisco, Netscreen, all Windows, FreeBSD, some Linux
• you can detect systems sniffing on your local network
• taking over as MLD
• side channels in IPv6
• techniques to identify remote systems (can't scan 16 octets!)

Tarjei Mandt / Kernel pool exploitation on Win7
This talk was too technical to go into details in a brief summary - see the slides for details.
The short version is as following: Win7 has protections in the kernel pool - these are not good enough. several tehcniques were invented by Tarjei to bypass these and two exploits for CVEs were shown.
CVE-2010-3939 (MS10-098) - Used Quota process pointer to exploit
CVE-2010-1893 (MS10-058) - Used Pool index overwrite to exploit
If you like kernel exploits on Windows you really should check out Tarjei's slides and blog.


Sebastien Tricaud / Capture me if you can!
Sebastien discussed handling incidents in large networks (country level). He talked about capture methods like: PCAP, netfilter queue, DAQ and others. And on data processing techniques - how to find meaningful information in logs. He made a very convincing point on log "0day" - due to not collecting or collecting information incorrectly you can lose important information,
In my view, the most interesting news in the talk were:

• Using visualization to handle large data sets (not a new idea, I admit):
• Several interesting projects:
• Secviz - http://www.secviz.org
• Circos - http://mkweb.bcgsc.ca/circos/software/download/circos/
• Using GPGPU for accelerating log processing:
• NetGPU - http://Code.google.com/NetGPU

Well, I hope you enjoyed; Please leave a comment. I will try to keep up in the next two days...