Understanding x86-64 Assembly for Reverse Engineering and Exploits

This class helps you bootstrap into the areas of reverse engineering, vulnerability exploitation, operating system design, code optimization, and compiler design. It's extremely rare to see any security conference where assembly language isn't mentioned in someone's slides. If you don't known assembly, you're missing out on full understanding of what people are trying to tell you!

Once you've taken this class, it will open the door to all the other specialty areas that depend on assembly knowledge. And this is the first time this class is being offered focusing on 64 bit rather than 32 bit assembly! Although x86 has hundreds of special purpose instructions, students will be shown it is possible to read most programs by knowing only around 20-30 instructions and their variations.

25% of the time will be spent bootstrapping knowledge of fully OS-independent aspects of Intel architecture. 50% will be spent learning Windows tools and analysis of simple programs. The final 25% of time will be spent learning Linux tools for analysis. This distribution is partially due to Windows' dominance of the marketplace, but also because the tools on Windows are more user-friendly than those on Linux, allowing for a more gradual introduction for the student.

Day 1:

• x86 variable-length register sizes and mnemonics
  
• Windows/Linux 32/64 bit compiler calling conventions
  
• Understanding the x86 stack, and how it is used by programs for argument passing and local variable storage (and how it's essential to the most basic exploits: stack-based buffer overflows.)
  
• How to see mixed assembly-source view in Visual Studio
  
• How to perform debugging in Visual Studio
  
• Stepping through a small program and watching the changes to the stack at each instruction (push, pop, call, ret (return), mov)
  
• Stepping through a slightly more complicated program (adds lea(load effective address), add, sub)
  
• Understanding the correspondence between C and assembly control transfer mechanisms (e.g. goto in C == jmp in ams)
  
• Understanding conditional control flow and how loops are translated from C to asm(conditional jumps, jge(jump greater than or equal), jle(jump less than or equal), ja(jump above), cmp (compare), test, etc.)
  
• Boolean logic (and, or, xor, not)
  

Day 2:

• Logical and Arithmetic bit shift instructions and the cases where each would be used (shl (logical shift left), shr (logical shift right), sal (arithmetic shift left), sar(arithmetic shift right))
  
• Signed and unsigned multiplication and division
  
• Special one instruction loops and how C functions like memset or memcpy can be implemented in one instruction plus setup (rep stos (repeat store to string), rep mov (repeat mov)
  
• Port Input/Output instructions (in & out) which are necessary to understand to reverse engineering low level code such as BIOSes as seen in our Advanced x86: BIOS & SMM class
  
• Misc instructions like leave and nop (no operation)
  
• How x86 hardware and software breakpoints work through the use of dedicated debug registers or through interrupt instructions (int 3) _ with some background on x86 interrupt handling architecture (Interrupt Descriptor Table _ IDT) and how it is often taken advantage of by attackers
  
• Taking the examples previously run in the Visual Studio debugger on Windows and showing how they are compiled with Gnu C Compiler (GCC), and how debugging is performed in the Gnu Debugger (GDB) on Linux
  
• How x86's variable-length instructions lead to problems for disassemblers and opportunities for code obfuscation.
  
• Class capstone of reverse engineering the famous "binary bomb" lab from the Carnegie Mellon University computer architecture class. This requires the student to do basic reverse engineering to progress through the different phases of the bomb by giving the correct input, determined by analyzing the assembly, to avoid it "blowing up".
  

Learning Objectives:

• Show students with programming experience that assembly is not an arcane art, but rather an API which can be learned like any other.
  
• Allow student to gain exposure to a core set of Intel x86 architecture and assembly so as to be able to read and understand short programs in disassembled form.
  
• Provide exposure to a hands-on environment for both Windows and Linux.
  
• Teach the techniques for analyzing binary programs with both disassemblers and debuggers. As well as providing a deeper understanding of how both disassemblers and debuggers actually work.

Must have familiarity with basic programming constructs such as loops, if/else statements. The class will use C as the reference syntax/language for compilation, but there is very limited use of C-specific elements such as pointers (e.g. char ** argv).

Hardware:

• Students should bring a laptop capable of running a 64 bit version of Windows as specified below, with at least 4GB of RAM, so it can comfortably run the provided Linux virtual machine.
  

Software:

• Windows 7 or Windows 8 64 bit OS
  
• Microsoft Visual C++ Express 2012
  
• VMWare Player or VMWare Workstation (for running a provided Linux VM)

Windows software necessary for compiling, disassembling, and debugging. A VMWare Linux virtual machine with all necessary software installed. A soft copy of the slides.