Reverse Engineering Software with Assembly Language
WHY REVERSE ENGINEER?
So you have some program/software and you don’t know exactly what it does, maybe the code is proprietary thus you cant read the code and figure it out, you want to inspect a piece of malicious software/virus. Here is why you might want to do some RE:
-
To achieve Interoperability Make some system work with software or hardware you already have
-
To figure out how it works
-
Keygen/Cracks
-
Exploit Development
-
Proprietary File Formats
Assumptions
Before Proceeding It is important that the reader have some knowledge of the following concepts:
-
Endianness
-
Data structures
-
Hexadecimal notation
-
Intel Architecture
Assembly Language
A very low level programming language generated by compilers and translatable to machine code directly. Offers more control but less abstraction and would require alot of typing.
The goal is not to write programs in assembly code though i have written some basic bootloaders with assembly you can check out the project here but to be able to read and understand disassembled code from a piece of software you want to reverse engineer.
The Stack Data Structure
- LIFO data structure
push ebp
mov esp,ebp
sub esp,0x08
mov eax,45
mov ebx,43
add eax,ebx
call sym_add
ret
The Heap
Associated with Dynamic memory allocation
The BSS Section
Contains all Uninitialized variables
The Text Section
Contains the actual executable instructions( code )
The Registers
General Purpose
- EAX return values
- EBX Base register for memory access
- ECX Loop Counter
- EDX Data Register for I/O
Segment Registers
Usually are named with Two letter abbreviations
-
CS Stores code segment
- DS Stores Data segment
- ES,FS,GS Far addressing (video mem etc)
- SS Stack segment usually same as EDX
Indexes and Pointers
- EDI Destination index register for array ops
- ESI Source index register array ops
- EBP Base Pointer bottom of stack frame
- ESP Stack Pointer top of stack frame
- EIP Instruction Pointer to next instruction to be executed
The E prefix is for 32-bit, 16-bit and 8-bit are without the E prefix and finally for 64-bit the prefix is R instead of E however, forward compatibility is maintained
Flags Register
Holds 32 registers in total One bit values
- ZF Zero Flag Set to 1 if result of previous op is 0
- SF Sign Flag Set to 1 if result of previous op is negative -
Calling Conventions
CDECL
Arguments are passed on the stack in Right-to-Left order
Return Values are passed to EAX
The Calling Function cleans the stack
Allowing for Variadic functions as caller knows the number of arguments
STDCALL (AKA WINAPI)
Arguments are passed on the stack in Right-to-Left order
Return Values are passed to EAX
The Called Function cleans the stack
FASTCALL
The first 2 or 3 (32-bit or smaller ) arguments are passed directly in registers with the most commonly used registers being EDX, EAX, and ECX .
The Calling Function (usually) cleans the stack
THISCALL (C++)
Only Non-Static Member Functions. Also Non-Variadic
The Pointer to the class object is passed in ECX, and return value is passed to EAX.
The Called Function cleans the Stack
OPERAND TYPES
-
Immediates :03xf
-
Registers :EAX,.…ECX the values themselves
-
Memory adrresses [0x80542a], [eax]
-
Offset Types by bytes [eax + 0x4]
-
Sibs which are offsets by multiplication and addition [ eax * 4 + ecx ] , [eax * 2 + ecx]
OPS
-
mov Move destination, source reg,mem,immediate any combination
-
add,sub addition and subtraction
-
cmp compare subtract source from destination and assign a flag if ZF is 1 the destination and source are equivalent
-
test test does bitwise and of source and destination and assigns a flag to ZF or SF depending on the result
-
jcc/jmp conditional and regular jumps jz/jnz if ZF is zero or not jump ja/jae jump above and jump above equal jb/jbe/bjnb jump below and jump below equal
-
push/pop one operand and operate on stack
-
bitwise ops : and, or , xor, not
Recognizing Programming Constructs
Function Prologue and Epilogue
push ebp
mov ebp, esp
sub esp, N
...
mov esp,ebp
pop ebp
ret
About CALL and RET
Have an Implicit Operation
Call will push EIP onto the Stack while Return will pop the EIP pointer from the stack
LOOPS
ECX is usually loop counter conditional jumps based on loop counter easier to spot in call graphs
SWITCH STATEMENTS
jmp dwords endian formatted mem addressess with controlled offsets
Winding Up
Hopefully now you are familiar with some basic assembly language syntax and are now able to identify common programming structures like loops, switches, functions etc from some disassembled binary code.
In the future we will look at how to get this assembly code from compiled binaries as well as how to do some practical reverse engineering.
