Simply Unstable - Reversing Writeup

$ file challenge

Starting with basic binary analysis:

Binary Information:
ELF 64-bit LSB executable, x86-64, not stripped
The binary is not stripped, making analysis easier with function names intact

$ ghidra challenge

Opening the binary in Ghidra for static analysis. Let's examine the main function:

undefined8 main(int param_1,undefined8 *param_2,undefined8 param_3,uchar *param_4,size_t param_5) {
  undefined8 uVar1;
  ulong uVar2;
  size_t *outlen;
  uchar *out;

  if (param_1 < 2) {
    printf("Usage: %s hex_key",*param_2);
    uVar1 = 1;
  } else {
    out = (uchar *)0x0;
    uVar2 = strtol((char *)param_2[1],(char **)0x0,0x10);
    decrypt((EVP_PKEY_CTX *)(uVar2 & 0xffffffff),out,outlen,param_4,param_5);

    if (flag_ciphertext == -0x1a76b7ab) {
      (*(code *)&flag_ciphertext)();
      putchar(10);
    } else {
      puts("Invalid key");
    }
    uVar1 = 0;
  }
  return uVar1;
}

Key Observations:

The program expects a hex key as command line argument
The key is parsed with strtol() in base 16
A decrypt() function is called with the key
If flag_ciphertext == -0x1a76b7ab, it executes the flag as shellcode

$ analyze decrypt_function

Now let's examine the decrypt function:

int decrypt(EVP_PKEY_CTX *ctx,uchar *out,size_t *outlen,uchar *in,size_t inlen) {
  int iVar1;
  long lVar2;
  uint local_28;

  lVar2 = sysconf(0x1e);
  iVar1 = mprotect((void *)(-lVar2 & 0x104020U),(long)(0x104020 - (int)(void *)(-lVar2 & 0x104020U)),7);
  
  if (iVar1 < 0) {
    perror("mprotect:");
  }
  
  for (local_28 = 0; local_28 < 0x18; local_28 = local_28 + 1) {
    (&flag_ciphertext)[(int)local_28] = (&flag_ciphertext)[(int)local_28] ^ (uint)ctx;
  }
  
  return local_28;
}

Function Analysis:

mprotect() makes memory executable (RWX permissions)
XOR operation on 24 bytes (0x18) of flag_ciphertext
The XOR key is the user-provided hex value

$ gdb --args challenge test

Using GDB to find the memory addresses and extract the encrypted flag:

pwndbg> info files
Symbols from "/home/kali/Documents/studsec/simply_unstable/Handout/challenge".
Local exec file:
    `/home/kali/Documents/studsec/simply_unstable/Handout/challenge', file type elf64-x86-64.
    . . .
    0x0000555555558020 - 0x0000555555558082 is .secret

Now let's examine the encrypted flag data:

pwndbg> x/24xb 0x0000555555558020
0x555555558020 <flag_ciphertext>:    0x81    0x8c    0x3c    0x41    0x9c    0x49    0xe0    0x44
0x555555558028 <flag_ciphertext+8>:  0x9c    0x4d    0x62    0xec    0x6d    0x85    0xf4    0xe5
0x555555558030 <flag_ciphertext+16>: 0x95    0x85    0xf4    0xe5    0x95    0x8c    0x0d    0xa6

$ python3 solve.py

Now we need to calculate the XOR key. We know:

Target condition: flag_ciphertext == -0x1a76b7ab
Converting: -0x1a76b7ab = 0xe5894855

Initial value: 0x413c8c81 (first 4 bytes, little-endian)

Calculate XOR key: initial_value ⊕ target_value

initial_value = 0x413c8c81
target_value = 0xe5894855
X = initial_value ^ target_value
print(f"XOR key should be: {hex(X)}")

# Result: 0xa4b5c4d4

$ ./challenge 0xa4b5c4d4

Testing our calculated XOR key:

./challenge 0xa4b5c4d4

$ echo "Analysis Complete"

The challenge was solved through a systematic approach:

Key Techniques Used

Static Analysis: Used Ghidra to reverse engineer the binary logic
Dynamic Analysis: Used GDB to extract runtime memory values
Cryptography: Applied XOR operations to decrypt the flag
Memory Analysis: Understanding little-endian byte ordering
Shellcode Execution: Recognized self-modifying code patterns

Exploitation Flow:
1. Binary accepts hex key → 2. XORs encrypted flag with key → 3. Checks result against magic value → 4. Executes decrypted shellcode → 5. Prints flag

This challenge demonstrates the importance of understanding both static code analysis and dynamic memory manipulation in reverse engineering tasks.