Automating Reverse Engineering: How CutterMCP+ Leverages LLMs to Crack CTF Challenges and Malware Analysis

“

Giving AI a sharper disassembler: The free reverse engineering tool that’s automating complex analysis tasks

The Reverse Engineering Revolution

Reverse engineering has traditionally been a painstaking manual process. Security researchers would spend hours staring at assembly code, tracing function calls, and deciphering obfuscated logic. But what happens when we combine cutting-edge large language models (LLMs) with powerful reverse engineering tools?

CutterMCP+ represents this exact fusion – integrating the free, open-source Cutter reverse engineering platform with modern AI capabilities. This innovative plugin enables automated analysis of:

• CTF challenge binaries
• Custom VM implementations
• Real-world malware samples
• Obfuscated shellcode loaders

The results? AI models can now automatically solve HackTheBox challenges, analyze VirusTotal 0-detection malware, and reconstruct VM instruction sets – all without human intervention.

Real-World Testing: Pushing AI to Its Limits

Test 1: Anti-Analysis Bypass (HTB: Behind the Scenes)

Challenge: Simple reverse engineering CTF with illegal ud2 instructions that intentionally crash decompilers
AI Approach:

1. Detect decompiler failure points
2. Switch to raw assembly analysis
3. Identify key comparison/jump logic

Model Results:

Claude-Sonnet-3.7/4: ✅ Solved in ~60 seconds
Gemini-2.5-Pro:       ✅ Solved successfully
GPT-O4-Mini:          ✅ Correct solution
GPT-4.1:              ❌ Failed (incorrect assumptions)
Gemini-2.5-Flash:     ❌ Failed (misidentified key instructions)

“

“The entire process took about a minute to find the correct answer without human intervention.” – Project Developer

Test 2: VM Analysis (HTB: Virtually Mad)

Challenge: Custom virtual machine with:

• Proprietary instruction set
• Pointer-based function calls (not detected by standard tools)
• Input validation filters

Analysis Process:

1. Identify VM entry points
2. Reconstruct opcode handlers
3. Map instruction patterns
4. Decode execution flow

Model Performance:

Claude-Opus-4: ✅ Independently solved
Claude-Sonnet-4: ❌ Failed to reconstruct VM
Gemini-2.5-Pro: ❌ Incomplete analysis
GPT-O4-Mini: ❌ Couldn't handle complexity

Test 3: Real Malware Analysis (ShellcodeEncrypt2DLL)

Sample: Sophisticated shellcode loader (VirusTotal 0/72 detection)
Task:

• Analyze core DLL functions
• Identify obfuscation techniques
• Determine payload delivery mechanism

Results Comparison:

“

“The entire process took a few minutes and required no human intervention.” – Project Developer

Technical Implementation: How CutterMCP+ Works

Core API Functions

CutterMCP+ exposes these key operations to AI models:

Installation Walkthrough

Step 1: Install Dependencies

pip install -r requirements.txt

Step 2: Configure Cutter Plugin

1. Launch Cutter
2. Navigate to Edit → Preferences → Plugins
3. Copy mcp_plugin.py to <cutter_plugins>/python
4. Restart Cutter

Step 3: MCP Host Configuration

{
  "mcpServers": {
    "cuttermcp-plus": {
      "command": "python",
      "args": ["/absolute/path/to/mcp_server.py"]
    }
  }
}

Step 4: Model Selection Guide

Critical Safety Considerations

Security Risks

1. Malicious String Injection: Data section strings may contain executable commands
2. Unintended Actions: Automatic function renaming/modification
3. Token Exploitation: High-cost operations without safeguards

Protection Measures

# Sample safety protocol
def execute_command(user_input):
    if contains_malicious_patterns(user_input):
        require_human_approval()
    elif high_token_cost_operation(user_input):
        notify_user_before_execution()
    else:
        execute_immediately()

Cost Management Strategies

• Set token budgets per analysis session
• Use disasm_text() instead of decompile() for large functions
• Limit analysis scope with address ranges
• Enable interactive confirmation for expensive operations

Technical Deep Dive: How AI Understands Assembly

Overcoming Decompiler Failures

When encountering anti-analysis techniques like ud2 instructions:

1. CutterMCP+ detects decompilation failure
2. Switches to disasm_by_func_text() for raw assembly
3. LLM parses instructions with contextual awareness:

   ; ud2 at 0x401050 blocks decompilation
   mov eax, [ebp-0xc]
   cmp eax, 0xdeadbeef
   jz 0x401072 ; Correct branch
   

Handling Advanced Obfuscation

For function pointer-based execution (as in VM challenges):

1. Identify global function pointer tables
2. Trace cross-references with xrefs_to()
3. Reconstruct dispatch logic:

   void (*handlers[256])(void);
   handlers[opcode](); // Indirect call
   

Malware Analysis Workflow

1. Detect suspicious imports (VirtualAlloc, CreateThread)
2. Identify XOR decryption loops
3. Track shellcode writing patterns
4. Reconstruct execution flow:

   graph LR
   A[Allocate Memory] --> B[Decrypt Payload]
   B --> C[Create Thread]
   C --> D[Execute Shellcode]
   

Practical Applications Beyond CTFs

Malware Research Acceleration

• Automatically label 1,000+ samples by behavior
• Identify novel obfuscation techniques
• Generate YARA rules from analysis patterns

Vulnerability Discovery

• Detect insecure function usage (strcpy, sprintf)
• Identify unprotected memory operations
• Flag dangerous permission combinations

Legacy System Analysis

• Reconstruct undocumented protocols
• Map proprietary file formats
• Identify hardware interaction points

Model Performance Analysis

Accuracy Benchmarks

Cost-Performance Tradeoffs

pie
    title Analysis Cost Distribution
    "Claude-Opus-4" : 42
    "Claude-Sonnet-4" : 28
    "Gemini-2.5-Pro" : 20
    "GPT-O4-Mini" : 10

Speed Comparison

Installation Troubleshooting Guide

Common Issues

1. Dependency Conflicts:

   python -m venv cutter-env
   source cutter-env/bin/activate
   pip install -r requirements.txt
   

2. Plugin Not Loading:

   • Verify file location: plugins/python/mcp_plugin.py
   • Check Cutter error logs
   • Ensure Python version compatibility

3. Connection Failures:

   • Verify MCP server path in config
   • Check firewall permissions
   • Test manual execution: python mcp_server.py

Future Development Roadmap

Version 2.0 Objectives

1. Token optimization algorithms
2. Split architecture for dependency isolation
3. Interactive analysis sessions
4. Result caching mechanism
5. Multi-tool support (IDA, Ghidra)

Conclusion: The Future of Reverse Engineering

CutterMCP+ demonstrates that AI isn’t replacing reverse engineers – it’s augmenting them. The key findings from our testing:

1. Simple Challenges: Fully automatable with mid-tier models
2. Medium Complexity: Requires high-end models (Claude-Opus level)
3. Real Malware: AI accelerates analysis 5x+ while maintaining accuracy

As project developer notes:

“

“Without [Amey Pathak’s] project, this project probably wouldn’t exist.”

The revolution isn’t coming – it’s already here. And it’s open-source.

Get Started Today
CutterMCP+ GitHub Repository
Cutter Reverse Engineering Platform
Original CutterMCP Project

“

“Give AI a sharp cutter!” – The CutterMCP+ Philosophy