Testing

Test suite documentation, running tests, and test data examples

Overview

RAPTOR includes a comprehensive test suite to validate all commands, workflows, and user scenarios. Tests cover command structure, security analysis modes, package architecture, and real vulnerability detection.

Tests use real vulnerable code samples to ensure RAPTOR’s detection capabilities work in practice.

---

Quick Start

Run All Tests

cd raptor
bash test/comprehensive_test.sh

Expected duration: 2-3 minutes for full suite

---

Test Structure

RAPTOR’s test suite is organized into multiple categories:

Test Categories

1. Command Structure Tests

Validates that all RAPTOR commands are properly registered and accessible.

Tests:

• Main launcher recognizes all modes (scan, fuzz, web, agentic, codeql)
• Core execution scripts exist (raptor_agentic.py, raptor_fuzzing.py, etc.)
• Main scripts have valid Python syntax
• Help information is accessible

Example:

python3 raptor.py --help
# Should list: scan, fuzz, web, agentic, codeql

2. Scan Mode Tests

Validates static analysis capabilities.

Tests:

• Scan mode help available
• Scan requires —repo argument
• Scan supports policy groups

Example:

python3 raptor.py scan --repo /test/data

3. Agentic Mode Tests

Validates autonomous workflow capabilities.

Tests:

• Agentic script exists
• Agentic accepts arguments
• CodeQL integration available
• Exploit/patch generation controls present

Example:

python3 raptor.py agentic --repo /test/data --skip-exploits

4. Fuzzing Mode Tests

Validates binary fuzzing capabilities.

Tests:

• Fuzzing script exists
• Requires —binary argument
• Supports —duration option
• Supports —parallel option
• Supports autonomous mode

Example:

python3 raptor.py fuzz --binary ./target --duration 3600

5. CodeQL Mode Tests

Validates deep semantic analysis.

Tests:

• CodeQL script exists
• CodeQL accepts arguments
• Language detection supported

Example:

python3 raptor.py codeql --repo /test/data --language python

6. Web Mode Tests

Validates web application security testing (alpha).

Tests:

• Web mode script exists
• Web mode listed in help

Note: Web mode is in alpha stage.

7. Package Architecture Tests

Validates internal package structure.

Tests:

• Core modules directory exists
• Packages directory exists
• LLM analysis package exists
• Static analysis package exists
• Fuzzing utilities exist

8. Test Fixture Tests

Validates that test data samples are present and contain real vulnerabilities.

Tests:

• Test data directory exists
• Sample Python vulnerable code present
• Sample JavaScript vulnerable code present
• Code samples contain vulnerability patterns

9. Workflow Availability Tests

Validates that all user-facing workflows are accessible.

Tests:

• scan —help accessible
• agentic —help accessible
• fuzz —help accessible
• codeql —help accessible

---

Test Data Examples

RAPTOR includes realistic vulnerable code samples for testing:

Python SQL Injection

test/data/python_sql_injection.py

import sqlite3
from flask import Flask, request

app = Flask(__name__)

# VULNERABLE: SQL injection in database query
@app.route('/user/<user_id>')
def get_user(user_id):
    db = sqlite3.connect(':memory:')
    cursor = db.cursor()
    
    # Direct string concatenation - SQL injection vulnerability
    query = "SELECT * FROM users WHERE id = " + user_id
    cursor.execute(query)
    
    return cursor.fetchone()

# VULNERABLE: Hardcoded credentials
DATABASE_PASSWORD = "admin123!SuperSecret"

# VULNERABLE: Weak cryptography (MD5)
import hashlib
def hash_password(password):
    return hashlib.md5(password.encode()).hexdigest()

# VULNERABLE: Command injection
import subprocess
@app.route('/convert', methods=['POST'])
def convert_file():
    filename = request.form.get('filename')
    result = subprocess.run(f"convert {filename} output.jpg", shell=True)
    return "Conversion complete"

# VULNERABLE: Path traversal
@app.route('/download/<file_path>')
def download_file(file_path):
    with open(f"/var/www/files/{file_path}", "rb") as f:
        return f.read()

Vulnerabilities in this file:

• SQL injection (string concatenation)
• Hardcoded credentials
• Weak cryptography (MD5)
• Command injection (shell=True)
• Path traversal (unsanitized file path)

JavaScript XSS

test/data/javascript_xss.js

// VULNERABLE: DOM-based XSS
function displayUserInput() {
    const userInput = document.getElementById('user_input').value;
    // Direct innerHTML assignment - XSS vulnerability
    document.getElementById('output').innerHTML = userInput;
}

// VULNERABLE: Reflected XSS via query parameter
function handleSearchQuery() {
    const query = new URLSearchParams(window.location.search).get('q');
    // Unsanitized query in HTML - reflected XSS
    document.write('<p>Search results for: ' + query + '</p>');
}

// VULNERABLE: Eval usage
function executeUserCode(code) {
    // eval() is extremely dangerous - arbitrary code execution
    eval(code);
}

// VULNERABLE: Hardcoded API key
const API_KEY = "sk-1234567890abcdef";
const SECRET_TOKEN = "super_secret_token_12345";

// VULNERABLE: Insecure localStorage usage
function storeUserCredentials(username, password) {
    // Storing sensitive data in localStorage - insecure
    localStorage.setItem('username', username);
    localStorage.setItem('password', password);
}

// VULNERABLE: Insecure random number generation
function generateSessionToken() {
    // Math.random() is not cryptographically secure
    return Math.random().toString(36).substring(7);
}

// VULNERABLE: Prototype pollution
function mergeObjects(target, source) {
    for (const key in source) {
        // No validation - allows prototype pollution
        target[key] = source[key];
    }
    return target;
}

// VULNERABLE: Regular expression DoS
function validateEmail(email) {
    // Inefficient regex - ReDoS vulnerability
    const pattern = /^([a-zA-Z0-9]+)*@([a-zA-Z0-9]+)*\.([a-zA-Z0-9]+)*$/;
    return pattern.test(email);
}

Vulnerabilities in this file:

• DOM-based XSS (innerHTML)
• Reflected XSS (document.write)
• Arbitrary code execution (eval)
• Hardcoded secrets (API keys)
• Insecure storage (localStorage)
• Weak random generation (Math.random)
• Prototype pollution
• ReDoS (inefficient regex)

---

Running Tests

Comprehensive Test Suite

Run all tests with detailed output:

Run Comprehensive Tests

cd raptor
bash test/comprehensive_test.sh

Output format:

╔═══════════════════════════════════════════════════════════╗
║          RAPTOR Comprehensive Test Suite                  ║
║     Testing actual workflows, commands, and scenarios     ║
╚═══════════════════════════════════════════════════════════╝

1. COMMAND STRUCTURE TESTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Main launcher lists all modes
✓ Core execution scripts exist
✓ Main scripts have valid Python syntax
✓ Help information accessible

2. SCAN MODE TESTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Scan mode help available
✓ Scan requires --repo argument
...

╔═══════════════════════════════════════════════════════════╗
║ SUMMARY
╚═══════════════════════════════════════════════════════════╝

Passed:  42
Failed:  0
Skipped: 3
Total:   45

Compliance: 93% (42/45 tests passed)

✓ All tests passed!

Test Result Indicators

• ✓ PASS - Test succeeded
• ✗ FAIL - Test failed (with error message)
• ⊘ SKIP - Test skipped (with reason)

---

Test Workflows (Claude Code)

RAPTOR includes a /test-workflows command for Claude Code:

Via Claude Code

claude
# Then in Claude:
/test-workflows

What it tests:

1. Basic scan (findings only, no exploits)
2. Full agentic workflow (scan + exploit + patch)
3. Binary fuzzing
4. Manual crash validation
5. Tool routing sanity checks

Output: Pass/Fail/Skip status with summary counts

---

Integration Tests

Integration tests verify RAPTOR works with external tools:

Run Integration Tests

bash test/integration_tests.sh

Requirements:

• Semgrep installed
• CodeQL installed (optional)
• AFL++ installed (optional)
• Test data present

Tests:

• Semgrep scanning works
• CodeQL database creation (if available)
• AFL++ fuzzing setup (if available)
• Python package imports

---

User Stories

Tests are designed around real user scenarios:

1. User Story 1: Quick Scan

   As a developer, I want to quickly scan my code for vulnerabilities.

   python3 raptor.py scan --repo ./myapp
   

   Tested by: Scan mode tests, fixture tests

2. User Story 2: Autonomous Analysis

   As a security researcher, I want RAPTOR to autonomously analyze, generate exploits, and propose patches.

   python3 raptor.py agentic --repo ./myapp
   

   Tested by: Agentic mode tests, workflow tests

3. User Story 3: Binary Fuzzing

   As a bug hunter, I want to fuzz a binary and analyze crashes.

   python3 raptor.py fuzz --binary ./target --duration 3600
   

   Tested by: Fuzzing mode tests, integration tests

4. User Story 4: Deep Analysis

   As an analyst, I want to use CodeQL for deep semantic analysis.

   python3 raptor.py codeql --repo ./myapp
   

   Tested by: CodeQL mode tests, integration tests

5. User Story 5: Custom Workflows

   As a power user, I want to verify RAPTOR works after I make changes.

   bash test/comprehensive_test.sh
   

   Tested by: All test suites

---

Writing New Tests

To add tests to the suite:

1. Choose Test Category

   Determine which test file to modify:

   • comprehensive_test.sh - Core functionality
   • integration_tests.sh - External tool integration
   • test_workflows.sh - End-to-end workflows
2. Add Test Function

   # Add test in appropriate section
   if [condition]; then
       test_result "Your test name" "PASS"
   else
       test_result "Your test name" "FAIL" "Error message"
   fi
   
3. Add Test Data (if needed)

   Create new vulnerable code samples in test/data/:

   # test/data/your_vuln.py
   # VULNERABLE: Description
   def vulnerable_function():
       # ... vulnerable code ...
   
4. Run and Verify

   bash test/comprehensive_test.sh
   

   Ensure your new test passes and doesn’t break existing tests.

---

Continuous Integration

RAPTOR uses GitHub Actions for CI. Tests run automatically on pull requests.

CI Workflow:

1. Install dependencies
2. Run syntax validation
3. Run comprehensive test suite
4. Run integration tests (if tools available)
5. Report results

Badge:

---

Troubleshooting Tests

Tests fail with 'command not found'

Problem: Required tools not installed.

Solution:

1. Use the devcontainer (all tools pre-installed)
2. Or install missing tools:

pip install semgrep
# Install other tools as needed

Tests skip with 'May require valid parameters'

Problem: Test requires actual execution, skipped for safety.

Solution: This is normal. Skipped tests indicate optional features or safety guards.

Integration tests fail

Problem: External tools not working.

Solution:

1. Verify tool installation:

semgrep --version
codeql version

2. Check test/data directory exists
3. Ensure PYTHONPATH is set correctly

Fixture tests fail with 'vulnerability patterns not found'

Problem: Test data files modified or missing.

Solution:

1. Restore test/data files from git:

git checkout test/data/

2. Ensure files contain documented vulnerabilities

---

Test Coverage

RAPTOR’s test suite covers:

Command Coverage

All major commands:

• /scan
• /agentic
• /fuzz
• /codeql
• /web
• /analyze

Vulnerability Coverage

Multiple vulnerability types:

• SQL injection
• XSS (DOM, reflected)
• Command injection
• Path traversal
• Secrets exposure
• Weak cryptography

Language Coverage

Multiple languages:

• Python
• JavaScript
• C/C++ (via binaries)

Package Coverage

All packages:

• llm_analysis
• static-analysis
• fuzzing
• codeql
• web

---

Next Steps

DevContainer

Set up pre-configured environment

Contributing

Add your own tests

Quick Start

Start using RAPTOR

FAQ

Common questions