Lesson 10
Detection and Red Teaming

Monitoring for attacks, testing your defenses, and building a security practice

Why Detection Matters

• Assume breach: No defense is perfect. Assume some attacks will succeed.
• Detect early: The faster you detect an attack, the less damage it causes.
• Learn and adapt: Detection data feeds back into improving defenses.

Prevention ──▶ Detection ──▶ Response ──▶ Recovery
(Lessons 7-9)   (This lesson)   (Incident response)   (Improvement)

What to Log

Comprehensive logging is the foundation of detection:

Monitoring Patterns

What signals indicate a potential attack in progress?

• Topic drift: Sudden shift from legitimate questions to injection-like content
• System prompt fragments in output: If the response contains words from your system prompt, extraction may have succeeded
• Unusual tool usage: Tools called in unexpected sequences or with abnormal parameters
• Response length anomalies: Unexpectedly long or short responses may indicate injection
• Repeated failures: Multiple blocked requests from the same user suggest active probing
• Encoding patterns: Base64, hex, or other encoding in user input

Anomaly Detection

Establish baselines for normal usage and flag deviations:

BASELINE_METRICS = {
    "avg_input_length": 85,     # characters
    "avg_output_length": 320,   # characters
    "avg_session_messages": 4,
    "tool_calls_per_session": 1.2,
    "topics": ["orders", "returns", "shipping"],
}

ANOMALY_THRESHOLDS = {
    "input_length": 3.0,   # 3x standard deviation
    "output_length": 3.0,
    "session_messages": 2.5,
    "tool_calls": 3.0,
}

• Compute rolling averages and standard deviations
• Flag sessions that exceed thresholds
• Combine multiple signals for higher confidence

Red Teaming Methodology

A structured approach to testing your LLM application's security:

1. Scope: Define what's being tested (system prompt, tools, architecture)
2. Threat model: Identify attack surfaces and attacker capabilities
3. Test categories: Systematic coverage of attack types
4. Execute: Run tests and document results
5. Report: Findings, severity, evidence, recommendations
6. Remediate: Fix issues and re-test

Red Team Tools

# Garak: scan for vulnerabilities
garak --model_type openai --model_name gpt-4 \
      --probes injection

# promptfoo: eval with injection tests
promptfoo eval --config injection-tests.yaml

Building a Prompt Injection Test Suite

Organize test cases by attack category for comprehensive coverage:

Target: 30-60 test cases for comprehensive coverage.

Continuous Testing in CI/CD

Integrate injection tests into your deployment pipeline:

# .github/workflows/prompt-security.yml
name: Prompt Injection Tests
on:
  push:
    paths:
      - 'prompts/**'
      - 'config/system_prompt.txt'

jobs:
  injection-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run injection tests
        run: |
          python -m pytest tests/injection/ \
            --tb=short -v
      - name: Run promptfoo eval
        run: |
          npx promptfoo eval \
            --config tests/promptfoo.yaml

• Run tests on every system prompt change
• Block deployment if injection tests fail
• Track test pass rates over time

Responsible Disclosure

How to handle prompt injection vulnerabilities you discover:

• Internal discovery: File a security incident, follow your organization's security response process
• External discovery: Report through the vendor's security contact or bug bounty program
• Public disclosure: Follow coordinated disclosure timelines (typically 90 days)
• Never: Exploit vulnerabilities in production for profit, access others' data, or cause damage

The Arms Race

• As models improve, so do attack techniques
• As defenses evolve, attackers find new bypasses
• This is an ongoing process, not a one-time fix

Security is a continuous practice, not a product.

Course Summary