Homework

Task 1: Latency Anomaly Detector

Goal: Build a rolling anomaly detector that flags requests in real time as they arrive, rather than in batch after all data is collected.

Instructions

1. Implement a RollingAnomalyDetector class that maintains a sliding window of the last N latency values (default N=50).
2. For each new request, compute the rolling p95 from the current window.
3. Flag the request as an anomaly if its latency exceeds 2x the rolling p95.
4. The class should expose:
   • add(latency_ms: float) -> bool — returns True if the value is anomalous
   • get_stats() -> dict — returns current p50, p95, p99, and anomaly count
5. Test with your synthetic log generator from Exercise 3: feed in 200 log entries and print every anomaly with its index and latency.

Deliverable

A working RollingAnomalyDetector class and a test run showing detected anomalies.

=== Rolling Anomaly Detection (window=50) ===
Request #67:  4,912 ms (rolling p95: 2,301 ms) -- ANOMALY
Request #112: 5,445 ms (rolling p95: 2,287 ms) -- ANOMALY
Request #158: 4,723 ms (rolling p95: 2,350 ms) -- ANOMALY
...
Total anomalies: 7 / 200 (3.5%)

Self-check

• ✓ The detector uses a sliding window, not all historical data.
• ✓ The rolling p95 updates as new values are added.
• ✓ The anomaly threshold is 2x the rolling p95, not a fixed value.
• ✓ The first N requests (before the window is full) are handled gracefully (e.g., no anomaly detection until the window has at least 10 values).

Task 2: Feedback Collection Endpoint

Goal: Implement a feedback system that links user ratings to trace IDs and computes aggregate quality scores.

Instructions

1. Create a FeedbackStore class with the following methods:
   • record(trace_id: str, score: int, comment: str = "") — store a feedback entry (score is 0 or 1 for thumbs down/up)
   • get_feedback(trace_id: str) -> dict — retrieve feedback for a specific trace
   • summary() -> dict — return overall positive rate, total count, and the 5 most recent negative entries
2. Integrate with your logging wrapper from Exercise 1: after each LLM call, simulate user feedback (e.g., 80% positive, 20% negative with random comments).
3. Print a feedback summary report showing:
   • Overall satisfaction rate
   • Number of feedback entries
   • The 5 most recent negative feedback entries with their trace IDs
4. Bonus: If you set up Langfuse in Exercise 2, also send scores via langfuse.score() and verify they appear linked to traces in the dashboard.

Deliverable

=== Feedback Summary ===
Total feedback:   50
Positive rate:    82.0%
Negative entries: 9

=== Recent Negative Feedback ===
  trace d4e5f6...: "Response was too verbose" (score: 0)
  trace a7b8c9...: "Incorrect date mentioned" (score: 0)
  trace f1g2h3...: "Missed key detail from context" (score: 0)
  trace j4k5l6...: "" (score: 0)
  trace m7n8o9...: "Tone was too formal" (score: 0)

Self-check

• ✓ Each feedback entry is linked to a trace ID.
• ✓ The summary() method computes the positive rate correctly.
• ✓ Negative feedback entries include trace IDs for debugging.
• ✓ The store handles edge cases: no feedback, all positive, all negative.

Task 3: Cost Anomaly Alert

Goal: Build an alerting function that detects when hourly spend deviates significantly from the trailing average.

Instructions

1. Using your logs (real or synthetic), bucket costs into hourly windows.
2. Compute a trailing 24-hour average hourly cost (or fewer hours if less data is available).
3. For each hour, check if the cost exceeds 150% of the trailing average. If so, emit a warning alert. If it exceeds 200%, emit a critical alert.
4. Generate at least 72 hours of synthetic data with a deliberate cost spike in hours 48–52 (e.g., double the normal request rate or token counts).
5. Print all alerts with the hour, actual cost, trailing average, and severity.

Deliverable

=== Cost Anomaly Alerts ===
Hour 48: $2.34 (trailing avg: $1.12) -- WARNING (209% of avg)
Hour 49: $2.87 (trailing avg: $1.17) -- CRITICAL (245% of avg)
Hour 50: $2.65 (trailing avg: $1.23) -- CRITICAL (215% of avg)
Hour 51: $2.41 (trailing avg: $1.28) -- WARNING (188% of avg)
Hour 52: $1.89 (trailing avg: $1.31) -- WARNING (144% of avg)

Total alerts: 5 (2 critical, 3 warning)

Self-check

• ✓ The trailing average uses a 24-hour (or available-hours) window, not the global average.
• ✓ Warning fires at 150%; critical fires at 200%.
• ✓ The spike hours are correctly detected as anomalies.
• ✓ Normal hours do not generate false alerts.
• ✓ The trailing average increases after the spike starts (showing adaptation).