Bot Detection | ML System Design Problem Breakdown

Bot Detection (ML system design) — Subtitles Summary

1) Framing the problem (business + ML objectives)

• Use a structured ML system design framework: problem framing → high-level design → data → modeling → evaluation → optional deep dives.
• Adversarial dynamics: bot behavior is evasive and evolves, so the solution must handle:
  • Distribution shifts
  • Continuous retraining/updates

Key questions to ask

• Prevalence of bots and what remains after basic heuristics (e.g., rate limiting may reduce bot activity substantially).
• Nature of bot behavior: often shows inhuman posting frequency and abnormal activity patterns (e.g., being active at “impossible” times).
• Scale: may involve hundreds of millions to billions of daily users/actions, driving efficiency requirements.
• Actions/consequences when a bot is detected
  • Ban vs. soft actions (e.g., rate limiting)
  • False positives: appeals exist, but bots can also “appeal,” so appeals can’t fully correct mistakes.
• Label data availability:
  • Human investigator labels are scarce and expensive
  • Class imbalance is high

Class imbalance emphasis

• After heuristics, legitimate actions dominate (bots are rare), but “hard” bots remain.

Business objective reframing

• Avoid naive goals like:
  • “maximize bots detected”
  • “maximize model accuracy”
• Prefer goals like:
  • Minimize impact on legitimate users, subject to constraints on false positives
  • Example constraint: “ensure 95% of actions are on true bots”

Need calibrated scores

• Convert model outputs to a calibrated probability so a threshold corresponds to a desired false-positive rate.
• Calibration is presented as necessary for threshold-based decisions.

---

2) High-level system design (components + decision policy)

• Trigger the bot classifier at action time (not necessarily offline), because impact matters when users take harmful actions.

Pipeline

1. Bot classifier produces a score
2. Calibration layer outputs calibrated probability
3. Decision logic
   • High probability → ban
   • Mid probability → soft enforcement (e.g., rate limits / posting restrictions)

Memory / reuse of signals after enforcement

• After banning, store removed content in a vector database / ANN index.
• Use similarity signals (e.g., distance to previously removed content) as later classification inputs.

Ensemble / multi-model possibility

• Combine multiple modalities/models (and potentially multiple models in an ensemble).

Outer loop for discovering new bots (adversarial learning)

• Labeling loop:
  • Sample low-confidence or strategically chosen accounts for investigator labeling
• Unsupervised anomaly detection can generate candidate bot accounts not covered by supervised labels
• New labels feed back into continuous improvement cycles

---

3) Data strategy (labels + feature sources)

• Ground-truth bot labels exist but are inexact and scarce.
• Additional weak supervision signals:
  • Weekly supervised labels: user reports, content reports, appeal outcomes
• Unsupervised / graph-based labels:
  • Network clustering signals (e.g., IP address clusters, behavioral similarity clusters)
  • Clusters help infer shared entities across accounts (not perfect due to IP changes/proxies)
• Sequence data from real users
  • With high legitimate prevalence, anomalies stand out as exceptions in behavior
• Feature engineering guidance
  • Don’t enumerate thousands of features—use a few strong categories with good examples

---

4) Feature examples (what the model might use)

• Activity / posting behavior patterns
  • posting cadence, variance of post times, circadian coverage
  • detected “sleep gaps”
  • burstiness (max frequency)
• Content-level signals
  • typo frequency / editing behavior
  • copying/pasting patterns
  • language quality proxies (bots may produce refined/PR-like content)
• Network / social graph features
  • follower graph growth rate
  • follower-to-following ratios
  • synthetic/automated network patterns
• Real-time behavioral counters
  • number of content flags
  • number of rate limits applied
  • (features that can change quickly and shift classification)

---

5) Modeling approaches (architecture + training strategy)

Start with a benchmark

• Example baseline: logistic regression with engineered features
  • Useful as a production-friendly safety net and benchmark.

Core model patterns discussed

• Organic multi-model ensembles (e.g., separate models for content vs. behavior) are common in production evolution, but hard to optimize cleanly from scratch.
• Avoid a “mega end-to-end model” because:
  • content is easy for attackers to change
  • content models can overfit and miss more robust signals (timing, IP ranges, action frequency)

Preferred approach: multi-modal fusion (“late fusion”)

• Graph branch (network/IP/social context)
  • Use Graph Neural Networks, specifically GraphSAGE (inductive) for cold-start new nodes/accounts.
• Sequence branch (user action sequences)
  • Use an RNN / GRU rather than a transformer due to inference cost at huge scale (runs whenever actions happen).
• Dense/counter features
  • via an MLP
• Fusion and output
  • combine embeddings using attention
  • then an MLP
  • then sigmoid for bot/not score

Training complexity due to scarce labels (~1,000 labeled examples)

• Use pre-training
  • Graph branch: self-supervised learning
    • predict whether an edge exists
    • censor edges during training
  • Sequence branch: next-step prediction
    • cross-entropy loss
• Loss weighting based on business impact
  • give more weight to bots that negatively impact users (e.g., lots of actions / public spaces)
• Multitask objective
  • auxiliary tasks such as “account was reported” (leveraging more abundant data)
• Fine-tuning plan
  • freeze early layers
  • train later layers for bot/not task
• Output
  • calibrated probability for downstream thresholding

---

6) Inference efficiency + evaluation

Inference efficiency

• Inference cost remains high even with RNNs.
• Techniques discussed:
  • pruning / quantization
  • caching stable graph embeddings until graph changes
• Expected speedups are limited (e.g., caching ~2x, not 10x)

Two-stage inference with distillation-like “filter model”

• Use a cheap filter model (e.g., logistic regression) to predict whether the heavy classifier would trigger ban/limits.
• Only high-likelihood candidates go to the expensive model.
• Risk: the filter can discard accounts, preventing labels and harming recall over time.
• Mitigation:
  • random sampling of discarded or high-score candidates to measure filter recall and refresh labels
  • retrain the filter periodically

Online evaluation

• Online metrics should closely reflect business goals:
  • A/B testing with precautions for leakage/contamination
    • e.g., graph inputs shared across model versions
  • Measure things like user reports / spam prevalence
  • Measure a counter metric tied to false-positive constraints
• Since precision constraints must be verified:
  • measure false positives by sampling and labeling them
• Use important sampling for expensive investigator labeling:
  • sample more around score regions where bots are likely (e.g., around ~50%)
  • sample fewer where scores imply low prevalence

Offline evaluation

• Use metrics aligned to class imbalance:
  • Area under the Precision-Recall curve (PR-AUC)
    • less sensitive than ROC-AUC under heavy imbalance
• Ensure offline sets reflect online conditions:
  • time-based split (train on older data, evaluate on the next two weeks)
  • address drift and generalization

---

7) Deep dive: unsupervised/anomaly detection for new bots

Purpose: find bots the supervised model misses, especially when bots change tactics entirely.

• Isolation Forest
  • Randomly partitions feature space using tree splits.
  • Anomalies tend to require fewer splits to isolate → higher anomaly score.
  • Fast on CPU; scores usable for ranking.
• Autoencoder
  • Learn a compressed latent representation and reconstruct input.
  • Higher reconstruction error → anomaly.
  • Failure mode warning: autoencoders might reconstruct anomalies too if capacity is high.
• Ensemble anomalies
  • Combine/average anomaly scores from both methods to rank candidates for labeling.
• Output of this pipeline
  • generate seed data for investigator labeling to improve supervised detection over time

---

Main speakers / sources

• Speaker/Host: The subtitles use first-person narration (“we,” “I will include a link,” “let’s go ahead”), likely from Hello Interview.
• Source/channel: Hello Interview (referred to directly as the channel; website mentioned as hello.com).