Designing Harmful Content Detection System for Instagram

Question

## Problem: Designing Harmful Content Detection System for Instagram

Instagram contains billions of pieces of user-generated content, including **images**, **videos**, and **text** (e.g., captions, comments). Some of this content may be **harmful**, including:
- Hate speech
- Harassment
- Nudity
- Self-harm or suicide
- Misinformation

### Objective:
Design a system that can efficiently detect harmful content **at scale**, across all modalities — text, image, video, or a combination.

### Requirements:
- Handle **multi-modal** content (image + caption, video + comments).
- Detect harmful content **in real time or near real time**.
- Support **model retraining** with human feedback and new data.
- Allow **auditing** and **manual review** of model decisions.

### Clarifications:
- The system should support **automated content moderation** with a fallback to human review.
- Discuss the **data pipeline**, **architecture**, **training strategies**, **model types** (LLMs/VLMs), **evaluation**, and **trade-offs**.

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.