System design: iterative data mining and model training for multi modality foundational model training

Question

## Topic
Design a system to mine novel, high-quality training data from a massive *noisy* unlabeled corpus (e.g., 100M+ screenshots and interaction logs from iOS/macOS devices) to train a multi-modal UI navigation agent.

The corpus may contain both correct and incorrect interaction sequences. Your goal is to denoise, bootstrap, and iteratively refine a dataset that enables robust model training.

---

## Your Task

Walk through the full lifecycle of the system — from raw noisy data to a production-quality dataset and evaluation loop. Discuss:

---

### 1. Data Ingestion & Initial Filtering

- How would you ingest and store large volumes of screenshot + interaction data?
- How would you define a *valid interaction sequence*? (e.g., tap, type, swipe)
- What heuristics, model predictions, or feedback signals could help you discard:
  - Random/misclicked/no-op actions
  - Out-of-bounds UI taps
  - Failed attempts, error dialogs
- Would you use LLMs or learned policies to score trajectory validity?

---

### 2. Data Denoising & Trajectory Validation

- Propose a method to separate useful from harmful data:
  - Rule-based trajectory validators (e.g., ends on success screen)
  - LLM-base

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.