LLM Questions: Training Systems, Hardware, and Scaling

Question

## Set: Training Systems, Hardware, and Scaling

1.What forms of parallelism are used during large language model training? Explain data parallelism, model parallelism (including tensor and pipeline parallelism), and GPU-level parallelism, and discuss when each is most appropriate.

2.What are the key architectural differences between CPUs, GPUs, and TPUs? How do these differences impact their suitability for large-scale machine learning workloads?

3.What lessons from small-model research can be applied to large-model research, particularly in the context of optimization, generalization, architecture design, and data quality?

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.