AI Concepts
Knowledge distillation trains a smaller model to follow a larger model’s outputs
Knowledge distillation trains a student model using a teacher’s behaviour. Understand soft targets, useful trade-offs and why the student still needs evaluation.