What Makes a Reward Model a Good Teacher? An Optimization Perspective - Razin, Noam; Wang, Zixuan; Strauss, Hubert; Wei, Stanley; Lee, Jason; Arora, Sanjeev (Advances in Neural Information Processing Systems, 2025)