The paper considers the simple problem of a squared-loss for a two-layer ReLU network. By relying on certain invariance properties of the loss, the authors derive an analytical expression of the Hessian and its spectrum. The analysis, in the context of the specific problem at hand, is rather novel and although it only yields results on a small shallow network, it has some potential to yield some new research directions. Although I recommend acceptance, I should say that the paper is seen by the reviewers as quite technical and I think the paper would much benefit from a revision where more intuition (maybe a proof sketch) is given in the main paper.