NeurIPS 2020

End-to-End Learning and Intervention in Games


Meta Review

At the end of the review, rebuttal, and discussion phases, three of the four referees (all of whom are knowledgeable in the field) recommended that the paper be accepted. The other referee (R2) felt the paper was not quite ready for publication at NeurIPS in its current form. From the reviews and the reviewer discussion after the author rebuttal, it appears that the four reviewer see similar strengths and weaknesses of the paper, but assessed the paper differently based on these reviews. I believe that R2 stated this well in the discussion phase: R2: ”After reading other reviews (and rebuttal) I feel that my concerns are actually similar to other reviewers, mainly that the paper has a decent idea but (a) under-developed as the evaluation is very basic and (b) over-claiming at a number of places in the paper. Then, I believe that the scores differ due to differing subjective opinion and IMO I maintain the negative view because 1) This paper needs to demonstrate more in terms of experiments, my belief is that an under-developed idea should get be forced to develop more. 2) Overclaiming will hopefully be fixed by the authors but overclaiming, if it stays in the paper, can be harmful for any future paper in this area." In the discussion phase, the other reviewers confirmed concerns about “overclaiming.” For example, R4 expressed concern that the authors did not effectively establish the boundaries of their claims (and indeed made the claims seem broader than they actually were). In R4’s words: “In the rebuttal, the authors appear to double-down on asserting that their approach makes sense as long as the problem is monotone: `Since our method unrolls the projection method, it only works when Fλ is monotone and the projection method converges’ despite my providing an example where the projection method diverges from the Nash equilibrium (min_{x in [-1,1]} max_{y in [-1,1]} xy), i.e., convergence does not imply Nash or epsilon-Nash. The authors need to back off the monotone assumption; their statements still hold with the strictly-monotone assumption which 1) does not weaken the impact of the paper significantly and 2) makes clear where future work needs to be done.” I do note that the authors sent a message asking that R2’s review be discounted due to a variety of incorrect claims about the paper. I agree with the authors' arguments that R2 does appear to have misunderstood or misjudged some aspects of the paper, but also believe (and the other reviewers confirm) that some of R2’s “criticisms” are justified. All this said, the reviewers did all see value in the paper. Even with the stated weaknesses of the submitted paper, three of the four reviewers still thought the paper should be accepted. I am somewhat on the fence, but edge toward recommending “accept (poster).” I hope that the authors pay special attention to making sure their claims are carefully and properly scoped in the final version of their paper, along with addressing the other comments made by the reviewers.