NeurIPS 2020

A/B Testing in Dense Large-Scale Networks: Design and Inference


Meta Review

This paper is concerned with AB Testing in real-world settings where the common SUTVA assumption does not hold: in the context of Internet marketplaces for instance, treatments applied to different producers might have effects on other producers as all producers are related via the consumers. The proposed approach considers the case where the treatment consists of modifying the probability of showing a content of producer i to consumer j. The proposed approach combines several steps: i) assuming an intervening variable (mediator) capturing the global effect of a treatment, ii) defining auxiliary variables attached to each edge (ij) to estimate their impact w.r.t. the mediator; iii) showing that these auxiliary variables are solutions of a quadratic problem optimization ; iv) defining a relaxation thereof to achieve scalability and studying the convergence error. The reviewers particularly appreciated the relevance of the problem, the generality of the approach, and the fact that the code is accessible. The AC congratulates the authors for this nice piece of work.