| Paper ID: | 2495 |
|---|---|
| Title: | Beyond Exchangeability: The Chinese Voting Process |
The paper suggests Chinese Voting Process (CVP) to model user response patterns in online community where users can write a response or vote helpfulness of existing responses. The response patterns are divided into two phase: 1) selection phase and 2) voting phase. In the selection phase, a user decides to write a new response or vote an existing response. And once the user decided to vote, the user decides to give a positive vote or negative vote in the voting phase. With Amazon review and StachExchange datasets, the model shows that each community reveals different response patterns characterised by trendiness and conformity.
The exchangeability under a probabilistic model gives a great flexibility in building and inferencing, while preventing the representation power of a model. This paper well captures two different levels of non-exchangeability occurring in online voting. Although the problem is interesting, the presentation of this paper makes hard to understand the proposed approach. Especially, the structure and notations hinder readers from understanding some key ideas. Below I listed some suggestions and comments to improve the paper: * a short description of the reinforcing mechanism between the CRP and ddCRP: ddCRP is an important concept of the proposed approach, which breaks the general exchangeability assumption. But It is quite hard to get an intuition about how this model works to compare with the CRP. A brief comparison between two processes would be good. * intrinsic quality: In the generative process described in table2, each item has an unique intrinsic quality over time. However, in equation (1), it seems the model jointly learns different intrinsic qualities at a different time. I assume that only one intrinsic quality is estimated in the experiment. Please clarify the notation across the manuscript. * Table 2 [generative process]: When table 2 is introduced in the main text, there is no explanation about the notations in this table, and it's impossible to understand the process. Maybe move the table after section 2.2? It would be better to combine table 2 and 4 since both tables are talking about the same thing. * Table 5: There is no description of table 5 in the main text. The paragraph between line 208 and 218 seems a relevant part, but the description of each table should be included in the main text. Also the voting is a binary classification problem, so it might be more intuitive to interpret the result if it's written in terms of accuracy(?). * Prediction: since the suggested model is fully generative, once the model learns parameters, it would be also possible to have a full prediction task which predict two phases together. Without this, the propose model seems just a combination of two independent models.
3-Expert (read the paper in detail, know the area, quite certain of my opinion)
In this paper, the authors propose a novel model for the votes given by users to the reviews and answers provided by other users in a review or Q&A system. The proposed model combines ideas from the ddCRP and the Poyla Urn model, and is referred by the authors as Chinese voting process CVP. The key aim of the proposed approach is to capture two key factors of the voting process: quality of the review/answer and presentation bias. For the proposed model, the authors learn the model parameters as a maximum likelihood estimation problem, which results in a convex program. Finally, they experiment on real-world data gathered from Amazon and Stack Exchange.
The paper is in general well written (find more comments below) and the proposed application is novel and interesting. In particular, the proposed model can be applied to find out and analyze the quality of the content in any crowd-sourcing platform where both the content and its evaluations are provided by the users, as well as measure the influence of the presentation of the content. In general, I believe that the paper is a good example of NIPS poster level. Comments: - When I first read the introduction, based mainly on the explanation of Table 1, I understood that the model account for the order in which votes are given to an item (review, answer...). However, after carefully reading Section 2, it turns out that the model only accounts for the ratios of positive/negative votes given to an item up to time t, but not for the order in which they are given. Is that the case? In such case, I would recommend the authors to change the intro to avoid misunderstandings. - In Section 2.1., the authors claim that the the function f_i^(t)(j) is parametrized by \tau, however this dependency is never written in the paper (neither in Table 2 or the second paragraph of Sec 2.1). As a consequence, it is hard to figure out what this parameter means and where Eq. 3 comes from. Could you please clarify? - I believe that in Eq. (1) the term that depends on the response length is missing. - It is not clear to me what u_ij^(t) means -- I believe that the length feature u_ij^(t) is the length of the review and, therefore, it is observable (or at least it is not inferred in Section 3). Moreover, u_ij^(t) is not in the graphical model in Table 4. - I believe that there is typo in line 132, "u_ij^(t) explains the length-wise preferential..." should be "\nu_i explains the length-wise preferential..." - In general, I believe that the authors could further improve the description of the model, to better explain what each parameter and variable means. - Minor question: I believe that, since \mu and \lambda are capturing complementary effects, they should have opposite signs, right?
2-Confident (read it all; understood it all reasonably well)
The paper presents the Chinese Voting Process (CVP) that models generation of responses and votes, formalizing the evolution of helpfulness under positional and presentational reinforcements. CVP shows significant improvements in predictive probability for helpfulness votes in the critical early stages of a voting process.
The study is well motivated and presented. The voting process presented shows significant improvements in estimating the trajectory in early stages of a voting process. Thus, the results could be interesting for e.g. online services providers that use helpfullness voting (crowd wisdom).
1-Less confident (might not have understood significant parts)
This paper propose a Chinese Vote Process to model the dynamics of voters voting process. The assumption is that sequence of voting events would impact the true helpfulness of user-generated contents. Empirical studies confirm this assumption.
The article is well written, easy to follow with sound theoretical setups. The novelty of this paper is significant as traditional helpfulness prediction did not consider the sequence of votings and this is the first time authors propose the Chinese Voting Process in the helpfulness prediction domain. In addition, the experiment section provides enough results for confirming the superiorty of the proposed model.
2-Confident (read it all; understood it all reasonably well)
The authors propose to incorporate the position and presentation biases in user-responses and develop the Chinese Voting Process. The proposed model is able to reveal helpfulness of a review. Although I am not an expert in Graphical models, the paper seems like a novel extension of the distance dependent CRP referenced in the paper.
I find the paper interesting which answers a relevant problem: How to evaluate the intrinsic score of a review? The ideas and choice of models are well explained and supported by toy examples. However, I am concerned about reproducing the results. Some pseudo-code in the supplement (especially for parameter inference) will make the work more accessible. One concern is the assumption that the entire trajectory is observable in the process. The authors state that noise can confound their estimation. This is very likely in real scenarios where malicious ratings are placed. How is this accounted for in the parameter inference? According to my understanding, the voting model is the key differentiation between CVP and ddCRP. Am I correct? This point should come out more clearly. Minor comments: The variables and parameters are defined in Table 4, after introducing a lot of notations making it very hard to follow the paper. This table should come earlier. The equations are stated directly (eg line 157). Maybe some intuition about the derivation in the supplement may help a non-expert like me to follow it better. Post rebuttal comments: I am not completely satisfied by the author rebuttal , and although I keep my ratings the same , I urge the authors to make their work more accessible by making pseudo code for inference available.
1-Less confident (might not have understood significant parts)