| 1 | Feb 17, 2011 2:02 AM | I think reviewers will always prefer non-anonimity as it is easier to review and it doesn't hurt such a rowd. But the potential aithors and the community at large will by large majority prefer anonimity as hey would like the most objective reviews. So although I'd rather have non-anonimty, I'd vote for what I believe the general commnity needs. I am sure a vote of the crowd in any of our conference will vott in favor of anonimity... |
| 2 | Feb 16, 2011 3:14 PM | I don't think author names should be revealed until after the rebuttal choice. I frequently know personally at least one author of papers I review, and I find that makes it more difficult to be objective. |
| 3 | Feb 16, 2011 2:44 PM | Obviously, it is fairer to authors; moreover, the anonymity is symmetric, which is another way in which it is fairer. |
| 4 | Feb 16, 2011 2:03 PM | As an author, I prefer single blind because I get to give more information such as links to examples and such in papers and I don't have to worry about advertising my work whenever and however I want. (however, I am in the advantageous white, male, from-a-top-institution group).
As a review I may prefer double-blind. |
| 5 | Feb 16, 2011 1:49 PM | Really: Neither of the above, but a more lightweight double-blind process that ensures that reviewer first impressions are blind but that permits PC members to easily find expert reviewers. |
| 6 | Feb 15, 2011 2:29 PM | We seem to have too little CS evidence that this matters, with traces of evidence to the contrary, and it still seems to have significant costs. It's not clear that results from other fields are applicable. If author's names are revealed immediately after the initial review as suggested here, then this becomes a very mild preference. |
| 7 | Feb 15, 2011 2:14 PM | I am not entirely decided about this. It would be interesting to redo this survey after the conference. Here are my thoughts.
I am worried that there will be a lack of thorough reviewing. I am currently subreviewing two papers for LICS; I think I have done a thorough job and it has taken more than a week of my time. I can't afford to review 5 POPL papers with this level of rigour. One solution to this would be to allocate one (or perhaps two) super-expert reviewer per paper, this would be the person who was supposed to report on the correctness of the results with a high degree of confidence.
I was initially worried that DBR would encourage the culture where research is published first, openly discussed second, which certainly leads to lower quality and premature writing. But I think the latest POPL protocol avoids this as much as possible.
I suspect bias appears mainly at two points: firstly in the selection of papers for review, and secondly in selecting between borderline papers. It is not clear to me that the POPL DBR process will eliminate the second form of bias.
I am on the more theoretical end of the spectrum -- perhaps that affects my opinion and my reservations.
(name redacted). |
| 8 | Feb 15, 2011 1:21 PM | I don't think this choice (DBR s SBR) makes much difference because of (1) and (2) below, and DBR only incurs extra cost.
(1) Achieving anonymity is very difficult;
(2) Either a paper is very good/very bad, in which case minor
biases do not affect its fate, or it is borderline, in which case its
fate is determined by subjective biases anyway (such as,
reviewers view about that area, reviewers measure of the
significance/long-term impact, etc). |
| 9 | Feb 15, 2011 1:06 PM | I think it is an interesting experiment to try, though my main worry is that the field is too small for DBR be practical. |
| 10 | Feb 15, 2011 12:40 PM | I am open to DBR, but am so far not convinced. In my experience from both sides of the process, the expertise of reviewers and program committee members, and the depth of reviews, are the most important factors in accurately judging a submission's merit. It seems that great care and effort may be needed with DBR to recover from the loss of flexibility of reviewer choice as compared to SBR. Perhaps with a well-chosen and optimally-sized ERC, this will not be a problem. Relative to the usual number of reviewers (even of at least 2 submissions), the POPL 2012 ERC seems relatively small, if each is hoped to be able to provide expert reviews for their assignment. |
| 11 | Feb 15, 2011 12:08 PM | Double-blind complicates everything without adding value. Some reviewing bias is unavoidable in both cases. As an author, getting feedback by expert reviews on the technical merit of my paper is my main motivation for submitting to a confrence; whether the paper is accepted or rejected is of secondary interest (since there are so many conferences, I don't care much about where my paper eventually appears). |
| 12 | Feb 15, 2011 9:02 AM | I like double blind especially because I can form opinions without letting personal feelings interfere. I have often reviewed papers while thinking what impact I'm having on someone's career (esp. junior people). It's much better to not have this concern at first read. One of the best arguments I've heard *against* double blind is that it makes the process useless for PC members: they put in all the work and still don't get informed about who is doing what in the community. Revealing the authors after submission of the review removes this issue.
At the same time, I suspect that revealing the authors before the discussion also partly negates the anti-bias benefits of double blind. |
| 13 | Feb 15, 2011 9:01 AM | Strongly prefer no unblinding until the very end. |
| 14 | Feb 15, 2011 8:50 AM | I strongly prefer emacs to vi.
:-P
[In case my point is lost in my attempted humor: I do have opinions on DBR vs. SBR, but having heard all the arguments on both sides many, many times, I'm comfortable having the PC Chair make a decision and has reasonable procedures. I do think it's important to have PC members suggest external reviewers that are particularly good matches for a paper.] |
| 15 | Feb 15, 2011 8:29 AM | I don't care that much, honestly. It seems like epsilon more hassle for an epsilon benefit. Basically, if the PC is willing to do the work for the increased perception of fairness, I'm okay with you doing the work. |
| 16 | Feb 15, 2011 8:02 AM | Marginal preference. I found Kathryn's article persuasive, but have had little experience with the negative impacts of DBR. |
| 17 | Feb 15, 2011 7:57 AM | I think the benefits of double-blind reviewing are worth the effort, as long as it is lightweight and doesn't interfere with the possibility of obtaining expert reviews. To me, the motivation for DBR is to incrase the likelihood that every submission is evaluated on a level playing field. If I know the authors' identities, I may be more likely to prioritize papers that I expect to be worthwhile reading, which affects the depth and quality of the review.
I don't think it is reasonable to constrain authors from making presentations, during the review period, etc.
I have had mixed experiences with double blind (one journal article that was required to be blinded even though it was an invited submission for a conference paper, one successful SIGMOD paper, one unsuccessful ICFP and PLDI submission). In all cases I don't have the impression that double-blinding hurt the submission, although in one case the submission was a longer version of a workshop paper and this awkwardness may have counted against it. In several cases is was probably possible to guess my or other authors' identities with a little effort.
I am concerned that the debate about blinding or about how POPL submissions should be evaluated tends to be led by people who who are already satisfied with SBR, and whose concern is to avoid imposing further bureaucracy on an already sometimes awkward system (and perhaps to avoid making changes that might make their lives harder), while not taking into account the possible benefits to our field that could accrue from encouraging reviewers to focus on the content of the paper *first* and making an effort to disregard the authors' identities when forming their opinion of it (consciously or otherwise). Simply put, I think that the PC of a major conference should have the community's interests in mind, not just the interests of its leaders, and I think it is in the community's interests to make the reviewing process as fair as practicable, and reasonably lightweight DBR gets us closer to this goal.
There are already plenty of informal heuristics that one can apply to the content of the paper to determine whether it is suitable for POPL. The names of the authors should not be in the mix. |
| 18 | Feb 15, 2011 7:53 AM | I don't think there is enough evidence to decide whether DBR reduces bias in reviews of CS conference papers. But, given evidence of its effectiveness in other scenarios, and since I haven't seen convincing evidence of DBR being harmful, I think DBR is preferable to single-blind reviewing to further avoid perception of bias or an unfair system. The situation is somewhat analogous to avoiding PC submissions, though perhaps not as clear-cut. |
| 19 | Feb 15, 2011 7:51 AM | I'm on the fence really. I'm open to double blind, in the light-touch version, but it's a pain and I'm not convinced the pain is worth it. |
| 20 | Feb 15, 2011 7:41 AM | Double-blind is too burdensome on authors and hinders their free and open dissemination of their research. The evidence given for double-blind is highly inconclusive. Furthermore, I don't think the main problem with conference reviews is bias, I think it's sloppiness. Recruiting more expert reviews (as was done for POPL'10) is the best way of addressing that problem. |