| 1 | Nov 1, 2011 2:53 PM | I am very close to a toss-up for this. I used to be a strong advocate of SBR, but I am gradually seeing more advantages to DBR. On balance, I still think DBR as implemented this year is more trouble than it's worth, but with a little more mitigation of negative factors I think I could be convinced. |
| 2 | Nov 1, 2011 1:43 PM | Reviewing the papers without knowledge of the authors is obviously fairer. |
| 3 | Nov 1, 2011 8:34 AM | Personally, I don't feel strongly about double-blind vs. single-blind. In the abstract, DBR seems more fair, but I'm not sure if this is really the case in practice. Also, DBR seems to make the reviewing process significantly more complicated. |
| 4 | Nov 1, 2011 7:45 AM | Perhaps DBR ok for POPL.
However, for example (paper identifier redacted) the authors were clearly identifiable. |
| 5 | Oct 30, 2011 8:33 AM | Double-blind reviewing at POPL'12 has had one unexpected consequence for me: with a list of all potential reviewers public, it was easier than usual for me, as an author, to identify the reviewers assigned to my paper. This should be seen as a disadvantage. |
| 6 | Oct 28, 2011 6:08 AM | I think DBR is fine (it did help me focus on the content not the reputations in reviewing). Oen suggestion is to simplify the process for submitting supplementary material - it is too confusing for both authors and reviewers. |
| 7 | Oct 26, 2011 1:24 PM | As the author of (paper number redacted), I thought it is better if a submission was reviewed by expert reviewers (no experts in our reviews). However as a [reviewer], DBR was good. |
| 8 | Oct 25, 2011 3:41 PM | I did not think it could happen, but, recently even, I have heard the phrase "so-and-so does good work" as an argument to accept a paper in a program committee of a major PL conference. Knowing the authors definitely influences the reviewers, and this is exactly the wrong kind of influence. |
| 9 | Oct 25, 2011 9:19 AM | Doesn't matter. Most of the time reviewers can correctly guess the authors. The key problem is that the reviewers are
not held accoutable for their reviews. The only fair process would be if one could get reviewers to sign their reviews because that would indicate that they had spent enough time
reviewing/understanding the contribution and are confident
enough to stand by their reviews instead of acting
like snipers. |
| 10 | Oct 24, 2011 4:08 AM | Because you can have your work judged on its quality, not by who you are or are not friends with. |
| 11 | Oct 24, 2011 3:10 AM | I don't have a strong preference. Our submission this year was building on our past work. It felt awkward to refer to our past publications anonymously. Being able to include supplemental, non-anonymous additional material was important; the process this year allowed for that nicely. |
| 12 | Oct 24, 2011 3:03 AM | I received more balanced reviews than in the past. I believe some reviewers are less tolerant of certain new ideas coming from a single author PhD student. |
| 13 | Oct 24, 2011 2:48 AM | DBR has modest advantages, but overall I don't think the benefits justify its costs |
| 14 | Oct 22, 2011 7:31 PM | We need academic integrity. I am doubtful it can be engineering through these processes. Reviewers should sign their reviews. |
| 15 | Oct 22, 2011 4:25 PM | Level playing field for authors. |
| 16 | Oct 22, 2011 2:32 PM | I like thinly veiled double blind, where author names are removed but authors don't have to take many awkward measures to avoid leaking their identity. |
| 17 | Oct 22, 2011 1:57 PM | I have anecdotal evidence that supports that reviewers are unfairly influenced, often consciously, by the identity of authors or their institution. Double blind reviewing thus seems most fair. |
| 18 | Oct 22, 2011 8:01 AM | I'm not sure I prefer one to the other. |
| 19 | Oct 22, 2011 5:03 AM | Conversations with PC and EPC members indicated that had they known the authors of some papers prior to their first review, they would have viewed them more favorably. |
| 20 | Oct 22, 2011 5:01 AM | Sorry, it is my first paper that I ever submitted so I cannot give a good answer about this. |
| 21 | Oct 21, 2011 9:04 PM | I felt our paper was significantly hurt by our inability to inform reviewers of the constellation of supporting work we have done (and others have done atop our work). As such it was viewed as a purely theoretical submission and we were unable to adequately explain the extensive empirical work that led to our designs. |
| 22 | Oct 21, 2011 6:02 PM | The double-blind is partial, it should be double-blind until the end of the selection process since otherwise, as observed, referees ask a question, claim they are not satisfied by the answer and downgrade the mark once the authors are known |
| 23 | Oct 21, 2011 3:56 PM | I find it cumbersome to make a paper suitable for double-blind review. If there is very strong evidence of bias, then I guess I support it. Otherwise, I find it rather awkward. |
| 24 | Oct 21, 2011 3:45 PM | I don't believe any type of blindness (in reviewing or otherwise) is a good think. As a corollary, double-blind is worse than single-blind. |
| 25 | Oct 21, 2011 3:35 PM | I don't feel that either is clearly superior to the other. The qualification for my choice (DBR is better) is that the double-blinding actually works. See the response to Q2 below. |
| 26 | Oct 21, 2011 3:09 PM | This is my first POPL submission, so I can't compare it to previous POPL submissions, but I like the idea of double-blind reviewing and I thought it was implemented well here. It did not inconvenience my writing much. |
| 27 | Oct 21, 2011 2:57 PM | Can have honest reviews. However, since each programming language subfields are not very big, I expect that reviewer may easily recognize author in most case. |
| 28 | Oct 21, 2011 2:43 PM | Actually I don't feel a difference from my previous experience. |
| 29 | Oct 21, 2011 2:30 PM | We really should use the inverse "single-blind" review. One of our reviews was topical, anecdotal and submitted last minute. |
| 30 | Oct 21, 2011 2:15 PM | I don't have a strong opinion about this, but double-blind reviewing might help increase impartiality in some cases. The downside is you can't provide context in terms of your own past experience; instead you have to say artificial things like "the people working on X have informed us that..." when you are in fact the people working on X. Maybe the extra impartiality outweighs this downside; I just don't know. I would say the downside is significant if it hinders what you want to communicate in the paper, rather than just forcing you to say more awkwardly; but that hasn't been my experience (this year or at other conferences). |
| 31 | Oct 21, 2011 2:13 PM | DBR should be enforced to make sure a more objective evaluation. |
| 32 | Oct 21, 2011 2:11 PM | This makes it easier for reviewers to assess novelty vs prior work of the authors |
| 33 | Oct 18, 2011 3:21 PM | I don't have a strong opinion either way. My preference for single blind is that I have had reviewers complain about the way we tried to anonymize the paper. For me double blind reviewing sets up another (largely irrelevant) reviewer target that focuses on the presentation of the paper rather than the substance. I'd like to see reviewers focus more on the content of the paper. |
| 34 | Oct 18, 2011 9:48 AM | What was implemented in POPL this year was the worst of both worlds: it prevented me from taking credit for prior work and product impact, while simultaneously revealing my identity to reviewers before the review cycle was done. |
| 35 | Oct 18, 2011 8:16 AM | The ERC bench was not deep enough; the reviews were not as good as last year . |
| 36 | Oct 17, 2011 9:45 PM | I do not think DBR has improved the reviews very much (there were _way_ too many major factual/technical errors, and our rebuttals were simply ignored, or, worse, one reviewer got offended and responded with even less civilized comments!), but it still seems better than nothing. |
| 37 | Oct 17, 2011 5:20 PM | Double-blind reviewing is a waste of time. In my experience, it's always the case that you know who the authors are. A quick Google search normally reveals it. I would prefer no-blind reviewing, where all reviewers are known to authors and vice-versa. |
| 38 | Oct 17, 2011 4:01 PM | I think DBR tends to be more fair than SBR. The cost of having to anonymize papers is fairly low. |
| 39 | Oct 17, 2011 3:56 PM | It was well done this year, but I don't see much of an advantage in DBR, anyway. |
| 40 | Oct 17, 2011 3:30 PM | I think double-blind reviewing is a good idea, but this is my first paper as lead author at a major conference so I don't have much to compare against. I will say that in our case it would have been useful for the reviewers to know that we wrote a previous, related system. |
| 41 | Oct 17, 2011 1:51 PM | The problem with double-blind reviewing is the lack of expert review. One of my submission had 5 reviews in total, and 3 of them did not even understand the basics (we could see mistakes in their summaries) since they were from a totally different domain. For that paper, two reviews were of any quality, even if they failed to understand half of the paper. |
| 42 | Oct 17, 2011 1:28 PM | Double blind reviewing is the fairest and least biased way to evaluate papers. In controlled comparisons, papers accepted with DBR get cited at a higher rate. |
| 43 | Oct 17, 2011 1:22 PM | Lots of work builds directly on previous work by the same authors and it is difficult to write about this without resorting to verbal contortions with DBR. I do think DBR has some advantages wrt the integrity of the reviewing process, but I think the first point outweighs this. |
| 44 | Oct 17, 2011 12:49 PM | I do not see the advantage of double blind, and single blind is easier. |
| 45 | Oct 17, 2011 11:35 AM | I found the reviews this year were very fair and I had no issue with the rejection. I attribute this (at least to some degree) to DBR. |
| 46 | Oct 17, 2011 11:06 AM | With traditional single-blind reviewing, reviewers check out the credentials of the authors. This is unfair to those authors who are outsiders or who are not (yet) established as top researchers. |
| 47 | Oct 17, 2011 10:26 AM | As a reviewer, a prefer double-blind. As an author, I prefer single-blind because I would prefer to feel unconstrained about promoting my work while it is under submission. Also, when writing a double-blind paper, I usually feel angst about how to refer to my own related work. I know I'm not supposed to change the way I write (except to use the 3rd person), but I often feel compelled to in one way or another anyway or else I'll wind up completely breaking the spirit of the anonymity rules. |
| 48 | Oct 17, 2011 9:57 AM | This issue cannot be resolved by an opinion poll. |
| 49 | Oct 17, 2011 9:09 AM | The merit of double-blind seems much smaller than the demerits.
Double-blind does not really hide the identity of authors; reviewers can often guess authors from the subject of the paper and the methods they use.
Demerits are numerous. First, reviewers are restricted to PC and ERC members, which represent only a fraction of the community, and are possibly biased (as ERC are also selected by PC). Second, double blind prevents authors from advertising their results until the review process completes. |
| 50 | Oct 17, 2011 9:04 AM | I believe DBR helps keep the focus where it should be during reviewing. The process at POPL'12, in which author names were shown after reviews were submitted, seemed to nicely counteract the normal, in my opinion minor, disadvantages of DBR (i.e., authors have no "shame" incentive to submit good work, it can be hard to check for plagiarism without author names, etc). |
| 51 | Oct 17, 2011 8:58 AM | My opinion is that DBR has the potential to be more fair, but, in my experience it appeared to make no difference. |
| 52 | Oct 17, 2011 8:56 AM | In principle, I support double-blind reviewing. However, I am not sure that its implementation at POPL'12 was effective. In particular, there seemed to be a considerable disparity between the (enthusiastic) blind initial reviews my paper received and the eventual outcome (rejection) after the reviewers were unblinded. I do not know if learning who the authors were contributed to the change of heart of the reviewers, but if it did then I think it would have been better if they had known from the outset. |
| 53 | Oct 17, 2011 8:54 AM | I prefer DBR, but with one major exception, which is I don't understand how DBR is compatible with the open source philosophy of "release early, release often". I think the policy should explicitly allow papers where the software has been publicly released, even if this reveals the author identities. |
| 54 | Oct 17, 2011 8:22 AM | I read a lot about the alleged benefits of DBR. Assuming the benefits are indeed positive, I still think they are small compared with the considerable extra work DBR imposes on the PC and reviewing process. It feels like diminishing returns. PCs should also strive for efficiency. |
| 55 | Oct 17, 2011 8:19 AM | It adds credibility to the review process. As a side effect, it probably forces authors to make submissions more self-contained. |
| 56 | Oct 17, 2011 8:12 AM | DBR imposes extra work at several stages in the process that would be better spent on real work.
It has no positive impact on the review process as far as I can tell. (This is based on more than POPL.) |
| 57 | Oct 17, 2011 8:12 AM | Usually is never too difficult to individuate authors. Hence, the blind review mechanisms is still another fig leaf under which reviewers may hide to commit their worst wickedness. We should do the contrary: singned reviews: this would be serious. |
| 58 | Oct 17, 2011 8:07 AM | I'm not sure the anonymity made that much difference to the overall judgement. |
| 59 | Oct 17, 2011 8:00 AM | I don't really mean that - I would prefer "lightweight double-blind", i.e. just like single-blind but with author names held anonymous for bidding and not written at the top of the first page. |
| 60 | Oct 17, 2011 7:53 AM | Reviewers typically know anyway, subject is not as broad and contentious as others, eg perhaps in medicine or biology so that perceived adv of double-blind might not be as big. |
| 61 | Oct 17, 2011 7:53 AM | 1. Openness: Authors should not be made to feel inhibited about freely talking about or disseminating their submitted work.
2. Fairness: For many papers, the reviewers will for one reason or another know or be able to guess who the authors are. So it is patently untrue that DBR affords every paper an unbiased look. At least with SBR, all reviewers have the same information about each paper.
3. Appendices: The treatment of appendices this year was counterproductive. As a reviewer, I was forced to submit stub reviews for nearly every paper, just so I could get to the appendix, which I frequently want to look at while I'm reviewing the paper, in order to form a clear opinion.
There are many more reasons, which I will be glad to discuss in person. |
| 62 | Oct 17, 2011 7:52 AM | At least for my paper the blinding did not work since it relies on own previous work. |
| 63 | Oct 17, 2011 7:51 AM | If anonymization is light-weight (as in POPL '12), I don't see any reason for having author names on papers. |
| 64 | Oct 17, 2011 7:49 AM | Previously, I could email some external reviewers working in the related area to clarify whether my description of their work is fair or correct. Now, I cannot email any of them. Some reviewers misunderstood some parts describing related work. |
| 65 | Oct 17, 2011 7:47 AM | My work requires an experimental infrastructure,
from which it is easy to derive where the paper is from. |
| 66 | Oct 17, 2011 7:42 AM | Writing a pearl about an existing system was not fit for double-blind reviewing. |
| 67 | Oct 17, 2011 7:41 AM | I don't really care.. |
| 68 | Oct 17, 2011 7:41 AM | In many cases, the reviewer can easily identify the authors. |