Flooded with submissions, several SE conferences went for a two-phase model to reduce workload: A paper first gets two reviews, and only if at least one of these favors acceptance, the paper gets a third review. As PC chairs of ASE 2013, we went back to a single-phase model, though. Several people have asked us why this was so. The answer is: time. With a one-phase review, you can cut the time required for reviews in half, thus allowing for more, better, and more recent papers.
The math is as follows. Let's assume your conference gets 314 papers submitted (as we had for ASE):
- In a two-phase model,
- 314 papers would have gotten two reviews, totaling 618 reviews.
- 1/3 of papers are eliminated in the first round, leaving 210 which get a third review.
- Total number of reviews: 618 + 210 = 828.
- In a one-phase model,
- 314 papers get three reviews, totaling 942.
By having the one-phase model, your conference will thus have 14% more reviews – in our case, 22–23 papers to review per PC member instead of 20. However, going for two phases requires a lot more administrative overhead – for the PC chairs, of course, but also for the reviewers. In particular, you need to allocate two phases in which to do the reviews – instead of, say, eight weeks, where reviewers would be free to allocate their load, you now have six + four weeks, each of which may be more easily blocked by travel, holidays, etc. Plus, as a PC chair, you need at least another week in between to reallocate and inform the new reviewers.
All in all, you thus trade 14% more reviews against an extension of the review period by three weeks. Not necessarily a good thing. In addition, this assumes that papers are all equal; but they’re not: The 14% extra papers are mostly papers that would be quick rejections anyway. So, not much damage either.
Assuming that more than a third of papers may be rejected in the first round favors the two-phase model. But then, there’s the concern that by having only two reviewers, you run a greater risk of one reviewer being incompetent, unwilling, lazy, etc. (It happens to anyone.) So you may be have to introduce rebuttals to compensate for potential misjudgments, and here go another two weeks of reviewing period. And then, someone will have to read these rebuttals, which adds more overhead.
To illustrate how two-phase reviews and rebuttals eat up time in contrast to one-phase reviews, let's take a look at ICSE 2014, whose papers are due September 13, 2013. Notification is January 17 – that is, more than four months later. For ASE (one phase, no rebuttals), this was May 17 to July 25 – two months and one week, or roughly half the time ICSE takes.
With its short reviewing period, ASE 2013 had more than three times the number of submissions compared to last year, which gave us a great choice of papers. ICSE will also have a record number of submissions; but if you miss the deadline, chances are you will find another conference which will publish your paper well before ICSE.
Summary: Get rid of frills, and allow authors six more weeks to prepare their submissions. You will get more, better, and more recent papers – and more, better, independent reviews at the same time.
Andreas, very interesting post. I agree with the points about time (your math is of course accurate).
ReplyDeleteI would like to add a small note on the quality of reviews (you are talking of getting better reviews), irrespective of the reviewing model. The key issue, IMHO, is that the task of reviewing isn't typically taken into account by employers as part of the daily duties of a researcher.
I've often heard sentences such as "I didn't do much progress this week, I had to do reviewing for conference/journal XYZ". It seems that we (as researchers) apologize for "wasting" our time doing reviews.
Changing this attitude is worth more than inventing new reviewing models... I admit this may be the utopia of a 'young' researcher
I think, the argument for one-phase review is very valid.
ReplyDeleteI am only wondering regarding the >20 papers review approach. In my opinion and experience this actually invites running the risk of inadequate reviews. (Even if distributed out over time, with many things going on - e.g., other conferences - this is a extremely high load.)
To me, this means, that the PC is too smal..
Of course, people often solve it in such cases by sub-distributing the reviews, but I think, this is not the right way.
Of course. As a PC chair, one should always strive to size the PC such that the load is bearable for all. If the review times were shorter, one would factor this in, and limit not only the absolute load, but also the number of reviews per week. Fortunately, ASE 2013 with a 300% increase in submissions is the exception.
Delete