The new ICSE Erdős penalty, or why we should create incentives for frequent reviewers

Ever heard of Paul Erdős?  The 20th century Hungarian mathematician is not only known for his numerous contributions to Mathematics, but also for his multiple collaborations, engaging more than 500 collaborators.  Frequently, he would just show up on their doorstep, work with them for some hours, and then get a joint paper out of that.  A low Erdős number indicates academic closeness to Erdős, and is something one can brag about at academic venues.  Yet, if today, Paul Erdős knocked on your door, and asked whether you would like to work with him, you should avoid any collaboration with him – if you work in Software Engineering, that is.  Why is that?

The International Conference on Software Engineering, or ICSE for short, is the flagship conference of the field of Software Engineering.  If you want to publish and present your greatest work, this is where you submit it.  An ICSE submission is reviewed by three peer researchers, whose assessment eventually determines whether your work is accepted or not.  Even if your work gets rejected, you at least get detailed reviews and high quality feedback.

Over the years, ICSE has observed that there were authors who apparently were way more interested in the reviews than in getting their papers accepted; authors who would submit up to ten papers, of which none got in; but the authors would at least get thirty reviews, all for free.  This motivated ICSE to install a new limitation: Any single author can now appear only on up to three papers.  If you have four papers ready for submission, then you are supposed to select the best three.

The ICSE program chairs argue that few authors and even fewer acceptances would be affected by this decision.  But the problem with this decision is not the factual impact.  It is the potential impact.  What if a modern Paul Erdős knocked on your door and offered you to work with him?  You'd have to say no, because he would have too many co-authored submissions already.  What if you could not submit to the past conference, because you were the one organizing it, and still have work that wants to get published?  What if four of your students all have great results at the same time, results that should be shouted out to the world?  Well, too bad: you can only submit three of them, causing depression in the fourth student how is left out.  None of these is likely to happen, but the fact that it could happen is causing concerns and anxiety, and rightly so.  An open petition asking ICSE to revert its new rules has gained dozens of supporters overnight. (Disclaimer: me too.)

The Software Engineering community has members who have literally devoted their lives to Software Engineering research.  They have no spouses, no kids, they work day and night.  The boys would be out for a skiing weekend, and the girls would be out in their summer clothes – these folks are busy on the paper that they hope will make them famous.  They serve on program committees, they write reviews, they organize conferences, they help others on their PhD theses.  They are amazingly productive both on their own work as well as on the work of others.  And these are the men and women whom the new ICSE rules send the message: Thank you, but no, thank you.

The problem of ICSE – and our community in general – is not so much an abundance of papers.  It is the lack of reviews.  It is our publications that determine our academic worth; much less so teaching; and even less so service. Great papers get you tenure and a raise, whereas great reviewing might get you a committee dinner.  Rationally thinking, why should one spend time on reviews while one might just write papers that would get reviewed by others?  Fortunately, the large majority of our community is still driven by the Categorical Imperative: We profit from the reviews of others, so we review their papers, too.  What we don't like is members who game the system by not only submitting lots of papers, but also not participating in the review process.

Therefore, what our community needs to do is twofold.  First, we need to think about reviewing processes that scale well and get high-quality reviews.  The ICSE program board model is a step in the right direction; a VLDB-like journal model might be even better.  Second, we should not penalize researchers for their own productivity; but instead create incentives for researchers who spend great effort on reviews and service.  Rule by the carrot, not by the stick.

Such incentives for service should not be monetary (these wouldn't motivate researchers anyway); nor should they result in a different reviewing or acceptance process (this would be perceived as unfair).  But how about raising the limit of submissions if you have a co-author who is also a frequent reviewer?  Or allowing reviewing volunteers to apply for a one-day extension to the conference deadline?  (You'd get plenty of applications on the last day :-)  Or provide "fast track" journal reviewing for those authors who sport a status of "distinguished reviewer"?  With such incentives, if a prolific reviewer like Paul Erdős knocks on your door, you would not boot him, but embrace him instead.