How to Be an Area Chair for CVPR

How to Be an Area Chair for CVPR Based on material by ICCV 2019 / CVPR 2019 Program Chairs

Guiding principles • We want to make the best decisions we can to serve the community. • We want these decisions to be transparent to the authors. While an author may not be happy with a decision, he/she should understand why the decision was made. • We want reasonable consistency across area chairs. • Much of your work will be routine administration; you're serving the community, and you might be dealing with relatively few papers whose fate you can change. HOWEVER: those are the most important cases, where your contribution is extremely valuable.

The Decision Process: Overview Program Chairs 7. Area chairs discuss with reviewers and each other, make accept/reject decisions and oral recommendations 1. PCs assign papers to ACs 2. Primary AC suggests reviewers 3. Papers are assigned to reviewers using global matching algorithm 8. Program chairs finalize oral decisions based on space/time constraints Primary Area Chair 6. Reviewers update final reviews Reviewers Secondary Area Chair 5. Authors provide rebuttal to reviewers and ACs Authors 4. Reviewers write reviews, which are released to authors (after AC checking for quality)

The Decision Process: In Detail 1. 2. 3. 4. 5. 6. 7. 8. Program chairs (PCs) assign papers to area chairs (ACs), usually not more than 35 papers per AC. ACs suggest 10 reviewers per paper, with help from Toronto Paper Matching Software (TPMS). Papers are assigned to reviewers (3 per paper) using an optimization algorithm that takes into account AC suggestions, paper load and conflict constraints, and the prayers of PCs that nothing goes wrong. Reviewers submit initial reviews, typically handling 6 -10 papers each. ACs check quality of reviews, chase late reviewers, and assign emergency reviewers as necessary. Authors receive reviews with a mixture of gasps, grimaces, grumbles, and the occasional grin. After much thought and re-reading of paper and reviews, authors submit rebuttals. Discussion ensues among reviewers and AC, based on all reviews, rebuttal, and paper. Reviewers update their ratings and justification. ACs make decisions and write meta-reviews. The decision and meta-review are recorded by the primary AC for each paper and checked/approved by the secondary AC. Primary and secondary ACs discuss borderline papers. Additional opinions may be sought from other expert ACs after checking for conflicts. In addition to accept/reject decisions, AC pairs provide a roughly ranked list of oral/spotlight nominations to the PCs make final determination of poster/spotlight/oral for accepted papers, almost entirely based on the recommendations of the ACs but taking into account time and space constraints and topic diversity.

Three phases of AC duties 1. Assign papers to reviewers • • In some ways, this is actually the most important step. If done well, the rest of the process is a piece of cake. If not, it’s a disaster. Do not rely solely on TPMS and topic matching. You are smarter than a weighted histogram matcher, right? The automated tools provide initial rankings, but ultimately you should rely on your judgment. 2. Guide the review/discussion process • • • Check that submitted reviews are complete and respectful. Catching bad reviews early can save you (and the authors) a lot of pain later. Post-review, start discussions where appropriate to clarify any conflicting opinions. Get reviewers to update their final ratings – especially to reconsider their positions once they have more context (rebuttal, other reviews) and time to reflect. 3. Decide, write meta-reviews; check decisions and meta-reviews of papers for which you are secondary meta-reviewer

How to choose reviewers 1. 2. 3. Spend a few minutes reading the abstract / intro / figures / citations to understand the key idea, main technical topics, and community involved. While reading the paper, jot down names of reviewers that come to mind. Look through the list of references for more ideas. Finally, look at top-ranked reviewers according to TPMS scores. Suggest a mix of 10 potential reviewers who have expertise in each main technical topic, are likely to enjoy reading the paper, and can give feedback on the big ideas as well as technical details. Pro tips: • • Do not trust TPMS blindly; many of its suggestions are bad. Verify the suggestions by checking the reviewers’ Google Scholar profiles. Some reviewers may not show up in the TPMS rankings because they did not create a TPMS profile. If you think of somebody who may be good to review a paper, search for their name directly.

Level of decisions • Award: major advances that will heavily impact the field; will be used by many people, create new capabilities, etc. • E. g. , Res. Net (CVPR 2016 Best Paper), Mask R-CNN (ICCV 2017 Best Paper) • Oral: potential to be very significant; worthwhile for the whole community to hear about • Poster: incremental steps that expand the sum of the community’s knowledge or add bricks to the cathedral of knowledge; papers introducing useful tools; papers of interest to a subcommunity • Also, creative ideas that are hard to judge but could be promising -- no one knows the future, so we should give the benefit of the doubt to plausible ideas • Reject: unlikely to be significant • Your job as an AC is mainly to decide acceptance/rejection and to provide PCs with recommendations for orals and awards

Why not accept everything? Papers can have a negative impact: ● Wrong or fraudulent results mislead the field and damage the reputation of the conference. ● Misleading evaluation makes it hard to compare with, kills follow-up. ● Creates bad precedent (weak paper X got in, so this one should too). ● Fatigue/overload of too many papers, wastes everyone’s time. Each weak or mediocre paper we accept hurts the conference a little (though not as much as rejecting a good paper).

The role of experimental evaluation “Extraordinary claims require extraordinary evidence. ” — Pierre Simon-Laplace, paraphrased Different papers require different levels of evaluation, for example: ● Potentially transformative idea: basic proof-of-concept ● Established problem, plausible idea: benchmark results ● Weird, overly complex, implausible, and/or seemingly incremental: extraordinary results (which need to be scrutinized carefully) ● Position piece or theory paper: no experiments Having a good idea is not enough to have a paper. Ideas are cheap.

How to decide • Judge each paper based on its own merits, not some ranking with respect to other submissions to the conference. There is no global quota on the number of papers we can accept. • For unanimous accepts or rejects, just follow the reviewers (but do a sanity check in case the reviews are completely off or missed something). • For all other cases: Read reviews, see if you agree with claims. • Which arguments did the reviewers make that you find compelling? • Pay more attention to the arguments than the ratings! • If you aren’t sure: get help from secondary meta-reviewer or another expert AC of your choice (contact the Program Chairs first to check for conflicts).

How to write meta-reviews • A meta-review is a well-founded argument that a paper does/doesn’t meet the criteria for acceptance. It is not simply a summary of the individual reviews. Especially in the case of difficult “reject” decisions, you must reconcile the reviews and give explicit justification for the final recommendation. • Key principle: would a reasonable author object to the meta-review as the basis for the decision? • Use impersonal language: AC, ACs, authors -- not “I”, “you”. • Take care when reviewers don’t agree and especially if your decision goes against the majority. • Secondary meta-reviewer must proofread every meta-review to make sure it meets standards.

Meta-reviews for easy cases • When all reviewers agree, a couple of sentences will suffice. • In cases of acceptance, do not mention oral vs. poster recommendation in your meta-review since this status will not be finalized until later by the PCs. • In cases of rejection, briefly mention the main reasons for rejection and acknowledge the rebuttal (if only to say that it was missing or did not change anybody’s mind). • Examples: • “Three reviewers agree that the paper contains technical errors and is not clearly written. There is no rebuttal. ” • “Three reviewers find that the paper is below ICCV standards due to a lack of novelty and insufficient experimental validation. The rebuttal did not assuage these concerns. ”

Meta-reviews for difficult cases If referees disagree or make borderline recommendations: 1. Summarize the main arguments of the reviewers. 2. Indicate whether the rebuttal removed any initial concerns (if there was no rebuttal, explicitly state “there was no rebuttal”). 3. Summarize any substantive points from the reviewer discussion (authors will not see the discussion). 4. Reconcile all the above information and explain the main factors for your decision. 5. Discuss your decision with the secondary meta-reviewer. Include “This paper was discussed by two ACs” or similar language.

Good meta-review Initially, R 1 and R 2 liked the main idea of the paper and gave it a positive recommendation, but R 3, who wrote the most in-depth review, found that the experimental results were insufficient. In particular, all the experiments in the paper are on one small dataset [anon]. In the rebuttal, the authors responded that there is no ground truth available for other datasets. But in the subsequent discussion, R 3 pointed out that other datasets provide ground truth, e. g. [anon] and [anon], and also that the comparisons given in the paper contradict the reported results in [anon]. R 1 and R 2 acknowledged the validity of these concerns, but kept the same rating. The AC finds the arguments of R 3 to be the most persuasive. To properly validate their approach, the authors need to address the discrepancies with prior work and evaluate on the datasets pointed out by R 3. This decision was discussed with and approved by the secondary meta-reviewer. 1. Summarize the main arguments of the reviewers 2. Indicate whether the rebuttal removed any of the original concerns 3. Summarize any substantive points from the reviewer discussion 4. Reconcile the above information and explain the main factors for your decision 5. Discuss your decision with the secondary metareviewer

Not good enough “R 1 liked the paper for reason X. R 2 disliked aspect Y. R 3 disliked aspect Z. Therefore the paper was rejected. ” • Good start, but you need to explicitly reconcile three reviews. “Reviewers all say that they are OK if the paper is rejected, and we have many good papers. ” • Really bad, reads as though the decision on this paper depended on decisions made on other papers, or as though the AC pressured the reviewers to reject.

Overruling reviewers • If all reviewers agree that a paper should be rejected, we expect the paper to be accepted ONLY if there are unusual circumstances, e. g. , three exceptionally poor or misguided reviews. • If all reviewers agree that a paper should be accepted we expect the paper to be rejected ONLY if there are unusual circumstances, e. g. , a major technical error, fraud, or plagiarism that are not detected by reviewers. • If you wish to overrule a unanimous reviewer recommendation: • Secondary meta-reviewer must agree with decision. • You must contact the PCs for approval.

Main take-aways 1. Invest your time in picking the right reviewers for each paper. Don’t rely solely on the paper matching tools. 2. Make a well-founded decision for each paper, considering the substantive points raised by the reviewers and authors. Don’t rely solely on reviewer ratings. 3. Write a meta-review that fully justifies the decision to the authors. They should understand the reasons for the decision even if they don’t like it. 4. Always be respectful to the authors and reviewers, who put much time and trust into the process. And complete your tasks on time!