Learning Conference Reviewer Assignments Adith Swaminathan Guide Prof

Learning Conference Reviewer Assignments Adith Swaminathan Guide : Prof. Soumen Chakrabarti Department of Computer Science and Engineering, Indian Institute of Technology, Bombay

Future Work (from BTP 1) Given WWW 2010’s assignments, learn Affinity_Param, Topic_Param and Irritation l Citations as edge features l Load-Constrained Partial Assignments l Better estimation of Assignment Quality l

Background l Conference Reviewer-Paper Assignment as a Many-Manymatching [1] l Minimum (MCF) Cost Network Flow

Conference Reviewer Assignment l Set of Reviewers, R, max #papers = L_i Set of Papers, P, min #reviews = K l Assumption : Only require #reviews, not quality l Suppose we have cost function A_ij(y) for <R_i, P_j> l

ILP -> Assumption -> MCF

Two problems l Integer Linear Programs are NP-Hard! – – l Relax? More assumptions? How to determine A_ij? – – M * N ~ 10000 Multimodal clues

ILP l Enforce structure on A_ij – – l -> Assumption -> MCF Better model multimodality Fewer parameters to fix “Learn” A_ij using Structured Learning Techniques l. A_ij = T w Φ(R_i, P_j, y_ij)

Ramifications of Structured Costs l Costs decompose over <R_i, P_j> pairs – – l Decomposable Preference Auction Polynomial Algorithms for DPAs [2] Restricted notion of optimality – – Per-reviewer/Per-paper constraint could be combinatorial Stability?

ILP -> Assumption -> MCF

Minimum Cost Network Flow Directed graph G=(V, E), capacities u(E)>= 0, costs c(E) l Nodes have numbers b(V) : Sum(b(V)) =0 l Task : Find a function f: E->R+ which satisfies the b-flow at minimum cost l Successive Shortest Path Algorithm l

Node features and Edge features Profile Affinity Contents s Cite Reviewer Bid Paper Topics Topic Overlap Topics

The Loss Function l l l L_ij = w_1 * exp(-Affinity_ij) + w_2 * [[1 – Topic_Overlap_ij]] + w_3 * Bid_Cost = Potential(R_i, P_j, y_ij) Irritation (I) and Disappointment (D) needs to be set

Assignment Quality Measures Number of Bids Violated? – Not a reliable measure. l +ve Bids Violated l –ve Bids Violated l Assignments satisfying Topic Match l Confidence? l

Confidence == Quality? l Very sparse Fewer than 5% observed – Extrapolated Confidence? – l Reliable Bids as a precursor of Confidence [3] – Confidence-Augmented Loss? –

Learning w’s l Transductive Ordinal Regression – – l Assume : Assignments are independent (Naïve) Heuristic : Augment observed dataset Extrapolate observed Confidence [4] Learn w over extrapolated dataset Support Vector Machine for Structured Outputs – – – Cast as soft-margin SVM formulation [5] Upper-bound objective with a convex fn (Optimality? ) Minimize, using Cutting Plane (Approximate)

Transductive Ordinal Regression [6]

SVM Struct. [7] Loss Augmented Inference ~ Most Violated Constraint Loss is decomposable -> Modified MCF

PARA : Paper Assignment to Reviewers Apparatus

Results

Bimodal Behaviour • Reviewer either gets few or L_i papers • Load Penalties [8] • Introduce more parameters • Infer using modified MCF • Learning parameters? • Load Rebalancing • Tradeoff between MCF optimum and old assignment

Penalise Reviewer Loads

Load Constrained Assignments

Avenues for Future Work • Document Modelling for Affinity Scores • Objective Assignment Evaluation • Transitive Citation Scores • Load Penalty Parameter Estimation

References 1. 2. 3. 4. The Conference Paper Assignment Problem, J. Goldsmith, R. H. Sloan, 2007 Multi. Agent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Y. Shoham, K. Leyton. Brown, 2009 Automating the Assignment of Submitted Manuscripts to Reviewers, S. T. Dumais, J. Nielson, 1992 Semisupervised Regression with cotraining algorithms, Z. Zhou, M. Li, 2007

References – contd. 5. Learning structured prediction models : A Large Margin Approach, B. Taskar, et al, 2005 6. Ologit : Ordinal Logistic Regression for Zelig, G. King, et al, 2007 7. SVM Learning for Interdependant and Structured Output Spaces, I. Tsochantaridis, et al, 2004 8. Word Alignment via Quadratic Assignment, S. Lacoste-Julien, et al, 2006