Network Lasso Clustering and Optimization in Large Graphs

Network Lasso: Clustering and Optimization in Large Graphs David Hallac, Jure Leskovec, Stephen Boyd Stanford University Presented by Yu Zhao

What is this paper about Lasso problem The lasso solution is unique when rank(X) = p, because the criterion is strictly convex.

What is this paper about Network lasso problem The variables are scalar variables is mp. ) Here is the cost function at node i, and associated with edge. , where. (The total number of is the variable at node i, is the cost function

Outline Convex problem definition Proposed solution(ADMM) Non-convex extension Experiments

Convex problem definition (1) (2)

Convex problem definition A distributed and scalable method was developed for solving the network lasso problem, in which each vertex variable xi is controlled by one “agent”, and the agents exchange (small) messages over the graph to solve the problem iteratively.

Convex problem definition General settings for different applications e. g. Control system: � Nodes: possible states � xi: actions to take when state i � Graph: state transitions � Weights: how much we care about the actions in neighboring states differing

Convex problem definition General settings for different applications The sum-of-norms regularization that we use is like group lasso, which encourages not just , for edge , but , consensus across the edge.

Convex problem definition

Convex problem definition

Convex problem definition

Proposed solution(ADMM) Alternating Direction Method of Multipliers(ADMM) S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine

Proposed solution(ADMM)

Proposed solution(ADMM) ADMM in network lasso � 2). Augmented Lagrangian M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4: 302– 320, 1969.

Proposed solution(ADMM) ADMM in network lasso � 3). ADMM updates

Proposed solution(ADMM)

Non-convex extension

Non-convex extension Heuristic solution: to keep track of the iteration which yields the minimum objective, and to return that as the solution instead of the most recent step.

Non-convex extension Non-convex z-Update � Compared to the convex case, the only difference in the ADMM solution is the z-update, which is now

Experiments 1. Network-Enhanced Classification � We first analyze a synthetic network in which each node has a support vector machine (SVM) classifier, � but does not have enough training data to accurately estimate it Idea: � “borrow” training examples from their relevant neighbors to improve their own results � neighbors with different underlying models has non-zero lasso penalties

Experiments

Experiments 1. Network-Enhanced Classification Objective function:

Experiments 1. Network-Enhanced Classification Results(regularization path):

Experiments 1. Network-Enhanced Classification Results(prediction accuracy):

Experiments 1. Network-Enhanced Classification Results(timing): Convergence comparison between centralized and ADMM methods for SVM problem

Experiments 1. Network-Enhanced Classification Results(timing):

Experiments 2. spatial clustering and regressors Attempt to estimate the price of homes based on latitude/longitude data and a set of features.

Experiments 2. spatial clustering and regressors Dataset: �a list of real estate transactions over a oneweek period in May 2008 in the Greater Sacramento area. Network: � build the graph by using the latitude/longitude coordinates of each house � connect every remaining house to the five nearest homes with an edge weight inversely proportional to the distance between the houses � 785 nodes, 2447 edges, and has a diameter of 61.

Experiments 2. spatial clustering and regressors Optimization Parameter and Objective Function: � At each nodes, solve for � Objective function:

Experiments 2. spatial clustering and regressors Results:

Experiments 2. spatial clustering and regressors Results:

Conclusion

Questions?