Artificial Intelligence Representation and Problem Solving Optimization 2
Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming 15 -381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu. edu Wean Hall 4126
Recap � 2 Fei Fang 11/3/2020
Recap � 4 Reference: Wikipedia; Berkeley CE 191 11/3/2020
Recap � 5 Reference: Wikipedia; Berkeley CE 191 11/3/2020
Outline �Linear Program �Simplex Algorithm �LP Duality 6 Fei Fang 11/3/2020
Linear Program: Definition � Note: can also be minimization 7 Fei Fang 11/3/2020
Linear Program: Example �Example: Maximize Profit in Manufacturing Price Labor Machine Product 1 $30 0. 2 hour 4 hour Product 2 $30 0. 5 hour 2 hour <=90 <=800 Total 8 Fei Fang 11/3/2020
Linear Program: Example �Example: Maximize Profit in Manufacturing Price Labor Machine Product 1 $30 0. 2 hour 4 hour Product 2 $30 0. 5 hour 2 hour <=90 <=800 Total 9 Fei Fang 11/3/2020
Quiz 1: Linear Program � 11 Fei Fang 11/3/2020
Linear Program: Definition � 12 Fei Fang 11/3/2020
Linear Program: Definition � 13 Fei Fang 11/3/2020
Linear Program: Definition � 14 Fei Fang 11/3/2020
Linear Program: How to Solve � 16 Fei Fang 11/3/2020
Linear Program: How to Solve �Naïve approach � Find all vertices How? � Pick the one with the best objective value 18 Fei Fang 11/3/2020
Linear Program: How to Solve � 19 Fei Fang 11/3/2020
Linear Program: How to Solve � 20 Fei Fang 11/3/2020
Linear Program: How to Solve � 22 Fei Fang 11/3/2020
Linear Program: How to Solve � how many choices? 24 Fei Fang 11/3/2020
Linear Program: How to Solve 25 Fei Fang 11/3/2020
Linear Program: How to Solve 26 Fei Fang 11/3/2020
Linear Program: How to Solve 28 Fei Fang 11/3/2020
Linear Program: How to Solve 29 Fei Fang 11/3/2020
Linear Program: How to Solve �Naïve approach � Find all vertices � Pick the one with the best objective value �Practical approach: Simplex algorithm � Start from one vertex � Move towards a neighboring vertex for improvement 30 Fei Fang 11/3/2020
Linear Program: How to Solve �Naïve approach � Find all vertices � Pick the one with the best objective value �Practical approach: Simplex algorithm � Start from one vertex � Move towards a neighboring vertex for improvement � Recall local search and gradient descent � May enumerate almost all the vertices in the worst case, but very efficient in most cases 31 Fei Fang 11/3/2020
Simplex Algorithm � 32 Fei Fang 11/3/2020
Simplex Algorithm � 33 Fei Fang 11/3/2020
Simplex Algorithm � 34 Fei Fang 11/3/2020
Simplex Algorithm � 35 Fei Fang 11/3/2020
Simplex Algorithm: Simplex Algorithm 36 Fei Fang 11/3/2020
Simplex Algorithm �Example 37 Fei Fang 11/3/2020
Simplex Algorithm �Example 38 Fei Fang 11/3/2020
Simplex Algorithm � 39 Fei Fang 11/3/2020
Quiz 2 � 40 LP-R LPL Fei Fang 11/3/2020
Two-Phase Simplex Algorithm � 41 Fei Fang 11/3/2020
Simplex Algorithm �Theorem: the simplex algorithm is guaranteed to find a globally optimal solution to a linear program Proof: If simplex returns, then it has found a point where we cannot improve the objective locally; since linear programs are convex, this is a global optimum Because the objective of the simplex improves at each iteration, and since there a finite (but exponential) number of vertices in the polytope, the algorithm must return after a finite number of steps 42 Fei Fang 11/3/2020
Linear Program: How to Solve �Naïve approach � Find all vertices � Pick the one with the best objective value �Practical approach: Simplex algorithm � Start from one vertex � Move towards a neighboring vertex for improvement � Recall local search and gradient descent � May enumerate almost all the vertices in the worst case, but very efficient in most cases �(Not covered) Ellipsoid algorithm � Polynomial-time in theory, poor performance in practice 43 Fei Fang 11/3/2020
Linear Program: How �Solve LPs in practice: � All the solvers/algorithms for Convex Optimization problems can be applied � Additional solvers/algorithms for LPs � linprog (MATLAB), linprog (in Python package Sci. Py) � Pu. LP (Python) � Cplex, Gurobi � https: //www. informs. org/ORMS-Today/Public-Articles/June- Volume-38 -Number-3/Software-Survey-Linear-Programming 44 Fei Fang 11/3/2020
Linear Program: Note � 45 Fei Fang 11/3/2020
Linear Program Duality �Dual problem of an LP: also a linear program � Each dual variable corresponds to a constraint in primal LP �Strong duality theorem: LP and its dual have the same optimal objective value (if feasible and bounded) 46 Fei Fang 11/3/2020
Linear Program Duality Primal LP in Standard Form Dual LP 52 Fei Fang 11/3/2020
Linear Program Duality 53 Fei Fang 11/3/2020
Write the Dual of an LP (Non-Standard form) Maximize Minimize ith constraint ≤ ith variable ≥ 0 ith constraint ≥ ith variable ≤ 0 ith constraint = ith variable unrestricted jth variable ≥ 0 jth constraint ≥ jth variable ≤ 0 jth constraint ≤ jth variable unrestricted jth constraint = 55 Fei Fang 11/3/2020
Other properties of LP Duality (not required) � 57 Fei Fang 11/3/2020
Summary Optimization Problems Convex Programs Gradient Descent Linear Programs Simplex 58 Fei Fang 11/3/2020
Linear Program: Additional Resources �Textbook � Applied Mathematical Programming, Chapters 2 -4 � By Bradley, Hax, and Magnanti (Addison-Wesley, 1977) � http: //web. mit. edu/15. 053/www/AMP. htm �Online course � https: //ocw. mit. edu/courses/electrical-engineering-and-computer -science/6 -251 j-introduction-to-mathematical-programming-fall 2009/index. htm � 59 Fei Fang 11/3/2020
Acknowledgment �Some slides are borrowed from previous slides made by J. Zico Kolter 60 Fei Fang 11/3/2020
- Slides: 46
