Planning Old Perspectives New Horizons Derek Long Scientific

Planning: Old Perspectives, New Horizons Derek Long Scientific Advisor, Schlumberger Cambridge Research Professor of Computer Schlumberger Technology London Science, King’s College

A “Real World” Problem… A decorating company is commissioned to paint a large number of walls, each requiring to be painted by some number of coats of paint (wall-specific) The company has some number of painters, who can each paint a single wall at a time, or move from one wall to another Given the initial colours of the walls and the initial locations of the decorators, what is the best plan to paint all the walls to their target colour? * 2 Schlumberger Technology

Findings FF: 140 steps, 137 of them use D 3 LPG: 145 steps, makespan 96 Alternative planner: 161 steps; makespan 43 Optimal? Acceptable? 3 Schlumberger Technology

Planning for Execution Plans have to be good quality (not necessarily optimal) Practically: plans have to be temporal, numeric and executable Challenge: Produce good quality, executable plans efficiently, being responsive to user requirements (and plan metrics) 4 Schlumberger Technology

Optimisation Classic form of an optimisation problem: More generally, these constraints can be any Minimise f(x) conditions on the possible combinations of values of s. t. g(x) c the vector of variables, x x D x is a finite vector of variables, where each xi has a domain Di of possible values f: D R g : D R m and c is an m-dimensional vector of real constants Any x D satisfying g(x) c is feasible Schlumberger Technology

Linear Programming When f and g are linear functions of the decision variables, linear programming methods can solve the problems optimally, very efficiently Schlumberger Technology

Beyond Linear… Lagrangian method: • Minimise f(x) + . (g(x) – c) for x D • Non-linear optimisation approach when f and g are twice-differentiable functions Interior point methods generalise this approach, using barrier functions to allow a feasible solution to be iteratively improved to converge towards the (convex) constraint boundary Not effective for discontinuous functions and non-numeric domains Schlumberger Technology

Discontinuous Functions Arise when a function or constraint involves disjunction • Eg: Step function – f(x) = 0, if x < 0 = 1, if x >= 0 That is, either x < 0 and f(x) = 0, or x >= 0 and f(x) = 1 Problems with finite-domain variables always involve disjunctions • Eg Sudoku Schlumberger Technology

Sudoku Each cell contains a digit (1… 9). All entries in each row different; similarly each column; similarly each marked subdivision. 9 8 5 3 3 8 8 6 2 4 5 8 7 4 Schlumberger Technology 2 2 7 3 9 4 8 1 3 1 2 1 7

Describing a Problem Variables c[i, j] for i, j = 1… 9, each with domain {1, …, 9} For all k = 1, …, 9: | { c[i, j] : i = k, 1 <= j <= 9} | = 9 • “For each k, the set of distinct column entries in row k must contain 9 values” For all i, j, k (1. . 9): j k c[i, j] c[i, k] • “Every pair of entries in different columns of row i must be different values” See CSP vs LP formulations… Schlumberger Technology

Integer-LP Sudoku Encoding xijk = 1 if row i, column j, contains symbol k = 0 otherwise 0 i . . . 8 0. k. . 8 * Schlumberger Technology j 0. . . 8

Findings… LP-solve (integer program): no solution in “short time” CSP: fast solution Challenge: Choose solving strategy Different solution strategies havethe different strengths and weaknesses according to the structure of the problem… automatically 12 Schlumberger Technology

Planning Finite set of actions, a 1…am Finite set of state variables, s 1. . . sn A state is an assignment of values to the state variables Each action has a precondition pre[ai] which is a condition on some subset of the state variables – the action is applicable in a state S if the precondition is true with the state variables assigned according to S (we write S ⊧ pre[ai]) Each action assigns new values to some subset of the state variables The (classical) planning problem is to find a sequence of states S 0 … Sk and actions ai 1 … aik such that aij is applicable in Sj and applying aij to Sj yields Sj+1 Schlumberger Technology

Planning Finite set of actions, a 1…am Finite set of state variables, s 1. . . sn A state is an assignment of values to the state variables Each action has a precondition pre[ai] which is a condition on some subset of the state variables – the action is applicable in a state S if the precondition is true with the state variables assigned according to S (we write S ⊧ pre[ai]) Each action assigns new values to some subset of the state variables The (classical) planning problem is to find a sequence of states S 0 … Sk and actions ai 1 … aik such that aij is applicable in Sj and applying aij to Sj yields Sj+1 Schlumberger Technology

Planning Finite set of actions, a 1…am Finite set of state variables, s 1. . . sn A state is an assignment of values to the state variables Each action has a precondition pre[ai] which is a condition on some subset of the state variables – the action is applicable in a state S if the precondition is true with the state variables assigned according to S (we write S ⊧ pre[ai]) Each action assigns new values to some subset of the state variables The (classical) planning problem is to find a sequence of states S 0 … Sk and actions ai 1 … aik such that aij is applicable in Sj and applying aij to Sj yields Sj+1 Schlumberger Technology

Planning as Optimisation For a given plan length, k, we can define a set of variables sij for i = 0, …, k and j = 1…n (the state variables at each time point) and action variables ai 1 … aik (the k actions forming the plan) This makes the problem of finding a plan into an optimisation problem But, we don’t know k in advance! So we have to try solving with multiple values of k… Iterative deepening (incremental exploration of values of k) or else generalize the finite problem description to a dynamic optimization problem, in which adding variables is possible – optimality has to be handled with care Schlumberger Technology

A Framework for Optimisation We solve an optimisation problem by using an iterative scheme in which steps are one of three types Search Infer Relax Schlumberger Technology Prof. John Hooker at CMU has developed this framework over the past two decades

Search-Infer-Relax General framework for optimisation approaches Search: • Branch the problem by adding a new constraint Negated constraint New constraint Minimise f(x) s. t. g(x) c g’(x) c’ x D Minimise f(x) s. t. g(x) c g’(x) > c’ x D If a problem is over-constrained, it will have no feasible solutions Otherwise, we find a solution x* and can add the constraint f(x) < f(x*) to the second branch (this is the pattern known as branch and bound) Schlumberger Technology

Search Example Five neighbours live in adjacent houses in a row. The neighbours are of different nationalities, have different coloured front doors, each keep a different pet and prefer a different beverage. 1: The Englishman lives in the red house. 2: The Spaniard owns a dog. 3: The man in the green house drinks cocoa. 4: The Ukranian drinks eggnog. 5: The green house is immediately to the right of the ivory house. 6: The owner of the Oldsmobile keeps snails. 7: The owner of the Ford lives in the yellow house. 8: The man in the middle house drinks milk. 9: The Norwegian lives in the first house on the left. 10: The owner of the Chevrolet lives next to the fox owner. 11: The owner of the Ford lives next door to the horse owner. 12: The Mercedes driver drinks orange juice. 13: The Japanese drives a Volkswagon. 14: The Norwegian lives next to the blue house. 5 th house 15: One of the neighbours prefers to drink water. * Who keeps the zebra? Schlumberger Technology is red 5 th house is not red

Planning Search: • Choose an action to add to the plan • Branch on ordering of actions • Search over alternative bindings of variables in action schemas Schlumberger Technology

Search: A Challenge for Planning Imagine a logistics problem in which passengers have to be picked up and dropped off at various locations, with passengers becoming available at specific times Temporal problem, using TILs to model availability… Challenge: Forward search faces a choice between advancing time to the next TIL, or not… Avoid the urge to do while also Often, the pressure to do something remaining list of things to be avoiding that the shortens urge to the delay done favours moving forward to the TIL, rather than embarking on a complex sequence of steps to make progress * 21 Schlumberger Technology

Backtracking When a search choice fails, the solver has to explore the alternative choice – backtracking When the solver recognizes that there is no solution, which choice is to blame? Systematic backtracking – return to the most recent choice Dependency Directed Backtracking – return to the choice that led to the failure 22 Schlumberger Technology

Garden-Path Problem Consider the problem of mending a gate with bolts The bolts are in a shed at the start of the path and the gate is a number of steps down the path There are two hinges, each needing a bolt * 23 Schlumberger Technology

Findings… FF: If the planner is allowed to use the same bolt on each hinge, then it solves a garden path of 1000 strides length by searching 2002 states (a 2001 step plan) If it must use a different bolt for each hinge, it searches 15063 states (2002 step plan) Challenge: The fatal error is to choose to leave dependency-directed the shed with only one bolt, then head all the Implement way down the path, backtracking before realizingin it will need two bolts –– how but then backtracking planner-search over different ways to reach the gate before finally realizing it must pick up the do we attribute failure? second bolt… 24 Schlumberger Technology

Search-Infer-Relax Inference adds explicit constraints to the problem that are implied by the existing constraints: If, for all x D, g(x) c g’(x) c’ then add g’(x) c’ to the problem (more general form for any constraint type) Simplex algorithm can be seen as a series of inference steps (LP is inferencecomplete) Interior point methods are also based on a series of inference steps Schlumberger Technology

Inference Example 9 8 5 3 3 8 Schlumberger Technology Must be 9 8 6 2 4 5 8 7 4 2 2 7 3 9 4 8 1 3 1 2 1 7

The Power of Inference A constraint can only be added to the problem if it can be expressed in the language recognised by the solver • Eg: c[5, 6] = 9 A sufficiently rich set of inference rules can solve Sudoku problems without search • Pruning values from domains is not sufficiently powerful to solve without search Schlumberger Technology

Planning Search: • Choose an action to add to the plan • Branch on ordering of actions • Search over alternative bindings of variables in action schemas Infer: • Add temporal constraints to the plan • Infer which actions must be in a plan (landmarks) Schlumberger Technology

Cyclic Actions A common feature of planning problems is the need to repeatedly cycle the state of one or more objects in the problem Consider a machine shop problem in which parts have to be machined with tools that must be cleaned between uses The tools must be turned on for use and off to clean, so there is a sequence of uses interspersed by cleaning, with the tools being turned on and off interleaved with the cleaning and use * 29 Schlumberger Technology

Findings… Typically, the planners can infer that a machine that is off will need to be turned on and one that is on will need to be turned off, but they cannot infer the number of times that these actions are required Challenge: The result is a poor heuristic estimate of the number of steps required to solve the problem and, often, a poor quality plan results Infer the number of repetitions of actions required for cyclic patterns – A frequent error is cycling state unnecessarily better still, infer the structure of the cycle and exploit it in plan 30 Schlumberger Technology

Search-Infer-Relaxation of a problem removes constraints to arrive at a simpler problem • Solution to relaxed problem gives us a bound on the possible value of an optimal solution to the original problem • Also often offers a guide to good search branches Mixed-Integer Programs can be solved using the continuous LP as a relaxation, offering good branching choices for the variables (ie those that are non-integer in the relaxation solution) and a bound on the solution value Schlumberger Technology

Integer Program Encoding of Zebra Problem Green house is either left or right of the yellow house yellow - green >= 1 -5 x 5; green - yellow >= 5 x 5 - 4; x 5 is 0 -1 decision variable (0 = yellow on right of green, 1 = green on right of yellow) In the solution to the LP relaxation: x 5 = 0. 2 … so this is not a solution to the original problem But… maybe good idea to branch first on x 5 = 0 or 1 * Schlumberger Technology

Planning Search: • Choose an action to add to the plan • Branch on ordering of actions • Search over alternative bindings of variables in action schemas Infer: • Add temporal constraints to the plan • Infer which actions must be in a plan (landmarks) Relax: • Reachability analysis and relaxed plan construction Schlumberger Technology

Relaxation: problem projections? One way to view relaxation of problems is as a projection of the original problem: removing constraints is one way to project Projection can be defined in this way: If g(x) c P(x) for x D then the relaxed P-projection is: Find x to minimize f(x) s. t. P(x) x D 34 Schlumberger Technology

Problem projections for planning Many planning domains contain sub-problems: • Route-planning (variants including vehicle-routing problems for multiple vehicles, pick-up and deliver problems, availability time-windows and so on) • Scheduling problems (variants including job-shop scheduling, multi-processor scheduling) Challenge: • Bin-packing and knapsack problems Identify sub-problems and solve as Any solution to the complete planning problem must, of course, solve the subrelaxations, offering helpful actions problem, so these sub-problems are relaxation projections of the planning problem and improved heuristics – and we have good heuristics for solving them (not necessarily optimally) 35 Schlumberger Technology

Challenge Summary Pressing challenges for planning: • Produce good quality, executable plans efficiently, being responsive to user requirements (and plan metrics) • Choose the solving strategy according to the structure of the problem… automatically • Avoid the urge to do while also avoiding the urge to delay (aka handle TILs better) • Implement dependency-directed backtracking in planner-search – how do we attribute failure? • Infer the number of repetitions of actions required for cyclic patterns – better still, infer the structure of the cycle and exploit it in plan • Identify sub-problems and solve as relaxations, offering helpful actions and improved heuristics 36 Schlumberger Technology