Combinatorial Algorithms Jessica INTRODUCTION What are Combinatorial Algorithms

Combinatorial Algorithms

Jessica INTRODUCTION

What are Combinatorial Algorithms? • Algorithms to investigate combinatorial structures • Generation: Construct all the combinatorial structures of a particular type – Ex: subsets, permutations, partitions, trees, Catalan families • Enumeration: Compute the number of different structures of a particular type – Ex: find number of k-subsets of an n element set • Search: Find at least one example of a structure of particular type (if it exists) – Ex: find a clique of a specified size in a given graph

Optimization Problems • Suppose we are trying to solve an optimization problem • There exist backtracking solutions for this reason

Backtracking Algorithms • Generate all possible solutions in a certain lexicographic order • Advantages: – Useful for finding one optimal solution and for counting or enumerating all optimal solutions • Disadvantages: – May be inefficient for finding just one solution – For optimal verification, may need to examine large part of the state space tree, even with pruning – Sometimes need to “back up” several levels when partial solution cannot be extended

Heuristic Algorithm • Definition: A heuristic algorithm tries to find a certain combinatorial structure or solve an optimization problem by the use of heuristics. • Heuristic: “serving or helping to find out or discover; proceeding by trial and error” • Types of problems can be applied to: – Find 1 optimal solution – Find “close to” optimal solution

Heuristic Characteristics and Methods Characteristics • The state space is not fully explored • Randomization is often employed • There is a concept of neighborhood search Methods • Hill-climbing • Simulated annealing • Tabu search • Genetic algorithms

General framework for heuristic search Generic Optimization Problem (maximization): Instance: A finite set X An objective function P : X Z m feasibility functions g : Xj Z, 1 ≤ j ≤ m Find: the maximum value of P(X) subject to Xj X and g (X) ≥ 0, for 1 ≤ j ≤ m

Designing a heuristic search • Define a neighborhood function N : X 2 X – Ex: N(X) = {X 1 , X 2 , X 3 , X 4 , X 5 } • Design a neighborhood search: – Algorithm that finds a feasible solution on the neighborhood of a feasible solution X. – 2 types of neighborhood searches: • Exhaustive (chooses best profit among neighbor points) • Randomized (picks a random point among the neighbor points)

Defining a neighborhood function N : X 2 X So, N(X) is a subset of X. • N(X) should contain elements that are similar or “close to” X. • N(X) may contain infeasible elements of X. • To be useful, want to get to Xopt from X 0 via a number of applications of N(▪) • ie. Graph G with V(G) = X and E(G) = {{X, Y} : Y N(X)} should ideally be connected, or at least have one optimal solution in each of its connected components • Computing N(X) should be fast, and in particular |N(X)| shouldn’t be too large

Sumanaruban, Yogesh and Lahiru HILL CLIMBING ALGORITHM

Hill-Climbing • Idea: Go up the hill continuously, stop when stuck.

Hill Climbing Algorithm 1. Pick a random point in the search space 2. Consider all the neighbors of the current state 3. Choose the neighbor with the best quality and move to that state 4. Repeat 2 thru 4 until all the neighboring states are of lower quality 5. Return the current state as the solution state

• Problem: it can get stuck in a local optimum • Simple (often effective) solution – Multiple random restarts

Uniform graph partition • Instance A complete graph on 2 n vertices, A cost function, cost : • Find the minimum value of

Example • n = 4; – cost(1, 2) = 1, – cost(1, 4) = 5, – cost(2, 4) = 5, cost(1, 3) = 2, cost(2, 3) = 0, cost(3, 4) = 1 1 X 2 x 1 2 5 5 x 4 1 0 x 3

• Only 3 feasible solutions: • X 0 = {1, 2}, X 1 = {3, 4} 1 X 2 x 1 2 5 5 x 4 • C([X 0, X 1]) = 12 1 0 x 3

• X 0 = {1, 4}, X 1 = {2, 3} 1 X 2 x 1 2 5 5 x 4 • C([X 0, X 1]) = 9 1 0 x 3

• X 0 = {1, 3}, X 1 = {2, 4} 1 X 2 x 1 2 5 5 x 4 1 • C([X 0, X 1]) = 7 (optimal) 0 x 3

Neighbourhood function: exchange x ∈ X 0 and y ∈ X 1. Algorithm UGP(Cmax) X = [X 0, X 1] ← Select. Random. Partition c← 1 while (c ≤ Cmax) do [Y 0, Y 1] ← Ascend(X) if not fail then {X 0 ← Y 0; X 1 ← Y 1; } else return c←c+1

Algorithm Ascend([X 0, X 1]) g← 0 for each i ∈ X 0 do for each j ∈ X 1 do t ← G[X 0, X 1](i, j) (gain obtained in exchange) if (t > g) {x ← i; y ← j; g ← t} if (g > 0) then Y 0 ← (X 0 ∪ {y}) {x} Y 0 ← (X 1 ∪ {x}) {y} fail ← false return [Y 0, Y 1] else {fail ← true; return [X 0, X 1]}

Other drawbacks local maximum plateau ridge Image from: http: //classes. yale. edu/fractals/CA/GA/Fitness/Fi tness. html • Ridge = sequence of local maxima difficult for greedy algorithms to navigate • Plateau = an area of the state space where the evaluation function is flat.

Hill-climbing variations • How do we make hill climbing less greedy? • Stochastic hill-climbing – Random selection among the better neighbors. – The selection probability can vary with the steepness of the uphill move. • What if the neighborhood is too large to enumerate? • First-choice hill-climbing – Randomly generate neighbors, one at a time – If better, take the move • Random-restart hill-climbing – Tries to avoid getting stuck in local maxima.

Hill-Climbing Algorithm for Steiner Triple Systems Stinson's Algorithm

Steiner Triple System

Examples STS(3), v = 3 (trivial case) V = {1, 2, 3} B = {123}

Examples STS(3), v = 3 (trivial case) V = {1, 2, 3} B = {123} STS(7), v = 7 V = {1, 2, 3, 4, 5, 6, 7} B = {1, 2, 3}, {1, 4, 5}, {1, 6, 7}, {2, 4, 6}, {2, 5, 7}, {3, 4, 7}, {3, 5, 6}

Examples STS(3), v = 3 (trivial case) V = {1, 2, 3} B = {123} STS(7), v = 7 V = {1, 2, 3, 4, 5, 6, 7} B = {1, 2, 3}, {1, 4, 5}, {1, 6, 7}, {2, 4, 6}, {2, 5, 7}, {3, 4, 7}, {3, 5, 6} STS(9), v = 9 V = {1, 2, 3, 4, 5, 6, 7, 8, 9} B = {1, 2, 3}, {4, 5, 6}, {7, 8, 9}, {1, 4, 7}, {2, 5, 8}, {3, 6, 9}, {1, 5, 9}, {2, 6, 7}, {3, 4, 8}, {1, 6, 8}, {2, 4, 9}, {3, 5, 7}.

A Graph Theoretic View Another way to look at Steiner Triple Systems Image source - http: //mathworld. wolfram. com/Steiner. Triple. System. html

A Graph Theoretic View Another way to look at Steiner Triple Systems Consider the complete graph on v vertices, Kv. A decomposition of Kv into edge disjoint triangles (K 3's) is equivalent to a Steiner Triple System. Image source - http: //mathworld. wolfram. com/Steiner. Triple. System. html

Fano Plane

Fano Plane The Fano plane is an STS(7) Steiner triple system. The blocks are the 7 lines, each containing 3 points. Every pair of points belongs to a unique line.

Fano Plane The Fano plane is an STS(7) Steiner triple system. The blocks are the 7 lines, each containing 3 points. Every pair of points belongs to a unique line. Has application like factoring integers via quadratic forms

Kirkman's Schoolgirl Problem A problem in combinatorics proposed by Rev. Thomas Penyngton Kirkman in 1850 as Query VI in The Lady's and Gentleman's Diary (pg. 48). Image source - http: //delphiforfun. org/programs/kirkman. htm

Kirkman's Schoolgirl Problem A problem in combinatorics proposed by Rev. Thomas Penyngton Kirkman in 1850 as Query VI in The Lady's and Gentleman's Diary (pg. 48). Fifteen young ladies in a school walk out three abreast for seven days in succession: it is required to arrange them daily so that no two shall walk twice abreast. Image source - http: //delphiforfun. org/programs/kirkman. htm

Kirkman Systems The solution to Kirkman’s 15 schoolgirl problem is a resolution of a block design on 15 points with blocks of size 3 Steiner Triple System on 15 points

Kirkman Systems The solution to Kirkman’s 15 schoolgirl problem is a resolution of a block design on 15 points with blocks of size 3 Steiner Triple System on 15 points Solution not unique

Kirkman Systems The solution to Kirkman’s 15 schoolgirl problem is a resolution of a block design on 15 points with blocks of size 3 Steiner Triple System on 15 points Solution not unique 7 possible solutions for STS(15)

1 2 If an STS(n) exists, so too does an STS(2 n + 1). 3 If an STS(n) exists, so too does an STS(2 n + 7). 1

1

1

1

1

1

1

1

1

1

1

2 If an STS(n) exists, so too does an STS(2 n + 1). 3 If an STS(n) exists, so too does an STS(2 n + 7). Reference - http: //www. math. mun. ca/~davidm/4341/sts. pdf

1 Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v)

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • For any x ∈ V, the triples containing x partition V-{x} into pairs,

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • For any x ∈ V, the triples containing x partition V-{x} into pairs, • thus, v-1 is even

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • For any x ∈ V, the triples containing x partition V-{x} into pairs, • thus, v-1 is even • and v is odd

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • For any x ∈ V, the triples containing x partition V-{x} into pairs, • thus, v-1 is even • and v is odd • Therefore, v≡ 1, 3 or 5(mod 6)

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • For any x ∈ V, the triples containing x partition V-{x} into pairs, • thus, v-1 is even • and v is odd • Therefore, v≡ 1, 3 or 5(mod 6) • From lemma 1. 2 (b-number of blocks) b=v(v-1)/6

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • However, if v = 6 k+5, computing |B| gives:

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • However, if v = 6 k+5, computing |B| gives: |B| = (6 k+5)(6 k+4)/6 |B| = (36 k 2 + 54 k + 20)/6

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • However, if v = 6 k+5, computing |B| gives: |B| = (6 k+5)(6 k+4)/6 |B| = (36 k 2 + 54 k + 20)/6 which is not an integer.

Theorem 1. 1 STS(v) ⇒ v ≡ 1, 3 (mod 6) PROOF: Let (V, B) be an STS(v) • However, if v = 6 k+5, computing |B| gives: |B| = (6 k+5)(6 k+4)/6 |B| = (36 k 2 + 54 k + 20)/6 which is not an integer. Therefore, v≡ 1, 3(mod 6)

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Base Case : • The base cases are v = 1, v = 3, v = 7 and v = 9, which have been illustrated

Examples STS(3), v = 3 (trivial case) V = {1, 2, 3} B = {123} STS(7), v = 7 V = {1, 2, 3, 4, 5, 6, 7} B = {1, 2, 3}, {1, 4, 5}, {1, 6, 7}, {2, 4, 6}, {2, 5, 7}, {3, 4, 7}, {3, 5, 6} STS(9), v = 9 V = {1, 2, 3, 4, 5, 6, 7, 8, 9} B = {1, 2, 3}, {4, 5, 6}, {7, 8, 9}, {1, 4, 7}, {2, 5, 8}, {3, 6, 9}, {1, 5, 9}, {2, 6, 7}, {3, 4, 8}, {1, 6, 8}, {2, 4, 9}, {3, 5, 7}.

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Base Case : • The base cases are v = 1, v = 3, v = 7 and v = 9, which have been illustrated • Induction Hypothesis : • Let v = k, (k ≡ 1, 3 (mod 6)) • Assume that there exist an STS(l) for all l < k (l ≡ 1, 3 (mod 6))

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Induction Step : (Prove STS(k) exist) • We consider four cases, depending on the congruence of k modulo 12 • • k = 12 m + 1 k = 12 m + 3 k = 12 m + 7 k = 12 m + 9

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Induction Step : (Prove STS(k) exist) Case 1 : k = 12 m + 1 • 6 m-3 < k • Therefore, by the induction hypothesis • STS(6 m - 3) exist • k = 12 m + 1 = 2(6 m - 3) + 7 • Lemma 3 says : If an STS(n) exists, so too does an STS(2 n + 7) • Using Lemma 3 there is an STS(k)

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Induction Step : (Prove STS(k) exist) Case 2 : k = 12 m + 3 • 6 m+1 < k • Therefore, by the induction hypothesis • STS(6 m + 1) exist • k = 12 m + 3 = 2(6 m + 1) + 1 • Lemma 2 says : If an STS(n) exists, so too does an STS(2 n + 1) • Using Lemma 2 there is an STS(k)

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Induction Step : (Prove STS(k) exist) Case 3 : k = 12 m + 7 • 6 m+3 < k • Therefore, by the induction hypothesis • STS(6 m + 3) exist • k = 12 m + 7 = 2(6 m + 3) + 1 • Lemma 2 says : If an STS(n) exists, so too does an STS(2 n + 1) • Using Lemma 2 there is an STS(k)

Theorem 1. 2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction) • Induction Step : (Prove STS(k) exist) Case 4 : k = 12 m + 9 • 6 m+1 < k • Therefore, by the induction hypothesis • STS(6 m + 1) exist • k = 12 m + 9 = 2(6 m + 1) + 7 • Lemma 3 says : If an STS(n) exists, so too does an STS(2 n + 7) • Using Lemma 3 there is an STS(k)

Partial Steiner Triple System Is a set system In which every block has size Every pair of points in at most one block If is contained in

Construct Steiner Triple System

Stinson’s Algorithm 6

Switch Heuristic

Remarks No termination guarantee In practice, it always terminates if the choices made by the heuristic are random Usually runs very quickly

Results The average number of iterations is

Shalinda , Ramanathan and Pham Nam GENETIC ALGORITHM

Genetic Algorithm • The knowledge that each specie gains is encoded in its chromosomes and they undergo transformation when reproduction occurs. • Genes from `good‘ parents will produce offspring that are better than either parent. • Thus each successive generation will become more suited to their environment. Developed by John Holland, University of Michigan (1970’s) • Provide efficient, effective techniques for optimization and machine learning applications

Generic Genetic Algorithm Incorporates a heuristic h. N based on a neighborhood function N and a recombination operation rec: x → x Chromosomes could be: Bit strings Real numbers Permutations of element Lists of rules Program elements. . . any data structure … (0101. . . 1100) (43. 2 -33. 1. . . 0. 0 89. 2) (E 11 E 3 E 7. . . E 15) (R 1 R 2 R 3. . . R 22 R 23) (genetic programming)

Generic Genetic Algorithm Incorporates a heuristic h. N based on a neighborhood function N and a recombination operation rec: x → x Find members that are similar or close to X

Generic Genetic Algorithm Incorporates a heuristic h. N based on a neighborhood function N and a recombination operation rec: x → x Reproduction 1. Number of generations 2. Until the algorithm converged Parents are selected to mate, on the basis of their fitness

Generic Genetic Algorithm Incorporates a heuristic h. N based on a neighborhood function N and a recombination operation rec: x → x Operator types are: • Crossover (recombination) • Mutation

Crossovers • In biology a crossover happens when chromosomes from different parents are split and recombine; the new cell has partial strands from both parents • Prime distinguished factor of GA from other optimization techniques

Partially matched crossover = {1, …. , n} Two parents are α and β, two children are γ and δ Two crossover points j and k where 1≤j≤k≤n

Example… Assume that α =[3, 1, 4, 7, 6, 5, 2, 8] and β=[8, 6, 4, 3, 7, 1, 2, 5] Crossover points are j=3 and k=6 i=3 i=4 i=5 i=6 No effects γ=[7, 1, 4, 3, 6, 5, 2, 8] and δ=[8, 6, 4, 7, 3, 1, 2, 5] γ=[6, 1, 4, 3, 7, 5, 2, 8] and δ=[8, 7, 4, 6, 3, 1, 2, 5] γ=[6, 5, 4, 3, 7, 1, 2, 8] and δ=[8, 7, 4, 6, 3, 5, 2, 1]

Mutations • Adopting terminology from molecular biology, changes to chromosomes are called mutations • Mutation alters one or more gene values in a chromosome from its initial state • Purposes: • Inhibit premature convergence. • User-definable mutation probability. • Should be a low probability. Before: After: * * (6 5 4 3 7 1 2 8) (6 5 2 3 7 1 4 8)

Generic Genetic Algorithm Incorporates a heuristic h. N based on a neighborhood function N and a recombination operation rec: x → x Evaluate population • Determined by an objective function

Genetic TSP

A Genetic Algorithm for TSP Ramanathan

Travelling Salesman Problem Instance: a complete graph Kn a cost function C(X) Find: a Hamiltonian circuit [x 0, x 1, … , xn-1] that minimizes C(X) = c(x 0, x 1) + c(x 1, x 2) +. . . + c(xn-1, x 0) Universe: X = set of all (n-1)!/2 permutations For 10 cities, 181440 tours are possible

Idea When we find a good tour, try to improve it using mutation and recombination Substring from parent 1’s path Substring from parent 2’s path The recombined path could be better than either parents’ path

Preliminaries Fitness: Overall cost of the tour Selection: Initial population drawn from a random set of tours Recombination: Partially matched crossover Mutation: Steepest ascent 2 -opt heuristic

2 -opt move Once a tour has been generated by some tour construction heuristic, we need to improve that solution 2 -opt move: 1. Remove two edges from the tour 2. Reconnect the two paths created to form a new tour We complete the move only if the new tour will be shorter

2 -opt move xi xi+1 xj xj+1 The gain in applying a 2 -opt move is the decrease in cost. G(X, i, j) = C(X) – C(Xij) = c(xi, xi+1) + c(xj, xj+1) – c(xi+1, xj+1) – c(xi, xj)

Steepest Ascent method N(x) = all Y ∈ X that can be obtained from x by a 2 opt move Mutation heuristic consists of a sequence of 2 -opt moves Using a steepest ascent strategy, iteratively apply 2 opt moves until no pair of edges can be found to yield a positive gain

Algorithm Input: Given tour X Output: Optimal tour gainmax = 0; while (gainmax > 0) do gainmax = 0; for i = 0 to (n-1) do for j = (i+2) to (n-1) do g = G(X, i, j); if (g > gainmax) then gainmax = g; imax = i; jmax = j; if (gainmax > 0) then X = Ximax, jmax;

Example Fitness function : Euclidean distance

Example Random tour Tour length = 1. 7938

Example Tour length = 1. 7938

Example Tour length = 1. 4329

Example Tour length = 1. 4329

Example Tour length = 1. 2676

Example Tour length = 1. 2676

Example Tour length = 1. 1259

Example Tour length = 1. 1259

Example Tour length = 1. 0436

Example Tour length = 1. 0436

Example Tour length = 1. 006

Example Tour length = 1. 006

Example Tour length = 0. 9874

Example Tour length = 0. 9874

Example Tour length = 0. 9701

Example Tour length = 0. 9701

Example Optimal tour length = 0. 9678

Partially Matched Crossover Algorithm Input: Parent tours A, B Minimum bound on the substring length, h Output: Offspring tours C, D j = random number (0, n-1); (start of the substring) k = random number (h, n/2); (length of the substring) (C, D) = Partially. Matched. Crossover(A, B, j, (j+k) mod n) Steepest. Ascent. Two. Opt(C); Steepest. Ascent. Two. Opt(D); return (C, D);

Select initial population

Unranking algorithm: intuition

Unranking algorithm: intuition vs algorithm r Π(1) dn-1 intuition Unroll recursive algorithm r’ Π(2) Π(n-1) Π(n) dn-2 d 1 1 Π

Unranking permutation generating j 0 1 2 3 d 0 1 1 2 r 15 14 12 0 Π 1 2 1 3 2 1 4 2 3

Genetic TSP

Applications Puneet

Applications • • • Speech Recognition Mobile Robotics Singapore MRT Computer Aided Design Economics

Speech Recognition • Language Model • Acoustic Model • Initially a simple language model and an acoustic model is used for speech recognition and results are stored in a speech lattice. • However it will be error prone. • Rescoring of the speech lattice is done using the sophisticated LM. • Neighborhoods are created using edit distance metrics. • Hill Climbing is applied to find the path with the highest probability (which is the final output).

Speech Recognition

• Edit Distance is used for defining the neighbourhood. • Hill Climbing is used to find the highest probable path in the lattice after rescoring.

Edit Distance

Artificial Intelligence

Mobile Robotics • Gradients are used to determine most likely pose of a robot given sensor measurements. • The computation of gradient gave real time performance using hill climbing algorithm.

Computer Aided Design

Computer Aided Design • Genetic Algorithm takes feedback from evaluation process to select fitter designs. • Generates new design through the recombination of selected parts of the design • Results in a population of high performance designs.

Satellite Truss Design

• Genetic Algorithms optimized satellite truss designs for NASA to maximize vibration control. • Evolving: Design structures • Fitness: Vibration resistance • Evolutionary “creativity”

Optimized Structure

Real Time Systems

Mass Rapid Transit • Genetic Algorithm was used for Time. Table Synchronization in Singapore MRT.

Economics • Game Theory

Game Theory • Study of multi person decision problems. • Each rival player has to consider what the others will do. • All players play rationally to maximize their reward • Genetic algorithms find the most optimal strategy for a player in a given situation.

Conclusion • Introduction • Hill Climbing – Uniform Graph Partition – STS • Genetic Algorithm – TSP • Applications