Swarm Collaborative Intelligence From Networked Control to Trust

Swarm Collaborative Intelligence: From Networked Control to Trust in MANET John S. Baras Institute for Systems Research Department of Electrical and Computer Engineering And Department of Computer Science University of Maryland, College Park, MD 20742 Workshop on Swarming in Natural and Engineered Systems Napa Valley, California August 3 -4, 2005

Thanks to n Collaborators: Tao Jiang, George Theodorakopoulos, Xiaobo Tan, Wei Xi, Pedram Hovareshti n Funding sources: ARL (CTA on C&N), ARO CIP URI (Wireless Network Security), ARO MURI (Networked Control Systems), DARPA (Dynamic Coalitions)

Outline n Autonomous collaborating vehicles q q A stochastic approach based on MRF Analysis n n q q n Simulation results A hybrid scheme to improve the performance Distributed trust in MANET q q n Convergence study of a simple case Convergence speed analysis Trust (and Mistrust) spreading and dynamics Effects of topology on convergence Spin glasses and cooperative games Collaboration via trust schemes Conclusions and future work

A Battlefield Scenario n Mission q n Constraints q q q n Autonomous, distributed maneuvering of a vehicle group to reach and cover a target area Desired inter-vehicle distance Obstacles avoidance Threats (stationary or moving) avoidance Requirement q Using only local or static information

Review of Deterministic Gradient-Flow Approach n Dilemma of the Deterministic gradient-flow approach q q n Potentials-based approach can accommodate multiple objectives and constraints in a distributed and computationally effective way The system dynamics could be trapped by the local minima Weighted sum of potential functions: q q q Target (attraction) potential Jg Neighbor (avoidance) potential Jn Obstacle potential Jo Potential Js due to stationary threats Potential Jm due to moving threats n Gradient flow:

Being Trapped by Local Minima Different initial conditions may cause vehicles to be trapped by local minimum

Markov Random Fields n Markov Random Fields (MRF) q q q n A collection of random variables X={Xs}, s∈ S with discrete values in phase space s A neighborhood system on S is a family N = {Ns}, s∈ S, where Ns ⊂ S, and r ∈ Ns ↔ s ∈ Nr The marginal probability depends only on neighbor’s phase value Gibbs Field (GF) q q A clique is a subset c ⊂ S, such at for all s, r ∈ c, r ∈ Ns The potential energy of a configuration x={xs} is defined as a sum of all clique potentials

Gibbs Sampler and Gibbs Distribution n n Gibbs distribution (global description) q The marginal probability is function of local potentials q Gibbs distribution is function of global potentials q Hammersley-Clifford theorem: A MRF on a graph is equivalent to a GF Gibbs sampler q q q Gibbs sampler (MCMC method)defines a Markov chain on a Gibbs Field The stationary distribution of the MC is the Gibbs distribution Using simulated annealing algorithm, final configuration converges to global minimum with probability 1

Modeling a Swarm as a GF 2 D mission space on discrete lattice cells n q q q Agent s can communicate with neighboring agents in Ns which stay within the interaction range Rs An agent can go at most Rm within one move, which defines the phase space s Gibbs potential is designed to reflect global objective Obstacle Agents Difficulties in applying classical results n q q Non-stationary neighborhood system Time-varying and state-dependent phase space target

Gibbs Sampler Based Algorithm n Algorithm for single vehicle q q q Step 1. Pick a cooling schedule T(n) and the total number N of annealing steps Step 2. At each annealing step n, conduct a location update for the vehicle by performing the following: n Determine the set L of candidate locations for the next move n For each l ∈ L, evaluate n Update location by sampling above distribution Step 3. Let n = n+1. If n = N, stop; otherwise go to step 2

Convergence Study n Single vehicle with limited sensing and moving range q Fixed temperature n n Assume accessible area is connected Unique stationary distribution where n q From any distribution v, Simulated annealing n Cooling schedule n Let Qn = (PT(n))τ where

Convergence Rate Study n Convergence rate of a single vehicle case q For the single-vehicle case, the convergence rate is characterized by where n Using convergence rate bound as a design indicator q q q Design λg* to maximize the convergence rate Potential function Empirical distribution distance

Parallel Sampling n Problems with sequential sampling q q n Global indexing is difficult in practice Long time for one sweep Parallel sampling q q q Agents update locations in parallel by sampling local characteristics Conflicts could be solved by coin-toss. Simulation showed the MAS achieve global objective with only local strategies.

Stochastic Path Planning Simulation n Parallel stochastic path exploration based on MRF can get around the local minima n Potential function Target (attraction) potential Jg q Neighbor (avoidance) potential Jn q q Obstacle potential Jo

Simulation : Gathering n Potential function q q The first term attracts nodes close to z 0 The second term tends to cluster nodes

Simulation: Gathering specified center Z 0=(25, 25) n n unspecified center 200 nodes on 50 by 50 grid； 1= 0. 05 , 2 =1, =103 Rm=2 2, Rs=6 2 ； T(n)=1/(4 log(400+n))

Simulation: Line Formation n Potential function q q q is scaling factor is a penalization for node with no neighbor mk is the number of neighboring nodes of node k k, k’ is the desired angle of the line segment dk, k’ /Rs puts more weight on farther neighbors, which encourages the formation of long lines

Simulation: Line Formation One line Three lines n n Two lines n 200 nodes on 50 by 50 grid =10 , =5 Rm=2 2 Rs=10 2, 6 2, 4 2 T(n)=1/(4 log(400+n))

A Hybrid Control Scheme n Deterministic potential approach q q n Stochastic approach based on MRF q q n Pro: Save traveling time Con: May be get trapped by some obstacles Pro: Trouble free. Converge to global minimum for sure. Con: Waste time for path exploration Hybrid control scheme combines both advantages and may strike the right balance

Hybrid Scheme Algorithm n Step 1. Each vehicle (node) starts with the deterministic gradient-flow method and goes to Step 2 n Step 2. If a vehicle stops moving for d consecutive time instants and its location is not within the target area, then it switches to the simulated annealing method with a predetermined cooling schedule n Step 3. After performing simulated annealing for N time instants, the vehicle switches to the gradient method and goes to Step 2

Impact of Memory n Hybrid scheme with memory q q Experience can help vehicle to learn the complex environments better and thus change its behavior to achieve better performance. Implementation: when a vehicle determines it is trapped , it increases the risk level R of that spot, and does local sampling as follows

Impact of Memory (cont. ) n Hybrid scheme with memory

Autonomic Wireless Networks ● Wireless networks, such as mobile ad hoc networks (MANET) and sensor networks: q q q No trusted centralized authority Resource (power, bandwidth, computation etc. ) constraints Rapidly and dynamically changing topology and connectivity Uncertainty & incompleteness of trust evidence: trust values in [-1, 1] Distributed trust computation and locality of trust information exchanges ● Unique properties q Each node is its own authority and it is selfish q Networking functions (route discovery, packet forwarding and etc. ) rely on cooperation between nodes q Cooperation utilizes local information and local interactions (between neighbors)

Cooperation and Games ● In distributed wireless networks q Cooperation is restricted to only local interactions q Decision is made by each node individually q Nodes are self-interested q Explain and analyze emergent properties ● Game theoretic methods q Provide a framework for modeling individual interactions q Understand complex global structures and dynamics of a system composed of a large number of agents with simple local interactions q Guide for analytical approach q Examples: Ising model, prisoner’s dilemma ● Goal: how to encourage nodes to collaborate in games? q Incentive: trust systems to promote cooperation and circumvent misbehaving nodes.

A Simple Distributed Trust Computation Policy n Based on simple voting methods n Voters: q q n Nodes that qualified as legitimate voters by certificates signed by offline servers (have trust evidence about node i) Assume uniformly distributed in the network Policy: decision based on threshold q q q is the total number of votes node i received (signed sum) is the decision threshold is the number of i’s neighbors

Simple Voting Scheme Trusted nodes Neutral nodes Positive votes Negative votes n n Number of positive votes on node i: Vp, i = 3 Number of negative votes on node i: Vn, i = 1 Effective votes: Vi = Vp, i - Vn, i= 2 Given η = 0. 3, Vi > η|Ni| = 1. 8, node i is designated “trusted”

Trust Dynamics • Trust spreading Initial “islands” of trusts Trust spreads Trust-connected network ● Trust revocation: q Changes in topology, membership, secure paths q Referees of a node may change, trust evidence for a node may change q Votes timeout or negative votes

Trust Graph n Trust graph: GT(VT, ET) q q q n Induced subgraph of G(V, E) by VT VT is the set of nodes which are designated “trusted” by the trust computation algorithm ET = {e | e in E and both ends of e are in VT} Trust metric Psp: percentage of trusted pairs that are connected by one or more secure paths, which are composed of trusted nodes q q NPsecure is the number of trusted pairs that are connected by one or more secure paths. It is dependent of the cluster size and connectivity of GT

Random Graph Model Erdos and Renyi random graphs (ER model) n n Simulation results of Psp as function of decision threshold η When η is small q Most of nodes are considered to be trusted q Psp is dominated by the edge present probability p in ER random graphs q Zero-one law in random graph theory is present Increasing the threshold η results in q Reducing the number of trusted nodes q Increasing critical values q Smaller Psp

Small-world Networks Psp vs. η after one iteration n n Psp vs. η in steady states Number of trusted paths increases as trust spreads with each iteration Different curves are with different rewiring probability Prw= 0 represents a regular lattice q Prw = 1 converges to a random graph Observe the transition from lattices to random graphs q With a relative small portion of shortcuts, small-world networks facilitate the formation of secure paths q n n The effects of topology are obvious, so any distributed trust computation model should take into account the topology properties

Trust Revocation n The trust revocation process is initiated: q q q n when topology, membership or secure paths change when referees or trust evidence for a node changes when positive votes are timeout or new negative votes are received Decision policy of the revocation process q q q Revocation on a specific node, say B, usually starts from few nodes that have negative observations on B; A node A accepts the revocation on B, if it finds that more than a threshold fraction Φ of its neighbors revoke node B; Question: can a revocation be accepted by a large fraction of nodes in the network? A Trust Revoke

Phase Transition of Revocation n Revocation is launched from a randomly chosen node in an Erdös. Rényi random graph with average degree set as the Y-axis. Global cascade: area that lie inside of the contour represents the percentage of nodes, which accept the revocation, is greater than the value corresponding to the contour (level surfaces of histogram) Phase transitions happen suddenly: the steep of the contours is very sharp, which represents phase transitions

Previous Work n Decentralized path-inference protocols q q n Local interaction q q q n Combination of trust along and across paths (Beth, 1994) Probability of finding a trust path from source to target (Maurer, 1996) Eigen. Trust (Kamvar, 2003) Peer. Trust (Xiong, 2004) Bayesian methods (Buchegger, 2003) Our work is similar with Eigen. Trust and Peer. Trust, which provided promising results. q q q However, results of Eigen. Trust and Peer. Trust are all based on simulations. We analyze our local interaction rule using graph theory. We also provide a theoretical justification for network management that facilitates trust propagation.

Voting Scheme n Voting rule: q q n is the trust value of node i is the voting value of node j about node i Local voting rule Function f should satisfy the following properties: n The range of f is [-1, 1]. n Votes from neighbors with higher trust value are more credible, so they should carry larger weights. Policy: threshold rule for trustworthiness of the target agent where is the threshold, which is a constant

Simple Voting Rule n We use the weighted average as the voting rule, where weights are trust values of voters q q q n is the degree of node i n represents discrete time Assume is a constant, i. e. it doesn’t change with time, which is true when considering the steady state The voting rule can be written in system equation where D = diag[d 1 , d 2 , …, d. N], T is a vector representing trust values of all nodes and V is the matrix for votes

Convergence of Simple Voting Rule n Voting without uncertainty q q q For each pair (i, j) , if i and j are neighbors, then vij = 1. V = A, where A is the adjacency matrix of graph G, and D-1 A is a stochastic matrix with the largest eigenvalue being 1. Let be the right eigenvector of D-1 A corresponding to eigenvalue 1. then q n If , all nodes are trusted, and none is trusted otherwise. The initial trust values are very crucial. Voting with uncertainty q q vij ≤ 1, D-1 A is a semi-stochastic matrix. We proved , so T 0. established at all!!! Trust cannot be

Voting with Headers n We have shown that using the simple voting scheme, trust can only be established under certain strict conditions: q q n All votes value are 1 and the initial configuration must satisfy A single vote with value less than 1 will result in failure of trust establishment. We introduce the notion of headers q q q Headers are pre-trusted agents and only vote for nodes that they fully trust. If node i is trusted with bi headers, it will get bi more votes with value 1. Let B = diag[b 1 , b 2 , …, b. N ]. The system equation changes to

Convergence of Voting with Headers n Voting without uncertainty q q n V = A, and define . The system equation changes to If there is at least one node i such that bi > 0, (D+B)-1 A goes to 0. Therefore T(n) 1 and all nodes are trusted. Voting with uncertainty q q q Using the same technique as above, let. We are able to find the condition such that If we let , then all nodes are trusted. Theorem: Given the threshold is η , the number of headers for each node must satisfy n This theorem proves, as well as provides, a network design method to establish a fully trusted network by introducing headers

Spreading Speed and Topology n n The time for updating rule to reach the steady state, i. e. , how fast the trust values converge. Perron-Frobenius Theorem in algebraic graph theory: For a stochastic matrix A q q q n is the largest eigenvalue of A, which is 1 and is the second largest eigenvalue of A. The convergence rate of An is of order Normalized adjacency matrices are stochastic matrices, therefore those with smaller converge faster. What kind of networks or which network topology has smaller second largest eigenvalue

Spreading Speed and Topology (cont’) n We consider the small-world model proposed by Watts and Strogatz in 1998 q q q High clustering coefficient and small average graphical distance between any pair. We use Φ-model, which is modeled by adding small number of new edges into a regular lattice. Adding just 1% more edges, spreading finishes in 10 times less rounds.

Ising and Spin Glass Models ● Statistical Physics models for magnetization q q q Orientation of each particle’s spin depends on its neighbors Ising Model: behavior of simple magnets Spin Glass Model: complex materials ● Math interpretation: q s = {s 1, s 2, …, sn} is a configuration of n particle spins, where sj = 1 or -1 , spin j is up or down q Hamiltonian, or Energy for configuration s – Ising Model: Jij = J for all i, j – Spin Glass Model: Jij depend on i, j and can be random processes

Ising/SG Models and Games ● Ising and Spin Glass models can be interpreted as dynamic (repeated) games: each particle selects its own spin to maximize its own payoff q q ● n High T, conservative agents, not willing to change, small payoffs n Low T, aggressive agents, larger payoffs Collection of local decisions reduces the total energy of the interacting particles Statistical Mechanics primary object of interest q q ● Ising model (Jij = J) : align its spin with the majority of neighbors spin Recent excitement: computation of ground state, partition function Z, NP - complete, Replica Method Application to: turbocodes, image restoration, neural networks, learning, associative memory, SAT, knapsack, SA, number parttioning, graph partitioning, CDMA, MIMO, … Inspires an approach where trust is used as an incentive for cooperation q q q si represents whether node i cooperates or not with neighbors Jij can be interpreted as the worth of player j to player i Cooperate or not based on benefit from cooperation and trust values of neighbors

Spin Glass Cooperative Game ● Spin Glass model as a cooperative game (spin glass game) In q , the weights wij frustrate the system Both positive and negative local feedback (e. g. wij {-1, 1}) Interaction topology (i. e. the matrix J = [Jij] ) moderates effects pos. and neg. fback S N = {1, 2, …, N} is a coalition, in which all nodesΓcooperate =(N, v) v(S) : value of characteristic function of the game , v: 2 N R; maximum payoff S 2 6 J can get without cooperation from other nodes N /S. q q 21 J 12 5 1 3 J 34 J 14 J 41 q 4 J 43 Subset S={1, 2, 3, 4} v(S)=J 12+J 21+J 14+J 41+J 43+J 34 -J 36 -J 15 Model can be used to find what form or policy for Jij can induce all (or most)

Cooperative Games and Dynamic Coalitions ● Have a number of players, some can be coalitions themselves ● How do they negotiate an “acceptable” DC security policies set? ● What are the properties of the final result: “the negotiated policy set”? ● Is there an efficient scheme that gets us there? ● Cooperative games allow us to set up different types of games between the players, examine different concepts of solutions and values ● Can prove mathematically properties of the solution and value: e. g. minimizes maximum dissatisfaction, is anonymous, is stable ● Can get iterative methods to get to solution (negotiation schema), can use all kinds of constraints, invariance to a. V + b scaling (preferences) ● Working on extensions to partial information, learning, robustness to uncertainties

Spin Glass Cooperative Game Properties ● Spin Glass game is a convex and superadditive game iff (net pos. effects) ● Shapley value : in the core ● Not well understood in the regime of both negative and positive net effects ● Effects of interaction matrix structure (sparsity, neighborhood structure, range of interactions, strength of interactions) not well understood; Topology effects in network analog ● Oriented Spin Glass Game Γ(N, v) where v now depends on both an interaction matrix J and a preference vector L ; a pair of char. fcns ● Replica method can be used to analyze various problems under various models and constraints on J and L

Cooperative Games with Negotiation ● Consider Γ = (N, v), N as before but with ● Γ = (N, v) convex, superadditive, if ● Theorem : Γ = (N, v) has a nonempty core. The payoff allocation to node i , is in the core. Compute q q as follows This payoff allocation indicates a way to encourage cooperation Players with positive gain can negotiate with their neighbors by sacrificing certain gain (offering their partial gain ijxji )

Trust as Mechanism to Induce Collaboration ● Trust is an incentive for collaboration q Nodes who refrain from cooperation get lower trust values q They will be eventually penalized because other nodes tend to only cooperate with highly trusted ones. ● Assume, for node i, that the loss for not cooperating with node j is a nondecreasing function of xji as f (xji), and the new characteristic function is ● Theorem : if q , the core is nonempty and is a feasible payoff allocation in the core. By introducing a trust mechanism, all nodes are induced to collaborate without any negotiation

Dynamics of Cooperation ● System model Two linked dynamics • Trust propagation • Game evolution ● The network is modeled as a discrete-time system j all neighbors of i vij trust value node i votes for node j

Game Evolution ● Strategy of node i: q γij= 1 (= 0) represents that i cooperates (does not cooperate) with its neighbor j ● Payoff for node i when interacting with j: xij = Jij γji q xij > 0 (< 0) positive link (negative link) q Node selfishness cooperate with neighbors on positive links ● Strategy updates: node i chooses γij= 1 only if all of the following are satisfied: q q q Neighbor j has not been revoked Neighbor j is cooperative xij > 0, or the cumulative payoff of i is less than the case when it unconditionally conducts γij= 1. ● Trust propagation: q The threshold is chosen to ensure global revocation propagation q Reestablishing period τ : once a node is revoked, in order to reestablish trust the revocation has to be nullified for τ consecutive time steps

Results of Game Evolution ● Theorem: a reestablishing period τ > τ0 q q , there exists τ0, such that for The iterated game converges to Nash equilibrium; In the Nash equilibrium, all nodes cooperate with all their neighbors. ● Comparison of games with (without) trust mechanism, strategy update: Percentage of cooperating pairs vs negative links Average payoffs vs negative links

Conclusions and Future Research n n n n n A stochastic potential based approach guarantees global objective can be achieved by simple local strategies The parallel sampling algorithm saves running time compared with the sequential sampling algorithm Emergent behaviors of self-organized swarms are observed in simulations A hybrid scheme is proposed to achieve better performance by combining deterministic gradient-flow approach and stochastic potential based approach Convergence study of the distributed parallel algorithm Tighter convergence rate bound estimation and parameters estimation of the hybrid scheme Cooperative learning to further improve the performance of the hybrid scheme Convergence analysis when only partially observed potential functions available due to imperfect sensors Schedule of measurements due to sensor power constraints

Conclusions and Future Research ● Analyzed and evaluated fundamental methods to induce collaboration ● ● ● ● in wireless networks with mobile nodes Focused on distributed schemes using only local interactions Developed analyzed a cooperative game framework and showed that negotiation between agents can induce collaboration We developed a distributed trust establishment, propagation and maintenance scheme for such networks and showed that it can also induce collaboration Showed that trust propagation displays phase transitions Investigated the linked dynamics of trust propagation and game evolution and showed the benefits in inducing collaboration Methods inspired from statistical physics of spin glasses Future directions include analysis of networks with dynamic topologies, robustness of these collaboration inducing mechanisms, identification of parameters (including topology types) that influence the dynamics and qualities of collaborative behavior

Publications § Tao Jiang and John S. Baras, “Ant-based Adaptive Trust Evidence Distribution in MANET”, Proceedings of 2 nd International Workshop on Mobile Distributed Computing, in conjunction with the Intern. Conference on Distributed Computing Systems, Tokyo, Japan, March 2004. § John S. Baras and Tao Jiang, “Dynamic and Distributed Trust for Mobile Ad-Hoc Networks”, Proceedings of 24 th Army Science Conference, Orlando, Florida, December 2004. § John S. Baras and Tao Jiang, “Cooperative Games, Phase Transitions on Graphs and Distributed Trust In MANET”, invited paper, Proceedings 2004 IEEE Conference on Decision and Control, December 2004, Bahamas. § John S. Baras and Tao Jiang, “Managing Trust in Self-organized Mobile Adhoc Networks”, invited paper, Wireless and Mobile Security Workshop, Network and Distributed Systems Security Symposium, February 2005, San Diego, USA. § Tao Jiang and John S. Baras, “Autonomous Trust Establishment”, 2 nd International Network Optimization Conference (INOC), February 2005, Lisbon, Portugal. § John S. Baras and Tao Jiang, “Cooperation, Trust and Games in Wireless Networks”, invited paper, in Proceedings of Symposium on Systems, Control and Networks, honoring Professor P. Varaiya, Birkhauser, June 2005. n Tao Jiang and John S. Baras, “Graph Algebraic Interpretation of Trust Establishment in Autonomic Networks”, submitted to Wiley Journal of Networks (special issue)

Publications n J. S. Baras, X. Tan and P. Hovareshti, Decentralized Control of Autonomous Vehicles, ” in Proc. of 42 nd IEEE Conference on Decision and Control, Hawai, Dec 2003. n W. Xi, X. Tan, and J. S. Baras, “A stochastic algorithm for self-organization of autonomous swarms, ” to appear in Proc. 44 th IEEE Conference on Decision and Control. n J. S. Baras and X. Tan, “Control of autonomous swarms using Gibbs sampling, ” in Proceedings of the 43 rd IEEE Conference on Decision and Control, Atlantis, Paradise Island, Bahamas, 2004, pp. 4752– 4757. n W. Xi, X. Tan, and J. S. Baras, “Gibbs sampler-based path planning for autonomous vehicles: Convergence analysis, ” in Proceedings of the 16 th IFAC World Congress, Prague, Czech Republic, 2005. n [4]W. Xi, X. Tan, and J. S. Baras, “A hybrid scheme for distributed control of autonomous swarms, ” 2005, in Proc. of 24 th American Control Conference.