Data Fusion A review Layout Data Fusion A

Data Fusion- A review

Layout Data Fusion- A review 1 -Benefits of multisensor devices 2 -Typical sensors used in data fusion 3 -Sensor performance 4 -Data fusion models 5 -Decision fusion in parallel sensor suite 6 -Comparison of mathematical tools in data fusion

Data Fusion- A review Benefits of Multiple sensor devices: -Reduction in measurement time -A downtime reduction and an increase in reliability -Redundant and complementary information -A higher signal-to-noise ratio -A reduction in measured uncertainty -A more complete picture of the environment

Data Fusion- A review Survey of typical sensors used in Data fusion Sensor Output format Applications Optical sensor Image Mobile robot guidance Radar Pulse signal Target detection and target tracking Infrared sensor Image Object identification Satellite Image Surveillance and pattern recognition Ultrasonic sensor Pulse signal Mobile robot guidance NDT sensor Voltage Materials examination Sonar Pulse echo Obstacle detection Laser Image Pattern recognition X-ray Image Medical

Data Fusion- A review

Sensor performance Data Fusion- A review Sensor performance can be statistically represented using: POD : Probability Of Detection PFC: Probability of False Call ROC: Receiver Operating Characteristic (POD versus PFC) Major advantage of ROC curves compared to POD curves: In ROC curves false calls are taken into account But… In practice, ROC curves are difficult to realise.

Sensor performance Data Fusion- A review The performance and potential of each used sensor needs to be established in order to assign weight of evidence, for example, in sensor data fusion. The most common sources of uncertainty -little or no knowledge about measurement -incomplete measurement (when data are approximated rather than waiting for complete data which may be time consuming and costly) -limitations of the system

Data Fusion- A review Sensor performance Common types of errors -ambiguous - practicality (environment) -Human error -Incorrect output -incorrect -Equipment malfunction -Unreliable -Calibration error -False negative -measurement -No output -Precision -False positive -Accuracy -random -incomplete -systematic -reasoning -Inductive error -Deductive error

Data Fusion Models Data Fusion- A review -Multisensor data integration and fusion center -Three-level fusion paradigm -Centralized signal detection system -Distributed (decentralized) signal detection system [X. E. Gros, NDT Data Fusion, 1997]

Data Fusion- A review Data Fusion Models Multisensor data integration and fusion center Sensor 1 X 1 Y 1 X 2 Y 2 Sensor 1 Raw Data Processing Processed Data Yn Xn Z 1 Assignment of Bayes or Dempster. Shafer Rule of Combination Z 2 Sensor Data Selection Zn Fused Data Optimum Estimation Decision Level Sensor 1 Integration Fusion Integration Measurements from n sensors are integrated, data is then processed with evidental reasoning, probabilistic and belief theories, the results are classified and selected before a decision on the optimum fused information is made.

Data Fusion- A review Data Fusion Models Three-level fusion paradigm Sensors Data Fusion Signal Level Features fusion Level of evidence Database Level of Dynamics Decision Fusion

Data Fusion- A review Data Fusion Models Distributed (decentralized) signal detection system Local Decision Sensor 1 Measurement Sensor 2 Measurement Sensor N Measurement Feature Extraction Level Local Decision Fusion Center Global Decision Level Local Decision Fuse identity declarations using Bayesian theory, the Dempster-Shafer paradigm or Thomopoulos generalized evidence processing (GEP). The output from each sensor is a decision which forms the inputs to a fusion center where association is performed.

Data Fusion- A review Data Fusion Models Centralized signal detection system Sensor 1 Measurement Sensor 2 Measurement Sensor N Measurement Fusion Center Decision Level More suitable for fusion of raw data but the association phase can be difficult.

Data Fusion Models Data Fusion- A review Four major sensor network types -Serial -Parallel-Serial-Parallel Sensor 1 Sensor 2 Sensor n

Data Fusion- A review

Data Fusion- A review Decision fusion in parallel sensor suite Data set Z 1 Sensor 2 Sensor j Sensor t Local Output O 1 processor 1 Data set Z 2 Local Output O 2 processor 2 Data set Zj Local Yes Decision Fusion Output Oj Processor Consistent decision across the suite? processor j Data set Zt No Local Output Ot processor t A recursive processing structure for enhanced performance with a parallel sensor suite. B. V. Dasarathy, 1991, IEEE

Data Fusion- A review Sensor output can be regarded as a decision array of n decisions. The efficiency of each sensor, , is the probability of correctness of the decision Dj from sensor j, a measure of the effectiveness of a sensor. Cj and Wj : the belief that the decisions from sensor j are correct and wrong (based on the Dempster-Shafer Theory) Uj: the ignorance (uncertainty) of a measurand

Decision fusion in parallel sensor suite Data Fusion- A review ck , wk , uk : incremental probabilities of the joint correct, incorrect decisions and nondecisions respectively

Decision fusion in parallel sensor suite Data Fusion- A review

Decision fusion in parallel sensor suite Data Fusion- A review

Data Fusion- A review Decision fusion in parallel sensor suite Second sensor Decision First sensor Decision Target Nontarget Undecided Target Undecided Nontarget Undecided Undecided A simple decision fusion rules matrix

Decision fusion in parallel sensor suite sensor 1 Local processor 1 sensor 2 Local processor 2 Data Fusion- A review Decision Fusion processor

Decision fusion in parallel sensor suite Data Fusion- A review Perfect Sensor Case: ηi = 1 ( i = 1, 2 ) η : the efficiency of the imperfect sensor The fused decision approaches the correct decision asymptotically even though one of the sensors may remain imperfect and the user does not know which one it is.

Decision fusion in parallel sensor suite Data Fusion- A review Bad Sensor Case: ηi = 0 ( i = 1, 2 ) Fusion leads to complete failure of the system. Therefore no totally faulty sensor can be allowed to operate indefinitely in a two-sensor fusion system of this type.

Decision fusion in parallel sensor suite Data Fusion- A review Equally Imperfect Sensor Case: ηi = η ( i = 1, 2 ) Minimum number of recursions needed for the fused decision to be better than the decision derived by the individual sensor:

Data Fusion- A review Decision fusion in parallel sensor suite u 1|max=0. 5 c 1, w 1, Ck |max, Wk |max c, w u η η Fused correct (c 1), incorrect (w 1) and non-decision (u 1) rates vs. sensor efficiency (η) Initial (c 1, w 1) and final (Ck |max , Wk |max ) fused decision rates vs. sensor efficiency

Decision fusion in parallel sensor suite Data Fusion- A review η=0. 1 η=0. 2 η=0. 3 η=0. 4 η=0. 5 η=0. 6 η=0. 7 η=0. 8 η=0. 9 Ck Ck η Fused correct decision rate (Ck ) vs. sensor efficiency ( η) at different numbers of recursions (k) k Fused correct decision rate (Ck ) vs. recursion number (k) at different sensor efficiencies (η)

Decision fusion in parallel sensor suite General Case Data Fusion- A review η 1 and η 2 are related by α Asymptotic fused correct decision Ck |max rate( Ck|max ) vs. sensor efficiency ( η ) at different sensor performance ratios (α ) η

Data Fusion- A review Decision fusion in parallel sensor suite η=0. 1 η=0. 2 η=0. 3 η=0. 4 η=0. 5 η=0. 6 η=0. 7 η=0. 8 η=0. 9 Ck |max t Asymptotic fused decision efficiency ( Ck | max ) vs. number of sensors ( t) for different sensor efficiencies (η )

Decision fusion in parallel sensor suite Data Fusion- A review Equally Imperfect Sensor Case: ηi = η ( i = 1, 2, …, t ) Minimum number of recursions needed for the fused decision to be better than the decision derived by the individual sensor:

Decision fusion in parallel sensor suite Data Fusion- A review

Decision fusion in parallel sensor suite Data Fusion- A review m: the number of hypothesis

Decision fusion in parallel sensor suite Data Fusion- A review The minimum number of sensors for the correct fused decision rate to exceed the incorrect fused decision rate: The asymptotic values of the fused decision rates:

Data Fusion- A review Decision fusion in parallel sensor suite Binary Decision making 1 w 1 : majority 0. 9 c 1: majority 0. 8 c 1, w 1 0. 7 0. 6 0. 5 w 1 : consensus 0. 4 c 1 consensus 0. 3 0. 2 0. 1 0 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 1 η Initial fused decision rates vs. sensor efficiency with three sensors (comparison of the consensus and majority based fusion methods)

Data Fusion- A review Decision fusion in parallel sensor suite Multihypothesis Decision making (m=3) 1 0. 9 w 1 : majority 0. 8 c 1, w 1 c 1 majority 0. 7 0. 6 0. 5 0. 4 w 1 : consensus c 1 consensus 0. 3 0. 2 0. 1 0 0 0. 1 0. 2 0. 3 0. 4 0. 5 1 η 0. 6 0. 7 0. 8 0. 9 Initial fused decision rates vs. sensor efficiency with three sensors (comparison of the consensus and majority based fusion methods)

Decision fusion in parallel sensor suite Data Fusion- A review

Data Fusion- A review Fusion Methodology The most common data fusion and integration methods Fusion method Applications Pixel level fusion Image processing, image segmentation Bayesian theory Decision making between multiple hypotheses Demspter-Shafer theory of evidence Decision making, Beliefs intervals Neural Network Signal interpretation Neyman-Pearson criteria Decision making Fuzzy Logic Handle vagueness Knowledge based system Pattern recognition Markov random field Image processing

Data Fusion- A review Fusion Methodology Classical Inference The most common inference approaches based on an observed sample of data for acceptance or rejection of a hypothesis: -Maximum a posteriori -Likelihood ratio criterion -Neyman-Pearson test -Bayes criteria

Data Fusion- A review Fusion Methodology Classical Inference Maximum a posteriori Compares two probabilities assigned to two hypothesis and favors either one or the other depending only on their chance of occurrence. y is an observation from a sensor and Hi a hypothesis i

Data Fusion- A review Fusion Methodology Classical Inference Likelihood ratio criterion A test to decide between hypothesis H 0 or its alternative Hi. If Λ(u)>t , H 0 is true otherwise, H 1 is true. . Likelihood ratio = (level of sufficiency) H 0 and H 1 are hypothesis 0 and 1, n the number of sensors, ui random observed sample data and t, the threshold (significance level) determined from experiment. Λ(u) : the degree to which the observation of evidence u influences the Prior probability of H

Data Fusion- A review Fusion Methodology Classical Inference Neyman-Pearson hypothesis test A general theory used to make a decision between two hypothesis. Hypothesis H 0 is rejected if the following equation is verified: The threshold t is chosen depending on the risk the user is prepared to take to accept or reject H. the smaller the value of t, the lower the risk.

Data Fusion- A review Fusion Methodology Classical Inference Bayes criteria A cost function based on false alarm and probability of detection is used to select between two hypotheses H 0 and H 1. P 0 and P 1 are a priori probabilities which govern the decision output. The cost function C for each decision outcome: -C 00 : the cost function assigned to the decision 0 when the true outcome is 0 P(H 0|H 0) : the probability associated with this decision - C 01 : the cost function assigned to the decision 0 when the true outcome is 1 P(H 0|H 1) : the probability associated with this decision - C 10 : the cost function assigned to the decision 1 when the true outcome is 0 P(H 1|H 0) : the probability associated with this decision - C 11 : the cost function assigned to the decision 1 when the true outcome is 1 P(H 1|H 1) : the probability associated with this decision

Data Fusion- A review Fusion Methodology Classical Inference Bayes criteria The expected values of the cost as the risk R is defined as : The decision intervals are defined as: Where the right hand side is the threshold of the test and should be such that the cost is as small as possible.

Data Fusion- A review Fusion Methodology Bayesian Inference Used to estimate the degree of certainty of multiple sensors providing information about a measurand. Uses a priori probability of a hypothesis to produce an a posteriori Probability of this hypothesis. Suppose there are n mutually exclusive and exhaustive hypotheses H 0…Hn that an event E will occur. The conditional probability p(E|Hi) states the probability of an event E that Hi is true and is given by: p(Hi) : a priori probability of the hypothesis Hi p(Hi|E): a posteriori probability of having E given that Hi is true

Data Fusion- A review Fusion Methodology Bayesian Inference If multiple sensors are used…

Fusion Methodology Data Fusion- A review Bayesian Fusion • Target location and tracking • Search formation of targets in a region • …

Data Fusion- A review Fusion Methodology Example: Two sensor data fusion x: to be identified (e. g. aircraft) Latest data set Old data set Current measurement

Data Fusion- A review Fusion Methodology Bayesian Inference Some limitations: -no representation of ignorance is possible -prior probability may be difficult to define -result depends on choice of prior probability -it assumes coherent sources of information -adequate for human assessment (more difficult for machine-driven decision making) -complex with large number of hypotheses -poor performance with non-informative prior probability (relies on experimental data only)

Fusion Methodology Data Fusion- A review Dempster-Shafer Evidental reasoning Often described as an extension of the probability theory or a Generalization of the Bayesian inference method. Frame of discernment Θ={X 0, X 1 , …Xn} Mass probability (basic probability assignment (bpa)) : m(X)

Data Fusion- A review Fusion Methodology Dempster-Shafer Evidental reasoning Bel(X) : the degree of support Properties of the belief function:

Data Fusion- A review Fusion Methodology Dempster-Shafer Evidential reasoning Dempster rule of combination:

Data Fusion- A review Fusion Methodology Dempster-Shafer Evidental reasoning Geometrical representation of Dempster rule of combination m 2(Xn) … 1 m 1, 2 (Xi, Xj) m 2(X 1) 0 0 m 1(X 1) 1 m 1(Xi) m 1(Xn)

Data Fusion- A review Fusion Methodology Dempster-Shafer Evidental reasoning Incertitude 0 Belief Disbelief Plausibility [Bel(X), Pls(X)] Decision [0, 1] Total ignorance, no belief in support of X [1, 1] Proposition X is completely true [0, 0] Proposition X is completely false [0. 4, 1] Partial belief, tends to support X [0, 0. 7] Partial disbelief, tends to refute X [0. 3, 0. 5] Both support and refute X 1

Fusion Methodology Data Fusion- A review Dempster-Shafer Fusion Gives a rule for calculating the confidence measure of each state, based on data from both new and old evidence. Assigns its masses to all of the subsets of the entities that comprise a system • Mobile robot map building (e. g. occupancy grid) {occupied, empty, unknown} : {O, E, U} m: confidence in each element ms: confidence from sensors mo: confidence from old existing evidence

Data Fusion- A review Fusion Methodology Dempster-Shafer Evidental reasoning Some features: -An overestimation of the final assessment can occur -Small changes in input can cause important changes in output -High efficiency with bodies of evidence in pseudo-agreement -Lower efficiency with bodies of evidence in conflict

Fusion Methodology Data Fusion- A review Bayesian Fusion vs. Dempster-Shafer Fusion Bayes: • Works with probabilities, numbers that reflect how often an event will occur • Less calculations. Dempster-Shafer: • Considers a space of elements that each reflect not what Nature chooses, but rather the state of our knowledge after making a measurement. • Calculations tend to be longer. • Allows more explicitly for an undecided state of our knowledge. (in military it is sometimes far safer to be undecided than to decide wrongly) • Sometimes fails to give an acceptable solution.

Data Fusion- A review Fusion Methodology Fuzzy Logic Inference Technique Is very flexible and there is no universal rule of formalism which can be associated with it. Fuzzy logic evaluates qualitatively a signal from a sensor and fuzzy sets associate a grade (numerical value) to each element. Typical associated values for different elements in fuzzy logic Element Associated value Associated reliability Signal high [1. 0, 0. 7] Certain Signal medium [0. 7, 0. 3] Uncertain Signal low [0. 3, 0. 0] Incorrect

Data Fusion- A review Fusion Methodology Fuzzy Logic Inference Technique Fuzzy logic methods can be very useful to represent uncertainty from multiple sensors and to handle vagueness. A multilevel system to handle vagueness: -sensor level -data fusion level -reasoning level Produce information Integrate information Generates a decision making use of artificial intelligent systems

Data Fusion- A review Fusion Methodology Fuzzy Logic Inference Technique Combining information from multiple images to improve classification accuracy of a scene where images are processed at the pixel level using segmentation algorithm. Can be performed for… image processing and image smoothing image segmentation to combine information perceived by visual sensors. Fusion center

Data Fusion- A review Fusion Methodology Artificial Intelligence AI techniques developed for data association make use of expert systems and neural networks Artificial Neural networks (NNs) are software simulated processing units or nodes, which are trained in order to solve problems. NNs can be very useful to solve problems in applications where it is difficult to specify an algorithm. They are composed of interconnected nodes that act as independent processing units

Data Fusion- A review Fusion Methodology Artificial Intelligence Weights wi Input xi node Output signal A two-layer neural network, Perceptron

Data Fusion- A review Fusion Methodology Artificial Intelligence Some NN applications in data fusion -For sensor data fusion for detection and correct classification of space object maneuvers observed by radar of different frequencies and resolution. -Used in decision systems for target tracking, object detection, recognition and classification in defence applications -In image processing operations such as filtering and segmentation -To select matching pixel based fusion from sensors for robotics application. To perform pixel-to-pixel image association for object identification -Applied to non-destructive examination for eddy current signal classification and automatic tube inspection, defect characterisation, classification of weld defects and signal interpretation.

A Review on Decision Fusion Strategies Data Fusion- A review Acknowledgements: • This powerpoint presentation was prepared by Miss Mahdavi and Miss Bahari former M. Sc. Students at School of ECE , University of Tehran in Dec. 2005 where is highly appreciated.