An Edge detection and HPFbased Intelligent Space A

An Edge detection and HPF-based Intelligent Space – A Network based Integrated Navigation System By, Rachana Ashok Gupta Under the direction of Dr. Mo-Yuen Chow 9/15/2020 ADAC, NC State University 1

NC STATE UNIVERSITY l Overview of presentation What are network based integrated navigation system (NBINS) in brief followed by Abstract for this research. l l l Introduction to i. Space – i. Space components and modules – Limitations and Scope of improvement as a NBINS The new structure of i. Space – New modules (edge detection and HPF planner) – Advantages and improvements achieved over the old structure Results and Discussions ADAC, NC State University 2

NC STATE UNIVERSITY Network based integrated navigation system Network based Integrated Navigation systems – Different modules combined together to guide a UGV (Unmanned Ground Vehicle) from one point to another in the space of interest ( 2), where the navigational intelligence lies on a main controller away from the UGV. l Advantages – – – l Applications – – – l Remotely control over a long-distance. Efficient to fuse global information. Scalability - Easy to add more sensors and UGVs with very little cost and without heavy structural changes. Manufacturing plant monitoring Nursing homes or hospitals Tele-robotics & Tele-operation etc Issues to be considered – – – Network and processing delay Data sharing and Interfacing ADAC, NC State University 3

NC STATE UNIVERSITY What is Intelligent Space l Human-machine interaction in i. Space l A new concept to effectively use distributed sensors, actuators, robots, computing processors, and information technology over a physically and/or virtually connected space. For examples, a room, a corridor, a hospital, an office, or a planet. It fuses global information within the space of interest to make intelligent operation decision such as how to move a mobile robot effectively from one location to another. ADAC, NC State University 4

NC STATE UNIVERSITY l Components – – l Overhead network camera Network controller with graphical user interface A Differential drive UGV as the navigator Computer Network (IP) On the main network controller – – l i. Space as a NBINS Image acquisition Image processing Path generation Path tracking controller Graphic User Interface (GUI) – Any remote Computing interface in the world (Internet) ADAC, NC State University 5

NC STATE UNIVERSITY l Image acquisition – l Template Matching Top view Image processing – – Image Thresholding – Black and white Template matching Circles used for rotation invariance Draws the circle around obstacle with safety margin radius of rsafe. f. B – Template image ADAC, NC State University 6

NC STATE UNIVERSITY l Find a path from the starting point A to the end point B for the UGV – – l Path generation The path of the UGV should be as short as possible (minimize time) The path of the UGV should not collide with any obstructions Fast Marching Method (by J. A. Sethian, Dept. of Mathematics, UC Berkeley) – A numerical technique that counts the shortest distance from a point to the original point with a shortest distance update algorithm ADAC, NC State University 7

NC STATE UNIVERSITY Path tracking The path tracking algorithm runs in every control loop and adjusts the speed and turn rate of the UGV to track the generated path. 1. Calculate the closest point on the reference path from the current UGV position (xc, yc, c). 2. Pick a reference point (xref, yref) on the generated path that is a set distance (d 0) 3. speed and turn rate for the UGV to reach the reference point given its current position and orientation. ADAC, NC State University 8

NC STATE UNIVERSITY l l Time delay issue Network delay component – Sensor to controller delay (Image Acquisition) – Controller to actuator delay (Commands from controller to the UGV) Processing or computational delay component – Non-real time Computational delay (Initial image processing, path planning) – Real time Computational delay (continuous image processing, motion control) ADAC, NC State University 9

NC STATE UNIVERSITY l Template matching output – Insufficient safety margin Conservative safety margin Location co-ordinates of » » l Limitations - Template matching UGV Obstacles Limitations – – Obstacles are from a priori set No knowledge about shape and size Either Insufficient or conservative rsafe Leading to inefficient or non-optimal path planning rsafe All these make the system restricted to operate in only a few environmental patterns. ADAC, NC State University 10

NC STATE UNIVERSITY l l Limitations with Fast Marching Implements and maintains a binary tree through out the path generation. O(N(Log. N)) problem and memory intensive. The time and the number of loop iterations are dependent on the respective position of the destination from the source. Operations involved are – search, distance calculations (squares and sqaure root functions) Every iteration – need of check whether the TRIAL point is outside the rsafe margin of recognized obstacle. ADAC, NC State University 11

NC STATE UNIVERSITY l l Limitations with path planning Navigation problem is looked upon as a path tracking problem and therefore the reference path generation is mandatory. The reference point generation on the path for the UGV (off the path) does not consider the obstacle avoidance. Quadratic curve controller needs current position and the reference point to calculate speed. Complete reference path is not needed. Real time computation to find the closest point on the reference path Point to point guidance function ADAC, NC State University 12

NC STATE UNIVERSITY l l Delay tolerance, efficiency, accuracy, generality and optimality of i. Space depend upon the following factors. Fast, efficient and generic enough Image processing algorithm with different category of obstacle maps for the navigation system. The path planning algorithm will decide – l Optimality and length of the path generated Path tracking algorithm has to continuously consider the obstacle avoidance for a navigation system – – l Important points Probability to hit an obstacle. The time required to track that path Compatibility and data flow between different modules is also a key factor for an integrated system. ADAC, NC State University 13

NC STATE UNIVERSITY New Structure for i. Space Points of emphasis – 1. Processing and computation delay, increase in the efficiency, generality, optimality with each added new module to create a suitable platform for NBINS. 2. Creating a homogenous structure by putting edge detection, HPF planner and quadratic curve fitting path tracking controller - three heterogeneous systems together for the first time to create a network based system is the novel contribution to the network based integrated navigation system area. ADAC, NC State University 14

Edge Detection NC STATE UNIVERSITY Image I Laplacian of Gaussian 2 G Gradient of Gaussian G Zero Crossing Magnitude E(xi, yj) = 1 if (xi, yj) B = 0 if (xi, yj) B Threshold (C) Edge map E for all (i, j) Where E(x, y) is the image representing the edge map and B is the set of edge points including the boundary points for all obstacles in workspace. ADAC, NC State University 15

NC STATE UNIVERSITY 2 -D Harmonic Potential Field Laplace equation for 2 is a Harmonic function. – 2 is the Laplace operator, is the workspace of the UGV ( 2), is the boundary of the obstacles (output of the edge detection stage), and is (x. T, y. T) the target point. The obstacles were represented by the repelling force and the point of destination was represented by the attractive force. – The potential function in closed contour of will converge to a constant potential – The obstacle free path to the target is generated by traversing the negative gradient( ) i. e. . The normalized gradient at each point represents the directional guidance at that point in the workspace. ADAC, NC State University 16

NC STATE UNIVERSITY Solving Laplace Equation Using finite difference method, Taylor series approximation, Thus with Laplace equation, this method simply consists of repeatedly replacing each grid points with the average of its neighbors using successive relaxation. Terminate when the array u contains a sampling of where every non-boundary condition node has a neighbor with a smaller value representing a negative gradient except the destination point. ADAC, NC State University 17

NC STATE UNIVERSITY HPF with synthetic data The destination point is represented by the lowest potential ( = -1) ADAC, NC State University 18

NC STATE UNIVERSITY HPF and Edge detection E(xi, yj) = 1 if (xi, yj) B = 0 if (xi, yj) B for all (i, j) Where E(x, y) is the edge map and B is the set of edge points including the boundary points for all obstacles in workspace is the boundary of the obstacles B nothing but boundaries of the obstacle, , raised to a high potential. Thus it provides the exact raw data required for HPF in Dirichlet’s setting to create the gradient direction matrix. The destination point is then represented by the lowest potential ( = -1) ADAC, NC State University 19

NC STATE UNIVERSITY HPF, a region to point guidance function We observe that the HPF plan of the workspace is the function of obstacle boundaries and the destination point. Thus HPF converts the edge map into a “Region to Point Guidance Function” ADAC, NC State University 20

NC STATE UNIVERSITY Goal Seeking with HPF The important feature of HPF planner is to convert the edge map into a region to point guidance function. the problem is converted to a “goal seeking” problem from a “path tracking” problem. ADAC, NC State University 21

NC STATE UNIVERSITY HPF with Motion Controller The reference position for each current position is calculated from the gradient array of the HPF ( ). L – discretized look-ahead distance. (ex, ey, e ) the error vector is calculated for (x 0, y 0, 0) and (x. R, y. R) y = A n x 2 ADAC, NC State University 22

NC STATE UNIVERSITY Effect of look-ahead distance L = 1, Network delay = 0. 1 s T = 27 seconds L = 8, Network delay = 0. 1 s L = 8, Network delay = 0. 6 s T = 16 seconds The magnitude of the speed v and the turn-rate is proportional to the distance d 0 between (x 0, y 0) and (xref, yref) Small L – UGV close to the path – Small distance error – more time Large L – Less time – large distance error – probability to hit the obstacle ADAC, NC State University 23

NC STATE UNIVERSITY Dynamic look-ahead distance Look-ahead distance (d 0) is function of curvature. From the Quadratic curve controller y = An x 2 GD – grid size distance Workspace image resolution – (m x n) = 320 x 240 Workspace size – (xa x ya) = (4 m x 3 m) High curvature point small d 0, small L The UGV runs slowly on the turn. Path is a straight line (low curvature point) large d 0, large L. UGV moves faster. Optimality between the path tracking accuracy and the time required to reach the goal. ADAC, NC State University Network delay = 0. 3 s T = 24. 3 s 24

NC STATE UNIVERSITY Edge detection Vs Template matching l Need of templates for the obstacles Shape and size restriction on these obstacles. l Knowledge about only the location of the obstacles l No knowledge about the actual boundaries of the obstacles l Many restrictive assumptions about the environment l Edge Detection l No need of templates No Shape and size restrictions on the obstacles l Capture a full representation of the environment l Boundaries of the obstacles are marked distinctively. l No restrictive assumptions regarding its contents. l ADAC, NC State University 25

NC STATE UNIVERSITY Fast Marching Vs HPF Fast Marching with Motion controller Source to destination Reference path l l Path tracking Region to point Reference array for the whole workspace. l l Goal seeking Reference point is calculated using the gradient of HPF planner, which always drive UGV away from obstacles. l Reference point calculation does not have obstacle vicinity knowledge. l Computationally and Memory intensive algorithm. l HPF with Motion controller Less computationally intensive (simple averaging operation) l ADAC, NC State University 26

NC STATE UNIVERSITY Real time computation Np – Number of points on reference path At each sampling instance ti – Each reference point (xref, yref) computation in real time will take L addition operations. Np real time checks on the path to find out the closest point and (xref, yref) Total ( L. n) real time computations. say n. ti – time required to reach the goal (n – number of loop iterations) Performace improvement factor Total (Np. n) real time computations p = Np L p 1 ADAC, NC State University 27

NC STATE UNIVERSITY Key Interface points Old structure of i. Space 1. After template matching, safety radius has to be decided before the points can be passed on to the fast marching method. 2. Fast marching method has to keep track of all the points using a binary tree, whether the point lies inside or outside the safety circle. 3. Reference point calculation in real time is mandatory due to path tracking problem. New structure of i. Space 1. The output from edge detection can be directly fed to HPF planner without preprocessing 2. Knowledge of boundaries (high potential) make it easy with successive relaxation to get the gradient array 3. Reference array calculation from the HPF planner makes it quick for the quadratic curve fitting controller. The new structure has efficient interfacing ADAC, NC State University 28

NC STATE UNIVERSITY Results - 1 For Comparison purpose, ideal path from source to destination was generated ADAC, NC State University 29

NC STATE UNIVERSITY Results - 2 Left – HPF planner and Right – fast marching planner for same grid size Image with a barrier separating two regions. (Blue dot is the source and red dot is the chosen goal. ) ADAC, NC State University 30

NC STATE UNIVERSITY l l l Conclusion Edge detection, a model based HPF planner and network based quadratic Controller go hand in hand to create an efficient and delaytolerant integrated navigation system. More generality and flexibility to the UGV workspace environment. A good edge map helps to build the correct HPF planner. The gradient array calculation from HPF planner decreases the computational burden in real time making - more suitable for networkbased control. The Dirichlet’s setting keeps the robot path, as much as possible, away from the obstacles making it efficient even in heavily cluttered environments. The combined effect of HPF and the quadratic curve controller display intelligent behavior such as no movement in case of goal unreachable problems. Thus the new i. Space structure suggested satisfies many requirement which are key for a network based integrated navigation system. ADAC, NC State University 31

NC STATE UNIVERSITY l l Future Research Improvement on edge detection as reliable edge detection is the backbone of the new structure of i. Space. Considering the dimensions of the robot before creating the HPF planner to take care of the safety margin around the wall. (Possible solutions - Dilation of the edge map) l l Using HPF for velocity control for the UGV. Dynamic obstacle avoidance for NBINS. ADAC, NC State University 32

NC STATE UNIVERSITY ADAC, NC State University 33