Autonomous Swarm of Heterogeneous Robots for Border Surveillance

Autonomous Swarm of Heterogeneous Robots for Border Surveillance Georgios Orfanidis Information Technology Institute (ITI) Center for Research & Technology (CERTH) 6 th km Harilaou - Thermi, 57001, Thessaloniki, Greece email: g. orfanidis@iti. gr This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 740593

Overview �Part 1: Introduction �Part II: Detection services �Part III: Navigation services �Part IV: Use case scenarios �Part V: Conclusions

Part I: Introduction � Advanced technological tools in surveillance have increased the operational effectiveness of Law enforcement agencies. � The surveillance of critical infrastructures is a complex objective: ◦ Diverse operational scenarios and environments. � Current approaches are time and resource consuming and frequently insufficient: ◦ Patrols with vehicles, guards and marine vessels. � The proposed architecture’s main objective is to develop a fully-functional autonomous surveillance system ◦ Controlling Aerial (UAV), water Surface (USV), Underwater (UUV) and Ground (UGV) Vehicles ◦ Operation as a single instance or in a swarm. ◦ Can cover disperse or restricted areas.

Part II: Detection services and User-Interface services �Complete overview of the monitoring area requires scene analysis and understanding. �Can aid the personnel into taking valuable decisions by augmenting the environment awareness. �Modules used within the system include: ◦ Detection techniques for pollution incidents ◦ Identification and tracking of suspicious activities ◦ Low-level fusion to increase the recognition capabilities ◦ Additional detection module for cyberphysical attacks

Detection techniques for pollution incidents � Dedicated module for detecting oil spills over sea surface near relevant critical infrastructure like harbors and sea oil refineries. � Oil spills identification by UAVs of the system equipped with Synthetic Aperture Radar (SAR) sensors. � SAR were chosen for their efficiency and ability to operate under adverse weather conditions. � The results is a semantic representation of the surveyed area emphasizing oil spill presence.

Detection techniques for pollution incidents � Oil spill size estimation feature. � A new dataset for oil spill detection was released. ◦ Visual data (SAR images) taken from Copernicus server (ESA). ◦ Oil spill incident information derived from EMSA. ◦ Manually annotated oil spills. ◦ Publicly available under conditions. � Results on dataset:

Identification and tracking of suspicious activities �A module which focuses on ◦ The identification, localization and tracking of object of interest. ◦ The detection of abnormal activities via behavior analysis. �Diverse and often harsh conditions are typically involved in critical infrastructures and surveillance applications.

Identification and tracking of suspicious activities �Utilization of a wide range of sensors including: ◦ visual, ◦ infrared and ◦ thermal sensors �Combined approaches of different sensors can reduce false positives in detection to avoid false alarms.

Identification and tracking of suspicious activities � For object detector a variation of Faster RCNN was used, chosen for its robustness and effectiveness. � Results on classes using Pascal Voc object detection metric: Object detection results - Average Precision Person Car Bus 0. 82152 0. 75726 0. 5731 5 Truck Boat 0. 53351 0. 7025 1 Ship 0. 8358 6 Helicopter 0. 71638

Low-level fusion to increase the recognition capabilities Combining multiple information from different sources comprises a significant advantage for increasing the accuracy of the recognition modules. � The fused information derives from visual and thermal cameras with the disadvantages of each camera being complemented by the merits of the other. � An early fusion approach was chosen as being more modular (same detectors can be used). �

Low-level fusion to increase the recognition capabilities � Visual cameras provide a clearer and easier to use representation in day-time � Thermal cameras can capture scenes of the operational area in light restricted conditions.

Low-level fusion approach & results � Approach used: ◦ FABEMD[1] decomposes both images into multiple hierarchical components. ◦ Fuses each component to its corresponding one. ◦ Use of PSO[2] to optimize the entropy of the fused result. [1]: Fast and Adaptive Bidimensional Empirical Mode Decomposition [2]: Particles Swarm Optimization

Detection module for cyber-physical attacks �A module to increase the situational awareness of agents against cyberphysical attacks and secure significantly the deployed equipment. �Emphasis on confidentiality breaches and jamming. �The module complements the visual identification, especially, in cases where they lead to incorrect actuation or unreliable sensing due to highjacking.

Part III: Navigation services Enhanced interface � 3 D virtual and augmented reality interface is used. ◦ Video streams projected to operator via virtual and augmented reality. �Description of missions via domain specific language. ◦ ◦ Metadata, Operation commands, Event oriented commands and Post-operation commands.

Resource Controller: Swarm Intelligence �A Resource Controller is used. ◦ An easy-to-operate, remote-control platform to control the swarm of autonomous, heterogeneous agents. ◦ Receives high level objectives and requirements from the human operator. ◦ Translates them to real-time, remoteaction commands for the agents. ◦ Optimum navigation to maximize the surveillance coverage

CISE-compliant framework �A CISE-compliant common representation framework was developed. ◦ Streaming data from different sensors ◦ Representation and mapping of data on semantic knowledge structures. �A visual analytics module ◦ Visualize information from various fusion modes and sensors.

Part IV: Use Cases scenarios In order to assess the developed architecture, the final system will be evaluated in three real scenarios: 1) Identifying pollution incidents in harbors and sea oil refineries: � Capability to track pollutants spilled at sea. � Determine key environmental conditions. � To forecast the fate of the pollutants within the territory of the maritime critical infrastructure.

Use Cases scenarios 2) Unauthorized trespassing in a maritime infrastructure: � Monitoring of large sea territories under the responsibility of the relevant authorities. � Heterogeneous autonomous vehicles equipped with a plethora of sensors like optical and thermal cameras. � Mobile devices interaction with static infrastructure to determine whether an alarming situation occurs. 3) Unauthorized trespassing in land territories: � Ability to patrol difficulty accessible territories within the area of a critical infrastructure � Optimized surveillance with a maximum coverage. � Agents will direct the patrols and track the illegal activities to mitigate personal risks and increase monitoring capabilities.

Part V: Conclusions � The platform provides an efficient tool to perform surveillance operations in both remote land maritime scenarios that include critical infrastructures. � The proposed architecture focuses on the exploitation of multi-modal cameras and identify potential threats within the operational scope. � Provides a flexible and interoperable approach. � State of the art technologies have been utilized that maximizes the operational efficiency and minimizes the required time to take action in critical incidents.

Thank you! Questions and clarifications…