Persistent Tactical Seeability Through Integrated Sensor Guidance Tim

Persistent Tactical Seeability Through Integrated Sensor Guidance Tim Mc. Lain Randy Beard Bryan Morse

BYU Team Introduction • Randy Beard – Professor, Electrical and Computer Engineering – autonomous systems, unmanned aircraft systems, multiple vehicle coordination and control • Bryan Morse – Associate Professor, Computer Science – Computer vision, image reconstruction and registration, applications in Wi. SAR • Tim Mc. Lain – Professor and Dept. Chair, Mechanical Engineering – UAS autonomy and control, vision-based guidance, cooperative control of UA teams, Wi. SAR 1/10/2022 Mosaic Team Proprietary 2

UAS Research at BYU Cooperative Control • • Cooperative timing problems Cooperative persistent imaging Cooperative fire monitoring Consensus seeking • 3 D Waypoint path planning • Wind compensation • Collision avoidance • Optic flow sensor • Laser ranger • EO cameras Path Planning Path Following Image Directed Control Autonomous Vehicles • Image stabilization • Geo-location • Vision aided tracking & engagement • • Autopilot design for small UA Attitude estimation Adaptive control Tailsitter guidance & control

Related Work • UAS-assisted wilderness search and rescue (Wi. SAR) – Funded for six years by NSF – Morse, Goodrich, Mc. Lain – Key take-away: “Just because you flew over it and imaged it, doesn’t mean you saw it. ” – Seeability metric conceived 1/10/2022 Mosaic Team Proprietary 4

Related Work • UAV/UGV cooperative tracking (Army SBIR Ph. II with SET Corp. ) – Enhancing probability of detection and persistent imaging of dismounts • SUAS wind energy extraction (Navy STTR Ph. I with Mosaic ATM) – Exploit energy available from Wx – Maintain mission effectiveness 1/10/2022 Mosaic Team Proprietary 5

Related Work • Aerial recovery of UAS (Air Force STTR Ph. II with Procerus) – Goal: Recover air-launched SUAS – Concept: Mothership/drogue – Challenge: Mothership airspeed significantly higher than SUAS airspeed 1/10/2022 Mosaic Team Proprietary 6

1/10/2022 Mosaic Team Proprietary 7

Three Product Families Kestrel TM Flight Systems • Kestrel Autopilot v 3. 0 Adaptive Control • Kestrel Cockpit TM • Kestrel SIM – For Real. Flight TM On. Point TM Vision Systems • On. Point On. Board TM Vision Suite v 3. 0 • On. Point Targeting v 2. 0 • On. Point VPU • On. Point GUI TM Perceptor TM Imaging Systems • Perceptor 88 x • Perceptor DG 2

Kestrel Autopilot v 3. 0 • • Fixed-wing / Heli / tri-rotor / quad-rotor 500 mhz DSP, 32 Mb flash, 32 Mb RAM 11 servos supported onboard Integrated 3 -axis magnetometer High quality pitot-static systems Factory calibrated, temp. compensated IMU 17 state EKF navigation solution High accuracy gimbal pointing / geo-location Kestrel Autopilot v 3. 0 2. 25” x 1. 4” x. 5” 21 grams

Kestrel Cockpit v 3. 0 • • • Improved user interface 3 D streaming maps/terrain 3 D waypoints Intuitive map elements VTOL controls/features Multi-agent support – Fixed wing and VTOL – Kestrel 2 x and 3 x autopilots • • Intuitive multi-function display & health monitor Agent & flight path rendered – Full-context 3 D environment • Tight integration with On. Point Targeting

On. Point Targeting v 2. 0 • • Target tracking and geolocation Video stabilization Click ‘n Fly operation New highly configurable GUI w/dockable windows TIVO-like video pause/playback/record Snap shots appear in Virtual Cockpit GCS as clickable icons Improved Kalman filter target position/velocity estimation Position uncertainty estimates shown on video overlay

On. Point On. Board – Vision Suite v 3. 0 • • On. Point VPUTM 1. 9” x 1. 35” 11 grams (. 4 oz) Target tracking and geolocation Video stabilization Click ‘n Fly operation On. Point VPUTM (Vision Processing Unit) Perceptor DG (e. PTZ) imager (gyros, 5 mp) OMAP DSP processor (2 Gb flash, 1 Gb DDR) SD card slot, USB 2, video in/out, SPI/GPIO/ethernet, serial Computer vision and inertial measurement for solid video e. PTZ - 5 mp Imager 1. 3" x 0. 9” Hardware 7/24/09

Perceptor 88 x • • • • 2 axis gyro stabilized, slip ring, continuous 360° pan rotation, 90° tilt Pointing resolution: 0. 05⁰ Stabilization: 0. 05⁰ error w/ 15⁰ 2 Hz disturbance Small & lightweight: 3. 5 in diameter turret, <0. 85 lbs (400 g) EO (10 x zoom) or IR Factory calibrated IMU sensor suite High bandwidth camera positioning for small platforms 17 state GPS/INS solution + targeting & motion control provides for standalone operation Graceful targeting degradation during GPS loss/denied environment RS 232, TTL serial, and CAN interfaces Compliance with popular protocols for easy integration Available soon in retractable version On. Point GUI: video record, playback, gimbal pointing Control Modes Include: rate, angle, and latitude longitude elevation Continuous 360° Stabilized Gimbal 3. 5” turret Sony FCB-IX, EO 10 x optical zoom 4 x digital zoom * Shown without IMU (KAP 3) or cover.

Procerus Perceptor 88 x 1/10/2022 Mosaic Team Proprietary 14

1/10/2022 Mosaic Team Proprietary 15

Simulation Architecture

Current Status of Simulation • TCP-IP connection to Virtual Cockpit ground station • Includes n-step look-ahead algorithm – Faster observation of targets than orbits, lawnmower paths, other traditional paths – Only needs to know search area – no other parameters needed • Configurable dynamics block to accommodate various UAV platforms • Plotting capabilities – Import or customize terrain – Paint imaged terrain with measure of seeability • Recently implemented vector field path following

Future of Simulation • Look-ahead algorithm is exponential, needs improvement – Chain-link model – Genetic algorithm • Gimbal model and pointing strategy enhancements needed to reduce ‘jumpiness’ of the camera • Need to establish metric to judge quality of real-time video imagery – Compare to simulated Tactical Seeability metric – Allow for user feedback – Consider amount of blur, contrast, wash-out, self-obstruction – Wavelet transforms or color histograms are possible options

1/10/2022 Mosaic Team Proprietary 19