Remote Sensing and Machine Vision Aerial Robotics Camp

Remote Sensing and Machine Vision Aerial Robotics Camp 2018 June 11, 2018 Sam Siewert

Prof. Sam Siewert NASA – Johnson – Jet Propulsion Lab Intel and Start-ups – HPC Networking & Storage – Big Data! College - Teaching – U. of Colorado Boulder – U. of Alaska – Embry Riddle Aeronautical Student Introductions Sam Siewert 2

Machine Vision Robotics Physics (Electromagnetic Spectrum) AI (Machine Learning) Machine Vision Computer Vision (Human Model) Image Processing “Photo Shop” https: //www. starwars. com/ Sam Siewert 3

Machine Vision Systems Cameras Playing Chess for “Real” – Extrinsic - mounting & pointing – Intrinsic - lens, detector, optical path Embedded Systems – Computer for processing Images – Continuous Objects in an Image – Segments – Foreground, Background You. Tube - Chess Automatic Duck Hunter Pixel to Real World Coordinates Stereo Vision – Distance Estimation Methods (Disparity) – Binocular Vision Sam Siewert You. Tube - Duck Hunter 4

Machine Vision and Video Machine Vision – Industrial Automation and Robotics – Controlled Environments – Inspection, Optical Navigation, Medical Aerial Robotics Overhead Camera RT Robotics Spitzer NASA JPL Aerial Robotics Camp - Overhead Satellite View – Remotely “there” - visual sensing – Vision is powerful – Use to create a map of a remote world rtsp: //192. 168. 0. 100/11 Sam Siewert 5

Why is Human Vision > Computer? Cortex 10 Billion Neurons > 1 Trillion Synapses Total=100 Billion Neurons 1. 2. 3. 4. Neuron > Transistor Better Programming? ROM? More Richly Interconnected Storage + Processing Red Epic 63 Mega-Pixel + Approximately 100+ Mega-Pixel (Rod & Cone Count) Neuroscience. 2 nd edition. Purves D, Augustine GJ, Fitzpatrick D, et al. , editors. Sunderland (MA): Sinauer Associates; 2001. http: //www. ncbi. nlm. nih. gov/books/NBK 10848/ 20 billion transistors, < 10 nm CPU Local Bus Memory Controller I/O Bus (x 16 32 Gbps = 64 GB/sec) USB 3 Gig. E Vision Sam Siewert 6

Biological Vision vs. Machine Vision Why A Honey Bee is Hard to Beat! Humans - 100 million Pixels – – 10 billion Neurons (Cerebral Cortex) Brain with 100 billion Neurons Millisecond Transfer Massively Parallel Analog + Digital Computation Synapse Match is a Challenge – 7000 Connections from 10 Billion Neurons – 3 Year Olds Have 1015 Synapses 960 K Neurons in flight: Learns locations, complex odors, colors, and shapes; with high efficiency (500 Watt/Kg), 0. 218 g Brain plasticity for learning, connectedness, concurrency, integrated sensing, power efficiency, and resiliency Intel/Altera Stratix-10 14 nm (30 billion) NVIDIA VOLTA 12 nm, (21 billion) CPU to Digital Camera/HDD – Connects 10’s of millions of pixels – to Several Billion transistors – Through Sequential Logic and I/O Bus Moore’s Law Sam Siewert 7

What is an Image? Ideas? Sam Siewert 8

What is Light? Small range of electromagnetic spectrum UV to IR (10 to 200, 000 nm = 0. 01 to 200 microns) Humans only see Red, Green, Blue (400… 700 nm) Sam Siewert 9

Can we “See” UV or Infrared? Not directly Indirectly – With instrumentation – Thermal and night vision – NIR vegetation index LWIR and NIR Camera Demos Fluorescence – Re-emission of visible-light – RGB from UV exposure https: //en. wikipedia. org/wiki/Fluorescence LWIR camera – – Measures thermal band 10 to 14 micron “light” Intensity False color visible display Sam Siewert Drone Net Video - Bugs, Drone, Aircraft 10

What is an Image? An array of numbers? A cube of numbers? Light Intensity at a Location? Pixels? Scene with foreground, background? 2 D projection of 3 D world? Information? All of the above? Sam Siewert 11

Artist Perspective and Physics of Light Painted, Traced Edge Silk Screen Illumination Observer Viewpoint Rays http: //en. wikipedia. org/wiki/File: Perspectiva-2. svg Sam Siewert 12

Brighten, Contrast Transform {P} = clamp [ {Q}*alpha + beta] – Scale Each Pixel by alpha – Brightness Increase/Decrease – Bias Each Pixel with beta – Contrast Increase/Decrease – Make Sure Result Does Not Exceed Saturation 0… 255 for 8 -bit Sam Siewert 13

Play with MV Cameras Stream an Image over Internet Canny Edge Finder Hough Lines Hough Circles How do we “Find” objects or features in an image? Sam Siewert 14

Using Overhead Camera - Remote Simulates Satellite view from Above Remote Sensing – Strategy – Use to map the game world – Verify position of … http: //worldview 3. digitalglobe. com/ Tellos Lego Robots Goals Hazards Sam Siewert 15

Planning … for Success Goals and Objectives (1920 x 1080 pixels) / 120 = 16 x 9 – Map world - Google Earth, Worldview – Tello air support – Navigate ground robotics for rescue Requirements – – Take off Tello and navigate to game Get to target circle Land on circle (partial credit) Navigate robot to circle in maze 16 x 9 World model Constraints – – – Flight time (battery) Maze walls Time Sam Siewert https: //www. draw. io 16

More Reading Siewert, S. , “Explore video analytics in the Cloud”, IBM, June “ 2013. Siewert, S. , “Machine data analytics - Drop-in-place security “ and safety monitors”, IBM, January, 2014. Siewert, S. , Andalibi, M. , Bruder, S. , Gentilini, I. , & Buchholz, J. , "Drone Net Architecture for UAS Traffic Management Multimodal Sensor Networking Experiments. " IEEE Aerospace Conference, Big Sky, Montana. 2018. Siewert, S. , Andalibi, M. , Bruder, S. , Gentilini, I. , Dandupally, A. , Gavvala, S. , . . . & Burklund, D. “DRONE NET, A PASSIVE INSTRUMENT NETWORK DRIVEN BY MACHINE VISION AND MACHINE LEARNING TO AUTOMATE UAS TRAFFIC MANAGEMENT”, AUVSI, May 2018. Sam Siewert 17