Iron Reign Computer Vision Rover Ruckus Season Outline

Iron Reign Computer Vision Rover Ruckus Season

Outline Need outline completed by tonight - Tuesday evening ● What is FTC - (backgrounder for Computer Visionaries - not needed for DPRG ● ftc_app - basic description ● Vuforia ● Open. CV 4 Android ● “Tensorflow Lite” - a black box mineral recognition engine for all teams to use without really having to understand computer vision ● Roll your own CNN in Tensorflow

ftc_app ● Android app framework published by ftctechnh on github ● Common framework for all ftc teams - this is our starting ground ● Robot controller - phone on robot connected to underlying hardware controllers ● Driver station - phone that operators use to: ○ ○ Send remote joystick commands Get telemetry on robot status Change the active opmode Restart the robot ● Opmodes ● Configurations

ftc_app control system Driver Side Robot Side

Robot Wiring

The REV Expansion Hub REV Robotics started by Greg Needel, former DPRG member

Game Elements Gold cubes Silver balls

Why do we need Computer Vision in the first place? Main reason: Sampling is a part of the FTC challenge this year. It awards 30 points to a team which can move only the gold mineral, not the silver mineral, during the autonomous portion of the match (i. e. no driver control).

Vuforia ● Augmented reality SDK for mobile devices, but includes image tracking ● Developed by Qualcomm, now owned by PTC, developers of Creo and Math. CAD ● What we use it for ○ ○ ○ Localization against reference targets - 1 slide Real-time target tracking - demo of cartbot Initial frame capture for hand off to downstream CV ● Not using Vuforia target tracking in this year’s challenge ● Demo - Cartbot tracking a target

What is Open. CV? ● ● ● Collection of open source Computer Vision algorithms This is the “standard” computer vision library Can be combined into powerful pipelines Been an FTC standard for many years Very stable and well-tested library Written in native code for maximum efficiency 11

Pros and Cons of Open. CV Pros Cons ● Easy to use and design ● Tools like GRIP make it even easier ● ● We can only test pipelines on lighting conditions that we have images for ● If a competition has different lighting conditions, our pipeline might fail ● Open. CV lacks the “wow factor” associated with Machine Learning/AI during judging. Iron Reign has a good track record with Open. CV - using it for the last 3 years ○ As an added benefit, this means that Open. CV is already integrated into the build so we don’t have to do any fiddling around with Gradle

Open. CV 4 Android ● ● ● ● ● The name describes it Java wrappers around opencv functionality tuned for on-phone use Native code is possible but a heavier lift Not integrated with the shipping ftc_app project - teams have to do it themselves High school teams struggle with learning the basic vision algorithms Then coding accurately becomes an issue We started with sample apps like Color. Blob. Detector and adapted to ftc_app Manually coded hybrids of Vuforia and Open. CV Now we use interactive vision pipeline explorers and code generators ○ ○ Robo. Realm - contributed licenses, closed source GRIP - based on Open. CV - generates pipeline code in Java, C++ and Python

But Open. CV looks too scary. .

Use GRIP to design your pipelines

Our Open. CV Pipeline

Our Open. CV Pipeline

Our Open. CV Pipeline

Our Open. CV Pipeline

Our Open. CV Pipeline

Handling anomalies

Our final pipeline

Tensorflow Lite Tensor. Flow is Google’s open source machine learning library. It models neural networks using “tensors, ” which are basically neurons. Tensor. Flow Lite is the solution for running ML models on mobile and embedded devices. Build (or retrain) in Tensor Flow Convert model to tflite format Deploy. tflite into mobile app

Refresher on Neural Networks

Tensor. Flow Object Detection Tensor. Flow’s Object Detection is implemented via a sliding window classifier. A basic sliding window classifier

“Tensorflow” as bundled in ftc_app ● Game specific solution to targeting game elements ● Easy to follow tutorial on getting it working ● Gives a recognition confidence level and location of detected elements ● Slow and sloppy on our phones (what’s the fps)

“Tensorflow” as bundled in ftc_app ● Black box ○ ○ Not sure what algorithms are involved, though likely a small CNN called a Mobile. Net Probably a “re-trained” Imagenet (transfer learning) But without knowing, not sure how to improve recognition or speed Trying to improve repeatability through on-bot lighting ● Sketchy performance ○ ○ ○ We have had trouble getting it to work right Higher accuracy with minerals on the left than the right during testing Our sister team said that “mounting their phone at a sideways angle helped improve accuracy” ¯_(ツ)_/¯

Pros and Cons of “Tensor. Flow” Pros Cons ● Easy to get started (sample code provided) ● Black box - we don’t know what is happening ● Already integrated into ftc_app build ● Very sketchy performance ● Little to no “wow-factor” as this is available to all teams

Rolling our own Convolutional Neural Network Instead of using a black box model, why not write our own model? We actually had the idea to train our own CNN before Tensor. Flow Lite integration was even released to ftc_app.

Step 0: Determine Training Objective of the model Given an image, we wished for the network to output 2 integers x and y, (0 < x < 320 and 0 < y < 240; our image is 320 x 240). These two integers would be the location of the gold mineral on the image.

Step 1: Capturing training data

Step 2: Label training data We could label coordinates by hand, but this is too difficult. Instead, we wrote a program to do help us label the data.

Step 2. 1: Write labeling program Available at github. com/arjvik/Mineral. Labler

Step 2. 2: Use labeling program to label images

Step 3: Train model

CNN Structure

Step 3. 1: Try again with Java/DL 4 J

Our CNN Structure (in Java)

Step 4: Continue adapting the model to improve it We are considering turning our model into a sliding window classifier, to potentially increase accuracy. We also need to capture and label more training data, to better fit our model.

Step 5: ? ? ? Step 6: Profit Of course, if we can get this to work, it will be a great benefit, both during the robot game, and also during judging. For now, we will keep trying to improve it, but we haven’t forgotten about the other two vision methods either. Iron Reign believes in parallel development, and we will continue working on all three before evaluating which one we wish to use later on in the season.

Summary Open. CV Tensor. Flow Lite Convolutional Neural Network Pros: - Easy to design Pros: - Easy to get started Pros: - Very adaptable - Easy to prototype in GRIP Cons: - Adaptability to different lighting conditions - Lacks “wow factor” - Already integrated into build Cons: - Black box - Very sketchy performance - “Wow factor” Cons: - We don’t have it working. Yet.

Resources GRIP pipeline generator: https: //wpilib. screenstepslive. com/s/4485/m/24194/l/463566 -introduction-to-grip Tensor. Flow Lite on Android: https: //medium. com/tensorflow/using-tensorflow-lite-on-android-9 bbc 9 cb 7 d 69 d Tensor. Flow for Poets codelab: https: //codelabs. developers. google. com/codelabs/tensorflow-for-poets Find out more about Iron Reign’s vision algorithms: https: //www. ironreignrobotics. com/tags/vision/index