CIS 694EEC 693 Android Sensor Programming Lecture 26

CIS 694/EEC 693 Android Sensor Programming Lecture 26 Wenbing Zhao Department of Electrical Engineering and Computer Science Cleveland State University w. zhao 1@csuohio. edu 10/28/2020 Android Sensor Programming 1

Outline n ML Kit: Machine Learning for Mobile Developers 10/28/2020 Android Sensor Programming 2

ML Kit: Machine Learning for Mobile Developers n n n https: //developers. google. com/ml-kit/ Image labeling Text recognition Face detection Barcode scanning Landmark detection 10/28/2020 Android Sensor Programming 3

ML Kit: Machine Learning for Mobile Developers n n You will need to register a Google account using your existing email (any email will be fine) Log in to the Firebase console, go to MLKit panel 10/28/2020 Android Sensor Programming 4

ML Kit: Machine Learning for Mobile Developers n n n For every app you build, you will have to register with the firebase After registration, you will be asked to download a json file googleservices. json to be placed into your project for compilation Add Firebase to your Android project q Add to the root-level build. gradle file: buildscript { //. . . dependencies { //. . . classpath 'com. google. gms: google-services: 4. 0. 1' // google-services plugin } } allprojects { //. . . repositories { //. . . google() // Google's Maven repository } } 10/28/2020 Android Sensor Programming 5

ML Kit: Machine Learning for Mobile Developers n Add Firebase to your Android project q Then, in your module Gradle file (usually the app/build. gradle), add the apply plugin line at the bottom of the file to enable the Gradle plugin: dependencies { //. . . implementation 'com. google. firebase: firebase-core: 16. 0. 1' // Getting a "Could not find" error? Make sure you have // added the Google maven respository to your root build. gradle } // ADD THIS AT THE BOTTOM apply plugin: 'com. google. gms. google-services' q q You should also add the dependencies for the Firebase SDKs you want to use. We recommend starting with com. google. firebase: firebase-core, which provides Google Analytics for Firebase functionality Available firebase libraries: https: //firebase. google. com/docs/android/setup#available_libraries 10/28/2020 Android Sensor Programming 6

ML Kit: Image Labeling n n n https: //firebase. google. com/docs/ml-kit/label-images With ML Kit's image labeling APIs, you can recognize entities in an image without having to provide any additional contextual metadata, using either an on-device API or a cloud-based API Image labeling gives you insight into the content of images When you use the API, you get a list of the entities that were recognized: people, things, places, activities, etc. Each label found comes with a score that indicates the confidence the ML model has in its relevance With this information, you can perform tasks such as automatic metadata generation and content moderation 10/28/2020 Android Sensor Programming 7

ML Kit: Image Labeling n The device-based API supports 400+ labels, such as the Example labels following examples: Category 10/28/2020 People Crowd Selfie Smile Activities Dancing Eating Surfing Things Car Piano Receipt Animals Bird Cat Dog Plants Flower Fruit Vegetable Places Beach Lake Mountain Android Sensor Programming 8

ML Kit: Image Labeling n The cloud-based API supports 10, 000+ labels 10/28/2020 Category Example labels Arts & entertainment Sculpture Musical Instrument Dance Astronomical objects Comet Galaxy Star Business & industrial Restaurant Factory Airline Colors Red Green Blue Design Floral Pattern Wood Stain Drink Coffee Tea Milk Events Meeting Picnic Vacation Fictional characters Santa Claus Superhero Mythical creature Food Casserole Fruit Potato chip Home & garden Laundry basket Dishwasher Fountain Activities Wedding Dancing Motorsport Materials Ceramic Textile Fiber Media Newsprint Document Sign Modes of transport Aircraft Motorcycle Subway Occupations Actor Florist Police Organisms Plant Animal Fungus Organizations Government Club College Places Airport Mountain Tent Technology Robot Computer Solar panel Things Bicycle Pipe Doll Android Sensor Programming 9

ML Kit: Google Knowledge Graph entity IDs n n In addition the text description of each label that ML Kit returns, it also returns the label's Google Knowledge Graph entity ID This ID is a string that uniquely identifies the entity represented by the label, and is the same ID used by the Knowledge Graph Search API q n https: //developers. google. com/knowledge-graph/ You can use this string to identify an entity across languages, and independently of the formatting of the text description 10/28/2020 Android Sensor Programming 10

ML Kit: Google Knowledge Graph entity IDs 10/28/2020 Android Sensor Programming 11

Label Images with ML Kit on Android n Configuration in Android Studio q In the project-level build. gradle, add: dependencies { classpath 'com. android. tools. build: gradle: 3. 1. 3' classpath 'com. google. gms: google-services: 4. 0. 1' } q In the app-level build. gradle, add: dependencies { implementation file. Tree(dir: 'libs', include: ['*. jar']) implementation 'com. android. support: appcompat-v 7: 27. 1. 1' implementation 'com. android. support. constraint: constraint-layout: 1. 1. 2’ implementation 'com. google. firebase: firebase-core: 16. 0. 1' implementation 'com. google. firebase: firebase-ml-vision: 16. 0. 0' implementation 'com. google. firebase: firebase-ml-vision-image-label-model: 15. 0. 0' test. Implementation 'junit: 4. 12' android. Test. Implementation 'com. android. support. test: runner: 1. 0. 2' android. Test. Implementation 'com. android. support. test. espresso: espresso-core: 3. 0. 2' } apply plugin: 'com. google. gms. google-services' 10/28/2020 Android Sensor Programming 12

Label Images with ML Kit on Android n Configuration in Android Studio q q If you use the on-device API, configure your app to automatically download the ML model to the device after your app is installed from the Play Store To do so, add the following declaration to your app's Android. Manifest. xml file: <application. . . > . . . <meta-data android: name="com. google. firebase. ml. vision. DEPENDENCIES" android: value="label" /> <!-- To use multiple models: android: value="label, model 2, model 3" --> </application> q If you do not enable install-time model downloads, the model will be downloaded the first time you run the on-device detector. Requests you make before the download has completed will produce no results 10/28/2020 Android Sensor Programming 13

Register with Firebase n n Go to https: //console. firebase. google. com/u/0/project/mlkitaa 998/overview Click “Add another app” link, then “Add Firebase to your Android app 10/28/2020 Android Sensor Programming 14

Register with Firebase n n Go to https: //console. firebase. google. com/u/0/project/mlkitaa 998/overview Click “Add another app” link, then “Add Firebase to your Android app 10/28/2020 Android Sensor Programming 15

Register with Firebase n Then, follow the steps as prompted 10/28/2020 Android Sensor Programming 16

On-device image labeling n Configure the image labeler q By default, the on-device image labeler returns at most 10 labels for an image. If you want to change this setting, create a Firebase. Vision. Label. Detector. Options object as in the following example Firebase. Vision. Label. Detector. Options options = new Firebase. Vision. Label. Detector. Options. Builder() . set. Confidence. Threshold(0. 8 f). build(); n Run the image labeler q Create a Firebase. Vision. Image object from a Bitmap object Firebase. Vision. Image image = Firebase. Vision. Image. from. Bitmap(bitmap); q You can also create a Firebase. Vision. Image from a file, from a Byte. Buffer or byte array, or a media. Image taken by the device’s camera 10/28/2020 Android Sensor Programming 17

On-device image labeling n Run the image labeler q Get an instance of Firebase. Vision. Label. Detector: Firebase. Vision. Label. Detector detector = Firebase. Vision. get. Instance(). get. Vision. Label. Detector(); // Or, to set the minimum confidence required: Firebase. Vision. Label. Detector detector = Firebase. Vision. get. Instance(). get. Vision. Label. Detector(options); q Finally, pass the image to the detect. In. Image method: Task<List<Firebase. Vision. Label>> result = detector. detect. In. Image(image) . add. On. Success. Listener( new On. Success. Listener<List<Firebase. Vision. Label>>() { @Override public void on. Success(List<Firebase. Vision. Label> labels) { // Task completed successfully //. . . }) . add. On. Failure. Listener( new On. Failure. Listener() { @Override public void on. Failure(@Non. Null Exception e) { // Task failed with an exception //. . . }); 10/28/2020 Android Sensor Programming 18

On-device image labeling n Get information about labeled objects q q q If the image labeling operation succeeds, a list of Firebase. Vision. Label objects will be passed to the success listener Each Firebase. Vision. Label object represents something that was labeled in the image. For each label, you can get the label's text description, its Knowledge Graph entity ID (if available), and the confidence score of the match for (Firebase. Vision. Label label: labels) { String text = label. get. Label(); String entity. Id = label. get. Entity. Id(); float confidence = label. get. Confidence(); } 10/28/2020 Android Sensor Programming 19

A complete on-device image labeling app n Preparation: find an interesting image and copy it to res/mipmap: 10/28/2020 Android Sensor Programming 20

A complete ondevice image labeling app n Setup the layout for the activity as shown in the previous slide, and here: 10/28/2020 <? xml version="1. 0" encoding="utf-8"? > <Relative. Layout xmlns: android="http: //schemas. android. com/apk/res/android" xmlns: tools="http: //schemas. android. com/tools" android: layout_width="match_parent" android: layout_height="match_parent" tools: context=". Main. Activity"> <Button android: text="Lable Image" android: layout_width="wrap_content" android: layout_height="wrap_content" android: layout_center. Horizontal="true" android: id="@+id/btn. Label. Image" /> <Image. Switcher android: id="@+id/image. Switcher" android: layout_width="match_parent" android: layout_height="250 dp" android: layout_align. Parent. Start="true" android: layout_align. Parent. Top="true" android: layout_margin. Top="50 dp" /> <Text. View android: id="@+id/txt. Labels" android: layout_width="match_parent" android: layout_height="172 dp" android: layout_align. Parent. Bottom="true" android: layout_align. Parent. End="true" android: layout_below="@+id/image. Switcher" android: layout_center. Horizontal="true" android: layout_margin. Top="5 dp" android: text. Color="#ffffff" android: background="#000000" /> </Relative. Layout> Android Sensor Programming 21

A complete on-device image labeling app n Populate Main. Activity import android. support. v 7. app. App. Compat. Activity; import android. os. Bundle; import android. graphics. Bitmap. Factory; import android. graphics. Bitmap; import java. util. List; import com. google. android. gms. tasks. On. Success. Listener; import com. google. android. gms. tasks. On. Failure. Listener; import android. support. annotation. Null; import android. widget. Image. Switcher; import android. support. v 7. app. Action. Bar; import android. widget. Text. View; import android. widget. View. Switcher; import android. widget. Image. View; import android. view. View; import android. widget. Button; import com. google. android. gms. tasks. Task; import com. google. firebase. ml. vision. Firebase. Vision; import com. google. firebase. ml. vision. common. Firebase. Vision. Image; import com. google. firebase. ml. vision. label. Firebase. Vision. Label. Detector; 10/28/2020 Android Sensor Programming 22

A complete on-device image labeling app n Populate Main. Activity public class Main. Activity extends App. Compat. Activity { @Override protected void on. Create(Bundle saved. Instance. State) { super. on. Create(saved. Instance. State); set. Content. View(R. layout. activity_main); Image. Switcher img. Switcher = (Image. Switcher) find. View. By. Id(R. id. image. Switcher); img. Switcher. set. Factory(new View. Switcher. View. Factory() { @Override public View make. View() { Image. View my. View = new Image. View(get. Application. Context()); my. View. set. Scale. Type(Image. View. Scale. Type. FIT_CENTER); my. View. set. Layout. Params(new Image. Switcher. Layout. Params( Action. Bar. Layout. Params. WRAP_CONTENT, Action. Bar. Layout. Params. WRAP_CONTENT)); return my. View; }); img. Switcher. set. Image. Resource(R. mipmap. windows); Button btn. Label. Image = (Button) find. View. By. Id(R. id. btn. Label. Image); 10/28/2020 Android Sensor Programming 23

n Populate Main. Activity btn. Label. Image. set. On. Click. Listener(new View. On. Click. Listener() { @Override public void on. Click(View v) { Bitmap bm = Bitmap. Factory. decode. Resource(get. Resources(), R. mipmap. windows); Firebase. Vision. Image image = Firebase. Vision. Image. from. Bitmap(bm); Firebase. Vision. Label. Detector detector = Firebase. Vision. get. Instance(). get. Vision. Label. Detector(); Task<List<Firebase. Vision. Label>> result = detector. detect. In. Image(image). add. On. Success. Listener( new On. Success. Listener<List<Firebase. Vision. Label>>() { @Override public void on. Success(List<Firebase. Vision. Label> labels) { // Task completed successfully for (Firebase. Vision. Label label: labels) { String text = label. get. Label(); String entity. Id = label. get. Entity. Id(); float confidence = label. get. Confidence(); Text. View tv = (Text. View) find. View. By. Id(R. id. txt. Labels); tv. append(text+"; entity id="+entity. Id+"; confidence="+confidence+"n"); } }). add. On. Failure. Listener(new On. Failure. Listener() { @Override public void on. Failure(@Non. Null Exception e) { // Task failed with an exception //. . . }); } }); } } 10/28/2020 Android Sensor Programming 24

A complete on-device image labeling app 10/28/2020 Android Sensor Programming 25

Homework#29 n Build the on-device image labeling app 10/28/2020 Android Sensor Programming 26