Tensorflow in Deep Learning Lecture 1 Tensor Flow

Tensorflow in Deep Learning Lecture 1: Tensor. Flow Basics JAHANDAR JAHANIPOUR jjahanipour@uh. edu www. easy-tensorflow. com https: //github. com/easy-tensorflow 30/04/2018 www. easy-tensorflow. com 1

About Me • B. S. Electrical Engineering – IKIU (Iran) • M. S. Electrical Engineering – IUT (Iran) • Ph. D Candidate at ECE department - UH • www. easy-tensorflow. com Bio-Analytics Lab Aryan Mohammad Jahandar “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, NIPS 2015 • Research Interests: • 30/04/2018 Deep Learning - Machine Learning – Computer Vision www. easy-tensorflow. com 2

What Do You Learn at This Workshop ? • Tensor. Flow Basics • Datatypes: Efficiently use your memory • Graph & Session: Save computation time by running needed operations • Tensor. Board: Flashlight to your Neural Network BLACK BOX • Neural Network • Auto. Encoder • Convolutional Neural Network 30/04/2018 www. easy-tensorflow. com 3

Outline • About Tensor. Flow • • • Tensor. Flow Basics • • What is Tensor. Flow? Why Tensor. Flow? Introduction Graph & Session Datatypes Logistic Regression (Linear Classifier) 30/04/2018 www. easy-tensorflow. com 4

What is Tensor. Flow? “Tensor. Flow™ is an open source software library for numerical computation using data flow graphs. ” “… software library for Machine Intelligence” • Created by Google • API available for multiple languages (Python, C++, Java, Go, etc. ) • Suitable for both production & research 30/04/2018 www. easy-tensorflow. com 5

Companies Using Tensor. Flow 30/04/2018 www. easy-tensorflow. com 6

Trends on Different Deep Learning Libraries 30/04/2018 www. easy-tensorflow. com 7

Drawing with Tensor. Flow A Neural Representation of Sketch Drawings (Ha et al. 2017) 30/04/2018 www. easy-tensorflow. com 8

Image Style Transfer with Tensor. Flow Image Style Transfer Using Convolutional Neural Networks (Gatys et. al. 2016) 30/04/2018 www. easy-tensorflow. com 9

Why Tensor. Flow? • Developed and maintained by Google • Very large and active community + Nice documentation • Python API • Multi-GPU support • Tensor. Board (A very Powerful visualization tool) • Faster model compilation than Theano-based options • High level APIs build on top of Tensor. Flow (Keras, TFlearn, …) 30/04/2018 www. easy-tensorflow. com 10

How to Set It Up? 1. Python – Programming language • Learn more from: ◦ ◦ 2. Stanford CS 231: Python Numpy Tutorial Corey Schafer Python Tutorial Anaconda – Package manager (Optional; instead of installing Python directly) • What is Anaconda? 3. Tensor. Flow 4. IDE – Editor (preferably Py. Charm) http: //www. easy-tensorflow. com/install 30/04/2018 www. easy-tensorflow. com 11

Introduction to Tensor. Flow • What is a Tensor? • Multi-dimensional array • • 0 -d tensor: scalar (number) 1 -d tensor: vector 2 -d tensor: matrix Importing the library • • Input tensor for images: [batch_size, image_height, image_width, channels] import tensorflow as tf “Computational Graph” approach 1. 2. 30/04/2018 Build the GRAPH which represents the data flow of the computation Run the SESSION which executes the operations on the graph www. easy-tensorflow. com 12

Graph and Session Graph ◦ 30/04/2018 Nodes = Operations www. easy-tensorflow. com 13

Graph and Session Graph ◦ ◦ 30/04/2018 Nodes = Operations Edges = Tensors www. easy-tensorflow. com 14

Graph and Session Graph ◦ ◦ Nodes = Operations Edges = Tensors Session ◦ ◦ 30/04/2018 Tensor = data Tensor + flow = data + flow www. easy-tensorflow. com 15

Graph and Session Example 1: Graph import tensorflow as tf c = tf. add(2, 3, name='Add') print(c) LAZY PROGRAMMING: Call-by-need DON’T BE LAZY !!!! Tensor. Flow names the node if you don’t ! 30/04/2018 www. easy-tensorflow. com 16

Graph and Session Example 1: Graph import tensorflow as tf a = 2 b = 3 c = tf. add(a, b, name='Add') print(c) Variables ? Tensor("Add: 0", shape=(), dtype=int 32) “Computational Graph” approach 1. Build the GRAPH which represents the data flow of the computation 2. Run the SESSION which executes the operations on the graph 30/04/2018 www. easy-tensorflow. com 17

Graph and Session Example 1: Graph import tensorflow as tf a = 2 b = 3 c = tf. add(a, b, name='Add') print(c) Variables sess = tf. Session() print(sess. run(c)) sess. close() 5 30/04/2018 www. easy-tensorflow. com 18

Graph and Session Example 1: Graph import tensorflow as tf a = 2 b = 3 c = tf. add(a, b, name='Add') print(c) Variables sess = tf. Session() with tf. Session() as sess: print(sess. run(c)) sess. close() 5 30/04/2018 www. easy-tensorflow. com 19

Graph and Session Example 2: import x = 2 y = 3 add_op mul_op pow_op Graph tensorflow as tf = tf. add(x, y, name='Add') = tf. multiply(x, y, name='Multiply') = tf. pow(add_op, mul_op, name='Power') with tf. Session() as sess: pow_out = sess. run(pow_op) 30/04/2018 Variables www. easy-tensorflow. com 20

Graph and Session Example 3: Graph import tensorflow as tf x = 2 y = 3 add_op = tf. add(x, y, name='Add') mul_op = tf. multiply(x, y, name='Multiply') pow_op = tf. pow(add_op, mul_op, name='Power') useless_op = tf. multiply(x, add_op, name='Useless') Variables with tf. Session() as sess: pow_out = sess. run(pow_op) 30/04/2018 www. easy-tensorflow. com 21

Graph and Session Example 3: Graph import tensorflow as tf x = 2 y = 3 add_op = tf. add(x, y, name='Add') mul_op = tf. multiply(x, y, name='Multiply') pow_op = tf. pow(add_op, mul_op, name='Power') useless_op = tf. multiply(x, add_op, name='Useless') with tf. Session() as sess: [pow_out, useless_out = sess. run([pow_op, useless_op]) 30/04/2018 www. easy-tensorflow. com Variables 22

Data types 1. Constants are used to create constant values tf. constant( value, dtype=None, shape=None, name='Const', verify_shape=False ) Example s = tf. constant(2, name='scalar') m = tf. constant([[1, 2], [3, 4]], name='matrix') 30/04/2018 www. easy-tensorflow. com 23

Data types 1. Constants are used to create constant values Graph Before: import tensorflow as tf a = 2 b = 3 c = tf. add(a, b, name='Add') Variables with tf. Session() as sess: print(sess. run(c)) 5 30/04/2018 www. easy-tensorflow. com 24

Data types 1. Constants are used to create constant values Graph Now: import tensorflow as tf a = tf. constant(2, name='A') b = tf. constant(3, name='B') c = tf. add(a, b, name='Add') Variables with tf. Session() as sess: print(sess. run(c)) 5 30/04/2018 www. easy-tensorflow. com 25

Data types 2. Variables are stateful nodes (=ops) which output their current value 1. 2. They Can be saved and restored Gradient updates will apply to all variables in the graph ⇒ Network Parameters (weights and biases) Example get_variable( name, shape=None, dtype=None, initializer=None, regularizer=None, trainable=True, collections=None, caching_device=None, partitioner=None, validate_shape=True, use_resource=None, custom_getter=None, constraint=None) s 1 = tf. get_variable(name='scalar 1', initializer=2) s 2 = tf. get_variable(name='scalar 2', initializer=tf. constant(2)) m = tf. get_variable('matrix', initializer=tf. constant([[0, 1], [2, 3]])) M = tf. get_variable('big_matrix', shape=(784, 10), initializer=tf. zeros_initializer()) W = tf. get_variable('weight', shape=(784, 10), initializer=tf. truncated_normal_initializer(mean=0. 0, stddev=0. 01)) 30/04/2018 www. easy-tensorflow. com 26

Data types Graph Variables 2. import tensorflow as tf # a b c create graph = tf. get_variable(name="A", initializer=tf. constant([[0, 1], [2, 3]])) = tf. get_variable(name="B", initializer=tf. constant([[4, 5], [6, 7]])) = tf. add(a, b, name="Add") # launch the graph in a session with tf. Session() as sess: # now we can run the desired operation print(sess. run(c)) Variables ? Failed. Precondition. Error: Attempting to use uninitialized value 30/04/2018 www. easy-tensorflow. com 27

Data types 2. Graph Variables import tensorflow as tf # create graph a = tf. get_variable(name="A", initializer=tf. constant([[0, 1], [2, 3]])) b = tf. get_variable(name="B", initializer=tf. constant([[4, 5], [6, 7]])) c = tf. add(a, b, name="Add") # Add an Op to initialize variables init_op = tf. global_variables_initializer() # launch the graph in a session with tf. Session() as sess: Variables # run the variable initializer sess. run(init_op) # now we can run the desired operation print(sess. run(c)) [[ 4 6] [ 8 10]] 30/04/2018 www. easy-tensorflow. com 28

Data types Placeholder is a node whose value is fed in at execution time. 3. 1. 2. Assemble the graph without knowing the values needed for computation We can later supply the data at the execution time. ⇒ Input data (in classification task: Inputs and labels) tf. placeholder( dtype, shape=None, name=None ) Example a b X Y = = tf. placeholder(tf. float 32, shape=[5]) tf. placeholder(dtype=tf. float 32, shape=None, name=None) tf. placeholder(tf. float 32, shape=[None, 784], name='input') tf. placeholder(tf. float 32, shape=[None, 10], name='label') 30/04/2018 www. easy-tensorflow. com 29

Data types Placeholder 3. Graph import tensorflow as tf a = tf. constant([5, 5, 5], tf. float 32, name='A') b = tf. placeholder(tf. float 32, shape=[3], name='B') c = tf. add(a, b, name="Add") with tf. Session() as sess: print(sess. run(c)) ? You must feed a value for placeholder tensor 'B' with dtype float and shape [3] 30/04/2018 Variables www. easy-tensorflow. com 30

Data types 3. Placeholder Graph import tensorflow as tf a = tf. constant([5, 5, 5], tf. float 32, name='A') b = tf. placeholder(tf. float 32, shape=[3], name='B') c = tf. add(a, b, name="Add") with tf. Session() as sess: # create a dictionary: d = {b: [1, 2, 3]} # feed it to the placeholder print(sess. run(c, feed_dict=d)) Variables [6. 7. 8. ] 30/04/2018 www. easy-tensorflow. com 31

Final Example • Logistic Classifier (linear classifier) D=784 30/04/2018 www. easy-tensorflow. com Out=10 32

MNIST Data 28 28 30/04/2018 www. easy-tensorflow. com 33

Epoch / Iteration Example: MNIST data - number of training data: N=55, 000 - Let’s take batch size of B=100 … First 100 images (=1 st iteration) … … 2 nd iteration - How many iteration in each epoch? 55000/100 = 550 30/04/2018 www. easy-tensorflow. com … Last iteration 1 epoch = 550 iteration 34

Logistic Classifier (linear classifier) • Bias (1, 10) Weight (784, 10) WX+b = y 28 (1, 784) 30/04/2018 28 0. 0 1. 5 0. 7 0. 4 0. 1 3. 5 0. 2 0. 1 Logits www. easy-tensorflow. com 35

Logistic Classifier (linear classifier) WXn+b 0. 0 1. 5 0. 7 0. 4 0. 1 3. 5 0. 2 0. 1 SOFTMAX (Superscript: index of elements) Logits (yn) 0. 02 0. 09 0. 04 0. 03 0. 02 0. 70 0. 03 0. 02 Probs. (S(yn)) Input (Xn) 30/04/2018 Cross-entropy www. easy-tensorflow. com 0 0 0 1 0 0 0 One-hot encoded labels (Ln) 36

Optimizer Gradient descent: Learning rate An overview of gradient descent optimization algorithms 30/04/2018 www. easy-tensorflow. com 37

Logistic Classifier (linear classifier) • Disadvangates: ◦ ◦ • Few parameters: #parameters = 10 x 784 + 10 = 7850 At the end, it’s a linear model! Advantages: ◦ 30/04/2018 Linear models are STABLE! www. easy-tensorflow. com 38