Matrix Calculus PoChen Wu Media IC and System

Matrix Calculus Po-Chen Wu Media IC and System Lab GIEE, NTU

Outline • • Introduction Derivatives in Numerator-Layout Notation Derivatives in Denominator-Layout Notation Identities Media IC & System Lab Po-Chen Wu (吳柏辰) 2

Outline • Introduction Notation Matrix Operations • Derivatives in Numerator-Layout Notation • Derivatives in Denominator-Layout Notation • Identities Media IC & System Lab Po-Chen Wu (吳柏辰) 3

Matrix Calculus • Matrix calculus is a specialized notation for doing multivariable calculus, especially over spaces of matrices. • Two competing notational conventions split the field of matrix calculus into two separate groups. Numerator-Layout Notation Media IC & System Lab Denominator-Layout Notation Po-Chen Wu (吳柏辰) 4

Notation • Media IC & System Lab Po-Chen Wu (吳柏辰) 5

Properties of Transpose • Media IC & System Lab Po-Chen Wu (吳柏辰) 6

Trace & Determinant • Media IC & System Lab Po-Chen Wu (吳柏辰) 7

Hadamard Product • Media IC & System Lab Po-Chen Wu (吳柏辰) 8

Kronecker Product • Media IC & System Lab Po-Chen Wu (吳柏辰) 9

Outline • Introduction • Derivatives in Numerator-Layout Notation List of Differentiation Derivative Formulas Chain rule The Matrix Differential • Derivatives in Denominator-Layout Notation • Identities Media IC & System Lab Po-Chen Wu (吳柏辰) 10

Vector-by-Scalar • Media IC & System Lab Po-Chen Wu (吳柏辰) 11

Scalar-by-Vector • Media IC & System Lab Po-Chen Wu (吳柏辰) 12

Vector-by-Vector • Media IC & System Lab Po-Chen Wu (吳柏辰) 13

Example 1 • Media IC & System Lab Po-Chen Wu (吳柏辰) 14

Example 2 • Media IC & System Lab Po-Chen Wu (吳柏辰) 15

Matrix-by-Scalar • Media IC & System Lab Po-Chen Wu (吳柏辰) 16

Scalar-by-Matrix • Media IC & System Lab Po-Chen Wu (吳柏辰) 17

List of Differentiation • Result of differentiating various kinds of aggregates with other kinds of aggregates. Notation Type scalar Media IC & System Lab Po-Chen Wu (吳柏辰) 18

Derivative Formulas • Media IC & System Lab Po-Chen Wu (吳柏辰) 19

Chain Rule (1/2) • Media IC & System Lab Po-Chen Wu (吳柏辰) 20

Chain Rule (2/2) • Media IC & System Lab Po-Chen Wu (吳柏辰) 21

Exercise 1 (Numerator-Layout) • Media IC & System Lab Po-Chen Wu (吳柏辰) 22

The Matrix Differential (1/2) • Media IC & System Lab Po-Chen Wu (吳柏辰) 23

The Matrix Differential (2/2) • Media IC & System Lab Po-Chen Wu (吳柏辰) 24

Summary of Numerator-Layout • Media IC & System Lab Po-Chen Wu (吳柏辰) 25

Outline • Introduction • Derivatives in Numerator-Layout Notation • Derivatives in Denominator-Layout Notation List of Differentiation Derivative Formulas Chain Rule • Identities Media IC & System Lab Po-Chen Wu (吳柏辰) 26

List of Differentiation • The results of operations will be transposed when switching from numerator-layout notation to denominator-layout notation. Notation Type scalar Media IC & System Lab Po-Chen Wu (吳柏辰) 27

Derivative Formulas • Media IC & System Lab Po-Chen Wu (吳柏辰) 28

Chain Rule • Media IC & System Lab Po-Chen Wu (吳柏辰) 29

Exercise 2 (Denominator-Layout) • Media IC & System Lab Po-Chen Wu (吳柏辰) 30

Outline • • Introduction Derivatives in Numerator-Layout Notation Derivatives in Denominator-Layout Notation Identities Vector-by-Vector Scalar-by-Vector-by-Scalar-by-Matrix-by-Scalar Media IC & System Lab Po-Chen Wu (吳柏辰) 31

Vector-by-Vector • Condition Media IC & System Lab Expression Numerator layout Po-Chen Wu (吳柏辰) Denominator layout 32

Scalar-by-Vector • Condition Media IC & System Lab Expression Numerator layout Po-Chen Wu (吳柏辰) Denominator layout 33

Vector-by-Scalar • Condition Media IC & System Lab Expression Numerator layout Po-Chen Wu (吳柏辰) Denominator layout 34

Scalar-by-Matrix • Condition Media IC & System Lab Expression Numerator layout Po-Chen Wu (吳柏辰) Denominator layout 35

Matrix-by-Scalar • Condition Media IC & System Lab Expression Numerator layout Po-Chen Wu (吳柏辰) 36

Reference • Matrix calculus, Wiki http: //en. wikipedia. org/wiki/Matrix_calculus • The Matrix Cookbook http: //www. imm. dtu. dk/pubdb/views/edoc_downlo ad. php/3274/pdf/imm 3274. pdf Media IC & System Lab Po-Chen Wu (吳柏辰) 37