Transformer Introduction Architecture Key Concepts Transformer on CV

• Transformer Introduction • • Architecture, Key Concepts Transformer on CV

Transformer Pipeline • • 由Encoder和Decoder两部分构成的网络。 Encoder和Decoder内部仅仅使用self-attention 和 fc 来执行特征学习。 Encoder Decoder

Machine Translation • • Decoder 部分采用auto regressive的策略。 训练时采用掩码策略。

Transformer Encoder

Transformer Encoder

Encoder-Decoder Attention K, V are derived from encoder feature, Q is derived from decoder feature.

Other Key Concepts in Transformer • Residual Connection. • Feed Forward Network. consists of two linear transformation layer and a nonlinear activation. • Input Element-wise Feature (d=512). 1. Embed into a vector with d=512 dimension. 2. Position encoding feature for each dimension i 3. Element-wise add. • Final Layer in the Decoder/Encoder

Positional Encoding

Visual Transformer: i. GPT(image GPT) by Open. AI Image Down. Sample 9 -bit color palette to represent pixels Decoder Image generation No positional Encoding! https: //openai. com/blog/image-gpt/ Classification Pre-Training Encoder Image Completion

Visual Transformer: i. GPT(image GPT) by Open. AI Image completion Image generation

Visual Transformer: i. GPT(image GPT) by Open. AI

Visual Transformer: Vi. T(visual Transformer) by Google . cls_token = nn. Parameter(torch. zeros(1, 1, embed_dim)) # positional encoding self . pos_embed = nn. Parameter(torch. zeros(1, num_patches + 1, embed_dim)) self

Visual Transformer: Vi. T(visual Transformer) by Google • 数据集越大，Transformer学的越好（inductive bais）。 Relational inductive biases, deep learning, and graph network

Visual Transformer: Classification Comparison 知识蒸馏

Visual Transformer: DERT(detection transformer) by Facebook H’ x W’ x 3 HW x D Hx. Wx. D Nx. D HW x D Magic Here! Nx. D Initially Random, Totally Learnt (c, x, y, h, w)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, Ground Truth w 1) w 2) w 3) w 4) w 5) (Bird, x 1, y 1, h 1, w 1) (Bird, x 2, y 2, h 2, w 2)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, Ground Truth w 1) w 2) w 3) w 4) w 5) (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, Ground Truth h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5) ? ? ? 1 to 1 (Bird, (None, Step 1: Find optimal assignment x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Bipartite Matching Loss Predefined N=5 Prediction Ground Truth With hat (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5) ? ? ? (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5) = Matching Cost Step 1: Find optimal assignment None in GT does not matter

Bipartite Matching Loss Predefined N=5 Prediction Ground Truth With hat (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5) ? ? ? (Bird, (None, = Matching Cost Step 1: Find optimal assignment x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, Ground Truth h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5) (Bird, (None, Step 1: Find optimal assignment x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, Ground Truth w 1) w 2) w 3) w 4) w 5) (Bird, (None, x 1, x 2, x 3, x 4, x 5, Step 2: Compute total loss for training y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Bipartite Matching Loss Predefined N=5 Prediction (Bird, (None, x 1, x 2, x 3, x 4, x 5, y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, Ground Truth w 1) w 2) w 3) w 4) w 5) (Bird, (None, x 1, x 2, x 3, x 4, x 5, Step 2: Compute total loss for training y 1, y 2, y 3, y 4, y 5, h 1, h 2, h 3, h 4, h 5, w 1) w 2) w 3) w 4) w 5)

Visual Transformer: Deformable DERT(detection transformer) by Sense. Time • • Deformable Self-Attention (10 x faster) Multi-scale Feature

Visual Transformer: Deformable DERT(detection transformer) by Sense. Time 对每个query，sample K*M个点

Visual Transformer: Deformable DERT(detection transformer) by Sense. Time C 3 C 4 不需要FPN，即可轻松实现不同scale feature的融合！

Visual Transformer: DERT(detection transformer) by Facebook

拍脑瓜 • 能利用Transformer Encoder-Decoder的结构吗？ Pair-wise输入问题(FR-IQA)、Shape Prior/Depth+分割 共同的embedding space: 多模态输入(RGB和光流)、 Q&A 循环的Transformer(RTT), Multi-stage结构 自监督任务 • 能利用Self-Attention无限的感受野吗？ 多尺度特征 • 如何处理输入？ • No inductive bias的对抗攻击？