Week 10 Presentation Predictive coding Dr Rawat Tamique

Week 10 Presentation Predictive coding (Dr. Rawat) Tamique de Brito

Summary of project concept ● Combining predictive coding with classification into a self-supervised approach ○ ○ Predictive coding has been used with predicting future representations; however, we are experimenting with using arbitrary predictions Using action-recognition dataset as labelled data ● The approach is as follows: ○ ○ ○ Take a group of video clips from different times and encode them (encoder is trainable) Pair clip encodings with temporal encodings and aggregate into global representation Take a target clip and encode it Use the global representation and the temporal encoding of the target to predict the target’s representation Can train jointly with classification for partially-labeled data

Standard Predictive Coding

Arbitrary Predictive Coding

What has been done this week ● Tested approach with pretrained weights ○ ○ Basic multiclip approach with I 3 D and transformer has 8% higher accuracy than I 3 D by itself There were difficulties with applying combined approach to pretrained weights ■ It helped to slowly increase the weight given to predictive coding component ■ However, the combined approach still seems to reduce accuracy with pretrained weight ● Tried semi-supervised learning: ○ Not working: adding in unlabelled data is making accuracy worse

What needs to be done ● See if semisupervised learning can be made to work ○ ○ Test different fractions of data labelled Test different coding loss and contrastive loss functions (as well as different weights)