CRCV REU Week 3 Arushi Rai What did

What did I do this week? • Background knowledge, read recommended papers • Ran

Project • Sub-Action Representation Learning in Long-Complex Activities • with Maddy and Dr. Rawat

Motivation • Supervised learning is expensive and there’s inconsistency in human annotation • Growing

Deep. Cluster (2018) • Caron et al. developed an end-to-end self supervised method to

https: //github. com/facebookresearch/deepcluster

Prec 1 Evaluate on Ci. FAR-10 Epochs Using Pretrained Alex. Net Deep. Cluster covnet

Understand Deep. Cluster and run on GPU and CPU Modified code to run with

Cross-Modal Deep Clustering (XDC) (2019) Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Dataset • Long-Complex Activities captured from How. To 100 M Dataset • 136 M

Transfer Task: COIN I started looking at the COIN dataset Madeline Schiappa , “Self-Supervised

Next Steps • Train an i 3 D network on the How. To subset

Slides: 16

Download presentation

CRCV REU Week 3 Arushi Rai

What did I do this week? • Background knowledge, read recommended papers • Ran Deep. Cluster + understand code base • Started looking at COIN dataset

Project • Sub-Action Representation Learning in Long-Complex Activities • with Maddy and Dr. Rawat How to Build a House Long-Complex Activity Put bricks Apply Cement Build Roof Sub-actions (Instructions)

Motivation • Supervised learning is expensive and there’s inconsistency in human annotation • Growing interest in doing unsupervised learning using clustering, alternative to GANs https: //www. gatevidyalay. com/k-means-clustering-algorithm-example/

Deep. Cluster (2018) • Caron et al. developed an end-to-end self supervised method to get image representation *implementation clarifies training

https: //github. com/facebookresearch/deepcluster

Prec 1 Evaluate on Ci. FAR-10 Epochs Using Pretrained Alex. Net Deep. Cluster covnet Trained on imagenet data Epochs Using Deep. Cluster covnet trained on Ci. FAR 10 data

Understand Deep. Cluster and run on GPU and CPU Modified code to run with Ci. FAR-10 and as well as fixing bugs due to Pytorch/faiss updates

Cross-Modal Deep Clustering (XDC) (2019) Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Dataset • Long-Complex Activities captured from How. To 100 M Dataset • 136 M video clips with captions sourced from 1. 2 M Youtube videos (15 years of video) • 23 k activities from 12 domains such as cooking, hand crafting, personal care, gardening or fitness https: //www. di. ens. fr/willow/research/howto 100 m/

Transfer Task: COIN I started looking at the COIN dataset Madeline Schiappa , “Self-Supervised Learning in Long. Complex Activities” Slides

Next Steps • Train an i 3 D network on the How. To subset • Implement a modified version of XDC • Extract features from COIN dataset