Social Sync Synchronization for a Smartphone Camera Network

Social. Sync: Synchronization for a Smartphone Camera Network Richard Latimer Rice University September 7, 2014

Intro 2 2

Why Smartphones? • 1 billion smartphones sold in 2014 • Growth in computer vision applications • Advanced development platform Best camera is the one with you! 3

What if Smartphones Could Collaborate? Lytro Refocusing Kilner et al. Free Viewpoint Television Pelican Philips 3 D 4 Depth 4

Prior Work: Camera Arrays High Performance Imaging Using Large Camera Arrays [Wilburn et al. ] High-Speed Videography Using a Dense Camera Array [Wilburn et al. ] Traditional camera arrays are assumed to be embedded and tightly synchronized. 5

A Smartphone Camera Array Involves the OS Stack Application Framework Library Runtime Linux Kernel/Drivers Hardware 6 6

Social Capture: Problem Statement If you use multiple smartphone cameras to capture video simultaneously, there is an inherit synchronization challenge that manifests (due to OS stack) when the scene is moving. 7 7

Scene Setup: How Bad is Misalignment? Resolution Target 1 0 0 Smartphone Camera Array 8 Scene Setup 8

Outcome: How Bad is Misalignment? Even when photos are captured simultaneously, misalignment of frames can cause problems when target is moving. Moving Resolution Target Captured by 4 Phones Simultaneously 9 9

Source of Problem 10 Sub-Frame Misalignment Camera 1: Camera 2: Camera 3: 10 Frame 1 Frame 2 Frame 3 Frame 0 Frame 1 Frame 2 Capture Request time

11 Goal of Social. Sync Objective: Align frame sequences from smartphone cameras within tolerance ranges less than t milliseconds, where t < frame duration. Sub-Frame Misalignment Camera 1: Camera 2: Camera 3: 11 Frame 0 Frame 1 Frame 2 Frame 3 Frame 0 Frame 1 Frame 2 Capture Request time

12 1. Characterization 2. Social. Sync Protocol 3. Evaluation 12

13 1. Characterization • Measurement Test Apparatus • Results 2. Social. Sync Protocol 3. Evaluation 13

What We Use: Android 4. 4 14 14

Network Setup Hub and Spoke Network 1 0 0 • Server broadcasts image capture request 1 0 0 15 • Server provides global reference clock 15

16 System for Measuring Capture Timestamps 16

17 1. Characterization • Measurement Test Apparatus • Results 2. Social. Sync Protocol 3. Evaluation 17

Random Variables Associated with Capturing a Frame Setup Time: Delay to setup camera before capture of first frame Delivery Time: After capture, there is delay to when frame reaches application and a timestamp can be recorded. 18 18

19 Camera Setup Frame 0 Frame 1 Frame 2 time 19

Camera 1: Is Setup Time Consistent? Setup time 20 20

Camera 1: Is Setup Time Consistent? Setup Frame 0 time 21 21

22 Variability in Camera Setup Makes Aligning Frames Challenging Each time camera started, the start of the image sequence will be delayed by a variable amount of time. • Setup can vary at least 35 ms • St. Dev = 9. 4 ms • Not deterministic 22

23 Frame Delay TD(0) TD(1) Setup Frame 0 Frame 1 . . . TD(n) Frame N time 23

Measurement of Frame Delay 24

Delivery Time of Frame to Application If we keep measuring capture times of frames, they are spaced evenly Delivery time of frame to app is a proxy for estimating capture time with delay: • μ = 67 ms • σ = 4. 5 ms 25 25

26 1. Characterization 2. Social. Sync Protocol • Estimation • Alignment 3. Evaluation 26

27 Goal of Social Sync Sub-Frame Misalignment Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 Sync Tolerance 27 time

28 Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 28

If Misalignment is Due to Camera Setup Sub-Frame Misalignment Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 time 29 29

Could We Estimate Capture Time By Calibrating for Setup Time? Sub-Frame Misalignment Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 time 30 30

Recall Wide Variability in Setup If setup time had little variance, could align capture timestamps by offsetting • Setup can vary at least 35 ms, when worst case frame misalignment was 33 ms (inverse of FPS) 31 • Can’t calibrate for setup time to reduce misalignment! 31

Does Frame Delay Provide a Better Estimate of the Capture Timestamp? TD(0) TD(1) Setup Frame 0 Frame 1 . . . TD(n) Frame N time 32 32

Measurement of Frame Delay 33

How does estimate of mean improve with number of samples? Stats 101: Standard error of a sample mean is Based on Central Limit Theorem, 95% confidence sample mean is within range: 34

35 Mean Frame Delay Recap TD(0) Setup Frame 0 time 35

36 Mean Frame Delay Recap Mean Delay Est. : TD(0) Setup Frame 0 TC(0) 36 time

37 Mean Frame Delay Recap Mean Delay Est. : TD(0) TD(1) Setup Frame 0 Frame 1 TC(0) TC(1) 37 time

38 Mean Frame Delay Recap Mean Delay Est. : TD(0) TD(1) Setup Frame 0 Frame 1 . . . TC(0) TC(1) 38 TD(n) Frame N T (n) time C

39 Samples Required for Estimate Since mean delay is , we can determine the capture image sequence position for milliseconds using: • 22 samples for 2 ms accuracy • 85 samples for 1 ms accuracy 39

40 Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 40

41 Synchronization Tolerance Goal: Want synchronization to a reference image sequence to be aligned within some tolerance Request Sync Tolerance Capture Request 41 time

Synchronization Attempt 1 Sync Tolerance Setup Frame 0 time 42 42

Synchronization Attempt 1 Sync Tolerance Setup Frame 0 . . Frame N time 43 43

Synchronization Attempt k Sync Tolerance Setup Frame 0 time 44 44

Synchronization Attempt k Sync Tolerance Setup Frame 0 . . Frame N time 45 45

46 Obtaining Synchronization • Let T be the worst case misalignment between frames • Let t be the range specified by the sync tolerance, where t ≤T Let p = t/T, be the probability of synchronization error being reduced for a single phone 46

47 Single Camera Sync Probability After k Attempts Pk = 1 -(1 -p)k Expected Number of Sync Cameras n*Pk 47

Expected Attempts to Sync N Cameras 48 48

49 • Characterization • Social. Sync Protocol • Evaluation 49

50 Evaluating Synchronization Naïve. Sync Social. Sync 4 Cameras 23 ms 5 ms 8 Cameras 35 ms 6 ms Smartphones in array to reduce artifacts not related to time synchronization error 50

51 8 Cameras Naïve Sync Social. Sync Reference Views Depth Map 51

52 4 Cameras Reference Naïve. Sync Social. Sync 52

53 4 Cameras Reference Naïve. Sync Social. Sync 53

54 Conclusion 1. 2. 3. Characterize Android Camera Developed Social. Sync Protocol Demonstrated Social. Sync Future Work: - Greater control of Android camera system - In wild implementation 54

55 Acknowledgements 1. 2. 3. 4. Ashok, Lin, Ashu and their students Co-Author Jason Holloway LF 4 CV Reviewers, Paper Accepted National Science Foundation 55