REU End to End Self Driving Car Farzain

REU - End to End Self Driving Car Farzain + Kishan

BEWARE

Quick Recap • Create an end-to-end system that is able to autonomously maneuver a car through a variety of scenarios. Only allowed to use CAMERAS. Extremely complex situations. • IMPOSSIBLE. • Deep learning can help us out.

Related • NVIDIA has released several reports on their work. • Works very well for simpler situations (staying in lane, not hitting cars in front) • Doesn’t work so well in more complex situations that are harder for a network to recognize (traffic signal, pedestrians) • This is partially because there isn’t enough training data and partially because the network may have a hard time differentiating between whats important and what isn’t.

Related • Princeton paper recently proposed an FCN-LSTM • Wanted to use ONLY dash cam videos, nothing else, to predict future movement. Almost turned the problem into a text prediction problem. • Semantic segmentation label side training to “learn a meaningful feature representation”. Here they are implicitly feeding it segmentation information. • Doesn’t go directly from action -> input, “future motion action feasibility distribution”

Other

Udacity • Labeled datasets. • Challenged programmers to create a deep network to predict steering angles. • First place used a 3 D CNN + LSTM + RNN • Second place used Optical Flow + NVIDIA CNN • Top 5 models are open source.

Model 2 D CNN DECISION

Model SEG LOSS FCN LSTM DECISION

Model 3 D CNN LSTM DECISION

Where do we go from here?

Model 2 D CNN

Model 3 D CNN

Model 3 D CNN LSTM

Model 3 D CNN LSTM DECISION

Model RGB 3 D CNN Segmentation Model Output LSTM DECISION 3 D CNN Issues: output of two stream CNN, memory, usage of another model Pre-training?

Model DECISION Output as probability distribution

Next 7 Days • Code up/train NVIDIA base model in Keras (half-done) • Code up Udacity dataset reader script (half-done). • Get results of NVIDIA model to establish a solid base. • Create 2 D CNN + LSTM and see results