OverviewIn this post, we will go over the work I did for project 3 of Udacity’s self-driving car project, behavior cloning for driving. The main task is to drive a car around in a simulator on a race track, and then use deep learning to mimic the behavior of human. This is a very interesting problem because it is not possible to drive under all possible scenarios on the track, so the deep learning algorithm will have to learn general rules for driving. We must be very careful while using deep learning models, because they have a tendency to overfit the data. Overfitting refers to the condition where the model is very sensitive to the training data itself, and the model’s behavior does not generalize to new/unseen data. One way to avoid overfitting is to collect a lot of data. A typical convolutional neural network can have up to a million parameters, and tuning these parameters requires millions of training instances of uncorrelated data, which may not always be possible and in some cases cost prohibitive. For our car example, this will require us to drive the car under different weather, lighting, traffic and road conditions. One way to avoid overfitting is to use augmentation.Augmentation refers to the process of generating new training data from a smaller data set such that the new data set represents the real world data one may see in practice. As we are generating thousands of new training instances from each image, it is not possible to generate and store all these data on the disk. We will therefore utilize keras generators to read data from the file, augment on the fly and use it to train the model. We will utilize images from the left and right cameras so we can generate additional training data to simulate recovery. Keras generator is set up such that in the initial phases of learning, the model drops data with lower steering angles with higher probability. This removes any potential for bias towards driving at zero angle. After setting up the image augmentation pipeline, we can proceed to train the model. The training was performed using simple adam learning algorithm with learning rate of 0.0001. After this training, the model was able to drive the car by itself on the first track for hours and generalized to the second track.All the training was based on driving data of about 4 laps using ps4 controller on track 1 in one direction alone. The model never saw track 2 in training, but with image augmentation (flipping, darkening, shifting, etc) and using data from all the cameras (left, right and center) the model was able to learn general rules of driving that helped translate this learning to a different track.IMPORTANT: These results were obtained on Titan X GPU machine I built earlier. Full specifications of the computer can be found here. Please note that computers with different performance will provide a different performance of the network.Augmentation helps us extract as much information from data as possible. We will generate additional data using the following data augmentation techniques. Augmentation is a technique of manipulating the incoming training data to generate more instances of training data. This technique has been used to develop powerful classifiers with little data. https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html . However, augmentation is very specific to the objective of the neural network.Brightness augmentationChanging brightness to simulate day and night conditions. We will generate images with different brightness by first converting images to HSV, scaling up or down the V channel and converting back to the RGB channel.Using left and right camera imagesUsing left and right camera images to simulate the effect of car wandering off to the side, and recovering. We will add a small angle .25 to the left camera and subtract a small angle of 0.25 from the right camera. The main idea being the left camera has to move right to get to center, and right camera has to move left.Horizontal and vertical shiftsWe will shift the camera images horizontally to simulate the effect of car being at different positions on the road, and add an offset corresponding to the shift to the steering angle. We added 0.004 steering angle units per pixel shift to the right, and subtracted 0.004 steering angle units per pixel shift to the left. We will also shift the images vertically by a random number to simulate the effect of driving up or down the slope.Shadow augmentationThe next augmentation we will add is shadow augmentation where random shadows are cast across the image. This is implemented by choosing random points and shading all points on one side (chosen randomly) of the image. The code for this augmentation is presented below.FlippingIn addition to the transformations above, we will also flip images at random and change the sign of the predicted angle to simulate driving in the opposite direction.2. PreprocessingAfter augmenting the image as above, we will crop the top 1/5 of the image to remove the horizon and the bottom 25 pixels to remove the car’s hood. Originally 1/3 of the top of car image was removed, but later it was changed to 1/5 to include images for cases when the car may be driving up or down a slope. We will next rescale the image to a 64X64 square image. After augmentation, the augmented images looks as follows. These images are generated using kera’s generator, and unlimited number of images can be generated from one image. I used Lambda layer in keras to normalize intensities between -.5 and .5.3. Keras generator for subsamplingAs there was limited data and we are generating thousands of training examples from the same image, it is not possible to store all the images apriori into memory. We will utilize kera’s generator function to sample images such that images with lower angles have lower probability of getting represented in the data set. This alleviates any problems we may ecounter due to model having a bias towards driving straight. Panel below shows multiple training samples generated from one image. The keras generator is presented below. The ‘pr_threshold’ variable is a threshold that determines if a data with small angle will be dropped or not.4. Model Architecture and trainingI implemented the model architecture above for training the data. The first layer is 3 1X1 filters, this has the effect of transforming the color space of the images. Research has shown that different color spaces are better suited for different applications. As we do not know the best color space apriori, using 3 1X1 filters allows the model to choose its best color space. This is followed by 3 convolutional blocks each comprised of 32, 64 and 128 filters of size 3X3. These convolution layers were followed by 3 fully connected layers. All the convolution blocks and the 2 following fully connected layers had exponential relu (ELU) as activation function. I chose leaky relu to make transition between angles smoother.Training:I trained the model using the keras generator with batch size of 256 for 8 epochs. In each epoch, I generated 20000 images. I started with pr_threshold, the chance of dropping data with small angles as 1, and reduced the probability by dividing it by the iteration number after each epoch. The entire training took about 5 minutes. However, it too more than 20 hours to arrive at the right architecture and training parameters. Snippet below presents the result of training.5. Model performance:Video below shows the performance of algorithm on the track 1 on which the original data was collected. The car is able to drive around for hours, we will next look into the case where either the camera resolution, video size or tracks are changed.Generalization from one image size to anotherVideo below presents generalization from one image size to another. I used the same pretrained model and tested it on all the other image sizes and found that the deep learning neural network was able to drive the car around for all image sizes.Generalization from one image resolution to anotherVideo below presents generalization from one image resolution to another. I used the same pretrained model and tested it on all the other image resolutions and found that the deep learning neural network was able to drive the car around for all image resolutions.I also tested different combinations of image size and image resolutions, and on track 1 the deep learning algorithm was able to drive the car around for all combinations of image resolution and sizes.Generalization from one track to anotherFigure below presents generalization from one track to another. This was perhaps the toughest test for the deep learning algorithm. In the second track, there were more right turns and u-turns, it was darker, and the road had slopes. All of which were absent in the original track. However, all these effects were artificially included into the model via image augmentation.6. Future DirectionsThis project is far from over. This project opened more questions than it answered. A few more things to try are,6. ReflectionsThis was perhaps the weirdest project I did. This project challenged all the previous knowledge I had about deep learning. In general large epoch size and training with more data results in better performance, but in this case any time I got beyond 10 epochs, the car simply drove off the track. Although all the image augmentation and tweaks seem reasonable n0w, I did not think of them apriori. I hope others find this post useful, and get inspried to try novel things. I havent used on-the fly training agile trainer by John Chen yet. I wanted to try and stretch the data as much as possible. Next thing to try is to experiment with parallel network using John’s trainer.Acknowledgements:I am very thankful to Udacity for selecting me for the first cohort, this allowed me to connect with many like-minded individuals. As always, learned a lot from discussions with Henrik Tünnermann and John Chen. I am also thankful for getting the NVIDA’s GPU grant. Although, its for work, but I use it for Udacity too.From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.Staff Software Engineer at Lockheed Martin-Autonomous System with research interest in control, machine learning/AI. Lifelong learner with glassblowing problem.Best place to learn about Chatbots. We share the latest Bot News, Info, AI & NLP, Tools, Tutorials & More.