For the last two week I have been dying to implement different kinds of Recurrent Neural Networks (RNN) and finally I have the time to implement all of them. Below is the list of different RNN cases I wanted to try out.Case a: Vanilla Recurrent Neural Network Case b: Multi-Stream Recurrent Neural NetworkCase c: Concatenated Recurrent Neural NetworkCase d: Internal Convolutional Recurrent Neural NetworkCase e: Lag 2 Recurrent Neural NetworkVanilla Recurrent Neural NetworkThere is in total of 5 different case of RNN I wish to implement. However, in order to fully understand all of the implementations it would be a good idea to have a strong understanding of vanilla RNN (Case a is vanilla RNN so if you understand code for case a you are good to go.)If anyone wishes to review simple RNN please visit my old blog post “Only Numpy: Vanilla Recurrent Neural Network Deriving Back propagation Through Time Practice ”.Case a: Vanilla Recurrent Neural Network ( Results)Red Box → 3 Convolutional LayerOrange → Global Average Pooling and SoftMaxGreen Circle → Hidden Unit at Time 0Blue Circle → Input in 4 Time StampBlack Box → Recurrent Neural Network with 4 Time StampAs seen above, the base network is simple RNN combined with convolutional neural network for classification. The RNN have time stamp of 4, which means we are going to give the network 4 different kinds of input at each time stamp. And to do that I am going to add some noise to the original image.Blue Line → Train Cost Over TimeOrange Line → Train Accuracy Over TimeGreen Line → Test Cost Over TimeRed Line → Test Accuracy Over TimeAs seen above our base network already performs well. Now the question is how other methods performs and would it be able to regularize better than our base network.Case b: Multi-Stream Recurrent Neural Network (Idea / Results)Red Box → 3 Convolutional LayerOrange → Global Average Pooling and SoftMaxGreen Circle → Hidden Unit at Time 0Blue Circle → Convolution Input Stream Yellow Circle → Fully Connected Network StreamBlack Box → Recurrent Neural Network with 4 Time StampThe idea behind this RNN is simply to give different representation of data to the RNN. In our base network we have the network either the raw image or image with some noise added.Red Box → Additional Four CNN/FNN layers to ‘process’ the inputBlue Box → Creating Inputs at each different time stampsAs seen below now our RNN takes in input of tensor size with [batch_size, 26, 26, 1] reducing the width and the height by 2. And I was hoping that different representation of the data would act as a regularization. (Similar to data augmentation)Blue Line → Train Cost Over TimeOrange Line → Train Accuracy Over TimeGreen Line → Test Cost Over TimeRed Line → Test Accuracy Over TimeAs seen above the network did pretty well, and have outperformed our base network by 1 percent on the testing images.Case c: Concatenated Recurrent Neural Network (Idea / Results)Red Box → 3 Convolutional LayerOrange → Global Average Pooling and SoftMaxGreen Circle → Hidden Unit at Time 0Blue Circle → Input in 4 Time StampBlack Box → Recurrent Neural Network with 4 Time StampBlack Curved Arrow → Concatenated Input for Each Time StampThis approach is very simple, the idea was that on each time stamp different features will be extracted and it might be useful for the network to have more features overtime. (For the Recurrent Layers.)Blue Line → Train Cost Over TimeOrange Line → Train Accuracy Over TimeGreen Line → Test Cost Over TimeRed Line → Test Accuracy Over TimeSadly, this was a huge failure. I guess the empty hidden values does not help (one bit) for the network to perform well.Case d: Internal Convolutional Recurrent Neural Network (Idea/Results)Red Box → 3 Convolutional LayerOrange → Global Average Pooling and SoftMaxGreen Circle → Hidden Unit at Time 0Blue Circle → Input in 4 Time StampBlack Box → Recurrent Neural Network with 4 Time StampGray Arrow → Performing Internal Convolution before passing onto the next time stampAs seen above, this network takes in the exact same input as our base network. However this time we are going to perform additional convolution operations in the internal representation of the data.Right Image → Declaring 3 new convolution layerLeft Image (Red Box) → If the current internal layer is not None, we are going to perform additional convolution operation.I actually had no theoretical reason behind this implementation, I just wanted to see if it works LOL.Blue Line → Train Cost Over TimeOrange Line → Train Accuracy Over TimeGreen Line → Test Cost Over TimeRed Line → Test Accuracy Over TimeAs seen above the network did a fine job at converging, however it was not able to outperform our base network. (Sadly).Case e: Lag 2 Recurrent Neural Network (Idea / Results)Red Box → 3 Convolutional LayerOrange → Global Average Pooling and SoftMaxGreen Circle → Hidden Unit at Time 0 (or Lag of 1)Blue Circle → Input in 4 Time StampBlack Box → Recurrent Neural Network with 4 Time StampPurple Circle → Hidden State Lag of 2In a traditional RNN setting we only rely on the most previous values to determine the current value. For a while I was thinking that there is no reason for us to limit the look back time (or lag) as 1. We can extend this idea into lag 3 or lag 4 etc. (Just for simplicity I took lag 2)Blue Line → Train Cost Over TimeOrange Line → Train Accuracy Over TimeGreen Line → Test Cost Over TimeRed Line → Test Accuracy Over TimeThankfully the network did better than the base network. (But with very small margin), however this type of network would be most suitable for time series data.Interactive Code / TransparencyFor Google Colab, you would need a google account to view the codes, also you can’t run read only scripts in Google Colab so make a copy on your play ground. Finally, I will never ask for permission to access your files on Google Drive, just FYI. Happy Coding! Also for transparency I uploaded all of the training logs on my github.To access the code for case a click here, for the logs click here. To access the code for case b click here, for the logs click here.To access the code for case c click here, for the logs click here.To access the code for case c click here, for the logs click here.To access the code for case c click here, for the logs click here.Final WordsI wanted to Review RNN for quite a long time now, finally I get to do it.If any errors are found, please email me at jae.duk.seo@gmail.com, if you wish to see the list of all of my writing please view my website here.Meanwhile follow me on my twitter here, and visit my website, or my Youtube channel for more content. I also implemented Wide Residual Networks, please click here to view the blog post.ReferenceFrom a quick cheer to a standing ovation, clap to show how much you enjoyed this story.https://jaedukseo.me | | | |  |Your everyday Seo, who likes kimchiSharing concepts, ideas, and codes.