What is Activation Function ?Why we use Activation functions with Neural Networks?The Activation Functions can be basically divided into 2 types-As you can see the function is a line or linear.Therefore, the output of the functions will not be confined between any range.Equation : f(x) = xRange : (-infinity to infinity)It doesn’t help with the complexity or various parameters of usual data that is fed to the neural networks.The Nonlinear Activation Functions are the most used activation functions. Nonlinearity helps to makes the graph look something like thisIt makes it easy for the model to generalize or adapt with variety of data and to differentiate between the output.The main terminologies needed to understand for nonlinear functions are:The Nonlinear Activation Functions are mainly divided on the basis of their range or curves-1. Sigmoid or Logistic Activation FunctionThe Sigmoid Function curve looks like a S-shape.The main reason why we use sigmoid function is because it exists between (0 to 1). Therefore, it is especially used for models where we have to predict the probability as an output.Since probability of anything exists only between the range of 0 and 1, sigmoid is the right choice.The function is differentiable.That means, we can find the slope of the sigmoid curve at any two points.The function is monotonic but function’s derivative is not.The logistic sigmoid function can cause a neural network to get stuck at the training time.The softmax function is a more generalized logistic activation function which is used for multiclass classification.2. Tanh or hyperbolic tangent Activation Functiontanh is also like logistic sigmoid but better. The range of the tanh function is from (-1 to 1). tanh is also sigmoidal (s - shaped).The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph.The function is differentiable.The function is monotonic while its derivative is not monotonic.The tanh function is mainly used classification between two classes.3. ReLU (Rectified Linear Unit) Activation FunctionThe ReLU is the most used activation function in the world right now.Since, it is used in almost all the convolutional neural networks or deep learning.As you can see, the ReLU is half rectified (from bottom). f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero.Range: [ 0 to infinity)The function and its derivative both are monotonic.But the issue is that all the negative values become zero immediately which decreases the ability of the model to fit or train from the data properly. That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.4. Leaky ReLUIt is an attempt to solve the dying ReLU problemCan you see the Leak? 😆The leak helps to increase the range of the ReLU function. Usually, the value of a is 0.01 or so.When a is not 0.01 then it is called Randomized ReLU.Therefore the range of the Leaky ReLU is (-infinity to infinity).Both Leaky and Randomized ReLU functions are monotonic in nature. Also, their derivatives also monotonic in nature.I will be posting 2 posts per week so don’t miss the tutorial.So, follow me on Medium, Facebook, Twitter, LinkedIn, Google+, Quora to see similar posts.Any comments or if you have any question, write it in the comment.Clap it! Share it! Follow Me!Happy to be helpful. kudos.....2. Epoch vs Batch Size vs Iterations3. Train Inception with Custom Images on CPU4. TensorFlow Image Recognition Python API Tutorial On CPUFrom a quick cheer to a standing ovation, clap to show how much you enjoyed this story.I am interested in Programming (Python, C++), Arduino, Machine learning :) I'm the editor of Arduino Community on Medium. I also like to write stuff.Sharing concepts, ideas, and codes.