C++Builder What Is A Rectified Linear Unit Activation ANN Function?

[FireWind]

What Is A Rectified Linear Unit Activation ANN Function?
By Yilmaz Yoru December 17, 2021

What is Rectified Linear Function in ANN? How can we use ReLU Activation Function Hyperbolic? Let’s refresh our memories on what activation functions are and explain these terms.

What is an artificial neural network (ANN) activation function?​

An Activation Function ( phi() ) also called as transfer function, or threshold function determines the activation value ( a = phi(sum) ) from a given value (sum) from the Net Input Function . Net Input Function, here the sum is a sum of signals in their weights, and activation function is a new value of this sum with a given function or conditions. In another term. The activation function is a way to transfer the sum of all weighted signals to a new activation value of that signal. There are different activation functions, mostly Linear (Identity), bipolar and logistic (sigmoid) functions are used. The activation function and its types are explained well Для просмотра ссылки Войди или Зарегистрируйся.

In C++ (in general in most programming languages) you can create your activation function. Note that sum is the result of Net Input Function which calculates the sum of all weighted signals. We will use some as a result of the input function. Here activation value of an artificial neuron (output value) can be written by the activation function as below,

By using this sum Net Input Function Value and phi() activation functions, let’s see some of activation functions in C++; Now Let’s see how we can use Binary Step Function as in this example formula,

What is a Rectified Linear Unit?​

In Artificial Neural Networks, the Rectifier Linear Unit Function or in other terms ReLU Activation Function is an activation function defined as the positive part of its argument. Can be written as f(x)= max(0, x) where x is sum of weighted input signals to an artificial neuron. ReLU Function is also known as a Для просмотра ссылки Войди или Зарегистрируйся and is analogous to Для просмотра ссылки Войди или Зарегистрируйся in electrical engineering.

This function called as a Parametric ReLU function. If Beta is 0.01 it is called Leaky ReLU function.

This is max-out ReLU function,

if Beta is 0 then f(x) = max(x, 0). This function will return always positive numbers. Let’s write maxout ReLU function in C programming language,

double phi(double sum)
{
  return ( std::max(sum, beta*sum) );   //  ReLU Function
}

Is there a simple ANN example using a Rectified Linear Unit activation function in C++?​

#include <iostream>
 
#define    NN 2   // number of neurons
 
class Tneuron    // neuron class
{
  public:
   double a;       // activity of each neurons
   double w[NN+1]; // weight of links between each neurons
 
   Tneuron()
   {
 a=0;
 for(int i=0; i<=NN; i++) w[i]=-1;  // if weight is negative there is no link
   }
 
   // let's define an activation function (or threshold) for the output neuron
    double activation_function(double sum)
    {
        return ( std::max(sum, 0) );   //  ReLU Function
    }
};
 
Tneuron ne[NN+1];   // neuron objects
 
void fire(int nn)
{
   float sum = 0;
 
   for ( int j=0; j<=NN; j++ )
   {
    if( ne[j].w[nn]>=0 ) sum += ne[j].a*ne[j].w[nn];
   }
 
   ne[nn].a = ne[nn].activation_function(sum);
}
 
int main()
{
 //let's define activity of two input neurons (a0, a1) and one output neuron (a2)
 
 ne[0].a = 0.0;
 ne[1].a = 1.0;
 ne[2].a = 0;
 
 //let's define weights of signals comes from two input neurons to output neuron (0 to 2 and 1 to 2)
 
 ne[0].w[2] = 0.3;
 ne[1].w[2] = 0.2;
 
 
 // Let's fire our artificial neuron activity, output will be
 fire(2);
 
 printf("%10.6f\n", ne[2].a);
 getchar();
 return 0;
}