Basis Of Neural Networks In Visual Basic .NET


In this article (hopefully, the first of a small series), we'll see how to implement a neural network in
Visual Basic .NET, i.e. a model capable of processing input data and adjust its internal mechanics to learn how to produce a desired result. We'll see more on this later. The present article will focus on generic definitions about neural networks and their behaviors, offering a simple implementation for the reader to test. In the final paragraph, we will code a small network capable of swapping two variables. 


A first definition

The term "neural network" is typically used as a reference to a network or circuit constituted by neurons. We can differentiate two types of neural networks: a) biological and b) artificial. Obviously, speaking about software development, we are here referring to artificial ones, but those kind of implementations get their basic model and inspiration from their natural counterparts, so it can be useful to briefly consider the functioning of what we intend when we speak of biological neural networks. 

Natural neural networks

Those are networks constituted by biological neurons, and they are typical of living creatures. The neurons/cells are interconnected into the peripheral nervous system or in the central one. In neurosciences, groups of neurons are identified by the physiological function they perform. 

Artificial neural networks

Artificial networks are mathematical models, which could be implemented through an electronic medium, which mime the functioning of a biological network. Simply speaking, we will have a set of artificial neurons apt to solve a particular problem in the field of artificial intelligence. Like a natural one, an artificial network could "learn", through time and trial, the nature of a problem, becoming more and more efficient in solving it. 


After this simple premise, it should be obvious that in a network, being it natural or artificial, the entity known as "neuron" has a paramount importance, because it receive inputs, and is somewhat responsible of a correct data processing, which end in a result. Think about our brain: it's a wonderful supercomputer composed by 86*10^9 neurons (more or less). An amazing number of entities which constantly exchange and store information, running on 10^14 synapses. Like we've said, artificial models are trying to capture and reproduce the basic functioning of a neuron, which is based on 3 main parts:

  • Soma, or cellular body
  • Axon, the neuron output line
  • Dendrite, the neuron input line, which receives the data from other axons through synapses
The soma executes a weighted sum of the input signals, checking if they exceed a certain limit. If they do, the neuron activates itself (resulting in a potential action), staying in a state of quiet otherwise. An artificial model tries to mimic those sub-parts, with the target of creating an array of interconnected entities capable of adjusting itself on the basis of received inputs, constantly checking the produced results against an expected situation. 

How a network learns

Typically, the neural network theory identifies 3 main methods through which a network can learn (where, with "learn", we intend - from now on - the process through which a neural network modifies itself to being able to produce a certain result with a given input). Regarding the Visual Basic implementation, we will focus only on one of them, but it's useful to introduce all the paradigms, in order to have a better overview. For a NN (neural network) to learn, it must be "trained". The training can be supervisioned, if we possess a set of data constituted by input and output values. Through them, a network could learn to infer the relation which binds a neuron with another one. Another method is the unsupervisioned one, which is based on training algorithms which modifies the network's weights relying only on input data, resulting in networks that will learn to group received information with probabilistic methods. The last method is the reinforced learning, which doesn't rely on presented data, but on exploration algorithms which produce inputs that will be tested through an agent, which will check their impact on the network, trying to determine the NN performance on a given problem. In this article, when we'll come to code, we will see the first presented case, or the supervisioned training. 

Supervisioned training

So, let's consider the method from a closer perspective. What it means to training a NN with supervision? As we've said, it deals primarily of presenting a set of input and output data. Let's suppose we want to teach our network to sum two numbers. In that case, following the supervision training paradigm, we must feed our network with input data (lets say [1;5]) but also telling it what we expect as a result (in our case, [6]). Then, a particular algorithms must be applied in order to evaluate the current status of the network, adjusting it by processing our input and output data. The algorithm we will use in our example is called backpropagation.


The backpropagation of errors is a technique in which we first initialize our network (typically with random values regarding the weights on single neurons), proceding in forwarding our input data, matching results with output data we expect. Then we calculate the deviation of the obtained values from the desired one, obtaining a delta factor which must be backpropagated to our neurons, to adjust their initial state according to the entity of error we've calculated. Through trials and repetitions, several sets of input and output data are presented to the network, each time repeating the match between the real and the ideal value. In a certain time, that kind of operation will produce increasingly precise outputs, calibrating the weight of each network's components, and ultimately refining its ability to process the received data. For an extensive explanation about backpropagation, please refer to the Bibliography section.

Creating a neural network

Now that we saw some preliminar concepts about neural networks, we should be able to develop a model which responds to the discussed paradigms. Without exceeding in mathematical over-explanations (which aren't really needed, unless you want to understand better what we will see), we will proceed step by step in coding a simple yet functional neural network, that we'll test when completed. The first thing we need to consider is the structure of a neural network: we know it's organized in neurons, and the neurons themselves are interconnected. But we don't know how.


And that's where the layers comes into scene. A layer is a group of neurons which share a more or less common function. For example, think about the entry point of input data. That will be the input layer, in which resides a group of neurons which share a common functionality, in that case limited to receiving and forward information. Then we'll surely have an output layer, which will group those neurons that receive the result of previous processing. Among those layers could reside many layers, typically called "hidden", because an user won't have direct access to them. The number of those hidden layers, and the number of neurons each of them will contain, depends heavily on the nature and complexity of the problem we want to solve. To sum up, each network will be constituted of layers, each of which will contain a certain predetermined number of neurons.

Neurons and dendrites

"Artificially" speaking, we could conceive a neuron like an entity which exposes a certain value, adjusted by trial iterations, and bind to other neurons through dendrites, which in our case will be represented by sub-entities possessing an initial random weight. The training process will consist in feeding the input layer neurons, which transmit their value through dendrites to an upper layer, which will do the same thing until the output layer is reached. Finally, we calculate a delta between the current output and the desired one, recurring through the network and adjusting the dendrites weights, the neuron values and each deviation value, to correct the network itself. Then we start again with another training round. 

Preparing network classes

Having saw how a network is structured, we could sketch down some classes to manage the various entities of a network. In the following snippet, i will outline Dendrite, Neuron and Layer classes, which we will use together in the implementation of a NeuralNetwork class. 

Dendrite class
  1. Public Class Dendrite  
  2.     Dim _weight As Double  
  4.     Property Weight As Double  
  5.         Get  
  6.             Return _weight  
  7.         End Get  
  8.         Set(value As Double)  
  9.             _weight = value  
  10.         End Set  
  11.     End Property  
  13.     Public Sub New()  
  14.         Me.Weight = r.NextDouble()  
  15.     End Sub  
  16. End Class  
First, the Dendrite class: as you can see, it's constituted by a sole property, named Weight. In initializing a Dendrite, a random Weight is attributed to our dendrite. The type of the Weight property is Double, because our input values will be between zero and one, so we need a strong precision when it comes to decimal places. More on this later. No other properties nor functions are required for that class.

Neuron class
  1. Public Class Neuron  
  2.     Dim _dendrites As New List(Of Dendrite)  
  3.     Dim _dendriteCount As Integer  
  4.     Dim _bias As Double  
  5.     Dim _value As Double  
  6.     Dim _delta As Double  
  8.     Public Property Dendrites As List(Of Dendrite)  
  9.         Get  
  10.             Return _dendrites  
  11.         End Get  
  12.         Set(value As List(Of Dendrite))  
  13.             _dendrites = value  
  14.         End Set  
  15.     End Property  
  17.     Public Property Bias As Double  
  18.         Get  
  19.             Return _bias  
  20.         End Get  
  21.         Set(value As Double)  
  22.             _bias = value  
  23.         End Set  
  24.     End Property  
  26.     Public Property Value As Double  
  27.         Get  
  28.             Return _value  
  29.         End Get  
  30.         Set(value As Double)  
  31.             _value = value  
  32.         End Set  
  33.     End Property  
  35.     Public Property Delta As Double  
  36.         Get  
  37.             Return _delta  
  38.         End Get  
  39.         Set(value As Double)  
  40.             _delta = value  
  41.         End Set  
  42.     End Property  
  44.     Public ReadOnly Property DendriteCount As Integer  
  45.         Get  
  46.             Return _dendrites.Count  
  47.         End Get  
  48.     End Property  
  50.     Public Sub New()  
  51.         Me.Bias = r.NextDouble()  
  52.     End Sub  
  53. End Class  
Next, the Neuron class. As one can imagine, it will expose a Value property (of type Double, for the same motivations saw above), and a series of potential Dendrites, the number of which will depend to the neuron number of the layer our current neuron will be connected to. So we have a Dendrite property, a DendriteCount (which returns the number of Dendrites), and two property which will serve in the process of recalibration, namely Bias and Delta.

Layer class
  1. Public Class Layer  
  2.     Dim _neurons As New List(Of Neuron)  
  3.     Dim _neuronCount As Integer  
  5.     Public Property Neurons As List(Of Neuron)  
  6.         Get  
  7.             Return _neurons  
  8.         End Get  
  9.         Set(value As List(Of Neuron))  
  10.             _neurons = value  
  11.         End Set  
  12.     End Property  
  14.     Public ReadOnly Property NeuronCount As Integer  
  15.         Get  
  16.             Return _neurons.Count  
  17.         End Get  
  18.     End Property  
  20.     Public Sub New(neuronNum As Integer)  
  21.         _neuronCount = neuronNum  
  22.     End Sub  
  23. End Class  
Finally, a Layer class, which is simply a container for an array of neurons. In calling upon the New method, the user must indicate how many neurons the layer is required to have. We'll see in the next section how those classes can interact in the contest of a full-fledged neural network.

NeuralNetwork class

Our NeuralNetwork can be seen as a list of layers (each of which will inherit the underlying layer properties, i.e. neurons and dendrites). A neural network must be launched (or put in running state) and trained, so we will likely have two methods which can be used for that means. The network initialization must specify, among other properties, a parameter that we will call "learning rate". That will be a variable we'll use in weights recalculation. As the name implies, the learning rate is a factor which determines how fast a network will learn. Since it's a corrective factor, the learning rate must be chosen accurately: if its value is too large, but there is a large multitude of possible inputs, a network may not learn too well, or at all. Generally speaking, a good practice is to set the learning rate to a relatively small value, increasing it if the effective recalibration of our network becomes too slow.

Lets see an almost complete NeuralNetwork class:
  1. Public Class NeuralNetwork  
  2.     Dim _layers As New List(Of Layer)  
  3.     Dim _learningRate As Double  
  5.     Public Property Layers As List(Of Layer)  
  6.         Get  
  7.             Return _layers  
  8.         End Get  
  9.         Set(value As List(Of Layer))  
  10.             _layers = value  
  11.         End Set  
  12.     End Property  
  14.     Public Property LearningRate As Double  
  15.         Get  
  16.             Return _learningRate  
  17.         End Get  
  18.         Set(value As Double)  
  19.             _learningRate = value  
  20.         End Set  
  21.     End Property  
  23.     Public ReadOnly Property LayerCount As Integer  
  24.         Get  
  25.             Return _layers.Count  
  26.         End Get  
  27.     End Property  
  29.     Sub New(LearningRate As Double, nLayers As List(Of Integer))  
  30.         If nLayers.Count < 2 Then Exit Sub  
  32.         Me.LearningRate = LearningRate  
  34.         For ii As Integer = 0 To nLayers.Count - 1  
  36.             Dim l As Layer = New Layer(nLayers(ii) - 1)  
  37.             Me.Layers.Add(l)  
  39.             For jj As Integer = 0 To nLayers(ii) - 1  
  40.                 l.Neurons.Add(New Neuron())  
  41.             Next  
  43.             For Each n As Neuron In l.Neurons  
  44.                 If ii = 0 Then n.Bias = 0  
  46.                 If ii > 0 Then  
  47.                     For k As Integer = 0 To nLayers(ii - 1) - 1  
  48.                         n.Dendrites.Add(New Dendrite)  
  49.                     Next  
  50.                 End If  
  52.             Next  
  54.         Next  
  55.     End Sub  
  57.     Function Execute(inputs As List(Of Double)) As List(Of Double)  
  58.         If inputs.Count <> Me.Layers(0).NeuronCount Then  
  59.             Return Nothing  
  60.         End If  
  62.         For ii As Integer = 0 To Me.LayerCount - 1  
  63.             Dim curLayer As Layer = Me.Layers(ii)  
  65.             For jj As Integer = 0 To curLayer.NeuronCount - 1  
  66.                 Dim curNeuron As Neuron = curLayer.Neurons(jj)  
  68.                 If ii = 0 Then  
  69.                     curNeuron.Value = inputs(jj)  
  70.                 Else  
  71.                     curNeuron.Value = 0  
  72.                     For k = 0 To Me.Layers(ii - 1).NeuronCount - 1  
  73.                         curNeuron.Value = curNeuron.Value + Me.Layers(ii - 1).Neurons(k).Value * curNeuron.Dendrites(k).Weight  
  74.                     Next k  
  76.                     curNeuron.Value = Sigmoid(curNeuron.Value + curNeuron.Bias)  
  77.                 End If  
  79.             Next  
  80.         Next  
  82.         Dim outputs As New List(Of Double)  
  83.         Dim la As Layer = Me.Layers(Me.LayerCount - 1)  
  84.         For ii As Integer = 0 To la.NeuronCount - 1  
  85.             outputs.Add(la.Neurons(ii).Value)  
  86.         Next  
  88.         Return outputs  
  89.     End Function  
  91.     Public Function Train(inputs As List(Of Double), outputs As List(Of Double)) As Boolean  
  92.         If inputs.Count <> Me.Layers(0).NeuronCount Or outputs.Count <> Me.Layers(Me.LayerCount - 1).NeuronCount Then  
  93.             Return False  
  94.         End If  
  96.         Execute(inputs)  
  98.         For ii = 0 To Me.Layers(Me.LayerCount - 1).NeuronCount - 1  
  99.             Dim curNeuron As Neuron = Me.Layers(Me.LayerCount - 1).Neurons(ii)  
  101.             curNeuron.Delta = curNeuron.Value * (1 - curNeuron.Value) * (outputs(ii) - curNeuron.Value)  
  103.             For jj = Me.LayerCount - 2 To 1 Step -1  
  104.                 For kk = 0 To Me.Layers(jj).NeuronCount - 1  
  105.                     Dim iNeuron As Neuron = Me.Layers(jj).Neurons(kk)  
  107.                     iNeuron.Delta = iNeuron.Value *  
  108.                                     (1 - iNeuron.Value) * Me.Layers(jj + 1).Neurons(ii).Dendrites(kk).Weight *  
  109.                                     Me.Layers(jj + 1).Neurons(ii).Delta  
  110.                 Next kk  
  111.             Next jj  
  112.         Next ii  
  115.         For ii = Me.LayerCount - 1 To 0 Step -1  
  116.             For jj = 0 To Me.Layers(ii).NeuronCount - 1  
  117.                 Dim iNeuron As Neuron = Me.Layers(ii).Neurons(jj)  
  118.                 iNeuron.Bias = iNeuron.Bias + (Me.LearningRate * iNeuron.Delta)  
  120.                 For kk = 0 To iNeuron.DendriteCount - 1  
  121.                     iNeuron.Dendrites(kk).Weight = iNeuron.Dendrites(kk).Weight + (Me.LearningRate * Me.Layers(ii - 1).Neurons(kk).Value * iNeuron.Delta)  
  122.                 Next kk  
  123.             Next jj  
  124.         Next ii  
  126.         Return True  
  127.     End Function  
  129. End Class  
New() Method

When our network is initialized, it requires a learning rate parameter, and a list of Layers. Processing that list, you can see how each layer will result in a generation of neurons and dendrites, which are assigned to their respective parents. Calling the New() method of neurons and dendrites, will result in a random assignation of their initial values and weights. If the passed layers are less than two, the subroutine will exit, because a neural network must have at least two layers, input and output.
  1. Sub New(LearningRate As Double, nLayers As List(Of Integer))  
  2.     If nLayers.Count < 2 Then Exit Sub  
  3.     Me.LearningRate = LearningRate  
  5.     For ii As Integer = 0 To nLayers.Count - 1  
  6.         Dim l As Layer = New Layer(nLayers(ii) - 1)  
  7.         Me.Layers.Add(l)  
  8.         For jj As Integer = 0 To nLayers(ii) - 1  
  9.             l.Neurons.Add(New Neuron())  
  10.         Next  
  11.         For Each n As Neuron In l.Neurons  
  12.             If ii = 0 Then n.Bias = 0  
  13.             If ii > 0 Then  
  14.                 For k As Integer = 0 To nLayers(ii - 1) - 1  
  15.                     n.Dendrites.Add(New Dendrite)  
  16.                 Next  
  17.             End If  
  18.         Next  
  19.     Next  
  20. End Sub  
Execute() Function

As we've said, our network must possess a function through which we process input data, making them move into our network, and gathering the final results. The following function does this. First, we'll check for input correctness: if the number of inputs are different from the input layer neurons, the function cannot be executed. Each neuron must be initialized. For te first layer, the input one, we simply assign to the neuron Value property our input. For other layers, we calculate a weighted sum, given by the current neuron Value, plus the Value of the previous layer neuron multiplied by the weight of the dendrite. Finally, we execute on the calculated Value a Sigmoid function, which we'll analyze below. Processing all layers, our output layer neurons will receive a result, which is the parameter our function will return, in the form of a List(Of Double). 
  1. Function Execute(inputs As List(Of Double)) As List(Of Double)  
  2.     If inputs.Count <> Me.Layers(0).NeuronCount Then  
  3.         Return Nothing  
  4.     End If  
  5.     For ii As Integer = 0 To Me.LayerCount - 1  
  6.         Dim curLayer As Layer = Me.Layers(ii)  
  7.         For jj As Integer = 0 To curLayer.NeuronCount - 1  
  8.             Dim curNeuron As Neuron = curLayer.Neurons(jj)  
  9.             If ii = 0 Then  
  10.                 curNeuron.Value = inputs(jj)  
  11.             Else  
  12.                 curNeuron.Value = 0  
  13.                 For k = 0 To Me.Layers(ii - 1).NeuronCount - 1  
  14.                     curNeuron.Value = curNeuron.Value + Me.Layers(ii - 1).Neurons(k).Value * curNeuron.Dendrites(k).Weight  
  15.                 Next k  
  16.                 curNeuron.Value = Sigmoid(curNeuron.Value + curNeuron.Bias)  
  17.             End If  
  18.         Next  
  19.     Next  
  21.     Dim outputs As New List(Of Double)  
  22.     Dim la As Layer = Me.Layers(Me.LayerCount - 1)  
  23.     For ii As Integer = 0 To la.NeuronCount - 1  
  24.         outputs.Add(la.Neurons(ii).Value)  
  25.     Next  
  26.     Return outputs  
  27. End Function  
Sigmoid() Function

The sigmoid is a mathematical function known for its typical "S" shape. It is defined for each real input value. We use such a function in neural network programming, because its differentiability - which is a requisite for backpropagation - and because it introduces non-linearity into our network (or, it makes out network able to learn the correlations among inputs that don't produce linear combinations). Plus, for each real value, the sigmoid function return a value between zero and one (excluding upper limit). That function has peculiarities which make it really apt when it comes to backpropagation.

Train() Function

A network is initialized with random values, so - without recalibration - the results they return are pretty random themselves, more or less. Without a training procedure, a freshly started network can be almost useless. We define, with the word training, the process through which a neural network keeps running on a given set of inputs, and its results are constantly matched against expected output sets. Spotting differences on our outputs from the returned values of the net, we then proceed in recalibrating each weight and value on the net itself, stepping forward a closer resemblance of what we want and what we get from the network.

In VB code, something like this:
  1. Public Function Train(inputs As List(Of Double), outputs As List(Of Double)) As Boolean  
  2.     If inputs.Count <> Me.Layers(0).NeuronCount Or outputs.Count <> Me.Layers(Me.LayerCount - 1).NeuronCount Then  
  3.         Return False  
  4.     End If  
  6.     Execute(inputs)  
  8.     For ii = 0 To Me.Layers(Me.LayerCount - 1).NeuronCount - 1  
  9.         Dim curNeuron As Neuron = Me.Layers(Me.LayerCount - 1).Neurons(ii)  
  10.         curNeuron.Delta = curNeuron.Value * (1 - curNeuron.Value) * (outputs(ii) - curNeuron.Value)  
  11.         For jj = Me.LayerCount - 2 To 1 Step -1  
  12.             For kk = 0 To Me.Layers(jj).NeuronCount - 1  
  13.                 Dim iNeuron As Neuron = Me.Layers(jj).Neurons(kk)  
  14.                 iNeuron.Delta = iNeuron.Value *  
  15.                                 (1 - iNeuron.Value) * Me.Layers(jj + 1).Neurons(ii).Dendrites(kk).Weight *  
  16.                                 Me.Layers(jj + 1).Neurons(ii).Delta  
  17.             Next kk  
  18.         Next jj  
  19.     Next ii  
  21.     For ii = Me.LayerCount - 1 To 0 Step -1  
  22.         For jj = 0 To Me.Layers(ii).NeuronCount - 1  
  23.             Dim iNeuron As Neuron = Me.Layers(ii).Neurons(jj)  
  24.             iNeuron.Bias = iNeuron.Bias + (Me.LearningRate * iNeuron.Delta)  
  25.             For kk = 0 To iNeuron.DendriteCount - 1  
  26.                 iNeuron.Dendrites(kk).Weight = iNeuron.Dendrites(kk).Weight + (Me.LearningRate * Me.Layers(ii - 1).Neurons(kk).Value * iNeuron.Delta)  
  27.             Next kk  
  28.         Next jj  
  29.     Next ii  
  31.     Return True  
  32. End Function  
As usual, we check for correctness of inputs, then start our network by calling upon the Execute() method. Then, starting from the last layer, we process every neuron and dendrite, correcting each value by applying the difference of outputs. The same thing will be done on dendrites weights, introducing the learning rate of the network, as saw above. At the end of a training round (or, more realistically, on the completion of several hundreds of rounds), we'll start to observe that the outputs coming from the network will become more and more precise.

A test application

In the present section we will see a simple test application, which we'll use to create a pretty trivial network, used to swap two variables. In the following, we will consider a network with three layers: an input layer constituted by two neurons, an hidden layer with four neurons, and an output layer with two neurons. What we expect from our network, giving enough supervision training, will be the ability to swap the values we present it or, in other words, to being able to shift the value of the input neuron #1 on the output neuron #2, and vice versa.

In the downloadable NeuralNetwork class, you'll find a couple of methods which we haven't analyzed in the article, which refers to the graphical and textual network rendering. We will use them in our example, and you can consult them by downloading the source code. 

Initializing the network

Using our classes, a neural network can be simply initialized by declaring it, and feeding it its learning rate and start layers. For example, in the test application you can see a snippet like this:
  1. Dim network As NeuralNetwork  
  3. Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load  
  4.     Dim layerList As New List(Of Integer)  
  5.     With layerList  
  6.         .Add(2)  
  7.         .Add(4)  
  8.         .Add(2)  
  9.     End With  
  11.     network = New NeuralNetwork(21.5, layerList)  
  12. End Sub  
I've defined my network as globally to a Form, then - in the Load() event - i've created the layers i need, passing them to the network initializer. That will result in having our layers correctly populated with the number of neurons specified, each element starting with random values, deltas, biases and weights.

Running a network

Running and training a network are very simple procedures as well. All they require is to call upon the proper method, passing a set of input and/or output parameters. In case of the Execute() method, for example, we could write: 
  1. Dim inputs As New List(Of Double)  
  2.    inputs.Add(txtIn01.Text)  
  3.    inputs.Add(txtIn02.Text)  
  5. Dim ots As List(Of Double) = network.Execute(inputs)  
  6.    txtOt01.Text = ots(0)  
  7.    txtOt02.Text = ots(1)  
Where txtIn01, txtIn02, txtOt01, txtOt02 are Form's TextBoxes. What we will do with the above code, is simply to get inputs from two TextBoxes, using them as inputs for the network, and writing on another couple of TextBoxes the returned values. The same procedure applies in case of training.

Swapping two variable

In this video, Neural Network Example in VB.NET, you could see a sample training session. The running program is the same you'll find by downloading the source code at the end of the article. As you can see, by starting with random values, the network will learn to swap the signals it receives. The values of 0 and 1 must not be intended as precise numbers, but rather as "signals of a given intensity". By observing the results of training, you could note how the intensity of the values will increase toward 1, or decrease to zero, never really reaching the extremes, but representing them anyway. So, at the end of an "intense" training to teach our network that, in case of an input couple of [0,1] we desire an output of [1,0], we can observe that we receive something similar to [0.9999998,0.000000015], where the starting signals are represented, by their intensity, to the closer conducive value, just like a neuron behave himself, speaking about activation energy levels.

Source code

The source code we've discussed so far can be download at the following link: Basis of Neural Networks in Visual Basic .NET.