The dot product

Dot product is the calculation used to determine a neuron’s output. Those who have learnt matrices/vectors may be familiar with it.

Matrices!

Matrices are defined as a list of numbers. More accurately, it’s a ‘table’ of numbers with rows and columns:

$\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$

The above matrix has a size of 2x2 - 2 rows and 2 columns. It can be rewritten in Python like so:

matrix = [["a", "b"], 
["c", "d"]]

Each list inside of the list is a row and each column is an index of a list within.

Although you can add and subtract matrices, there is no ordinary multiplication or division of matrices. This is when we use dot product.

$\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\times \left[\begin{array}{cc}e& f\\ g& h\end{array}\right]$

Important

Multiplying matrices is only possible if the number of columns in the first matrix is equal to the number of rows in the second matrix. This is important and will come up later.

For example, the matrices below cannot be multiplied because the column size of the first matrix is not equal to the row size of the second.

$\left[\begin{array}{cc}a& b\end{array}\right]\times \left[\begin{array}{cc}c& d\\ e& f\\ h& i\end{array}\right]$

$1\times 2,3\times 2$

$2\ne 3$

If it does multiply, the resulting matrix’s size will have the column size of the first matrix and row size of the second matrix.

It is also important to realize that the order of which you multiply in is important. Matrix A multiplied by B may not be the same as matrix B multiplied by A. This will also come in later.

$\left[\begin{array}{cc}(a\times e)+(b\times g)& (a\times f)+(b\times h)\\ (c\times e)+(d\times g)& (c\times f)+(d\times h)\end{array}\right]$

Essentially, the output is given by the value of the inputs multiplied by their input’s weight. For example, the following is a random ‘neuron’ with 3 inputs (and subsequently 3 weights) and a bias.

The output is equal to the first input value (input_values[0]) multiplied by its respective weight (weights[0]). Repeat that with all input values.

input_values = [0.2, 0.5, 1.2]
weights = [1.0, 0.5, 1.3]
bias = 1.0
 
# The brackets used below are unnecssary and just used for clarity.
output = (input_values[0] * weights[0]) + (input_values[1] * weights[1]) + (input_values[2] * weights[2]) + bias

The sum of those weighted values plus the bias will be the output of the neuron.

In code

In my project here, I used Numpy(2.2.3), a package used to simplify this formula. Numpy has a method numpy.dot() that does the dot product for you. The functionality is the same however! The math applied above is what that .dot() method does.

import numpy as np
 
inputs = [0.2, 0.5, 1.2]
weights = [1.0, 0.5, 1.3]
bias = 1.0
 
output = np.dot(inputs, weights) + bias

However, as of right now, I am only dealing with 1D arrays - 1D inputs and 1D weights. This becomes slightly more complicated when dealing with 2D arrays and weights, like so:

import numpy as np
 
inputs = [[0.2, 0.5, 1.2],
			[1.2, 1.1, 0.6]]
 
weights = [[1.0, 0.5, 1.3],
			[2.0, 3.2, -0.8]]
 
bias = [1.0, 1.2]
 
output = np.dot(inputs, weights) + bias

Running the code above however will result in an ValueError. This is because of the inconsistent size of the matrices, as I’ve mentioned in the callout above.

$\left[\begin{array}{ccc}0.2& 0.5& 1.2\\ 1.2& 1.1& 0.6\end{array}\right]\times \left[\begin{array}{ccc}1.0& 0.5& 1.3\\ 2.0& 3.2& -0.8\end{array}\right]$

$2\times 3,2\times 3$

$3\ne 2$

Therefore, you are unable to multiply these matrices and is the reason of the ValueError. To fix this, all you need to do is transpose the weights (flip columns and rows). This can be easily done by using numpy’s transpose method np.array().T:

import numpy as np
 
inputs = [[0.2, 0.5, 1.2],
			[1.2, 1.1, 0.6]]
 
weights = [[1.0, 0.5, 1.3],
			[2.0, 3.2, -0.8]]
 
bias = [1.0, 1.2]
 
output = np.dot(inputs, np.array(weights).T) + bias

This should fix the error and give you a matrix of outputs, ready for the next layer of processing!