Radial Basis Function

A radial basis function(RBF) is a real-valued function whose value depends only on the input and its distance from some fixed point (c) or the origin. The function satisfies the criteria below:

The distance is usually the Euclidean distance between two points. The Euclidean distance is calculated as follows:

The sum of radial basis functions is typically used to approximate the given function.

But what does this mean?

||x – x_n|| is the radial part of the function, since its value depends on some distance from a fixed center. The basis function is the mathematical representation of the gaussian function. Now, you have points (x_n,y_n) ∈ D, where D is a dataset. All the points in D affect a hypothesis h(x). This means that all the points in the dataset affect and influence a machine learning model. But in our case, it affects h(x) specifically based on ||x – x_n|| , i.e the distance

What do we do with the equation?

We find the w_ns or weights that minimize the error between h(x) and y_n, so we basically try to equate them. This is called interpolation.

This is a representation of the gaussian distribution. You can observe above, an RBF is symmetric around the center if we take the topmost point as the center.

Applications of Radial Basis Function

Radial basis functions serve as a glue for many topics of machine learning. RBFs are often used for classification.

Relationship to nearest neighbor method

In the nearest neighbor(NN) method, you classify points according to their features. But is not going to be a smooth process. You thus consider only K nearest neighbors(KNN) to smoothen the classification boundaries. When we use RBF with K centers, we can assign more weight to the nearest points and less weight to the points that are far away. This makes the classification smoother than NN and KNN.

RBF with K centers

How to choose the centers?

We choose K number of clusters, where K << N. We get K number of centers for K clusters.

Our main objective is to minimize the distance between x_n and the closed center u_k. We choose a cluster around the center u_k and split the data points D into K clusters. We choose points that are nearest to the center. We minimize ||x_n-u_k||² of all the points in each cluster where x_n are points nearest to the center u_k.

Mathematically, we express the equation as follows:

w is the weight we aim to calculate.

How to choose the weights?

We minimize the equation for h(x) and approximately equate it to y_n.

We can treat gamma as a learnable parameter and use the EM approach(see wiki) to solve it. i.e

• We fix the value of gamma and solve for w_n
• We fix w_n, and try to minimize the error by adjusting gamma. We can calculate different gamma for different centers

Relationship to Neural Networks

Radial Basis Function network is an artificial neural network with an input layer, a hidden layer, and an output layer. It is similar to 2-layer networks, but we replace the activation function with a Gaussian radial basis function. RBF centers can be selected randomly from a set of data by using supervised learning, or they can be defined by using k-means clusters using unsupervised learning[1]. We take each input vector and feed it into each basis. Then, we do a simple weighted sum to get our approximated function value at the end. This is the approximation is compared to the training data and then assigned a value of similarity. We train these using backpropagation like any neural network.

Some points about RBF Networks:

1. Radial basis function networks are distinguished from other neural networks due to their universal approximation and faster learning speed
2. They are used in image classification 
3. It is used for the non-linear classification of data

You can see the code in python in this Python Machine Learning post.

RBF in Kernels

Radial Basis Function kernels are popular kernels in machine learning for non-linear data. It is the most commonly used kernel in Support Vector Machines(see our post).

References

[1] Xanthoula Eirini Pantazi, Dimitrios Moshou, Dionysis Bochtis, Chapter 2 – Artificial intelligence in agriculture, Intelligent Data Mining and Fusion Systems in Agriculture, Academic Press, 2020, Pages 17-101, ISBN 9780128143919, https://doi.org/10.1016/B978-0-12-814391-9.00002-9.

Links to Resources