Machine Learning with ArrayFire: Linear Classifiers

Pavan ArrayFire, CUDA, OpenCL Leave a Comment

Linear classifiers perform classification based on the linear combinition of the component features. Some examples of Linear Classifiers include: Naive Bayes Classifier, Linear Discriminant Analysis, Logistic Regression and Perceptrons.

ArrayFire's easy to use API enables users to write such classifiers from scratch fairly easily. In this post, we show how you can map mathematical equations to ArrayFire code and implement them from scratch.

Naive Bayes Classifier

Naive bayes classifier is a probabilistic classifier that assumes all the features in a feature vector are independent of each other. This assumption simplifies the bayes rule to a simple multiplication of probabilities as show below.

First we start with the simple Baye's rule.

Where:

Since p(x) is the same for all classes, it can be safely ignored. The classification is then done using the following formula

Classification based on Naive Bayes method uses the following phases:

  • Training Phase: The probability ditribution of each feature variable is modeled
  • Prediction Phase: The probability of a sample belonging to each class is calculated based on the distributions from Phase 1.

In our example, we are assuming all the input features fit a normal distribution. This requires us to calculate the mean, mu and the variance Sig2 in the training phase.

The necessary ArrayFire code for the training phase can be seen below:

With the mean and variance of each class known, we can check the probabiltiy of a sample belonging to a particular class using the CDF of the distribution. The class with the highest probability wins.

However when working with a large number of feature variables, multiplication can lead to underflow errors. We can get around this by using log and sum instead of exp and mul.

The prediciton phase can be seen below.

Perceptron

Perceptron is one of the first classification algorithms to use supervised learning. The perceptron transforms the weighted input features into the activation in the range [0, 1].

A visual representation of a perceptron

To perform classification on K classes, we use K perceptrons and choose the class with the maximum activation. For the perceptron k, the activation function can be seen below:

To get an activation function that is continuous we use sigmoid function show below:

The equivalent ArrayFire code can be seen here:

The weights of the perceptron are trained in an iterative process. The trained weights are then used for prediction on test data. Training a perceptron includes the following steps:

  1. Initialize the weights
  2. Predict the output based on training data
  3. Update the weights in proportion to the folloing

  4. if error greater than treshold, go to step 2.

The training process of the perceptron can be implemented in ArrayFire using the following code.

Results

The complete source code for testing and benchmarking these algorithms can be found here:

We tested these algorithms on a small subset of the MNIST database with 6100 training images, 3900 testing images.

The performance and accuracy results can be seen below:

Algorithm Hardware Accuracy Train Time Predict Time
Naive Bayes NVIDIA GTX 680 80.69% 0.003s 0.011s
Naive Bayes AMD Radeon 7970 81.19% 0.007s 0.007s
Perceptron (1000 iter) NVIDIA GTX 680 83.64% 2.005s 0.0003s
Perceptron (1000 iter) AMD Radeon 7970 85.29% 4.935s 0.0005s

Conclusion

We have shown how simple mathematical expressions can easily translate into ArrayFire code. If you are interested in learning more about how ArrayFire can be used to develop Machine Learning applications, follow this blog!

If are going to GTC 2015, you may also be interested in the following tutorials.

  • S5803: Simplified Machine Learning for CUDA
  • S5722: Deep Belief Networks using ArrayFire
Facebooktwitterredditlinkedinmail