7 Types of ML Classification Algorithms.

An overview of Machine Learning classification algorithms. The best algorithm and "No free lunch theorem".

Intro

This article is part of my big project. The main idea is to teach everybody Python, Data Science, and Machine Learning (ML) despite their educational background. I try to explain everything in simple terms but not compromise the quality.

If ML algorithms were a tasty soup, it is still possible to eat it without knowing what is inside. However, I prefer to shed light on the ingredients and the way it was cooked.

We already covered the reasons for learning Machine Learning, went through general terms/definitions, explored which common Python libraries are used for Data Science and ML, and loaded our first dataset!

Here, we are going to look into one specific ML approach for predictive modeling, called Classification. We will go through the definition of the classification modeling and a range of real-life applications.

My idea is to make this overview concise but broad at the same time. Thus, I will cover both a general overview of the topic and provide the links to each specific algorithm for practical exercises.

So, by the end of this article, you will get a general idea about different classification algorithms and will be able to dive deeper with a few of them right away*.

* I am adding the new articles as I write them (so, please check out for updates).

It is said “Give one a fish and it will feed one for a day. Teach one how to catch a fish and it will feed one for the rest of one's life.”

My purpose in writing this article is to teach you how to catch that fish by combining theory and hands-on examples to kick-start your ML journey.

So, let us begin.

What is Classification?

Generally speaking, Classification is an act or result of classifying (according to this dictionary).

In ML, by Classification, we mean a supervised learning approach referring to classifying or categorizing some unknown items into a discrete set of classes.

Talking in terms of feature and target variables, classification attempts to learn the relationship between them.

Simply speaking, it is all about predicting a label (a categorical variable with discrete values).

Why do we care about classification algorithms in Data Science?

As mentioned above, the point of classification is categorizing a given dataset into a set of labels/classes. What does it mean for real life?

It means, using the classification technique, we can assign a set of labels to any dataset that is possible to divide into classes or assign the labels to.

Depending on your area of interest, classification is used in a wide range of industries. Some of the real-world examples are:

• customer churn (leave/stay, for predicting if the customer will leave to another company/provider or not);
• email filtering (spam/no spam, to filter out annoying spam emails);
• document classification (to classify the documents into various types);
• biometrical identification (to assign a label representing a particular person);
• handwriting recognition (such as recognizing the digits and letters as separate classes);
• breast cancer identification (benign/malignant, to predict the state of cancer and assign a specific treatment);
• mushroom types (edible/poisonous, classification of mushrooms);

… to name a few.

As you can see, the range of possible applications of the classification approach is extremely wide. That should be enough by itself to motivate anyone to learn classification techniques.

Now, you might be thinking “That sounds great, Ruslan. I see your point. Just show me how to use that classification and I can apply it to any of the above-mentioned problems!”.

Hold on, my friend! I am not talking about classification in analogy to the Lord of the Rings:

One approach to rule them all.

Rather one approach = various algorithms.

I am talking about the whole range of topics that can be addressed by the classification approach. That approach encompasses different algorithms within itself.

Of course, there are many types of classification algorithms. Different kinds of data might and do require different approaches. So, one algorithm that might be good for one specific dataset, might be of no use to another.

Note: Usually, there is no way to know which algorithm will perform better for a given dataset — so, try them out, tune the parameters, and see what works better!

Let us talk about the 7 most popular classification algorithms.

The 7 Types of Classification Algorithms

Here is a list of the most common classification algorithms:

• K-nearest neighbors (K-NN)
• Logistic Regression
• Support Vector Machines (SVM)
• Decision Trees
• Naive Bayes
• Neural Networks (NN)
• Linear Discriminant Analysis (LDA)

While each classifier has its pros and cons, it is possible to train and use multiple classifiers. Then, make a final decision based on their results.

For example, the K-NN classification works well in the low-dimension feature space but it fails in the high-dimension one. This is called the curse of dimensionality. In this case, the SVM algorithm works better and is more efficient with memory. However, the SVM is prone to overfitting and does not provide a probability estimation, which is usually desirable for classification algorithms (such as, we want to know how confident is our algorithm in assigning one or another class/label).

The Best Classification Algorithm?

In their paper with the beautiful name "No Free Lunch Theorems for Optimization", David H. Wolpert and William G. Macready (1997) stated:

Roughly speaking, we show that for both static and time-dependent optimization problems, the average performance of any pair of algorithms across all possible problems is identical.

But we need to choose the best solution for our problem, right?

There might not be an ideal algorithm (in comparison to all the other algorithms) but there is a better algorithm for a given type of data, given task, and the proper tuning of the algorithm’s parameters.

In general, to address the success of a classification algorithm, one can use some metric of its performance, such as accuracy, F1-score, misclassification rate, etc.

To be continued

To make this overview as short and as useful as possible, we are concluding it here. The next natural question I would like to address is a variety of performance metrics. That is how we can judge and choose the best classification algorithm.

Here is an article describing the 9 Types of Performance Evaluation for Classification Machine Learning Modeling.

Want to learn more?

If you are interested in brushing up your knowledge of Python, here is a brief tutorial: from Hello Wolrd to Functions. As I mentioned earlier, here are a few of my projects explained from the beginning until the end. It can be a great opportunity to learn something new and practice some of the Python skills:

• tracking your weight from scratch in Python
• analyzing COVID-19 scientific papers
• creating a productivity app (Pomodoro) from scratch

Thank you for reading until the end. I hope you have learned something new. If you would like to connect or have any questions please do not hesitate to contact me.

Are you curious about the emerging field of Prompt Engineering? Grab my new e-book! You will learn and master everything from fundamental concepts to practical tips and real-world applications. Additionally, you will receive a bonus of 300 prompts and some of the free resources to kick-start your AI-driven journey. With all this value packed into one e-book, what is the price? The cost of a cup of coffee! Do not miss out on this opportunity to take your skills to the next level!

Contact

LinkedIn

I recently started a YouTube channel where I talk about different topics, including data science and AI news, research, and life in general among others. It is a steep learning curve for me but I invite you to check it out here.

Never miss a story, join my mailing list!

GitHub