Documents embeddings and text classification without coding

What are document embeddings and how to classify text without a single line of code?

Text is described by the sequence of character. Since every machine learning algorithm needs numbers, we need to transform the text into vectors of real numbers before we can continue with the analysis. To do this, we can use various approaches. The most known approach before the evolution of deep learning was the bag of words which is still widely used because of its advantages. The recent boom in the deep learning brought us new approaches such as word and document embeddings. In this post, we explain what document embedding is, why it is useful, and show its usage on the classification example without coding. For the analysis, we will use the Orange open-source tool.

Word embedding and document embedding

Before we can understand document embeddings, we need to understand the concept of word embeddings. Word embedding is a representation of a word in multidimensional space such that words with similar meanings have similar embedding. It means that each word is mapped to the vector of real numbers that represent the word. Embedding models are mostly based on neural networks.

Document embedding is usually computed from the word embeddings in two steps. First, each word in the document is embedded with the word embedding then word embeddings are aggregated. The most common type of aggregation is the average over each dimension.

Why and when should we use embedders?

Compared to bag-of-words, which counts the number of appearances of each token (word) in the document, embeddings have two main advantages:

• They do not have a dimensionality problem. The result of bag-of-words is a table which has the number of features equal to the number of unique tokens in all documents in a corpus. Large corpora with long texts result in a large number of unique tokens. It results in huge tables which can exceed the computer memory. Huge tables also increase the learning and evaluation time of machine learning models. Embeddings have constant dimensionality of the vector, which is 300 for fastText embeddings that Orange uses.
• Most of the preprocessing is not required. In the case of the bag-of-words approach, we solve the dimensionality problem with the text preprocessing where we remove tokens (e.g. words) that seems to be less important for the analysis. It can also cause the removal of some important tokens. When using embedders, we do not need to remove tokens, so we are not losing accuracy. Also most of the basic preprocessing can be omitted (such as normalization) in case of fastText embedding.
• Embeddings can be pretrained on large corpora with billions of tokens. That way, they capture the significant characteristics of the language and produce the embeddings of high quality. Pretrained models are then used to obtain embeddings of smaller datasets.

The shortcoming of the embedders is that they are difficult to understand. For example, when we use a bag-of-words, we can easily observe which tokens are important for classification since tokens themselves are features. In the case of document embeddings, features are numbers which are not understandable to human by themselves.

Orange

For the demonstration in this story, I will use my favourite tool for data analysis — Orange. Orange is one of the most known tools for interactive data analysis, visualization, and machine learning. It is open-source and you can download it from its website. Orange consist of the canvas where you place widgets and connect them together to create the workflow. Each widget represents one step in the data analysis.

Document Embedding widget

Orange offers document embedders through the Document Embedding widget. It uses fastText pretrained embedders, which support 157 languages and maps every document in the vector with 300 elements. Orange’s Document Embedding widget currently supports 31 most common languages.

The Fake News dataset

In this tutorial, we use the sample of Fake News dataset. The dataset sample is available here. It contains two datasets: training set including 2725 text items and testing set with 275 items. Each item is an article which is labelled as a real or fake.

Fake news identification

Here we present how to use document embeddings for fake news identification step by step. First, we will load a training part of the dataset with the Corpus widget.

The widget loads table which contains three columns: text, title, and label. After the dataset is loaded, we make sure that the text feature is selected in the Used text features field. It means that the text in this feature is used in the text analysis (tokens from this variable will be embedded), while the title feature is not used. When the dataset is loaded, we connect the Corpus widget to the Document embedder widget which will compute text embeddings. Our workflow should look like this:

In the document embeddings widget, we check that language is set to English since texts in this dataset are English. We will use mean (average) aggregation in this experiment — it is the most standard one. After a minute, documents are embedded — embedding progress is shown with the bar around the widget.

When embeddings are ready, we can train models. In this tutorial, we train two models — Logistic regression and Random forest. We will use default settings for both learners.

When our models are trained, we prepare the testing data to see how our models perform on new data. To load testing data, we use another Corpus widget and connect it to the Document embedder widget. Settings are the same as before. The only difference is that this time we load testing part of the dataset in the second Corpus widget. To make predictions and inspect the prediction results on the testing dataset, we use the prediction widget.

In the bottom part of the widget, we inspect accuracies. In the column with name CA (classification accuracy), we can see that both models perform with around 80 % accuracy. In the table above, we can find cases where models made mistakes. If we select rows, we can check them in the Corpus Viewer widget which is connected to the Predictions widget. We have also connected the confusion matrix widget to our workflow, which shows the proportions between the predicted and actual classes.

We can see that Logistic regression is slightly more accurate in cases of real news while Random forest model is better for predicting fake news.

In this tutorial, we explained what the document embeddings are and showed an example of how to use them on a dataset with articles, that are real or fake news. You can try a similar analysis with your own documents or you can also use embeddings for other tasks such as clustering, regression or other types of analysis.

References

[1] Grave, Edouard, et al. Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893, 2018.

[2] Demšar, Janez, et al. Orange: data mining toolbox in Python. the Journal of machine Learning research, 2013, 14.1: 2349–2353.

[3] Zhang, Yin; JIN, Rong; Zhou, Zhi-Hua. Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 2010, 1.1–4: 43–52.

[4] Wikipedia: Bag-of-words model. Available at https://en.wikipedia.org/wiki/Bag-of-words_model

[5] Joulin, Armand, et al. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016.