TF-IDF Vectorizer Explained

The Term Frequency-Inverse Document Frequency (TF-IDF) vectorizer is a widely used technique in text processing that reflect the importance of words in a document relative to a collection of documents or corpus. It is useful in tasks like document classification, information retrieval, and topic modeling.

The TF-IDF method weighs a term’s frequency (TF) and its inverse document frequency (IDF), producing a numerical statistic that reflect how important a word is to a document in a collection.

Term Frequency (TF)

Term Frequency measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in longer documents than shorter ones.

Thus, the term frequency is often divided by the document length (the total number of terms in the document) as a way of normalization:

Inverse Document Frequency (IDF)

Inverse Document Frequency measures how important a term is. While computing TF, all terms are considered equally important.

However, certain terms, like “is”, “of”, and “that”, may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scaling up the rare ones, by computing the following:

where:

• N is the total number of documents in the corpus D.
• |{𝑑∈𝐷:𝑡∈𝑑}| is the number of documents where the term 𝑡 appears.
• If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the denominator to 1+∣{𝑑∈𝐷:𝑡∈𝑑}∣.

TF-IDF Calculation

TF-IDF score is calculated by multiplying these two statistics:

TF-IDF(𝑡,𝑑,𝐷) = 𝑇𝐹(𝑡,𝑑)×𝐼𝐷𝐹(𝑡,𝐷)

Vectorization

When transforming text to vectors using TF-IDF, each document is represented as a vector. Each dimension of the vector corresponds to a separate term from the corpus vocabulary.

The value in each dimension is the TF-IDF score of the term in the document. To handle the vast size of the vocabulary and the sparsity of individual document vectors, implementations usually use sparse matrix representations.

Let’s Understand it using an example —

Assume there are three short text documents:

1. “Dog barks loudly.”
2. “Cat meows softly.”
3. “The cat and the dog play together.”

First I will manually calculate the TF-IDF values for some terms across these documents and then explain how this might look in a Python implementation using scikit-learn.

Step 1: Calculate Term Frequency (TF)

Calculating the TF for a few selected words: “dog”, “cat”, and “the”.

• Document 1: “Dog barks loudly.”
• dog: 1/3
• cat: 0
• the: 0
• Document 2: “Cat meows softly.”
• dog: 0
• cat: 1/3
• the: 0
• Document 3: “The cat and the dog play together.”
• dog: 1/6
• cat: 1/6
• the: 2/6 = 1/3

Step 2: Calculate Inverse Document Frequency (IDF)

Number of documents 𝑁=3.

Step 3: Calculate TF-IDF

Multiply TF by IDF for each term in each document.

Final Output —

Explanation of the DataFrame Values:

• Columns: Each column represents a unique word found in the entire corpus of documents.
• Rows: Each row corresponds to a document.
• Values (TF-IDF scores):
• These values are normalized TF-IDF scores. The exact values depend on the specific settings of the TfidfVectorizer(e.g., whether sublinear TF scaling and smooth IDF are applied).
• Higher values indicate a term is more relevant for a document, considering both the term frequency in the document and its inverse document frequency across all documents.

Note:

• “barks” and “loudly” have higher scores in Document 1, emphasizing that these words are unique to this document.
• Similarly, “meows” and “softly” are distinctive in Document 2.
• In Document 3, “the” has a relatively higher score than it might otherwise because, in scikit-learn, the default behavior uses a smoothed IDF, making less frequent words across documents more prominent in their respective documents.
• Words like “and”, “play”, and “together” are specifically relevant to Document 3 but generally less informative across the corpus.

This example highlights how TF-IDF vectors provide a numerical representation of text data, suitable for feeding into various types of machine learning models, particularly in natural language processing tasks.

Step 4: Python Implementation with scikit-learn

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "Dog barks loudly",
    "Cat meows softly",
    "The cat and the dog play together"
]

# Create a TfidfVectorizer object
vectorizer = TfidfVectorizer()

# Fit and transform the documents
tfidf_matrix = vectorizer.fit_transform(documents)

# Get feature names to use as dataframe columns
feature_names = vectorizer.get_feature_names_out()

# Display the matrix
import pandas as pd
df = pd.DataFrame(tfidf_matrix.toarray(), columns=feature_names)
print(df)

Application in Machine Learning

In machine learning, TF-IDF vectors can be used as features for algorithms that require numerical input, such as clustering and classification algorithms and also for calculating cosine similarity between documents.

For example, in document classification, each document is transformed into a TF-IDF vector; these vectors then serve as input features for a model like a Support Vector Machine or Naive Bayes classifier to classify documents into categories.

How the output of normalized TF-IDF vector is different from original output (Extra) —

The explanation till this part is enough to understand the concept briefly. Here I’ll go deeper into the mathematics and show how normalization might have an effect on the output of the vector.

Incase you were calculating the tf-idf value of different words in a document, you might have noticed that the value I have written for “barks” in the final output is not correct according to our calculations.

Let’s calculate the tf-idf value of barks —

Term frequency of “barks” —

• Document 1: “Dog barks loudly”
• “barks” appears 1 time
• Total number of words = 3
• 𝑇𝐹 = 1/3

Inverse document frequency IDF “barks”:

• Total number of documents, 𝑁=3
• Number of documents containing “barks”, ∣{𝑑∈𝐷:”𝑏𝑎𝑟𝑘𝑠”∈𝑑}∣ = 1
• 𝐼𝐷𝐹(“𝑏𝑎𝑟𝑘𝑠”) = log⁡(3/1)

= log⁡(3)≈0.477

TF-IDF for “barks” in Document 1

TF-IDF(“𝑏𝑎𝑟𝑘𝑠”, Doc 1) = 𝑇𝐹(“𝑏𝑎𝑟𝑘𝑠”, Doc1) × 𝐼𝐷𝐹(“𝑏𝑎𝑟𝑘𝑠”)

= 1/3×0.477≈0.159

But I have written it as 0.564 previously. Why is that?

The number 0.564 for “barks” in the example I described was intended to represent a possible output from a TF-IDF vectorizer like scikit-learn’s.

The exact value generated by tools like scikit-learn can differ from manual calculations due to a few reasons, such as the use of different normalization methods and smoothing in the IDF calculation.

Let’s clarify and perform a more precise manual calculation:

Recalculation using Logarithmic Scale and Smoothing —

If we consider the log scale and use smoothing (as done typically by libraries such as scikit-learn), the calculations adjust slightly:

1. Term Frequency (TF) of “barks” in Document 1:

• TF(“barks”) = 1 occurrence / 3 total words = 1/3

2. Inverse Document Frequency (IDF) considering smoothing:

𝐼𝐷𝐹(𝑡) = log⁡((𝑁+1)/(𝑛ₜ+1))+1 where 𝑛ₜ is the number of documents containing the term 𝑡, and 𝑁 is the total number of documents.

• For “barks”, appearing in 1 document, 𝑁=3:
• 𝐼𝐷𝐹(“𝑏𝑎𝑟𝑘𝑠”) = log⁡((3+1)/(1+1))+1 = log⁡(2)+1 ≈ 0.693+1=1.693

3. TF-IDF for “barks” in Document 1:

• TF-IDF(“barks”) = (1/3) * 1.693 ≈ 0.564

Given this corrected approach, the TF-IDF value is around 0.564, and not 0.159 as in the simpler example. This difference arises due to additional smoothing and normalization factors that scikit-learn and other libraries might apply.

Scikit-learn, for example, often uses the Euclidean norm (L2 norm) to normalize the TF-IDF vectors, which could further change the absolute values while retaining the relative importance of terms within and across documents.

This highlights how different settings and methodologies in vectorization tools can impact the numerical results, and why it’s crucial to understand the specific configurations of these tools when interpreting or comparing TF-IDF scores.

Incase we also have the Normalization term -

Calculating the L2 Norm for Each Document:

• For Document 1:

• Repeat this calculation for each document.

1. Divide Each TF-IDF Score by its Document’s L2 Norm:

• For Document 1: Normalize each term by dividing by 0.873.

Here’s how the final normalized TF-IDF scores might look after applying L2 normalization:

This DataFrame shows the L2 normalized TF-IDF vectors for each document. Each row sums up to 1 in squared terms, making the vectors unit vectors (each vector has a length of 1).

This normalization helps in reducing bias towards longer documents and focuses the similarity computation solely on the directions of the vectors rather than their magnitudes.

Incase something is not clear, please feel free to write in the comments.