How to Extract Relevant Keywords with KeyBERT

Yet another application of BERT

There are many powerful techniques that perform keywords extraction (e.g. Rake, YAKE!, TF-IDF). However, they are mainly based on the statistical properties of the text and don’t necessarily take into account the semantic aspects of the full document.

KeyBERT is a minimal and easy-to-use keyword extraction technique that aims at solving this issue. It leverages the BERT language model and relies on the 🤗transformers library.

KeyBERT is developed and maintained by Maarten Grootendorst. So go check his repo (and clone it) if you’re interested in using it.

In this post, I’ll briefly present KeyBERT: how it works and how you can use it

PS: If you want to see a video tutorial on how to use KeyBERT and how to embed it in a Streamlit app, you can have a look at my video:

KeyBERT: a BERT-powered keyword extraction technique

You can install KeyBERT with pip.

pip install keybert

If you need embeddings from other sources than 🤗transformers, you can install them as well:

pip install keybert[flair]
pip install keybert[gensim]
pip install keybert[spacy]
pip install keybert[use]

Calling KeyBERT is straightforward: you initialize a keyword extraction model based on a 🤗transformers model and apply the extract_keywords method on it.

How does KeyBERT extract keywords?

KeyBERT extracts keywords by performing the following steps:

1 — The input document is embedded using a pre-trained BERT model. You can pick any BERT model your want from 🤗transformers. This turns a chunk of text into a fixed-size vector that is meant the represent the semantic aspect of the document

2 — Keywords and expressions (n-grams) are extracted from the same document using Bag Of Words techniques (such as a TfidfVectorizer or CountVectorizer). This is a classical step that you may be familiar with if you’ve performed keywords extraction in the past

3 — Each keyword is then embedded into a fixed-size vector with the same model used to embed the document

4 — Now that the keywords and the document are represented in the same space, KeyBERT computes a cosine similarity between the keyword embeddings and the document embedding. Then, the most similar keywords (with the highest cosine similarity score) are extracted.

The idea is pretty simple: you can think of it as an enhanced version of a classical keyword extraction technique in which the BERT language model comes in to add its semantic capability.

This doesn’t stop here: KeyBERT includes two methods to introduce diversity in the resulting keywords.

1 — Max Sum Similarity (MSS)

To use this method, you start by setting the top_n argument to a value, say 20. Then 2 x top_n keywords are extracted from the document. Pairwise similarities are computed between these keywords. Finally, the method extracts the most relevant keywords that are the least similar to each other.

Here’s an example from the KeyBERT’s repository:

2 — Maximal Marginal Relevance (MMR)This method is similar to the previous one: it adds a diversity argument

MMR tries to minimize redundancy and maximize the diversity of results in text summarization tasks.

It starts by selecting the keywords that are the most similar to the document. Then, it iteratively selects new candidates that are both similar to the document and not similar to the already selected keywords

You can choose a low-diversity threshold:

or a high one:

So far so good, but…

One limitation that KeyBERT may suffer from though is the execution time: if you have large documents and need real-time results, KeyBERT may not be the best solution (unless you have dedicated GPUs in your production environment). The reason being that BERT models are notoriously huge and consume a lot of resources especially when they have to process large documents.

You can probably find some hacks to speed up the inference time by picking smaller models (DistilBERT), using mixed precision or even convert your model to ONNX format.

If this still doesn’t work out for you, check other classical methods: you’d be surprised by their efficiency despite their relative simplicity.

Thanks for reading!

That’s it for today. I hope you’ll find this small method useful for your NLP projects if you’re performing keywords extraction.

You can learn more about KeyBERT here:

and here:

Take care,

New to Medium? You can subscribe for $5 per month and unlock unlimited articles — click here.