How to do basic full-text searches in MongoDB

Search your text data by the text index in MongoDB

As MongoDB is a document-oriented NoSQL database, it’s common to store plain text in some fields. To search against a string field, we can use the regular expression operator $regex directly. However, $regex can only work for simple search queries and cannot use indexes efficiently.

In MongoDB, there are better ways to search against string fields. A classical way is to create a text index and search against it. Even though MongoDB now supports a “premium” full-text solution, however, it only works if you host your data with Atlas. As it can be common to use self-managing MongoDB servers in our work, especially for some small and simple projects, it’s worthy to learn and use the classical Text Search solution which can boost your search efficiency dramatically with simple queries. As will be demonstrated later, most common search problems can be solved with the text index together with classical MongoDB search and aggregation queries.

For demonstration, we will search against a list of laptops that will be stored in a MongoDB database. Please download the JSON file (generated by the author) containing some laptop data from a fictional online shop. Note that the data was generated randomly based on some common laptop brands. It can be used freely and won’t have any license issues. Then use the following commands to import the data:

When the code above is run, we will have a laptops collection in the products database containing 200 documents of laptop data. The documents have content as follows:

Now that the data is ready, we can start to create a text index and do some basic full-text searches.

In this tutorial, we will use mongosh to run the queries directly. If you need to write some complex queries, you may find a MongoDB IDE helpful which provides command autocompletion and error highlighting. For simplicity, we will use the mongosh shipped with the Docker container so we don’t need to install anything separately:

$ docker exec -it mongo-server bash

$ mongosh "mongodb://admin:pass@localhost:27017"
test> use products
products > show collections
laptops

Create a text index

Before we get started, there is something important we should remember, namely, there can be only one text index for a collection.

Let’s create a text index on the name field, which is done with the createIndex() method of a collection:

products> db.laptops.createIndex( { name: "text" } )
name_text

name is the string field for which we want to create an index, and the “text” value indicates that we want to create a text index that supports basic full-text searches. In comparison, to create a regular index in MongoDB, we specify 1 or -1 for a field to indicate if the field should be sorted in ascending or descending order in the index.

Before we start to search against the text index, we should know that even though there can only be a single text index for a collection, that index can cover multiple fields. Let’s drop the text index created above and create a new one covering both the name and attributes fields.

Note that the text index can be named differently but there can only be one text index in a collection.

Basic full-text searches using a text index

Now let’s do some basic full-text searches with the text index just created. We will use the $text query operator to perform text searches. For example:

The $text operator uses the $search field which accepts a string value to do text searches. Under the hood, the search string is tokenized using whitespace and punctuations as delimiters. For the generated tokens, each of them is searched independently and joined with a logical OR operator. Besides, the search is by default case insensitive. If you want to do case-sensitive searches, you can specify the $caseSensitive field for the $text operator.

Therefore, with the search query above, we get documents containing either “HP” or “ProBook”, but not necessarily both.

[
  { _id: 19, name: 'HP ZBook Model 19' },
  { _id: 20, name: 'HP ZBook Model 20' },
  { _id: 3, name: 'HP EliteBook Model 3' },
  { _id: 18, name: 'HP ProBook Model 18' },
  ...
]

Sort by textScore

Importantly, with the $text operator, there is a score assigned for each document indicating how well the document matched the search string. If both “HP” and “ProBook” match a document, the document gets a score higher than the one that only matches “HP” or “ProBook”. We can sort the documents by the scores and only get the top ones with the limit() method.

As it may seem strange, the score is returned with the {$meta: "textScore"} expression. Also, something that can be even weirder at first sight:

• The field name given (score) is not important. You can give a different name and it will still work.
• The sorting by score is always in descending order. This makes sense, as normally we want to find the most relevant matches.

With this query, we can get the most relevant results we want:

[
  { _id: 15, name: 'HP ProBook Model 15' },
  { _id: 16, name: 'HP ProBook Model 16' },
  { _id: 18, name: 'HP ProBook Model 18' }
]

Search by a phrase

If we only want to find the documents which contain exactly “HP ProBook”, we can search by a phrase, which is simply to put the search string in a nested pair of quotes. We can use alternate single and double quotes or escape the quotes with backslashes. The below queries will give the same results:

Use negation in the search query

We can also use negation in our search query which requires the documents not to match some token. Let’s search for the laptops that are “HP” but are not “ProBook”:

In the result list, we cannot see “ProBook” anymore:

[
  { _id: 19, name: 'HP ZBook Model 19' },
  { _id: 20, name: 'HP ZBook Model 20' },
  { _id: 3, name: 'HP EliteBook Model 3' },
  ...
]

Text search in a nested document

Let’s now search with an attribute as well to see if the text index covers both the name and attributes fields:

Note that we need to sort by score here otherwise the top results may not be the ones you expect. This is because “HP 1TB” is not a phrase. Actually, “HP” occurs in the name fields and “1TB” in the attributes.attribute_value field. As an OR logical operator is used by default, the documents returned will contain either “HP” or “1TB”, but not necessarily both. With the sort()and limit() methods, we will return the top most relevant results which are normally what we want.

Combine $text operator with other operators

The $text operator can be used together with regular MongoDB operators. For example, let’s find the HP ProBooks whose prices are below 10000 SEK:

And this is what we get:

[
  { _id: 13, name: 'HP ProBook Model 13', price: 9994 },
  { _id: 9, name: 'HP ProBook Model 9', price: 9980 }
]

However, it should be noted that there should only be one $text operator in the search query, otherwise only the last one will be effective. This is because the query document (a dictionary in Python) cannot have duplicate keys.

Use $text operator in aggregation

The $text operator can also be used in an aggregation pipeline. However, there are three major restrictions:

• The $text operator can only be used in the $match stage.
• The $match stage containing a $text operator must be the first stage of the pipeline.
• The $text operator can only occur once in the $match stage and in the whole pipeline.

Let’s write an aggregation pipeline to count the number of laptops grouped by RAM sizes for “HP ProBook”:

[
  { _id: '16GB', count: 2 },
  { _id: '8GB', count: 4 },
  { _id: '4GB', count: 1 }
]

It shows the $text operator works just like any other regular operator in the aggregation pipeline.

In this post, we introduced the classical Text Search in MongoDB using a text index. The full-text search solution provided by the text index and corresponding $text operator is simple but also quite powerful. It should be sufficient for most small projects that only need to search by simple conditions. If you want to have more advanced searches where you need to have indexes for multiple string fields and use complex should (not)/must (not) conditions, you would want to use a more advanced search engine, such as Elasticsearch, or the “Premium” Atlas Search.