Save money on Firestore read

How to create a news feed using Firestore ?

Creating a news feed using Firestore can be a challenging task, especially when it comes to filtering posts based on a user’s friends.

In this article, I will share a solution that I have used in my personal app, TheBoard, to overcome this challenge.

I’ll discuss the limitations of using Firestore for creating a news feed and offer a potential solution that can make the process smoother, more efficient and cost-effective. If you’ve ever struggled with creating a news feed using Firestore, this article is for you.

Introduction

🔥 Firestore is a powerful and versatile tool that can greatly aid in the development and scaling of your app. However, as with any tool, it has its limitations. In this article, I will be focusing on a specific limitation I encountered when working with Firestore, and how I overcame it.

📊 Firestore is a NoSQL database that is simple to integrate into your app. The setup process is straightforward, and the documentation is comprehensive. When I first began experimenting with Firestore, using a small test database, everything seemed to be going well. However, as I delved deeper into the capabilities of the tool, I encountered an issue that nearly caused me to give up on using Firebase altogether…

The issue

Imagine you have a collection in your Firestore database called “posts” where each document is structured like this:

documentID: {
  userId: str,
  publishedDate: timestamp,
  content: {
    ...
  }
}

If you want to retrieve the most recent documents from this collection, you can use the following code:

firestore()
  .collection('posts')
  .orderBy('publishedDate')
  .limit(10)
  .get()

This will work, but it will return all the documents regardless of their publisher. In a social network app, for example, you may only want to show posts that were published by the user’s friends. To accomplish this, you can use the following code:

const friendList = [...]

firestore()
  .collection('posts')
  .where('userId', 'in', friendList)
  .orderBy('publishedDate')
  .limit(10)
  .get()

However, there’s a problem with this approach. When filtering your query, Firestore only accepts arrays with a maximum length of 10. This means that if a user has more than 10 friends, the query will not work as expected…

How I DIDN’T solve it

Initially, one approach I considered to overcome the issue of Firestore’s array length limitation for filtering queries was to slice the list of friends into smaller arrays of length 10 and then query the top 10 posts on all slices. For example, if a user has 100 friends, this method would result in 100 documents being retrieved (10 * 10 = 100).

However, this approach presents a few problems. If one of the slices contains friends who have not been active on the platform for a long time, their 10 “most recent” posts will be old and will be mixed with the more recent posts. When scrolling down, triggering the same function again, the more recent posts (from other friends in another slice) will be loaded after those old posts, which is not ideal for the user.

Another issue with this approach is that if you want to display the posts in the correct order, you can only trust the top 10 documents from the 100 you just fetch (most recent ones) to be displayed as the most recent ones. This means that with 100 friends, YOU WILL HAVE TO THROW AWAY 90% OF ALL THE DOCUMENTS YOU FETCH, WHICH MEANS YOU PAY 90% OF FIRESTORE READS FOR NOTHING (200 friends -> 95%) 💸.

In summary, this approach didn’t solve the issue and it was not efficient or cost-effective, so I had to look for another solution.

How I finally solved it

After encountering the limitations of the initial approach, I came up with a new solution that ultimately solved the issue. My idea was simple: instead of querying a limited number of documents, I would query a limited time window.

What you have to do is set a time window (e.g. 1 day) and query all posts of you friends on this time window (and now you can slice your friends array as much as you without having to worry about old posts anymore).

That way, every document read will be displayed to your user (nothing will be trashed 🗑️) and you will eventually SAVE MONEY 💸. The only thing you have to decide, is the time window.

The solution involves setting a time window (e.g. 1 day) and querying all posts of the user’s friends within that time frame. This way, you can slice your friends array as much as you want without having to worry about old posts appearing in the feed. Additionally, every document read will be displayed to the user, nothing will be discarded and you will eventually save money on Firestore reads. The only thing you have to decide is the time window.

The code I use is the following:

const loadPosts = async (friends, startDate, endDate) => {
    // friends = [user1, user2, user3, user4, ...]
    const friendSliced = sliceFriends(friends);
    // friends = [[user1, user2, ...], [user11, user12, ...], ...]

    let posts = []
    for (let index = 0; index < friendSliced.length; index++) {
        const subFriends = friendSliced[index];
        const snapshot = firestore()
            .collection('posts')
            .orderBy('date', 'desc')
            .where('date', '<', startDate)
            .where('date', '>=', endDate)
            .where('userId', 'in', subFriends)

        const data = await snapshot.get()

        if (!data.empty) {
            for (const doc of data.docs) {
                const docData = doc.data()
                posts = [...posts, docData]
            }
        }
    }
    
    // sort all posts you just read
    posts = posts.sort((a, b) => (a.date.toDate() < b.date.toDate()) ? 1 : ((b.date.toDate() < a.date.toDate()) ? -1 : 0))
  
    return posts
}

This function, loadPosts, takes three inputs: a list of the user's friends, a starting date, and a finishing date. It's important to note that this function alone is not sufficient to create a fully functioning news feed. To ensure that the most recent posts are displayed, you'll also need to implement a logic to update the startDate and endDate variables each time the user reaches the bottom of their scrollview.

For example, if your time window is set to 1 day, you’ll need to decrease both startDate and endDate by 1 day each time the user reaches the bottom of the page, in order to load the following posts. This way, each time the function is triggered, it will retrieve the posts from the next day, and so on. This ensures that the user is always seeing the most recent posts from their friends.

This solution solved the issue in an efficient and cost-effective way and I was able to show my user the most recent posts from their friends.

Conclusion

In conclusion, Firestore is a powerful and versatile tool that can greatly aid in the development and scaling of your app. However, as with any tool, it has its limitations. In this article, we discussed the limitation of Firestore when it comes to filtering posts based on a user’s friends, and offered a potential solution that can make the process smoother and more efficient. By querying a limited time window instead of a limited number of documents, we were able to overcome the limitation of Firestore’s array length and display the most recent posts to the user without discarding any data and save on Firestore reads.

With a little creativity, I am confident you can overcome these limitations and develop an efficient and cost-effective app.

If you have better ideas, I would be more than happy to discuss them in the comments ! Also I will try to post more regularly in the future.