Firestore Pagination

Cloud Firestore, aka Firestore, is an NoSQL database released by Google on October 2017, currently it is on beta stage. It’s the successor of old Firebase database (Realtime Database), and in my opinion, the biggest change would be querying. Firestore allows you to do query based on multiple fields, which is very useful to handle data.

When I was carrying out my project Iridescent, I realized that I needed to paginate data. As the project grows up, I can’t load 100+ data all by once. Firestore has a daily limit for document reads and bandwidth, so I have to try to reduce the usage.

At the same time, a long waiting time is not part of good user experience, and loading all data by once might cause lag on low-performance devices.

In order to build a traditional pagination (with page numbers and previous and next buttons), we’ll need to know how many data are there. Sadly, Firestore doesn’t offer a method similar to MySQL’s COUNTThe only viable way I figured out was downloading all data once, get array’s length and build pagination.

But that’s exactly what I didn’t want to achieve, so I changed my plan.

All codes in this post are extracted from my project Iridescent, which is a platform used to submit questions under certain topics.

Questions are storied under Firestore collection questions, and topic data under topics.

It’s built using Vue.js, so I can’t guarantee that those code work properly on another framework.

Plan A: Infinite Scrolling

We don’t know data size, but if we don’t need to?

If we build a infinite scrolling, we only need to know last loaded data’s Firestore reference, and apply it to Firestore’s .startAfter() method.

Combine it with .limit(), we can request only the next x data.

The data that I want to fetch are storied under questions collection.

import * as firebase from "firebase/app";
import "firebase/firestore";

export default {
    name: 'QuestionList',
    data: () => ({
        questions: [],
        
        paging: {
            question_per_page: 20, 
            end: false,
            loading: false
        },
        ref: {
            questions: null,
            questionsNext: null
        }
    }),
    created () {
        /* Set common Firestore reference */
        this.ref.questions = firebase.firestore().collection('questions').where('topic', '==', this.ref.topic.id).orderBy("date", 'desc');
        
        /* Load first page */
        const firstPage = this.ref.questions.limit(this.paging.question_per_page);
        this.handleQuestions(firstPage);
    },
    methods: {
        loadMore () {
            if (this.paging.end) {
                return;
            };
            
            this.paging.loading = true;
            this.handleQuestions(this.ref.questionsNext).then((documentSnapshots) => {
                this.paging.loading = false;
                
                if (documentSnapshots.empty) {
                    /* If there is no more questions to load, set paging.end to true */
                    this.paging.end = true;
                }
            })
        },
        handleQuestions (ref) {
            /*
                Fetch questions of given reference
            */
            return new Promise((resolve, reject) => {
                ref.get().then((documentSnapshots) => {
                    /* If documentSnapshots is empty, then we have loaded all of pages */
                    if (documentSnapshots.empty) {
                        this.paging.end = true;
                        resolve(documentSnapshots);
                    };
                    
                    documentSnapshots.forEach((doc) => {
                        let questionData = doc.data();
                        questionData.id = doc.id;
                        this.questions.push(questionData);
                    });
                    
                    /* Build reference for next page */
                    const lastVisible = documentSnapshots.docs[documentSnapshots.size - 1];  
                    
                    if (!lastVisible) {
                        return;
                    };
                   
                    this.ref.questionsNext = this.ref.questions
                        .startAfter(lastVisible)
                        .limit(this.paging.question_per_page);
                    
                    resolve(documentSnapshots);
                });
            });
        }
    }
}

The idea is to load first page after the component is created, and build Firestore reference for next page (this.ref.questionsNext).

There should be a load more button. When user clicks on it, call function loadMore()to request next page data.

Eventually, there will be no more data to load, and document snapshot returned after request should have property empty: true, and that’s when we set this.paging.endto true, and disable load more button.

Plan B: Real Pagination

The infinite scrolling solution fits my need: reduce resource & data usage, and speed up first time loading.

But there are a few inconveniences:

  • Can’t jump pages. For example, load directly last page’s content without going through intermediate pages
  • Can’t know how many pages are left to be loaded

A few days ago I tried to solve those issues, and realized that they can be fixed easily using Firebase Cloud Functions.

Cloud Functions lets application developers spin up code on demand in response to events originating from anywhere.

I started to dig into Cloud Function’s document, and found out that it can listen to events of Firestore (and Realtime database too).

Cloud Functions

Cloud Functions supports four different events from Firestore:

  • onCreate: Triggered when a document is written to for the first time.
  • onUpdate: Triggered when a document already exists and has any value changed.
  • onDelete: Triggered when a document with data is deleted.
  • onWrite: Triggered when onCreateonUpdate or onDelete is triggered.

As you can see, onWrite event includes the previous three. My laziness made me wrote all codes inside one function:

'use strict';

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();

exports.questionChange = functions.firestore.document('questions/{questionID}').onWrite(
	(change) => {
	
		const oldQuestion = change.before.data(),
			newQuestion = change.after.data();

		let oldTopic,
			oldTopicRef,
			newTopic,
			newTopicRef;

		/* Load variables */
		if (change.before.exists) {
			oldTopic = oldQuestion.topic;
			oldTopicRef = db.collection('topics').doc(oldTopic);
		};
		if (change.after.exists) {
			newTopic = newQuestion.topic;
			newTopicRef = db.collection('topics').doc(newTopic);
		};

		if (!change.before.exists) {
			/* New document Created : plus one to count.total */

			return newTopicRef.get().then(snap => {
				return newTopicRef.set({
					count: {
						total: snap.data().count.total + 1
					}
				}, {
					merge: true
				});
			});
		};
		
		if (!change.after.exists) {
			/* Question deleted : subtract one from count.total */
			
			return oldTopicRef.get().then(snap => {
				return oldTopicRef.set({
					count: {
						total: snap.data().count.total - 1
					}
				}, {
					merge: true
				});
			});
		};
	});

I created a new field count for topic’s document (/topics/${topiciD}), to store amount of questions under that topic.

Function will be triggered each time a document storied under path questions/{questionID} is added / modified / deleted. And it should update the counter.

Also, I added a second parameter to .set() in order to not overwrite existing datausing merge: true.

When the function is being executed for the first time, it could take a while.


New questions should be counted correctly. But, what about existing data? Do I have to modify counter manually?

Of course no. I wrote a HTTP function which does the recount. In my case, it requires topicID GET parameter to be able to filter data out.

/* HTTP Function to initialize / reset question counter manually */
const express = require('express');
const cors = require('cors')({
	origin: true
});
const app = express();
app.use(cors);

app.get('/', (req, res) => {
	const topicID = req.query.topicID,
		topicRef = db.collection('topics').doc(topicID);
		
	if (!topicID) {
		res.send("Topic ID is required");
		return;
	};

	const allQuestions = db.collection('questions').where('topic', '==', topicID);
	
	allQuestions.get().then(snap => {
		const totalCount = snap.size
		
        /*
    		Set counter
    	*/
		topicRef.set({
			'count': {
				total: totalCount
			}
		}, {
			merge: true
		}).then(() => {
            /*
            	Send response
            */
			res.json({
				total: totalCount
			});
		});
	})
});

exports.reCount = functions.https.onRequest(app);

Once deployed, firebase-cli should return function’s URL. Make a request and try it.

I used express and cors to allow making cross domain requests (I added a button on front-end to calibrate counter).

I do not recommend you running this function frequently, as it will request all data under a topic (in my case), and waste a lot of quota if there are many documents. So run it only when needed.

Client side

Here’s a example of how implement pagination knowing data size (topicData.count.total)

import * as firebase from "firebase/app";
import "firebase/firestore";

export default {
    name: 'QuestionList',
    props: {
	    topicData: Object
    },
    data: () => ({
        questions: [],  
        
        paging: {
            question_per_page: 20,   /* Number of questions per page */
            loading: false, 
            loaded: []   /* Loaded pages, to avoid querying data again */
        },
        
        ref: {
            questions: null,   /* Old questions Firestore Reference */
        }
    }),
    created () {
        this.init();
    },
    methods: {
        init () {
            const count = this.topicData.count;
            
            /* Fill array with placeholders, to build pagination */
            this.questions = new Array(count.total).fill({
                loading: true
            });
            
            /* Set common Firestore reference */
          	this.ref.questions = firebase.firestore().collection('questions')
            	.where('topic', '==', this.topicData.id)
                .orderBy("date", 'desc');
                
            /*
                Load questions of first page
            */
            this.onPageChange(0);
        },
        handleQuestions (ref, index = 0) {
            /*
                Fetch questions of given reference
            */
            
            return new Promise((resolve, reject) => {
                console.log('Questions reference: ', ref);
                ref.get().then((documentSnapshots) => {
                    let _questions = [],
                        _index = index || 0;
                    
                    console.log('Questions document snapshots: ', documentSnapshots);
                    
                    documentSnapshots.forEach((doc) => {
                        let questionData = doc.data();
                        questionData.id = doc.id;
                        _questions.push(questionData);
                        this.$set(this.questions, _index, {
                            loading: false,
                            ...questionData
                        });
                        _index++;
                    });
                    resolve(documentSnapshots);
                });
            });
        },
        async onPageChange (toPage, fromPage) {
            const currentPage = toPage,
                per_page = this.paging.question_per_page;
            
            let startAfter = null,
                limit = per_page,
                index = per_page * (currentPage - 1),   /* Start after the question before that page */
                startAfterAvailable;
            
            if (this.paging.loaded.includes(currentPage)) {
                return;
            };
            
            /* Display progress spinner */
            this.paging.loading = true;
            const questionBefore = this.questions[index - 1];
            if (questionBefore && !questionBefore.loading) {
                /* 
                    If the question before that page is loaded, 
                    we can build it's documentSnapshot to query the following page only
                */
                const questionBeforeRef = firebase.firestore().collection('questions').doc(questionBefore.id),
                    questionBeforeSnapshot = await questionBeforeRef.get();
                startAfter = this.ref.questions.startAfter(questionBeforeSnapshot).limit(per_page);
                if (index < 0) {
                    index = 0;
                };
                startAfterAvailable = true;
            }
            else {
                /* 
                    But if it's not loaded, we'll have to request all questions between first page and current page
                */
                limit = currentPage * per_page;   /* Load all question from first page to current page */
                if (limit <= 0) {
                    /* 
                        Limit can not be less or equal to zero
                        This happend when currentPage == 0 
                    */
                    limit = per_page;
                };
                startAfter = this.ref.questions.limit(limit);
                index = 0;   /* Start loop from first page */
                startAfterAvailable = false;
            };
            
            this.handleQuestions(startAfter, index).then((documentSnapshots) => {
                this.paging.loading = false;
                this.loading.questions = false;
                if (startAfterAvailable) {
                    /*  startAfterAvailable = true
                            => Only requested current page 
                        Add current page to paging.loaded to avoid requesting the data again
                    */
                    this.paging.loaded.push(currentPage);
                }
                else {
                    /* 
                        Request from 0 to current page 
                        Add those pages to paging.loaded 
                    */
                    this.paging.loaded = Array.from(Array(currentPage).keys());
                };
            })
        }
    }
}

Main idea:

  • Initialize questions array with a length of topicData.count.total, in order to build pagination link correctly
  • Load first page
  • Listen to page change event. I used Vue-Paginate for my project, and it has @change listener.
    • If the question before that page is loaded, we can build it’s documentSnapshot to query the following page only. (Using .startAfter(questionBeforeSnapshot).limit(page_size))
    • Otherwise, we’ll have to request all questions between first page and current page.

Conclusion

Currently, Cloud Firestore and Cloud Functions are both in beta stage, so maybe in the future this post’s code will become unusable, please remind me in comment section.

Seems like Cloud Functions takes a while to process if there’s multiple onWriteevent being triggered. I haven’t tested it on high QPS situation yet.

I hope Firestore will release a better solution for collection counter.

References

Photo by Olia Gozha on Unsplash

4 thoughts on “Firestore Pagination”

  1. Thank you for this blog
    I have a question on the real pagination, what if we jump to page 100,
    are we going to download all 99 just to get to 100?

    1. Hi Ray,

      I’m afraid it has to be like that. I know it’s kind of expensive, but I haven’t found better way to do it yet. 😥

      I’ve been thinking about saving all doc IDs of a collection inside an array, to use `.startAfter (docID)` method, so it can avoid downloading all those datas.

      But I haven’t got time to experiment it. 😅

Leave a Reply

Your email address will not be published. Required fields are marked *