Implementing pagination with MongoDB, Express.js & Slush

MongoDB accepts and provides access to data in the Javascript Object notation (JSON) format. This makes MongoDB a perfect fit when dealing with javascript based REST services. In this post, we will take a look at Pagination using MongoDB. We will scaffold a simple Express/Mongojs application using slush-mongo. Then we will use skip() and limit() to fetch the required records from a set of data.

Pagination is one of the simplest ways to increase UX when dealing with average to huge data sets. We split the entire data into x records per page and the we will have (total records/x) pages and then we show a pagination with the number of page. As the user clicks on the page number, we seek and fetch the set of records for that particular view only.

Pagination

You can find a live demo of the app here and the complete code for this app here.

Setup the Project

Create a new folder named mongoDBPagination. Open terminal/prompt here. Next, we will install gulp, slush and slush-mongo modules. Run


$ [sudo] npm i -g gulp slush slush-mongo

Once this is done, run


$ slush mongo

You will be asked a few questions and you can answer it as follows


[?] Which MongoDB project would you like to generate? Mongojs/Express
[?] What is the name of your app? mongoDBPagination
[?] Database Name: myDb
[?] Database Host: localhost
[?] Database User:
[?] Database Password:
[?] Database Port: 27017
[?] Will you be using heroku? (Y/n) n

This will scaffold a simple Express/Mongojs app for us. Once the installation is done, run


$ gulp

Then open http://localhost:3000 in your favorite browser and you should see a table with list of routes configured in the application. This confirms that you have installed everything correctly.

Setup Test DB

We will create a new collection named ‘testData‘ and then populate some test data in it. Then we will show this data in a paginated table. Open a new Terminal/prompt and run


$ mongo

Then run


use myDb

to select our DB. Next copy the snippet below and paste it in the mongo shell and hit return.

for(var i = 1; i <= 999; i++) {
 db.testData.insert({

 name: Math.random()
           .toString(36)
           .substring(7),

 age: Math.floor(Math.random() * 99),

 random: Math.random()
             .toString(36)
             .substring(7)
 });
}

This will generate 999 sample records with some random data. A sample record will look like


{
 "_id":"5392a63c90ad2574612b953b",
 "name": "j3oasl40a4i",
 "age": 73,
 "random": "vm2pk1sv2t9"
}

This data will be paginated in our application.

Configure DB

Since we added a new collection, we need to update our Mongojs DB config to read from ‘testData‘. Open mongoDBPagination/config/db.js and update line 17 from


var db = mongojs(uristring, ['posts']);

to


var db = mongojs(uristring, ['posts', 'testData']);

Build Pagination End Point

Now, we will build our server code. We will create a REST endpoint, where the client will let us know what data it wants.

Pagination Logic

The logic for pagination is pretty simple. Our database collection consists of records and we want to retrieve and show only a few at a given instance. This is more of a UX thing to keep the page loading time to minimum. The key parameters for any Pagination code would be

  1. Total Records – The total number of records in the DB
  2. Size – Size of each set of records that the client wants to show
  3. Page – The page for which the data needs to be fetched

Lets say that the client wants 10 records from the first page, it will request


{
   page : 1,
   size : 10
}

And the server will interpret this as, the client needs 10 records that starts from index 0 (page : 1).

To get the data from third page, the client would request


{
   page : 3,
   size : 10
}

And the server will interpret as, the client needs 10 records that start from index 20 (page – 1 * size).

So, looking at the above pattern, we can conclude that if page value is 1, we will start fetching the data from record zero and if the page value is greater than one, we will  start fetching data from page times size (page*size).

MongoDB support

Now, we have an understanding of the logic. But, how do we convey the same to MongoDB?

MongoDB provides us 2 methods to achieve this

  1. skip – Once the query is completed, MongoDB will move the cursor to the value of skip
  2. limit – Once MongoDB starts populating records, it will collect only the limit number of records

Simple right? We will use these 2 methods along with find() to fetch the records.

Continue Development

Now, let us create a new file named paginator.js inside mongoDBPagination/routes folder. We will configure our Pagination end point here. Open paginator.js and add the below code

module.exports = function (app) {
   var db = require('../config/db')
   api = {};

   api.testData = function (req, res) {
      var page = parseInt(req.query.page),
         size = parseInt(req.query.size),
         skip = page > 0 ? ((page - 1) * size) : 0;

      db.testData.find(null, null, {
         skip: skip,
         limit: size
      }, function (err, data) {
         if(err) {
            res.json(500, err);
         }
         else {
            res.json({
               data: data
            });
         }
      });
   };
   app.get('/api/testData', api.testData);
};

On line 6,7 we get the page number and page size from the request params. On line 8, we configure the skip value. On line 10, we query the DB using the find method, passing null as the first 2 arguments to satisfy the method signature of find(). In the third argument of the find method, we will pass the filter criterion. And then once the results come back, we respond with a JSON.

To test this, make sure your server is running and navigate to

http://localhost:3000/api/testdata?page=1&size=2

And you should see the first 2 records in the collection. You can alter the values of page and size to see different results.

Build the client

Now, we will build the client that will implement the pagination. We will use Bootstrap tables to show the data and bootpag plugin to handle the pager.

First, we will install Bootstrap. Run,


$ bower install bootstrap

Next, we will download jquery.bootpag.min.js from here to public/js folder. Update views/index.html as

<!DOCTYPE html>
<html>
<head>
	<title><%= siteName %></title>
	<link rel="stylesheet" href="/css/style.css">
  <link rel="stylesheet" href="/bower_components/bootstrap/dist/css/bootstrap.min.css">
</head>
<body>
    <div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
      <div class="container">
        <div class="navbar-header">
          <a class="navbar-brand" href="javascript:"><%= siteName %></a>
        </div>
      </div>
    </div>
    <div class="container">
    <h1>My Data</h1>
        <table class="table">
          <thead>
            <tr>
              <th>_id</th>
              <th>Name</th>
              <th>Age</th>
              <th>Random</th>
            </tr>
          </thead>
          <tbody>
            <!-- we will populate this dyanmically -->
          </tbody>
        </table>
        <div id="pager" class="text-center"></div>
        <input type="hidden" value="<%= totalRecords %>" id="totalRecords">
        <input type="hidden" value="<%= size %>" id="size">
    </div>
  <script type="text/javascript" src="bower_components/jquery/dist/jquery.min.js"></script>
  <script type="text/javascript" src="/js/jquery.bootpag.min.js"></script>
	<script type="text/javascript" src="/js/script.js"></script>
</body>
</html>

And finally, we will write the logic to populate the table. Open js/script.js and fill it as

// init bootpag
$('#pager').bootpag({
    total: Math.ceil($("#totalRecords").val()/$("#size").val()),
    page : 1,
    maxVisible : 10,
    href: "#page-{{number}}",
}).on("page", function(event, /* page number here */ num) {
    populateTable(num);

});

var template = "<tr><td>_id</td><td>name</td><td>age</td><td>random</td>";

var populateTable = function (page) {
	var html = '';
	$.getJSON('/api/testdata?page='+page+'&size='+ $("#size").val(), function(data){
		data = data.data;
		for (var i = 0; i < data.length; i++) {
			var d = data[i];
			html += template.replace('_id', d._id)
							.replace('name', d.name)
							.replace('age', d.age)
							.replace('random', d.random);
		};
		$('table tbody').html(html);
	});
};

// load first page data
populateTable(1);

Now, navigate to


http://localhost:3000

and you should see the table and the pager component. You can click through the page numbers to browse through the data.

Simple and easy! Hope you got an idea on how to implement pagination using MongoDB.

You can find the code for this app here.

For more details on the performance on the paging operation refer to our other blog post - https://scalegrid.io/blog/fast-paging-with-mongodb/


Thanks for reading. Do comment.
@arvindr21


Arvind Ravulavaru is a passionate web developer, likes experimenting with new and upcoming technologies. He also blogs at http://thejackalofjavascript.com


  • Eric Hill

    Great sample! However, when I get to the end and hit http://localhost:3000, I get a reference error from index.html that totalRecords is not defined (and if I take that line out, then size is not defined). If I replace the lines in index.html that reference the javascript variables totalRecords and size with hardcoded values:

    I can get it to work. I’ll keep debugging and see if I can figure out how to get totalRecords and size to be defined, but I will give up at some point and get back to my own code.

    Thanks!

    Eric

1 Shares
+1
Tweet
Share1
Share
Pin