How to Build a URL Shortener with Node.js and MongoDB

In this post, we’ll show you how to build a URL shortening service like bit.ly or goo.gl using Express.js (Node.js) and MongoDB. Here’s a demo of the final product we’ll be building through our MongoDB hosting platform.

How Does a URL Shortener Work?

At a very high level, the URL shortener works by taking an entered URL and creating a relatively shortened version simplified into an easy to share format. The shortened hash will be generated by base-encoding an auto-incremented counter, and creates a minimum three-character hash that increases as the number of stored URLs go up.

When the shortened version of the URL is visited, the service will decode the hash to fetch the original URL stored in MongoDB and then redirect your user to it.

Getting Started

Here’s a list of the technologies we’ll use to build the URL shortener in this tutorial:

• Express.js (Node.js backend)

  A web application framework for Node.js. We’ll use it to build an API for shortening URLs and redirect users to the original URL.

• MongoDB (storing URLs)

  A NoSQL database perfect for this application. It provides a flexible schema design and is easy to get started with. In this tutorial, we’ll be using a Shared MongoDB Cluster on ScaleGrid. It takes less than 5-minutes to set up, and you can create a free 30-day trial here to get started.

• HTML, CSS, JavaScript (front-end)

  HTML, CSS and JavaScript will be used to build the front-end of the application that your users will use to shorten URLs.

URL Shortener Tutorial

1. Setup the MongoDB Database Structure

   Let’s start by creating a Shared MongoDB Cluster on ScaleGrid. This is the easiest way to create a quick cluster, but you can also install MongoDB on your machine and get started there.

   

   Once the cluster is created, you’ll be provided with a connection string that can be copied with a single click from your Cluster Details page. We’ll need this string to connect to the cluster from our application. Remember, never share your connection string with anyone.

   We’ll need two collections for the URL shortener:

   • Collection 1

     A collection to store the URL and dynamically generated ID:

     

   • Collection 2

     A collection to maintain the counter that will be auto-incremented when a new URL is stored in the previous collection. A new document is created in the previous collection with this newly incremented counter:

     

   It’s important to note that we’re not storing the hashes anywhere in the database. The hash will be base-encoded and decoded dynamically with the general algorithm that will result in the unique ID stored in the first collection. This ID will then fetch us the original URL that the user will redirect to.

   For this tutorial, we’ll be using the common base64 encoding and decoding mechanism for generating our unique shortened hash. For more information on encoding/decoding strings using base64, check our the following MDN Web Doc.

2. Setup the Express.js Backend

   Here’s a list of dependencies required to set up our Node.js backend:

   • express (base app)
   • body-parser (add-on for parsing data sent over HTTP requests)
   • btoa (base64 encoding)
   • atob (base64 decoding)
   • dotenv (storing connection string in a .env file for development purposes)
   • mongoose (adapter for MongoDB on Node.js)

   Here is a sample version of the package.json that you can use to set up the app:

   {
     "name": "sg-url-shortener",
     "version": "1.0.0",
     "description": "A simple URL shortener built with Node.js and MongoDB",
     "dependencies": {
       "atob": "^2.0.3",
       "body-parser": "^1.15.2",
       "btoa": "^1.1.2",
       "dotenv": "^4.0.0",
       "express": "^4.10.2",
       "mongoose": "^4.13.7"
     },
     "main": "index.js",
     "scripts": {
       "start": "node index.js"
     },
     "engines": {
       "node": "4.8.4"
     }
   }
   

   Run “npm install” to install all the required dependencies.

   Once all of our dependencies have been set up, we need to connect to our Shared MongoDB Cluster. Create an .env file in the root of the project and add the connection string to it. You can get the connection string from your Cluster Details page under the Overview tab on the ScaleGrid Console.

   connectionString=mongodb://user:password@devservers

   Before we start writing code, it’s a good practice to visualize the app flow so we have a good understanding of how the shortening process will work. Here’s a diagram showing the process of URL shortening:

   

   Here’s a diagram showing the process of redirection when a shortened URL is visited:

   

   Now that we’ve visualized the entire process, it’s time to translate the above flowcharts into code.

3. Initializing the Application

   Before we start writing business logic, we need to initialize our application with our node modules and set up a server.

   Load .env files only in dev mode. Since the demo application is hosted on Heroku, an environment variable has been created from the Heroku dashboard that already contains the connection string there:

   if(process.env.NODE_ENV !== 'production') {
       require('dotenv').load();
   }

   Application initialization, server and middleware setup. Note that we are also getting the connection string from the environment variable:

   var express = require('express'),
       bodyParser = require('body-parser'),
       app = express(),
       http = require('http').Server(app),
       mongoose = require('mongoose'),
       btoa = require('btoa'),
       atob = require('atob'),
       promise,
       connectionString = process.env.connectionString,
       port = process.env.PORT || 8080;
   
   http.listen(port, function() {
       console.log('Server Started. Listening on *:' + port);
   });
   
   app.use(express.static('public'));
   app.use(bodyParser.urlencoded({
       extended: true
   }));

   Base-route for loading up the front-end of our app:

   app.get('/', function(req, res) {
       res.sendFile('views/index.html', {
           root: __dirname
       });
   });
   

4. Storing URLs in MongoDB

   Let’s start by creating the collection schemas for storing data. As discussed above, we need two collections: one for storing the auto-incremented counter and the other for storing the URLs.

   var countersSchema = new mongoose.Schema({
       _id: { type: String, required: true },
       count: { type: Number, default: 0 }
   });
   
   var Counter = mongoose.model('Counter', countersSchema);
   
   var urlSchema = new mongoose.Schema({
       _id: {type: Number},
       url: '',
       created_at: ''
   });
   
   urlSchema.pre('save', function(next) {
       console.log('running pre-save');
       var doc = this;
       Counter.findByIdAndUpdate({ _id: 'url_count' }, { $inc: { count: 1 } }, function(err, counter) {
           if(err) return next(err);
           console.log(counter);
           console.log(counter.count);
           doc._id = counter.count;
           doc.created_at = new Date();
           console.log(doc);
           next();
       });
   });
   
   var URL = mongoose.model('URL', urlSchema);
   

   The above code creates the two collections and sets up our database for storing these collections. We’re also using a pre-save hook for the URL schema since we need to auto-increment the counter and log the date and time at which the URL was created.

   Next, we need to make sure that we start our application fresh and all previous entries are deleted. Once we reset, we’ll initialize our counter with a starting value of 10,000 to set up the URL shortening process. You may start with any value. This was chosen at random and will auto-increment by a value of one.

   promise = mongoose.connect(connectionString, {
       useMongoClient: true
   });
   
   promise.then(function(db) {
       console.log('connected!');
       URL.remove({}, function() {
           console.log('URL collection removed');
       })
       Counter.remove({}, function() {
           console.log('Counter collection removed');
           var counter = new Counter({_id: 'url_count', count: 10000});
           counter.save(function(err) {
               if(err) return console.error(err);
               console.log('counter inserted');
           });
       });
   });
   

   Our application is now ready to start accepting and shortening URLs! Let’s create a POST API that our front-end will use to send the URL:

   app.post('/shorten', function(req, res, next) {
       console.log(req.body.url);
       var urlData = req.body.url;
       URL.findOne({url: urlData}, function(err, doc) {
           if(doc) {
               console.log('entry found in db');
               res.send({
                   url: urlData,
                   hash: btoa(doc._id),
                   status: 200,
                   statusTxt: 'OK'
               });
           } else {
               console.log('entry NOT found in db, saving new');
               var url = new URL({
                   url: urlData
               });
               url.save(function(err) {
                   if(err) return console.error(err);
                   res.send({
                       url: urlData,
                       hash: btoa(url._id),
                       status: 200,
                       statusTxt: 'OK'
                   });
               });
           }
       });
   });

   As outlined in the flow diagram, once a valid URL is received, we check for its existence in the database.

   If found, we decode the corresponding _id field and return the hash back. Our front-end constructs the shortened URL and presents it to the user for redirection.

   If no URL is found, we save a new document in the collection. Remember, a pre-save step is run everytime the URL is saved. This will auto-increment the counter and log the current date and time. After the document is added, we send the hash to our front-end which constructs the shortened URL and presents it to the user for redirection.

5. Redirecting Users

   We’re almost done! Once our shortened URLs have been created, we need a way to redirect the user when a shortened URL is visited.

   app.get('/:hash', function(req, res) {
       var baseid = req.params.hash;
       var id = atob(baseid);
       URL.findOne({ _id: id }, function(err, doc) {
           if(doc) {
               res.redirect(doc.url);
           } else {
               res.redirect('/');
           }
       });
   });

   The above code looks for a hash in the shortened URL, base64 decodes it, checks if that ID is present in the collection, and redirects the user accordingly. If no ID is found, the user is redirected to the homepage of the URL shortener.

   For front-end code, please check out the GitHub repository mentioned at the end of this post. It’s essentially a text box field with a button to send the URL to the back-end and is out of the scope of this article.

More URL Shortener Enhancements

And we’re done! We have a bare-bones URL shortener that can be used internally to simplify your links. If you’d like to add more bells and whistles, here is a list of things you can additionally implement:

• Better code splitting
• Better/custom shortening algorithm for a smaller character hash (e.g. base52)
• Sharing shortened URLs on social media
• One-click URL copy
• Custom hashes
• User registration and associate shortened URLs

The entire code is available here: ScaleGrid URL Shortener Code Samples – Github A demo application is hosted on Heroku: ScaleGrid URL Shortener Demo

As always, if you build something awesome, do tweet us about it @scalegridio. If you need help with hosting for MongoDB® or hosting for Redis™*, reach out to us at [email protected].