snapJob Part III : Storing and loading data with Apache CouchDB and node.js

Part II of this project ("Managing application logs using LogStash") can be found here.

How cool and relaxed you think this guy is, lazing on it's couch ?

In this article, we will try to access our couchDB NOSQL database from our node.js application with a realy simple example.

Why is couchDB good in our case ? Because it keeps track of revisions on a document ! I have the idea that beeing able to watch previous versions of a curriculum vitae on the web site, for an applicant, may be a really nice feature to have. On top of this, this is Apache... this is open source !

But first of all, let's install it ! As I am on Ubuntu, let's just type the following command in a terminal :

sudo apt-get install couchdb

Now let's open our favorite browser and go to http://localhost:5984/

Fantastic ! It works !
Event better, browse http://localhost:5984/_utils/ :

Let's not create any databases, to see what happens next.

In our node.js application, we will use craddle, a node.js API that wraps CouchDB calls.

The very first thing we'll do will be to manage API Keys that will allow access to our RESTFul service. Each time an application tries to access the RESTFul web service, the API key must be provided. This will allow us to keep track of which application tries to access which data, and also block access to an unauthorized application. The API Key (a guid) must be provided each time any application tries to access to the service via http request headers.

The only two functions that will not require any API Key will be the one for requesting a key, and the one to activate a key.

So the logic will be the following :

• A user calls the requestAPI function, providing a email and an application name;
• The system generates a new key, and a confirmation key, and stores the key with the confirmation status as "pending";
• An email is send to the developer, with an htlm link to the method that activates the key. This method takes two arguments, the API key itself, and the confirmation key. This way, we can be sure that the email provided while requesting the key is valid;
• Once the developer opens this mail an clicks the link, the key is activated, and he can start playing with the api.

During that time, when the developer accesses the API via swagger, he provides no key. So only the two functions (request key and activate key) are available.
If he provides the key without having it activated, the same behavior applies.
If the key has been properly activated, then he can access to any of the RESTFul API functions.
On top of this, we want to be able to run the node.js application with a --masterAPIKey parameter. This master key will have access to any functions we want from the RESTFul web service. This may be handy later on, and will not be that difficult to implement.

... But first, let's do a brief refactoring...

We have a new folder, called "api'. This folder will contain singleton classes for each "application domain". Here, we have a very first one : API Keys management. So this class will contain our two methods : requestAPIKey, and confirmAPIKey. The methods in these kind of classes are not intended to actually send a response to the browser. It just do the job of storing data and perform other logical operations.

In the "models" folder, we expose only the swagger methods and here we do the job of sending back the response to the browser or any other application that calls the RESTFul api.

We also have a few new files in the "util" folder :
- database.js : a class that will wrap all of our database calls
- globals.js : a class that will contain miscellaneous application parameters such as the application domain and port, ...
- mail.js : a class to wrap email creation and performs sending (this thing does not work yet, but we'll see that later on)

The database.js file
This is the main focus for this article. So take a look at it, it's not that hard to understand :)

var cradle = require('cradle');
var logger = require('./log');
var async = require('async');

var Database = function Database() {
    var _this = this;
    // Connect to the apikeys database...
    this.ConnectApiKeysDb = function(callback) {
        _this.apiKeysDb = new (cradle.Connection)('127.0.0.1', 5984).database('apikeys');
        if(callback) callback();
    };

    this.ConnectAllDb = function(callback){
        async.parallel([
            _this.ConnectApiKeysDb
        ], callback)
    };

    this.ensureApiKeysDbExists = function(callback) {
        // if the apikeys database does not exists, let's create it...
        _this.apiKeysDb.exists(
            function (err, exists) {
                if (err) {
                    logger.logError(err);
                    if(callback) callback();
                } else if (!exists) {
                    logger.logWarning('database apikeys not exists and will be created.');
                    _this.apiKeysDb.create();
                    // ... and create a view to request all API keys, and confirmed ones...
                    _this.apiKeysDb.save('_design/apikeys', {
                        all: {
                            map: function (doc) {
                                if (doc.name) emit(doc.apiKey, doc);
                            }
                        },
                        confirmed: {
                            map: function (doc) {
                                if (doc.apiKey && doc.confirmationStatus == 'confirmed') {
                                    emit(doc.apiKey, doc);
                                }
                            }
                        }
                    }, callback);
                }
            }
        );
    };

    this.ensureAllDatabasesExists = function(callback){
        async.parallel([
            _this.apiKeysDb.ensureApiKeysDbExists
        ], callback)
    };

    // Saves an entry in the apikeys database.
    this.saveToApiKeys = function(key, data, callback) {
        _this.apiKeysDb.save(key, data,
            function (err, res) {
                if (err) logger.logError(err);
                if (callback) callback(err, res);
            }
        );
    };

    // Gets an entry from it's key from the apikeys database.
    this.getFromApiKeys = function(key, callback){
        _this.apiKeysDb.get(key, function (err, doc) {
            if (err) logger.logError(err);
            if(callback) callback(err, doc);
        });
    };

    // Gets entries from a view in the apikeys database.
    this.viewFromApiKeys = function(view, callback){
        _this.apiKeysDb.view(view, function (err, res) {
            if (err) logger.logError(err);
            if(callback) callback(err, res);
        });
    };
};

Database.instance = null;

/**
 * Singleton getInstance definition
 * @return singleton class
 */
Database.getInstance = function(){
    if(this.instance === null)
        this.instance = new Database();
    return this.instance;
};

module.exports = Database.getInstance();

As you can see, a new reference to the "craddle" api is used, which means we need to update our package.json file, and perform the "mpn install" command to download this new dependency.

In the app.js file, I just call what is needed to ensure that the databases exists, are connected, and event load api keys using the view we created by calling the refreshConfirmedApiKeys function :

database.ConnectAllDb(function(){
    database.ensureAllDatabasesExists(function(){
        apikey.refreshConfirmedApiKeys();
    });
});

So it you do run the app, you can see this :

You can even see that we already have one document... but what is this document ?!?

Let's dig a little bit :

It's the view we've created !

Let's have a closer view :

Do recognize that code ? It's the code we have in the database.js file, where we created the view !
So views are considered as documents !

So now, let's use our database.js file...

The api/apikey.js file

var globals = require("./../util/globals")
    , uuid = require('node-uuid')
    , database = require("./../util/database")
    , mailer = require("./../util/mail");

var ApiKey = function ApiKey() {
    // Sends a request for a new API key.
    this.requestAPIKey = function(applicationName, email, callback) {
        var data = {
            apiKey: uuid.v1(),
            confirmationKey: uuid.v4(),
            confirmationStatus: 'pending',
            applicationName: applicationName,
            email: email
        };

        var confirmationUrl = globals.applicationUrl + '/apikey/' + data.apiKey + '/' + data.confirmationKey;
        console.log(confirmationUrl);

        mailer.sendMail(
            'noreply@snapjob.com',
            data.email,
            'snapJob - Your API Key is here !',
            'Dear developer, as requested, here is your api key. This will not be valid until you activate it at the following address : ' + confirmationUrl,
            'Dear developer, as requested, here is your api key. This will not be valid until you activate it at the following address : <a href="' + confirmationUrl + '">' + confirmationUrl + '</a>'
        );

        database.saveToApiKeys(data.apiKey, data,
            function(err, res) {
                if(err) {
                    if (callback) callback(503, JSON.stringify(err));
                } else {
                    if (callback) callback(200, JSON.stringify('ApiKey for application "' + data.applicationName + '" is "' + data.apiKey + '". A confirmation email has been sent to "' + data.email + '".'));
                }
            }
        );
    };

    // Confirms an API key
    this.confirmAPIKey = function(apiKey, confirmationKey, callback) {
        var _this = this;
        database.getFromApiKeys(apiKey,
            function(err, doc) {
                if (err) {
                    if (callback) callback(503, err);
                } else {
                    if (doc == undefined) {
                        if (callback) callback(404, 'api-key not found');
                    } else {
                        if (doc.confirmationKey !== confirmationKey) {
                            if (callback) callback(403, 'Confirmation key is not correct');
                        }else {
                            switch (doc.confirmationStatus) {
                                case 'pending':
                                    doc.confirmationStatus = 'confirmed';
                                    database.saveToApiKeys(apiKey, doc,
                                        function (err) {
                                            if (err) {
                                                if (callback) callback(503, err);
                                            }else {
                                                if (callback) callback(200, 'API key is now active');
                                                _this.refreshConfirmedApiKeys();
                                            }
                                        }
                                    );
                                    break;
                                case 'banned':
                                    if (callback) callback(403, 'API key has been banned and cannot be reused');
                                    break;
                                case 'confirmed':
                                    if (callback) callback(403, 'API key has been already been confirmed');
                                    break;
                            }
                        }
                    }
                }
            }
        );
    };

    this.confirmedApiKeys = [];

    // Refreshes all confirmed API keys and puts them in the confirmedApiKeys array.
    this.refreshConfirmedApiKeys = function(){
        var _this = this;
        var newResult = [];
        database.viewFromApiKeys('apikeys/confirmed',
            function(err, res) {
                if(res !== undefined) {
                    res.forEach(
                        function (row) {
                            newResult.push(row.apiKey);
                        }
                    );
                    _this.confirmedApiKeys = newResult;
                }
            }
        );
    }
};

ApiKey.instance = null;

/**
 * Singleton getInstance definition
 * @return singleton class
 */
ApiKey.getInstance = function(){
    if(this.instance === null)
        this.instance = new ApiKey();
    return this.instance;
};

module.exports = ApiKey.getInstance();

There you can see that we have 3 functions :
- One that saves an API key request;
- Another one that confirms an API key (called after clicking on the link in the mail sent to the developer);
- And the last one takes advantages of the couchDB view we created to load all confirmed API keys in the memory.

app.js, the swagger ugly validator function

Swagger has been described in the previous article. It has a feature that is as interesting as it is ugly : The validator function.

Lets take a look at this function :

// Adding an API validator to ensure the calling application has been properly registered
swagger.addValidator(
    function validate(req, path, httpMethod) {
        // refresh all confirmed api keys...
        apikey.refreshConfirmedApiKeys();

        //  example, only allow POST for api_key="special-key"
        if(path.match('/apikey/*')){
            logger.logInfo('API call for the /apikey/ path allowed', req);
            return true;
        }

        var apiKey = req.headers["api_key"];
        if (!apiKey) {
            apiKey = url.parse(req.url,true).query["api_key"];
        }

        if(!apiKey) return false;

        if (apiKey === globals.masterAPIKey) {
            logger.logInfo('API call allowed by master key', req);
            req.clientName = 'master api-Key';
            return true;
        }

        return apikey.confirmedApiKeys.indexOf(apiKey) > -1;
    }
);

What I find ugly here, is that you provide a synchronous method, that returns true or false. If you try to access to a database or any other function that takes a callback as a parameter, then the asynchronous philosophy of node.js is broken. Here, for example, if I try to access my database, the result of my database call will come after we exit the validation function, which will not allow us to return true or false. This is why, here, I just take a look at pre-loaded confirmed API keys.

But if you think about it, this is a RESTFul web service, that is intended to be load balanced on several servers. Even if the confirmAPIKey in the api/apikey.js file updates the array, you may not be ensured that if the next call, using the confirmed api key, will be processed on the same server, which means the confirmedApiKeys array you can find in that file may not be properly updated every where... This is why, to contain this problem, I just reload the confirmed API keys asynchronously at every call. So in therory, this problem may occur a very few times, but I have no other solutions at that time.

Another thing you can see in this method, is that it returns true for every path that starts with "/apikey/" (yes, it's a regular expression).

If the masterKey (provided within the application parameters in the command line) is provided, we return always true.

If the key provided is in the array containing the confirmed keys, then we also return true.

Testing

Lets start the node.js application this way :

node app.js --masterAPIKey=06426e19-d807-4921-a668-4708287d8878

If you browse to http://localhost:8080/, you can see that you can expand the apikey methods, but not the test method.

If you put your master key in the field at the top right of the window and click "explore", then you can see all functions.

Now let's request a new API key. Call the POST method and provide a name and an email :

Click "Try it out!".

Your node.js console should provide you a link. For me, it is http://localhost:8080/apikey/12440de0-079c-11e4-81ff-db35b05bd397/0762362b-a7dd-4309-9e4e-e52f89ab4ec4, but it will be something alse for you, as it includes generated keys.

Also, in your database, you should see your pending key request :

Now if you follow the link that appeared in your node.js console, it should confirm the API key, your browser should say : API key is now active

Yeay ! We did it ! That wasn't so bad, isn't it ? :)

Presentation of the project can be found here.
Source code for this application can be downloaded from here.

Next part : snapJob Part IV : Scalling node.js, and ensure high availability using "cluster"