A frecuent case of double callbacks in node.js

A double-callback is in javascript jargon a callback that we expect to be called once but for some reason is called twice or more times.

Sometimes it is easy to discover, as in this example:

function doSomething(callback) {
  doAnotherThing(function (err) {
    if (err) callback(err);
    callback(null, result);
  });
}

The obvious error here is that when doAnotherThing fails the callback is called once with the error, and once with result.

However there is one special case that is very hard to reproduce and to discover, moreover it has happened to me several times.

Yesterday, my friend and co-worker Alberto asked me this:

"Why does this test hangs on the assertion line?" expect(foo).to.be.equal('123');

The test look like this

it('test something', function (done) {
  function_under_test(function (err, output) {
    expect(foo).to.be.equal('123');
  });
});

After some debugging I found out that it didn't hang only on expect but when we throw any error inside the callback.

A bunch of calls before in the stack, there was a little function with a bug like this:

function (callback) {
  another_function(function (err, some_data) {
    if (err) return callback(err);
    try {
      callback(null, JSON.parse(some_data));
    } catch(err) {
      callback(new Error(some_data + ' is not a valid JSON'));
    }
  });
}

The intention of the developer with this try method is clear: to catch JSON.parse errors. But the problem is that it also catch errors thrown inside callback and execute the callback with a wrong error.

The solution is trivial, parse outside the try as follows:

function (callback) {
  another_function(function (err, some_data) {
    if (err) return callback(err);
    try {
      var parsed = JSON.parse(some_data)
    } catch(err) {
      return callback(new Error(some_data + ' is not a valid JSON'));
    }
    callback(null, parsed);
  });
}

Introducing these errors is very easy I've done several times, throubleshooting is very hard, so be careful and do not wrap callbacks call in try/catch blocks.

Posted in node | 0 Comments and 0 Reactions

The Architecture we use to Deploy to Public and Private Clouds

Originally posted on inside.auth0.com.

How we use Puppet, GitHub, TeamCity, Windows Azure, Amazon EC2 and Route53 to ship Auth0

Introduction

There are two ways companies deploys web applications to the cloud: PaaS and IaaS. With Platform as a Service you usually deploy your applications to the vendor's platform. Infrastructure as a Service is the basic cloud-service model: you get servers, usually vms.

We started auth0 using Heroku's Platform as a Service but soon we decided to provide a self-hosted option of Auth0 for some customers. In addition, we wanted to have the same deployment mechanism for the cloud/public version and the appliance version. So, we decided to move to IaaS.

What I am going to show here is the work after several iterations at a very high level. It is not a silver bullet and you probably don't need this (yet?) but it is another option to consider. Even if this is exactly what you are looking for, the best advice I can give you "don't architect everything from start", this come from the work of several weeks and took several iterations but we never ceased to ship, everything evolved and keeps evolving, and we keep tuning the process a lot on the run.

The big picture

What's Puppet and why is so important when using IaaS?

Picture yourself deploying your application to your brand new vm today. What is the first thing you will do? Well, if you have a node.js application installing node will be a good start. But probably you will need to install 10 other things as well, and changing the famous ulimit, configuring logrotate, ntp and so on. Then you will copy your application somewhere in the disk and configure it as a service and so on. Where do you keep this recipe?

Puppet is a tool for configuration management. Rather than an install script you describe at a high level the state of the resources in the server. When you run puppet it will check everything and then it will do whatever it takes to put the server in that specific state, from removing a file to installing a software.

There is another tool similar to puppet called chef. One of the things regarding Chef that I would like to test in the future is Amazon OpsWorks.

After you have your configuration in a language like this, deploying to a new server is very easy. Sometimes I modify the configuration via ssh to test and then I update my puppet scripts.

There is another concept emerging called InmutableServers, it is a very interesting way and there seems to be some companies using it.

Sources and repositories

Auth0 is very modularized and it is not a single web application but a network of less than ten. Every web application is a node application. Our core is a web app without ui which handles the authentication flows and provide a rest interface, dashboard is another web application which is just an interface to our core where you can configure and test most of the settings, docs is another app full of markdown tutorials to name a few.

We use github private repositories because we already had a lot of things opensourced there.

We use branches to develop new features and when it is ready we merge to master. Master is always deployable. We took some of the concepts from a talk we saw; "How Github uses Github to build Github". When something is ready? is a tricky question but we are very responsible and self organized team, we do pull-requests from branch to master when we want the approval of our peers. Teamcity automatically run all tests and will mark the pull requests as OK, this is a very useful feature of TC. But the most important thing we do in this stage are code reviews.

Very often we send a branch to one of our 4 tests environments with our hubot (a personal bot on the chat room):

  • ui is our dashboard application
  • template-scripts was a branch
  • proto is the name of te environment

with that in place we can review a living instance of the branch in a environment similar to production.

Then we iterate until we finally merge.

This is what works for us now, anyone of the team can merge or push directly to master and we consciously decide when we should do the pull-request ceremony.

Continuous integration

We used Jenkins for six months but it failed a lot, I had to rebuild few of the plugins we were using. Then I had a short fantasy to build our own CI server but we choose teamcity since I had use it before, I knew how to set it up and it is a good product.

Every application is a project in teamcity, when we push to master teamcity does:

  1. pull the repository
  2. install the dependencies (in some repos node_modules versioned)
  3. run all the tests
  4. bundle node dependencies with carlos8f/bundle-deps
  5. bump package version
  6. npm pack

1,2,3 are very common even in non-node.js projects. In the 4th step what we do is to move all "dependencies" to "bundleDependencies" in the package.json by doing this, the npm pack will contain all the modules already preinstalled. The result of the task is the tgz generated by npm pack.

This will automatically trigger the next task called "configuration". This taks pull our configuration repository written in puppet and all the puppet sub modules, then it will take the last version of every node project and build one ".tgz" for every "site" we have in puppet. We have several "site" implementations like:

  • myauth0 used to create a quick appliance
  • auth0 the cloud version you see in app.auth0.com. It is very different from the previous one, for instance it will not install mongodb since we use MongoLab in the cloud deployment.
  • some-customer some customers need some specific settings or features, so we have configurations with our customers name.

The artifact of the configuration task is a tgz with all puppet modules including auth0 and the site.pp. All the packages are uploaded to Azure Blob storage in this stage.

The next task called "cloud deploy" in the CI pipeline will trigger immediately after the config task, it updates the puppetmaster (currently in the same CI server) and then runs the puppet agent in every node of our load balanced stack via ssh. After it deploys to the first node it does a quick test of the node and if there is something wrong it stop it and it will not deploy to the rest of the nodes. Azure load balancer then will take the node out of rotation until we fix the problem in the next push.

We have a backup environment where we continuously deploy, it is on Amazon and in a different region. It has a clone of our database (max 1h stale). This node node is used in case that azure us-east has an outage or something like that, when this happens Route53 will redirect the traffic to the backup environment. We take high availability seriously, read more here. When running in backup mode, all the settings become read-only, this means that you can't change the properties of an Identity Provider however your users will be able to login to your application which is Auth0 critical mission.

Adding a new server to our infrastructure take very few steps:

  • create the node
  • run an small script that installs and configure puppetagent
  • approve the node in the puppetmaster

Assembling an appliance for a customer is very easy as well, we run an script that install puppetmaster in the vm, download the last config from the blob storage and run it. We use Ubuntu JeOS in this case.

Final thoughts

I've to skip a lot of details to make this article concise. I hope you find it useful, if there is something you will like to know regarding this don't hesitate to put your question in a comment.

Posted in puppet git continuous delivery | 0 Comments and 0 Reactions

Promises A+ and Q

Two years ago I published a blog post about jQuery promises I have got lot of feedback since then but even if this still valid for jQuery I want to drag focus to the great job some of the javascript gurus are doing.

The important specification here is Promises A+

promisesalogo

The specification is very short, readable and useful. Go read it. It specify the interface for a Promise despite how it is created.

There are several frameworks and libraries that follow this specification and this is a GOOD thing, because it means that you can pass a promise from some library to other one and everyone speak the same interface.

So, what's a promise again?

A promise represents a value that may not be available yet.

There is another definition I heard that I like a lot:

A promise is an asynchronous value.

If you have done any javascript you know that when you need to call an asynchronous function you have to pass a callback which is a function that will be called after it finish doing its job. So, the function doesn't return anything and this make it harder to compose asynchronous code sometimes.

The Q library

Q is a library that implements the standard and has some extra helpers. Q works in the browse and in node.js.

From now on, I will use Q to show some examples, but keep in mind that the very basic things are part of the Promise/A+ and Q adds some friendly helpers on top of that.

Basic usage

In this first example I have called Q.delay(2000) this method returns a promise that will be fulfilled after two seconds. You can think of this method as a setTimeout that instead of having callback parameter it returns a promise.

Every promise has a then method that receive two arguments (or two callbacks) in order to access the fulfilled value and rejected value. Either callbacks could be null or undef.

Chaining

then returns a new promise, this allow Promises A+ to be chained

In this example I'm returning a value in the first then/onfulfilled function this makes the returned promise to be fulfilled with that value (3.2.6.1 section in spec).

Because this is something you do a lot, Q promises have a helper thenResolve:

The most interesting thing about chaining promises is that you can serialize work:

In this example we first get the user with getUser and then we get his tweets getTweets. The result of then(getTweets) becomes a new promise that will be fullfiled when the two things are fulfilled and it will be fulfilled with tweets.

Can you read that thing "getUser then getTweets then forEach tweet alert tweet message"? This is important. We are working with asynchronous code in javascript yet the code is still very readable and easy to compose.

Deferred

At this point we have used only the promise returned from the delay method. Another way to create promises is using Q.defer. A Defer has two important methods resolve and reject, and it has a property promise. It goes without saying that this is not part of the specification and different frameworks might have different ways to create deferred.

The delay method in Q could be implemented with defer as follows:

At the point I'm writing this jQuery promises are not compatible with Promises/A and Promises/A+, so an easy way to fix this is as follows:

Despite the specification doesn't work with jQuery Promises, the Q implementation does in a straightforward way:

Q($.get('/something'))

You can wrap a jQuery promise with Q to convert it to Promise/A+.

Parallelism

What if you need to do several asynchronous tasks that doens't depend on each other? Use Q.all.

Q.all converts an array of promises into a single promise that will be fulfilled when all the promises are fulfilled with an array of all the values or rejected with the first reason a promise is rejected.

in this example I'm calling getUsers three time for the three ids I have in the array. Then I wait the three promises to be fulfilled (this will happen after 1s aprox.) and then I show a message.

A more complex example here:

In this case the spread method (from Q- not standard) works like then but "spread" all the values in arguments thus we can give the mergeProfiles function directly.

Error throwing and handling in asynchronous code

Another interesting thing about promises is error handling. In node.js land it happens a lot that you end with a code like this:

doFoo(function (err, r1) {
  if (err) return handleError(err);

  doBar(r1, function (err, r2) {
    if (err) return handleError(err);

    doBaz(r2, function (err, r3) {
      if (err) return handleError(err);

      callback(r3);
    });

  });
});

I want you to notice this line three times:

if (err) return handleError(err)

With promises you can write this same code as follows:

doFoo()
  .then(doBar)
  .then(doBaz)
  .then(null, handleError);

Because the two first then calls doesn't have a onreject handler they will pass the rejection reason to the next promise until someone handles that error. More interesting if a promise is rejected none of the fulfill handlers here will be called.

The other interesting thing about this is that if you throw an exception inside a then call the promise will be rejected.

node.js

node.js api and modules follow a convention for asynchronous code, functions usually have callback parameter as the very last parameter and this callback get called with error and value.

So, Q make it easy to convert this style to promises as follows:

var Q = require('q');
var readdir = Q.nfbind(require('fs').readdir);

//usage
readdir('./path')
  .then(function (files) {

  }, function (err) {

  });

This nfbind method has an alias denodeify.

There are lot more helpers but the other one interesting is nodeify. Do you feel shame letting the world know that you use promises and want to expose an standard-old node.js api? Use nodeify:

module.exports = function (callback) {
  mysuperpromise()
    .then(blabla)
    .nodeify(callback); 
};

Tests

This is not that important but it is something I found and I like a lot. When writing unit tests against asynchronous code, typically you do something like this:

function test (done) {
  getSomething(function (err, result) {
    if (err) return done(err);
    Assert.areEqual(result, 123);
    done();
  });
}

As I cited before "promises are asynchronous values". What if the assert and test framework could handle promises as well? You could easily write something like this:

function test () {
  return getSomething().should.eventually.equal(123);
}

This is already done and you can use it today, have a look to chai-as-promised.

More material

Watch this video:

Follow @domenic.

Read his blogpost.

Q Api Reference is very helpful.

Conclusion

Promises are the future (of JavaScript asynchronous code). I put JavaScript there because I am sure some people are working on better languages with better syntax for asynchronous flows but that doesn't feel is going to change in the short term for javascript.

Posted in | 0 Comments and 0 Reactions

  • Categories

  • Archives