How To Create Node.js Background Tasks Using Kue Workers

Hi Folks,

If you have read the previous article on web workers, you know that they are awesome.
However if the user refreshes the page, your jobs are gone ...

In order to compensate this weakness, I'll show you how to quickly add a Server Side Worker Mechanism using Kue.js!
In order to do this, we need to install Redis first to store our tasks.

Once your Redis database is launched:

$ npm install kue

Ok we are ready to code now!

Let's start with the Front End.

A very simple template to trigger a job on a button click:

<button ng-click="triggerJob()">Trigger job</button>

In the JavaScript, we just send an http request to the server at the url /triggerJob.

controller('MyCtrl', function ($scope, $http) {
  $scope.triggerJob = function () {
    $http({
      method: 'POST',
      url: '/triggerJob'
    });
  }
}

Now the Back End. We first initialise Kue and create a Queue:

var kue = require("kue"),
  queue = kue.createQueue();

We then use the process method in order to create a ... process. We only do something simple here: displaying the data passed to accomplish this job. Generally this is the place where you type your code that requires a lot of processing (db insertion, csv manipulations, etc).

queue.process("test", (job, done) => {
  console.log("job data", job.data);
});

We could use the API's in order to visualise and manipulate the jobs but Kue already has a nice user-friendly UI to do this.

You just need to add:

app.use("/kue-ui", kue.app);

(I put this after every app.use lines).

Finally we receive the http request from the client here and create a new job:

app.post("/triggerJob", function(req, res) {
  var job = queue
    .create("test", {
      title: "job ran at " + Date.now()
    })
    .save(function(err) {
      if (!err) console.log(job.id);
    });

  res.send(200);
});

We then click on our button and head to /kue to see the results:

As you can see our job is here, stuck at 0%. That's normal, let's update our code to notify when the job is done:

queue.process('test', (job, done) => {  
  console.log('job data', job.data);
  // Do your task here
  .
  .
  .
  // And we are done
  done();
});

And voila!

By the way, you can manually delete the Active Job otherwise it will never complete.

Conclusion

Web workers are great solutions to outsource the heavy lifting but the Server Side Workers are at the moment a more reliable solution.
Kue.js is a very flexible library which will give you many features like priority handling, delaying jobs, great UI.