Btukfyl

I am running a cron job every 5 mins to get data from 3rd party API, It can be N number of request at a time from NodeJS application. Below are the details and code samples:

1> Running cron Job every 5 mins:

const cron = require('node-cron');

const request = require('request');

const otherServices= require('./services/otherServices');

cron.schedule("0 */5 * * * *", function () {

  initiateScheduler();

});

2> Get the list of elements for which I want to initiate the request. Can receive N number of elements. I have called request function (getSingleElementUpdate()) in the forEach loop

var initiateScheduler = function () {

  //Database call to get elements list

  otherServices.moduleName()

    .then((arrayList) => {

      arrayList.forEach(function (singleElement, index) {

        getSingleElementUpdate(singleElement, 1);

      }, this);

    })

    .catch((err) => {

      console.log(err);

    })

}

3> Start initiating the request for singleElement. Please note I don't need any callback if I received a successful (200) response from the request. I just have to update my database entries on success.

var getSingleElementUpdate = function (singleElement, count) {

  var bodyReq = {

    "id": singleElement.elem_id

  }

  var options = {

    method: 'POST',

    url: 'http://example.url.com',

    body: bodyReq,

    dataType: 'json',

    json: true,

    crossDomain: true

  };

  request(options, function (error, response, body) {

    if (error) {

      if (count < 3) {

        count = count + 1;

        initiateScheduler(singleElement, count)

      }

    } else{

      //Request Success

      //In this: No callback required

      // Just need to update database entries on successful response

    }

  });

}

I have already checked this:

request-promise: But, I don't need any callback after a successful request. So, I didn't find any advantage of implementing this in my code. Let me know if you have any positive point to add this.

I need your help with the following things:

I have checked the performance when I received 10 elements in arrayList of step 2. Now, the problem is I don't have any clear vision about what will happen when I start receiving 100 and 1000 of elements in step 2. So, I need your help in determining whether I need to update my code for that scenario or not or is there anything I missed out which degrade the performance. Also, How many maximum requests I can make at a time. Any help from you is appreciable.

Thanks!

AFAIK there is no hard limit on a number of request. However, there are (at least) two things to consider: your hardware limits (memory/CPU) and remote server latency (is it able to respond to all requests in 5 mins before the next batch). Without knowing the context it's also impossible to predict what scaling mechanism you might need.

The question is actually more about app architecture and not about some specific piece of code, so you might want to try softwareengineering instead of SO.