Improvement On The Control Flow Of Node.js Scraper For JSON Output On Large File

I am new to the asynchronous control flow of Node.js, my scraper works, but I can't help thinking that there must be a more optimal (elegant?) way of doing it, I am open to the use

Solution 1:

It is highly recommended that you take a look on https://github.com/caolan/async - especially its forEachSeries method - it looks like it is exactly what you need.

I can also recommend to use fs sync methods in this particular case. It is not recommended to use the blocking methods for the services, but for the shell-like scripts it is ok.

Solution 2:

What your're trying to achieve with your code is a Finite State Machine (FSM), a common pattern used in asynchronous programming. Some languages have built-in support for. For example, C# 5.0 have async/await, which dramatically simplifies asynchronous programming by providing us with familiar liner code flow.

There have already been some attempts to bring async/await to JavaScript. I believe, full support for it in Node.js and all major web browsers is just a matter of time.

Until then, the most common pattern for asynchronous code flow in JavaScript is Promise. It represents the result of an operation which will be completed in the future, and allows to take an action upon its completion, with a JavaScript callback function. I suggest you stick to this pattern with your code.

More resources:

• http://wiki.ecmascript.org/doku.php?id=strawman:deferred_functions
• http://howtonode.org/promises
• https://stackoverflow.com/q/17351494/1768303
• What nodejs library is most like jQuery's deferreds?