outten.net - random thoughts

First experiences with "async" library in Node.js

I have been hacking on some JavaScript code recently in particular with Node.js. With the hype around Node.js, I wanted to see what it was like to develop in JavaScript on the server. With Node.js’s heavy use of event-driven programming, I can see how there is real power there. It reminds me of some of the limited use of actors I have worked on or tinkered with.

This article might be more of a reference for me, but I did think it was an example of clever use of scopes in JavaScript. The problem I was trying to solve was running an SQL query to see if a row already existed in a table based on the key value and if it didn’t exist, go ahead and insert the row. I know I could have done it with a few callbacks or nesting functions, but I went in search of something better.

I had found several references to the async library. In particular, I thought the waterfall function would be a good fit with the exception of one thing. It wasn’t immediately obvious how to pass the key parameter to the first function which checked to see if the key was already used. I figured I could put all of the functions within a function and then reference some variables in the scope of the surrounding function. It would turn out something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
  addEntry: (value, callback) ->
    key = value.key
  
    # function to find if the key exists
    doesKeyExists = (callback) ->
      sql = """
        select count(*) as cnt
        from entries
        where key = ?
        """
      @db.get sql, key, (err, row) ->
        callback(null, row)

    # function to insert te row if it doesn't already exist
    addEntry = (exists, callback) ->
      @log.debug "_addEntry exists: #{util.inspect(exists)}"
      params = {$key: key, $value: JSON.stringify(value)}
        if exists.cnt > 0
          callback(null, false)
        else
          sql = """
        INSERT INTO entries
        (created_at, key, value)
        values (date('now'),$key, $value)
        """
          @db.run sql, params
          callback(null, true)
  
    # do methods in order and pass the callback values
    async.waterfall [ doesKeyExists, addEntry ], (err, result) =>
      if err
        @log.error "Error: #{err}"
        callback(err)
      else
        callback(result)

I didn’t actually try this, but I was thinking of something similar. Unfortunately, testing the inner functions is difficult because they rely on the context of the surrounding function. I was in search of a better solution. In searching, I found this answer on Stack Overflow. It really related to what I was trying to do. I wanted to pass a parameter to the functions that got passed to the async.waterfall tasks, but I didn’t want to have to embed them. Here is an excerpt of what I came up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
  addEntry: (value, callback) ->
    key = value.key
    async.waterfall [ @_doesKeyExists(key), @_addEntry(key, value) ], (err, result) =>
      if err
        @log.error "Error: #{err}"
        callback(err)
      else
        callback(result)

  _doesKeyExists: (key) ->
    (callback) =>
      sql = """
            select count(*) as cnt 
            from entries
            where key = ?
            """
      @db.get sql, key, (err, row) ->
        callback(null, row)

  _addEntry: (key, value) =>
    (exists, callback) =>
      @log.debug "_addEntry exists: #{util.inspect(exists)}"
      params = {$key: key, $value: JSON.stringify(value)}
      if exists.cnt > 0
        callback(null, false)
      else
        sql = """
              INSERT INTO entries
              (created_at, key, value)
              values (date('now'),$key, $value)
              """
        @db.run sql, params
        callback(null, true)

By calling a function with the parameters and then returning a function with the signature expected by the async.waterfall, the variables are available because they are in the scope. This reminded me of the information hiding examples in JavaScriptThe Good Parts.

Let’s breakdown the _doesKeyExists function:

_doesKeyExists(key) (callback) => …

With some of CoffeeScript’s goodness (last expression returned and shorter function declaration), an anonymous function is returned with the callback signature. key is available within the anonymous function because it is in the outer scope.

By separating _doesKeyExists and _addEntry functions, these can now be tested separately, but can also be used in the sync.waterfall call:

async.waterfall [ @_doesKeyExists(key), @_addEntry(key, value) ], …

This seemed like a clever approach to use within Node.js, or in the browser, to pass parameters to a series of async calls. Again I’m relatively new to evented programming so there is probably a better way to do this, but I could still see this being useful in future hacking.