Abe Massry

Web Development and Building Businesses

Building a Webapp With LevelDB

Building a webapp with LevelDB. I had heard LevelDB was useful as a key value store with good performance, and I thought Express Polls might be a good application to use this type of database. I’ll go through a few of the steps I used to get it to work. The code is available on GitHub at express-polls

Motivation

I wanted an online poll that didn’t require a login. I had used a flash based one before but that was years ago. I also wanted the values to update as people were voting so I knew I wanted to use Socket.IO to handle that. I thought LevelDB might be a good solution to store the data. This was my first attempt at using LevelDB so I thought I would document it here. The only feature of the database I haven’t looked into is deleting data, so this post will only cover writing and reading.

Writing to LevelDB

I started with node-levelup, I heard it was a really good library so I thought I would start there. The readme is very well written with examples and is very clear to understand. When I actually wrote the webapp the syntax is almost exactly like the examples here.

To start a poll a question and a list of answers are required, this is taken from the user and submitted via POST. Since no login is required a UID is generated and stored with HTML5 LocalStorage on the client.

In order to store different types of values with LevelDB you have to namespace the keys. This is like different tables in SQL. LevelDB organizes the keys in alphabetical order. So the first namespace I used was the poll. Each poll is going to be unique so in LevelDB there are a few libraries to help namespace the keys properly. I found doing the manual way worked best for me so I spaced them with a ‘!’. There are a few downsides to this such as it’s a typeable character and you might open yourself up to database injection if you take user data and write to a key with it. I don’t do that for this webapp but it should be considered an area for improvement.

pact is javascript file that contains most of my requires and then I export them using the nodejs builting export.

Save a key and value to LevelDB
1
2
3
4
5
var value = { question: question, answers: answers, user: uid};
var key = pact.getId();
pact.db.put('poll!'+key, JSON.stringify(value), function() {
  res.redirect('/poll/'+key);
});

I store all the data as strings in JSON format. I found this was the easiest way for me to interact with javascript objects.

The entire express route is here in the create.js file.

Reading from LevelDB

Reading from LevelDB is a little more involved for this application so I will be showing the entire poll route here.

Votes are stored with the ‘vote!’ namespace, which I will go over writing them in Reading and Writing from/to LevelDB.

  1. Get the poll that you are asking for, once you do that you can get votes associated with that poll.
  2. create a readStream to get a range of values. This is in node-levelup. Each item is handled with the .on('data' event. We push each vote to an array that will be then added to a JSON string and sent to the browser.
  3. When the .on('end' event comes through we total everything up because theres no more votes and send everything to the browser.
Read from LevelDB and send it to the browser
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
var pact = require('../pact.js')

exports.main = function(req, res){
  pact.db.get('poll!'+req.params.id, function (err, value) {
    if (err) return console.log(req.params.id+' does not exist in poll.js');
    console.log(req.params.id+' found');
    var pollId = req.params.id;
    //voteData: voteData,
    var data = JSON.parse(value);
    var question = data.question;
    var stats = new Array();
    pact.db.createReadStream({start: 'vote!'+pollId + '!',
                                end: 'vote!'+pollId + '!~'
                            })
      .on('data', function(data){
        var vote = JSON.parse(data.value).vote;
        stats.push(vote);
      })
      .on('end', function() {

        var statsCount = {};
        var stat;
        for (var i=0; i<stats.length; i++) {
          stat = statsCount[stats[i]];
          if (stat) {
            statsCount[stats[i]] = stat + 1;
          } else {
            statsCount[stats[i]] = 1;
          }
        }
        var pollAnswers = JSON.parse(value).answers;

        var voteData = [];
        var total = 0;
        for (var i=0; i<pollAnswers.length; i++) {
          if(!statsCount[pollAnswers[i]]) {
            statsCount[pollAnswers[i]] = 0;
          }
          voteData[i] = { label: pollAnswers[i],
                          data: statsCount[pollAnswers[i]]
                        };
          total = total + statsCount[pollAnswers[i]];
        }
        var jsonPayload = {total: total, voteData: voteData};
        var jsonPayloadString = JSON.stringify(jsonPayload);
        if (question == '') {
          question = 'Untitled';
        }
        res.render('poll', { title: question,
                             pollId: req.params.id,
                             data: data,
                             jsonData: jsonPayloadString,
                             voteData: voteData,
                             render: true
                           }
        );
      })
    }); // db.get poll end
};

The jsonPayload is used to render the plot of votes. It’s turned into a string so it can be sent over HTTP.

The namespace for each vote key takes the format of ‘vote!’ to signify it is a vote pollId which is the poll that vote belongs to and the last part of the createReadStream function is the start which is ‘!’ and has a low ascii value to the end ‘~’ which has a high ascii value. The ids I picked to use are all hexadecimal (0-9a-f) so ~ does not show up in the key structure and ! can safely be used as the delimiter for this application.

The documentation goes over createReamStream, but I understand it as it cycles through each value and sends out events. The .on statement listens for those events, when there is data it is added. When there is no more data the end event comes through and theres no more data and it is sent to the client. In SQL this might be equivalent to the SELECT statement.

Reading and Writing from/to LevelDB

The vote route does a lot so the entire file will be listed here as well.

  1. The same as last time we need to get the pollId
  2. We have already checked if this UID had voted before because the view would not display the vote buttons to them and would switch straight to the rendering of the poll graph if they had, so we can store this vote right to the database.
  3. Storing the vote is simple, it’s the db.put which we went over.
  4. After the new vote is stored the data has to be retrieved on the current voting and pushed out to any clients listening on Socket.IO. This data is also used in rendering the poll after the vote has taken place.
  5. The data is tallied like it was before, it is pushed out to Socket.IO clients and returned to the user to render the poll.
Reading and Writing with LevelDB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
var pact = require('../pact.js')


exports.main = function(req, res){
  if (req.param('poll_id')) {
    var pollId = req.param('poll_id');
    var uid = req.param('uid');
    pact.db.get('poll!'+pollId, function (err, value) {
      if (err) return console.log(req.params.id+' does not exist in vote.js');
      var voteId = pact.getId();
      var answer = req.param('answer');
      var putValue = { user: uid, vote: answer};
      var stats = new Array();
      pact.db.put('vote!' + pollId + '!' + voteId, JSON.stringify(putValue),
        function(err) {
          if (err) return console.log('db error', err);
          // get current poll stats
          pact.db.createReadStream({start: 'vote!'+pollId + '!',
                                      end: 'vote!'+pollId + '!~'
                                  })
            .on('data', function(data){
              var vote = JSON.parse(data.value).vote;
              stats.push(vote);
            })
            .on('end', function() {

              var statsCount = {};
              var stat;
              for (var i=0; i<stats.length; i++) {
                stat = statsCount[stats[i]];
                if (stat) {
                  statsCount[stats[i]] = stat + 1;
                } else {
                  statsCount[stats[i]] = 1;
                }
              }
              var pollAnswers = JSON.parse(value).answers;

              var voteData = [];
              var total = 0;
              for (var i=0; i<pollAnswers.length; i++) {
                if(!statsCount[pollAnswers[i]]) {
                  statsCount[pollAnswers[i]] = 0;
                }
                voteData[i] = { label: pollAnswers[i],
                                data: statsCount[pollAnswers[i]]
                              };
                total = total + statsCount[pollAnswers[i]];
              }
              console.log(' ');
              console.log('total');
              console.log(total);
              console.log(' ');
              var jsonPayload = {total: total, voteData: voteData};
              var jsonPayloadString = JSON.stringify(jsonPayload);
              pact.io.emit(pollId, jsonPayloadString);
              res.end(jsonPayloadString);
              //var voteData = [
              //  { label: "Polls",  data: 20},
              //  { label: "Learning",  data: 50},
              //  { label: "Fun",  data: 30}
              //];
              //res.end
              // update with websockets



            })

        }
      );
    });
  }
};

This route is called with AJAX.

Future work

There are a lot of other moving parts to the webapp so the examples aren’t as simple as they could be, but I wanted to show what LevelDB looks like inside of a functioning webapp with other things going on as well.

Using a different namespace separator, something like ‘x/00’. Couldn’t get that to work at the time but I would like to look into it more.

This is not related to LevelDB but I have seen some inconsistencies with the HTML5 LocalStorage UID scheme and I would like to look into that further.

Comments