Real-time Web Apps with Server-Sent Events (pt 2)

This is the second part of a two part series about building real-time web applications with server-sent events.

• Building Web Apps with Server-Sent Events - Part 1

Reconnecting

In this post we are going to look at handling reconnection if the browser loses contact with the server. Thankfully the native JavaScript functionality for SSEs (the EventSource) handles this natively. You just need to make sure that your server-side implementation supports the mechanism.

When the server reconnects your SSE end point it will send a special HTTP header Last-Event-Id in the reconnection request. In the previous part of this blog series we looked at just sending events with the data component. Which looked something like this:-

data: The payload we are sending\n\n

Now while this is enough to make the events make it to your client-side implementation. We need more information to handle reconnection. To do this we need to add an event id to the output.

E.g.

id: 1439887379635\n
data: The payload we are sending\n\n

The important thing to understand here is that each event needs a unique identifier, so that the client can communicate back to the server (using the Last-Event-Id header) which was the last event it received on reconnection.

Persistence

In our previous example we used Redis Pub/Sub to inform Node.js that it needs to push a new SSE to the client. Redis Pub/Sub is a topic communication which means it will be delivered to all connected clients, and then it will be removed from the topic. So there is no persistence for when clients reconnect. To implement this we need to add a persistence layer and so in this demo I have chosen to use MongoDB.

Essentially we will be pushing events into both Redis and MongoDB. Redis will still be our method of initiating an SSE getting sent to the browser, but we will also be be storing that event into MongoDB so we can query it on a reconnection to get anything we've missed.

The Code

OK so let us look at how we can actually implement this.

Update ServerEvent

We need to update the ServerEvent object to support having an id for an event.

function ServerEvent(name) {
    this.name = name || "";
    this.data = "";
};

ServerEvent.prototype.addData = function(data) {
    var lines = data.split(/\n/);

    for (var i = 0; i < lines.length; i++) {
        var element = lines[i];
        this.data += "data:" + element + "\n";
    }
}

ServerEvent.prototype.payload = function() {
    var payload = "";
    if (this.name != "") {
        payload += "id: " + this.name + "\n";
    }

    payload += this.data;
    return payload + "\n";
}

This is pretty straightforward string manipulation and won't impress anyone, but it is foundation for what will follow.

Store Events in MongoDB

We need to update the post.js code to also store new events in MongoDB.

app.put("/api/post-update", function(req, res) {
    var json = req.body;
    json.timestamp = Date.now();

    eventStorage.save(json).then(function(doc) {
        dataChannel.publish(JSON.stringify(json));
    }, errorHandling);

    res.status(204).end();
});

The event-storage module looks as follows:

var Q = require("q"),
    config = require("./config"),
    mongo = require("mongojs"),
    db = mongo(config.mongoDatabase),
    collection = db.collection(config.mongoScoresCollection);

module.exports.save = function(data) {
    var deferred = Q.defer();
    collection.save(data, function(err, doc){
        if(err) {
            deferred.reject(err);
        }
        else {
            deferred.resolve(doc);
        }
    });

    return deferred.promise;
};

Here we are just using basic MongoDB commands to save a new event into the collection. Yep that is it, we are now additionally persisting the events so they can be retrieved later.

Retrieving Events on Reconnection

When an EventSource reconnects after a disconnection it passes a special header Last-Event-Id. So we need to look for that and return the events that got broadcast while the client was disconnected.

app.get("/api/updates", function(req, res){
    initialiseSSE(req, res);

    if (typeof(req.headers["last-event-id"]) != "undefined") {
        replaySSEs(req, res);
    }
});

function replaySSEs(req, res) {
    var lastId = req.headers["last-event-id"];

    eventStorage.findEventsSince(lastId).then(function(docs) {
        for (var index = 0; index < docs.length; index++) {
            var doc = docs[index];
            var messageEvent = new ServerEvent(doc.timestamp);
            messageEvent.addData(doc.update);
            outputSSE(req, res, messageEvent.payload());
        }
    }, errorHandling);
};

What we are doing here is querying MongoDB for the events that were missed. We then iterate over them and output them to the browser.

The code for querying MongoDB is as follows:

module.exports.findEventsSince = function(lastEventId) {
    var deferred = Q.defer();

    collection.find({
        timestamp: {$gt: Number(lastEventId)}
    })
    .sort({timestamp: 1}, function(err, docs) {
        if (err) {
            deferred.reject(err);
        }
        else {
            deferred.resolve(docs);
        }
    });

    return deferred.promise;
};

Testing

To test this you will need to run both apps at the same time.

node app.js

and

node post.js

Once they are running open two browser windows http://localhost:8181/ and http://localhost:8082/api/post-update

Now you can post updates as before. If you stop app.js but continue posting events, when you restart app.js within 10 seconds the EventSource will reconnect. This will deliver all missed events.

Conclusion

This very simple code gives you a very elegant and powerful push architecture to create real-time apps.

Improvements

A possible improvement would be to render the events from MongoDB server-side when the page is first output. Then we would get updates client-side as they are pushed to the browser.

Download

If you want to play with this application you can fork or browse it on GitHub.

By Simon Baynes