UI with ReactJS and ImmutableJS

The reader is assumed to be familiar with kent/src and its CGI programs, and have a working knowledge of Javascript.

Background

Evolution of Javascript in kent/src

Before JS: just the CGI

• complete page render for every click
• the browser's back button worked because each page's content was static
• simple top-down thinking about page content, but low performance to hit server and redraw entire page every click

JS inlined in HTML

• onClick="..." enabled some client-side actions... but escaping inlined javascript inside printf'd HTML strings gets ugly quick
• the browser's back button usually still worked as expected, but not always

AJAX and JQuery: manipulating the DOM

• using jQuery made it much easier to inspect and change the DOM
• using javascript files enabled real code to be written, not just inlined statements
• using jQuery supported some bad habits such as using DOM elements as global variable storage
• authors of javascript code tended not to adhere to kent/src coding standards... it got ugly.
• after JS changes the page, the back button behavior is almost never what you'd expect

MV*

Meanwhile, most of the Javascript / front-end community was working with a more structured paradigm: Model-view-controller (MVC). That separates roles into modeling the application in terms of data structures and logic, presenting a graphical view (think HTML / CSS), and responding to input from the user. Many libraries and frameworks arose to support MVC (or more broadly, MV*) architectures. TodoMVC offers a platform for investigating and comparing the more popular frameworks by hosting each framework's implementation of the same simple todo-list app.

An abandoned experiment

After hearing good things about MV* and BackBone.js from Brian Craft, and comparing several MV* frameworks' code in TodoMVC, I decided to build a dynamic single-page app using Backbone's Model and View classes. Based on an evaluation of templating systems conducted by LinkedIn's engineering team, I chose dust.js for generating HTML from templates (compiled to JS by dust).

I made a static page htdocs/hgAi/index.html, various dust templates for dynamically generated HTML, and model & view subclasses to implement clade/genome/db selection, position search w/autocomplete, and a list of sections for selecting and configuring data sources. At first I was very pleased by the separation of model code from view code, but as the app structure became more complicated, maintaining parallel collections of submodels and subviews that changed in response to UI events became rather complex and unwieldy.

Starting with a static file, and filling in everything including the nav bar after an initial ajax request, caused things to jump around unpleasantly on the page when starting up. We already have code in place to spit out the nav bar immediately; no harm in using it, just for the sake of avoiding any HTML written by C code.

ReactJS & JSX

Simplicity of top-down render, made efficient

React home page: http://facebook.github.io/react/

Video of JSConf presentation about the thinking behind React: https://www.youtube.com/watch?v=DgVS-zXgMTk (HT Brian Craft)

JSX: HTML-ish with the power of JS

http://facebook.github.io/react/docs/jsx-in-depth.html

Special methods of React components

When constructing a new React component, at a minimum you must provide a render method that returns a React component object (e.g. a React div containing all of your cool elements). There are several other methods or properties with special meaning for React. For more info, see the React doc: http://facebook.github.io/react/docs/component-specs.html

React mixins

http://facebook.github.io/react/docs/reusable-components.html#mixins

The Flux architecture: why not?

React is awesome for rendering in the browser and receiving browser events, but it says nothing about how the rest of the system should be architected. It provides the V in MVC, but M and C are up to you.

The React team has been espousing an architecture called Flux: http://facebook.github.io/flux/docs/overview.html It focuses on a unidirectional flow of control and data. Instead of model and view, Flux has ActionCreator, Action, Dispatcher, Store and View. This division leads to several distinct objects for each type of data that will feed into the view. I find it harder to reason about than a model that passes data to the view and receives events from the view, and I'm not the only one: a significant portion of the questions on the ReactJS Google Group are not about React per se but are about Flux. Also, there are several subtly different implementations. There may be some advantages of isolating so many roles as objects for social apps with multiple asynchronous incoming streams of data, but for a user-driven querying interface, I think Flux is overly complex.

ImmutableJS

Video in which David Nolen describes the benefits of using efficiently implemented immutable data structures with React: https://www.youtube.com/watch?v=DMtwq3QtddY (HT Brian Craft)

• simplify reasoning about state changes; avoid bugs caused by out-of-sync mutable state
• very fast change detection (=== instead of traversing objects) to avoid re-rendering when unnecessary
• super-easy implementation of undo/redo

The new architecture

The new UI architecture is very simple: on the client (web browser), there is a model that is responsible for maintaining the entire state of the user interface, communicating with the server/CGI, and triggering the view to render, and a view that renders the model's state to the DOM and calls the model back when the user does something.

The CGI's role shifts from doing most of the rendering to acting as a back end for the UI. When the UI needs info from the cart, database, hub etc., it requests data from the CGI. When the user makes a change, the UI tells the CGI to update the user's cart (and possibly send refreshed data).

Bootstrapping: CGI prints minimal HTML, Javascript takes it from there

Instead of writing out HTML for the complete initial page display, the CGI now writes out just the toolbar, an empty <div> to be filled in by React components, and <script src="..."> tags to include the necessary javascript files. For performance, it's also a good idea to add a <script> that creates a javascript variable containing initial state so the the React code doesn't need to do an ajax request before it can display the initialized components.

The final javascript file instantiates the model, which in turn calls the view's top-level render method. Subsequent ajax responses or user actions invoke the model, which updates the state and in turn invokes the view to re-render.

See doMainPage in hgAi.c for an example.

ImModel: monolithic UI state model using ImmutableJS

The model's representation of the UI state is an immutable tree of objects, lists and scalars (strings/numbers). Every time the state changes in response to incoming ajax data or a user action, a completely new immutable data structure is created -- but thanks to the ImmutableJS library, this is done in a very space-efficient manner. This allows us to keep a stack of successive UI states, making undo/redo extremely easy to implement. Also thanks to the use of efficiently implemented immutable data structures, a simple === comparison will tell if there has been a change in any descendant of that node, so React components can avoid re-rendering if none of the state that affects them has changed.

After advancing to a new immutable state, the model calls the top-level React component's render function, passing in its immutable state data structure, and the user sees the updated UI.

ImModel is just a base class; an app model is a subclass of ImModel. ImModel provides a constructor function and several utility methods (e.g. registration and dispatching of handler functions, undo/redo, ajax requests, error reporting). Subclasses add handler functions for specific ajax response keys and UI events, and register those handlers in an initialize method. Subclasses can also add mixins (more below).

Building an app model on top of ImModel

Handling user actions

To handle a particular user action such as clicking a checkbox or changing a selection from human to mouse, an app model defines a handler function and registers it with this.registerUiHandler, a method provided by ImModel. The handler function takes one or two arguments: a path (i.e. a sequence of identifiers and optionally keywords that identifies the particular input/action) and, if applicable, the new value. For example:

var MyAppModel = ImModel.extend({
...
    changeOrg: function(path, newOrg) {
        // Update the state data structure and notify the server
    },
...
    initialize: function() {
    ...
        this.registerUiHandler(['org'], this.changeOrg);

When the user clicks a button or changes the value of an input, the view's React components call the model's update method with a path and the changed value if applicable. The ImModel base class provides an update method that dispatches to one or more handlers; it calls a handler if the incoming path matches the path with which the handler was registered.

Handling AJAX responses

ImModel dispatches ajax responses by the response object's top-level keys (no path). If no handler is registered for a key, the key and its value are added to the app's state. The app model can register handlers if more action is required when a particular item arrives from the server. For example:

   handleCartVar: function(cartVar, newValue) {
       //  Some updates have side effects...
       if (cartVar === 'trackDbInfo') {
            // Change default track too

   initialize: function() {
       ...
       this.registerCartVarHandler(['trackDbInfo', 'tableFields'], this.handleCartVar);

Suspension of immutability for performance (and easy undo)

It's common for an ajax response or user action to affect multiple page elements. For example, if the user changes database, not only the database select input but also the group/track/table options will change. That means that multiple parts of the immutable state tree will change in response to a single action that should correspond to a single undo/redo step. While ImmutableJS has an efficient implementation of immutable objects, its authors recognize that it's needlessly inefficient to create a series of 6 immutable objects for a single batch of 6 changes, so they provide an interface for making an object temporarily mutable. ImModel handles the details of this; the upshot for subclasses is that their handler functions modify the mutable incarnation of state, this.mutState, and don't touch the actual state. For example:

        this.mutState.set('species', newSpecies);

View: React components using ImmutableJS

The view is a hierarchy of React components; at the leaves are strings of text or React virtual DOM elements such as div, img, input etc. The render function returns a tree of React components constructed according to the immutable state data structure passed in from the model; in turn, those components' render methods are called if their part of the data structure has changed. Ultimately, React detects which DOM elements need to change, and changes only those.

Every component that has at least one possible user action receives two special properties: update, a function for notifying the model of any change, and path, a list of indices within the state data structure to the component's state, optionally followed by keywords to indicate the kind of action. Every user action has a corresponding callback which calls update with a possibly augmented path and optional data.

Avoid internal state when possible

Most view components have no internal state, and make no changes in direct response to user actions; they rely on the model to handle everything. One exception is TextInput; while the user is typing, instead of having a complete rendering cycle, it maintains its own state and notifies the model of the new value only when the user is done typing. Another exception is any component that uses a JQueryUI widget such as Autocomplete or Sortable; JQuery works by directly changing the DOM, so it completely avoids the model and React render cycle. When a widget completes an action, the view tells the widget to cancel its DOM changes and notifies the model of the change. Then the model triggers the usual React render.

Using UI state from ImModel

The model passes its immutable state data structure to the top-level React component's render method via the prop appState. Since appState is a tree of ImmutableJS objects (as opposed to plain old JS objects), it is traversed and accessed using ImmutableJS methods. This does bulk up the code a bit, and trying to access a plain old object property instead of using a method can result in silent bugs, i.e. you get undefined from the object property, which might be an acceptable result from using the method.

Below is an illustration of how plain old JS object dereferences translate to ImmutableJS methods. First, here are some example JS object dereferences:

// If we were using plain old JS objects for state (but we aren't!)
var db = appState.db;
var firstTrack = appState.trackDbInfo.trackList[0];

Here is how we use ImmutableJS methods instead:

// Here's what our React components need to do:
var db = appState.get('db');
var firstTrack = appState.getIn(['trackDbInfo', 'trackList', 0]);

Often, a branch of the appState tree will naturally correspond to a React component instantiation. In that case, it's appropriate to pass down just that branch as a prop instead of appState:

render: function() {
    return (
    ...
               <CladeOrgDb menuData={appState.get('cladeOrgDb')}
                           ...

Passing user actions to ImModel

When the user clicks on a button, changes a select input, etc., it's up to the model to change the UI state and trigger a re-render. The React component is notified of user actions by event callbacks:

              <input type='button' value='Add' onClick={this.onAdd} />

In turn, the callback method needs to notify the model. This is done by calling the function this.props.update with a path (list of property names and/or index numbers optionally followed by keywords) that uniquely identifies the UI element and the kind of action. Each component is instantiated with special props update and path, and uses those to alert the model. For example, for a click on the simple button above, the event handler calls update with a path that begins with the path passed down from above and ends with a keyword that tells the model what particular button was pushed:

   onAdd: function() {
       // Send path + 'addThingie' to app model
       this.props.update(this.props.path.concat('addThingie'));
   },

Sometimes data needs to be extracted from the (React virtual) event object and passed along with the path. For example, when the user changes a select input (with full path passed down from above):

   onChange: function(e) {
       var newValue = e.target.value;
       this.props.update(this.props.path, newValue);
   },

The CGI speaks JSON now

One nice feature of Javascript is its object literal notation because it is able to express nested data structures reasonably compactly and, with pretty-printing, in a fairly human-readable form. Javascript data structures without loops or function references can easily be serialized into JSON, a more restricted notation that is easy to parse. That makes JSON an ideal interchange format for data that's not too big when working with Javascript and some other language, in our case C.

The UI model sends JSON-encoded requests to the server/CGI, and expects JSON responses. For example, if the user selects a different clade, the model formulates this command as a Javascript object:

{ changeClade: { newValue: "rodent" } }

js/model/lib/cart.js makes a request to the CGI with the param cjCmd set to the CGI-encoded JSON string representing that object. Now the CGI needs to

1. recognize the cjCmd param -- that means output will be JSON, not HTML or text
2. decode and parse cjCmd's JSON value into a C data structure
3. act on the command(s) as specified in the data structure and write out any reponse data as JSON.

The new lib module hg/inc/cartJson.h provides a function cartJsonExecute that does part 2 (using src/inc/jsonParse.h) and also part 3 for common commands like changing clade/genome/database or retrieving track & table menu data (using hdb, cart, trackDb etc and src/inc/jsonWrite.h). (For a list of those common commands, see the source code for cartJsonNew.) It also pushes & pops a warning handler that accumulates warning messages into a string that will be added to the response JSON, e.g. warning: "ftp server timed out etc...", that the UI can then present to the user or not. CGIs create a cartJson object, and can plug in their own part 3's by writing handler functions with this signature:

typedef void CartJsonHandler(struct cartJson *cj, struct hash *paramHash);
/* Implementation of some command; paramHash associates parameter names with
 * jsonElement values. */

and then registering them with cartJsonRegisterHandler.

Here is an example use of cartJson by a CGI that needs one app-specific command, getThingie. First the CGI defines a handler function:

static void getThingie(struct cartJson *cj, struct hash *paramHash)
/* Write JSON for an object with some info */
{
char *foo = cartJsonOptionalParam(paramHash, "foo");
// ... compute ... use cj->cart if we need the cart ...
jsonWriteObjectStart(cj->jw, "thingie");
jsonWriteString(cj->jw, "title", fooDerivative);
jsonWriteString(cj->jw, "url", etc);
jsonWriteObjectEnd(cj->jw);
}

Then when the CGI detects the special parameter cjCmd, it creates a cartJson object, registers the handler function, and calls cartJsonExecute to do the rest (including printing out the Content-Type header).

static void doCartJson()
/* Perform UI commands to update the cart and/or retrieve cart vars & metadata. */
{
struct cartJson *cj = cartJsonNew(cart);
cartJsonRegisterHandler(cj, "getThingie", getThingie);
cartJsonExecute(cj);
}

void doMiddle(struct cart *theCart)
/* Depending on invocation, either respond to an ajax request or display the main page. */
{
cart = theCart;
if (cgiOptionalString(CARTJSON_COMMAND))
    doCartJson();
else
    doMainPage();
}