View source for UI with ReactJS and ImmutableJS

The reader is assumed to be familiar with kent/src and its CGI programs, and have a working knowledge of Javascript.

= Background =

== Evolution of Javascript in kent/src ==

=== Before JS: just the CGI ===

* complete page render for every click
* the browser's back button worked because each page's content was static
* simple top-down thinking about page content, but low performance to hit server and redraw entire page every click

=== JS inlined in HTML ===

* onClick="..." enabled some client-side actions... but escaping inlined javascript inside printf'd HTML strings gets ugly quick
* the browser's back button usually still worked as expected, but not always

=== AJAX and JQuery: manipulating the DOM ===

* using jQuery made it much easier to inspect and change the DOM
* using javascript files enabled real code to be written, not just inlined statements
* using jQuery supported some bad habits such as using DOM elements as global variable storage
* authors of javascript code tended not to adhere to kent/src coding standards... it got ugly.
* after JS changes the page, the back button behavior is almost never what you'd expect

== MV* ==

Meanwhile, most of the Javascript / front-end community was working with a more structured paradigm: [http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller Model-view-controller (MVC)].  That separates roles into modeling the application in terms of data structures and logic, presenting a graphical view (think HTML / CSS), and responding to input from the user.  Many libraries and frameworks arose to support MVC (or more broadly, MV*) architectures.  [http://todomvc.com/ TodoMVC] offers a platform for investigating and comparing the more popular frameworks by hosting each framework's implementation of the same simple todo-list app.

=== An abandoned experiment ===

After hearing good things about MV* and BackBone.js from Brian Craft, and comparing several MV* frameworks' code in TodoMVC, I decided to build a dynamic single-page app using Backbone's Model and View classes.  Based on an evaluation of templating systems conducted by LinkedIn's engineering team, I chose dust.js for generating HTML from templates (compiled to JS by dust).

I made a static page htdocs/hgAi/index.html, various dust templates for dynamically generated HTML, and model & view subclasses to implement clade/genome/db selection, position search w/autocomplete, and a list of sections for selecting and configuring data sources.  At first I was very pleased by the separation of model code from view code, but as the app structure became more complicated, maintaining parallel collections of submodels and subviews that changed in response to UI events became rather complex and unwieldy.  

Starting with a static file, and filling in everything including the nav bar after an initial ajax request, caused things to jump around unpleasantly on the page when starting up.  We already have code in place to spit out the nav bar immediately; no harm in using it, just for the sake of avoiding any HTML written by C code.  

== ReactJS & JSX ==

=== Simplicity of top-down render, made efficient ===

React home page: http://facebook.github.io/react/

Video of JSConf presentation about the thinking behind React: https://www.youtube.com/watch?v=DgVS-zXgMTk  (HT Brian Craft)

=== JSX: HTML-ish with the power of JS ===
http://facebook.github.io/react/docs/jsx-in-depth.html

=== Special methods of React components ===

When constructing a new React component, at a minimum you must provide a <code>render</code> method that returns a React component object (e.g. a React div containing all of your cool elements).  There are several other methods or properties with special meaning for React.  For more info, see the React doc: http://facebook.github.io/react/docs/component-specs.html

=== React mixins ===

http://facebook.github.io/react/docs/reusable-components.html#mixins

=== The Flux architecture: why not? ===
React is awesome for rendering in the browser and receiving browser events, but it says nothing about how the rest of the system should be architected.  It provides the V in MVC, but M and C are up to you.

The React team has been espousing an architecture called Flux: http://facebook.github.io/flux/docs/overview.html  It focuses on a unidirectional flow of control and data.  Instead of model and view, Flux has ActionCreator, Action, Dispatcher, Store and View.  This division leads to several distinct objects for each type of data that will feed into the view.  I find it harder to reason about than a model that passes data to the view and receives events from the view, and I'm not the only one: a significant portion of the questions on the [https://groups.google.com/forum/#!forum/reactjs ReactJS Google Group] are not about React per se but are about Flux.  Also, there are several subtly different implementations.  There may be some advantages of isolating so many roles as objects for social apps with multiple asynchronous incoming streams of data, but for a user-driven querying interface, I think Flux is overly complex.

== ImmutableJS ==
Video in which David Nolen describes the benefits of using immutable data structures with React:  https://www.youtube.com/watch?v=DMtwq3QtddY  (HT Brian Craft)

= The new architecture =

== Bootstrapping: the CGI prints minimal HTML ==

Instead of writing out HTML for the complete initial page display, the CGI now writes out just the toolbar, <code>&lt;script src="..."&gt;</code> tags to include the necessary javascript files, and an empty <code>&lt;div&gt;</code> to be filled in by React components.  For performance, it's also a good idea to add a <code>&lt;script&gt;</code> that creates a javascript variable containing initial state so the the React code doesn't need to do an ajax request before it can display the initialized components.

See doMainPage in [http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=blob;f=src/hg/hgAi/hgAi.c hgAi.c] for an example.

== ImModel: monolithic UI state model using ImmutableJS ==

=== Handling user actions ===

=== Handling AJAX responses ===

=== Undo and redo ===

=== ImModel mixins ===


== View: React components using ImmutableJS ==

=== Avoid internal state when possible ===

=== Using UI state from ImModel ===

=== Passing user actions to ImModel ===

=== JQueryUi: oil & water with React, but still too useful to drop ===


== The CGI speaks JSON now ==

One nice feature of Javascript is its object literal notation because it is able to express nested data structures reasonably compactly and, with pretty-printing, in a fairly human-readable form.  Javascript data structures without loops or function references can easily be serialized into [http://www.json.org/ JSON], a more restricted notation that is easy to parse.  That makes JSON an ideal interchange format for data that's not too big when working with Javascript and some other language, in our case C.  

The UI model sends JSON-encoded requests to the server/CGI, and expects JSON responses.  For example, if the user selects a different clade, the model formulates this command as a Javascript object:

 { changeClade: { newValue: "rodent" } }

js/model/lib/cart.js makes a request to the CGI with the param <code>cjCmd</code> set to the CGI-encoded JSON string representing that object.  Now the CGI needs to 
# recognize the <code>cjCmd</code> param -- that means output will be JSON, not HTML or text
# decode and parse <code>cjCmd</code>'s JSON value into a C data structure
# act on the command(s) as specified in the data structure and write out any reponse data as JSON.

The new lib module hg/inc/cartJson.h provides a function <code>cartJsonExecute</code> that does part 2 (using src/inc/jsonParse.h) and also part 3 for common commands like changing clade/genome/database or retrieving track & table menu data (using hdb, cart, trackDb etc and src/inc/jsonWrite.h).  (For a list of those common commands, see the source code for <code>cartJsonNew</code>.)  It also pushes & pops a warning handler that accumulates warning messages into a string that will be added to the response JSON, e.g. <code>warning: "ftp server timed out etc..."</code>, that the UI can then present to the user or not.  CGIs create a cartJson object, and can plug in their own part 3's by writing handler functions with this signature:

 typedef void CartJsonHandler(struct cartJson *cj, struct hash *paramHash);
 /* Implementation of some command; paramHash associates parameter names with
  * jsonElement values. */

and then registering them with <code>cartJsonRegisterHandler</code>.  

Here is an example use of cartJson by a CGI that needs one app-specific command, getThingie.  First the CGI defines a handler function:

 static void getThingie(struct cartJson *cj, struct hash *paramHash)
 /* Write JSON for an object with some info */
 {
 char *foo = cartJsonOptionalParam(paramHash, "foo");
 // ... compute ... use cj->cart if we need the cart ...
 jsonWriteObjectStart(cj->jw, "thingie");
 jsonWriteString(cj->jw, "title", fooDerivative);
 jsonWriteString(cj->jw, "url", etc);
 jsonWriteObjectEnd(cj->jw);
 }

Then when the CGI detects the special parameter <code>cjCmd</code>, it creates a cartJson object, registers the handler function, and calls <code>cartJsonExecute</code> to do the rest (including printing out the Content-Type header).

 static void doCartJson()
 /* Perform UI commands to update the cart and/or retrieve cart vars & metadata. */
 {
 struct cartJson *cj = cartJsonNew(cart);
 cartJsonRegisterHandler(cj, "getThingie", getThingie);
 cartJsonExecute(cj);
 }
 
 void doMiddle(struct cart *theCart)
 /* Depending on invocation, either respond to an ajax request or display the main page. */
 {
 cart = theCart;
 if (cgiOptionalString(CARTJSON_COMMAND))
     doCartJson();
 else
     doMainPage();
 }