A couple weeks ago I was looking at file uploads with Node, so I had a small test app lying around that seemed perfect for a comparison test. Here’s the code on GitHub. The instructions I used for Heroku are here, and the steps I followed for Nodejitsu are here.

Generally, the process for deploying your app on either platform looks like this:
1. User pushes code to cloud
2. Cloud service reads package.json file
3. Dependancies are managed (`npm install` step)
4. Environment is set up
5. App is started

Push
Just like their Rails deployments, Heroku requires that you have your code commit-ed to a local repo and then you push changes to Heroku (`git push heroku master`). Nodejitsu does this slightly differently and requires that you install their ‘jitsu’ app (`npm install jitsu -g`), but this means you don’t have to commit before deploying. At any time in your development you can just type `jitsu deploy` and expect your app to be pushed to the cloud. Jitsu also modifies your package.json with every deployment so they can do snapshotting. This is kind of annoying, but I think there’s a way to turn it off.

package.json
There are slight differences in how the package.json is used on the two platforms. Nodejitsu uses the package.json in the npm standard way allowing you to specify dependencies, bundledDependencies, and the start script. You can use it to select node version to run by adding an ‘engine’ block but you can’t specify which version of npm to use, which has some importance.

Dependencies
I had some trouble with dependencies on Nodejitsu. Their engine couldn’t find one of the modules in my test project. I got some really great support from #nodejitsu on IRC and one of their support engineers figured out the problem for me. It turns out the latest version of npm doesn’t like symlinks in modules so the index.js file had to be copied in place. You’ll notice in the github repo that the node_modules directory is uncharacteristically checked in. This is because I had to make this unfortunate change to the connect-form module. I happened to be running a slightly older version of npm on my dev boxes so things worked fine, but Nodejitsu is cutting edge and the workaround was necessary. This all worked fine on Heroku, so either they’re running an older version of npm like I am, or they have some other workaround.

Environment
Heroku only lets your app run on one port specified in their environment. To listen to the port, you have to do something like this : “var port = process.env.PORT || 4000;”. This is probably something you should be doing anyway, but one of node.js’s greatest features is listening to several ports and piping data between them. Being limited to one port does limit the type of apps that can be deployed there. Nodejitsu lets you listen to any number of ports you like and you can happily hard-code them into your application without trouble. Also, Nodejitu has support for websockets, which is missing from Heroku and critical for me. Since most apps that use websockets with node are using more than one port as a workaround for nginx’s missing websocket support, these apps will need some minor refactoring to be done to deploy on Heroku (that is, when Heroku supports websockets).

Startup
The script startup is where I like Nodejitsu a bit better. Just run `npm start` and the script will start as you expect. Heroku requires that you use the Foreman gem (`gem install foreman`) which seems out of place for node.js development. You then create a file named “Procfile” and specify your startup script there “web: node foo.js”. Unless you are doing ruby development in parallel with node.js, you may not have gem installed (and correctly configured using RVM or rbenv like you should). There’s a node version of Foreman called Forewoman that might be a better choice, but just using the standard npm “script”:{“start”:”node foo.js”} block would be my first choice.

Overall
This sample app was deployed to Heroku in less than 5 minutes. I’m clearly more familiar with Heroku than I am with Nodejitsu, but it took me several hours to work through my dependency troubles on Nodejitsu and that was with professional help. Even so, I think Nodejitsu is the way to go. They are clearly dedicated to the node.js community, provided fantastic support, and offer the best features.

Forward
Now, there is a new player that should be considered. RedHat’s new OpenShift platform is another option that deserves some further reading. The advantage of being able to run a bunch of different stacks all in one place might make it the choice for legacy apps deployed in the cloud.

//config.js
conf = {
    port: 8000,
    name: 'app name',
    otherSetting: 'etc.',
};

if( typeof module !== "undefined" && ('exports' in module)){
    module.exports = function(){return conf;};
} else {
    callback( conf );
}

The config file can be accessed like any other CommonJS module.

//config_test.js
var configuration = require('config.js')();
console.dir( configuration );

or included as a JSONP-like call in an HTML page like this

<html>
    <head>
        <script type="text/javascript">
            function callback(data){
                alert(data);
            }
        </script>
    </head>
    <body>
        body text
        <script src="config.js" type="text/javascript"></script>
    </body>
</html>

startup_message off
term screen
defscrollback 10000
escape ^Oo
vbell off
bind h prev
bind C hardcopy
hardstatus off
bell_msg ""
shell -$SHELL

The important parts here are ‘escape ^Oo’ to move the command key to something on my right hand and ‘defscrollback 10000′ so I can find things in scrolling log messages. Moving the command key to the right hand is important to avoid the extended finger yoga generally required by frequent keyboard shortcuts. The rest are just to remove some of the annoying bells and whistles and are well documented.

RANT TO ADMINS : Don’t change the default etc/screenrc! This will annoy the hell out of anyone trying to use screen on your box. Just because YOU like colorized numbered tab lists that flash and beep when you change windows, doesn’t mean all of your users will. Do you know how much time it takes to disable all that junk in a ~/.screenrc? [/rant]

This is being served statically from Nginx without writing any server-side logic. This page could come from any machine in a cluster without needing to share state, added to an HTML5 manifest, or served from a CDN. This is the purest form of separation between data and display. The only source of dynamic data is a single ajax call.

<script type="text/javascript">
    jQuery.ajax(
        "/twitter/search.json?q=%23haiku", {
        success: function(data, textStatus, jqXHR){
            twitterData = JSON.parse( jqXHR.responseText );
            $('body').append( $('#template').jqote(twitterData) );
        }
    });
</script>

The important line here is the call to jQote (pronounced like Chicote from Star Trek: Voyager). jQote is a client side templating system in the spirit of John Resig’s now famous Micro-Template code. To use it, you create a script block with type text/html:

<script type="text/html" id="template">
...
</script>

The browser will ignore this block and not parse it as javascript. You can add javascript code and data just like you might do with a PHP, Mason, or JSP page using <% %> and <%= %> syntax. This lets you push CPU expensive page creation to the client and conserve your resources for API calls.

In this particular example, I’m pulling the content from Twitter (using a proxy to get around XSS rules), so this takes next to nothing in terms of memory and CPU. I could improve the example further by adding a manifest file and whitelisting the /twitter/* network call. Then only the first request would be served from my server and all subsequent requests would only make one call out of the browser.

YQL offers a server-side proxy service, and I’ve seen some PHP proxies on the web, but if you run Nginx it’s dead simple to make your own. I’ll use the Twitter API as an example.

Say you want to show twitter search results in a web page. You could register with Twitter and get an API key, make the search, cache the results and serve them to the user. It’s much easer to just proxy the request with Nginx like this:

http {
    include       mime.types;
    default_type  application/octet-stream;
    server {
        listen       8080;
        server_name  *.codebudo.com codebudo.com;
        location /twitter {
            proxy_pass          http://search.twitter.com/;
            proxy_redirect      http://localhost:8080 http://search.twitter.com/;
        }
    }
}

This let’s us take any request to http://codebudo.com:8080/twitter/search.json?q=%23haiku and have it mimic the results you’d get from http://search.twitter.com/search.json?q=%23haiku. So, it’s a proxy. Not a big deal, but has some really useful applications.

Drag the boxes around to see the effect.

Yahoo Pipes uses the CANVAS tag to draw Bezier curves. While this looks the best and can be hardware accelerated, it’s not supported by IE. Also, in this application I would potentially have hundreds of these links and that could get quite heavy. Canvas would also require that I write using the canvas API, which is not something many developers are acquainted with (myself included). To avoid having a large number of tags, I could include all my content in a single canvas tag and draw everything with canvas. But, then I have zero separation between my content and display which just didn’t seem maintainable.

SVG is another possible option, but has some browser compatibility issues. I would have to implement the solution differently depending on the browser rendering the page and that was more complicated than I was hoping for. I played around with it, but unfortunately IE’s lack of support and increased complexity killed this as an option. I fully expect that to change in the future.

JSDraw is a 2D drawing library that will draw hundreds of exactly positioned DIVs to draw straight lines or any mathematical function you like (Bezier included). It’s quite easy to use and has the flexibility and gives me the separation of content and display that I want. It’s main disadvantage is that is pollutes the DOM with hundreds of DIVs. This could slow down page rendering if any CSS selectors have a DIV as their rightmost element (CSS selectors are evaluated from right to left). The results aren’t bad, but when moving on a diagonal the lines become drastically aliased.

There’s also the Raphael library that has some nice examples of drawing continuously updating lines. The example is exactly what I want, but again this uses the canvas tag and that puts it outside my requirements.

This leaves the option I settled on. It’s not perfect, but using CSS border radius I could achieve an effective simulacrum of the intended curve. Setting the border radius larger than the width of the element using the Mozilla and Webkit extensions (and the W3C standard naming convention for Opera, and a behaviour URL used for IE support) results in a nearly circular radius. Combining two DIVs, one for the first half the curve and another for the second half, a “double spherical” line is created. I have no idea how to represent this mathematically, but the result is pleasing enough to the eye for my purposes.

Several CSS styles are needed to accommodate the different quadrants where a line could be drawn: upper-right, upper-left, lower-left, and lower-right. You can see the full CSS in the iframe above, but here’s an excerpt.

            .bottom_right .first_half {
                border-bottom-left-radius: 1000px;
                -webkit-border-bottom-left-radius: 1000px;
                -moz-border-radius-bottomleft: 1000px;
                border-left: 2px solid green;
                border-bottom: 2px solid green;
            }
            .bottom_right .second_half {
                border-top-right-radius: 1000px;
                -webkit-border-top-right-radius: 1000px;
                -moz-border-radius-topright: 1000px;
                border-top: 2px solid green;
                border-right: 2px solid green;
                left: 50%;
                top: -2;
            }

That’s pretty much it. With a little javascript to move a div containing these curved DIVs and to change which CSS class to use based on the difference in location between the source and destination object, the draggable DIVs can be followed by the curved lines like above. Enjoy!

The obvious solution is to compare every member to every other member and that would give you a solution that runs in O(n^2). If we’re given a function called ‘Knows’ that accepts two arguments (and returns in O(1)), we can write a PHP function like this:

function theSlowMayor(){
    $townSize = 5;

    $candidates = array();

    // 1) The mayor must know no one.
    for( $i=0; $i < $townSize; $i++ ){
        for( $j=0; $j < $townSize; $j++ ){
            if( Knows($i,$j) && $i != $j )
                break;

            if( $j == $townSize - 1)
                $candidates[] = $i;
        }
    }

    // 2) Every one must know the mayor.
    foreach( $candidates as $c ){
        for( $i=0; $i<$townSize; $i++ ){
            if( !Knows($i,$c) && $i != $c )
                break;

            if( $i == $townSize - 1 )
                return $c;
        }
    }
}

While this may work, it's hardly ideal. We need a way to disqualify a candidate with every iteration over the list. This way, with one pass over the list we can remove anyone that is not the mayor / celebrity.

function theMayor(){

    $townSize = 5;
    $mayor_set = array();
    for( $i=0; $i < $townSize; $i++ )
        $mayor_set[] = $i;

    for( $i=0; $i < count($mayor_set); $i++ ){
        for( $j=0; $j < count($mayor_set); $j++ ){
            if( $i == $j )
                continue;
            if( Knows($i, $j) ) {
                // 1) The mayor must know no one.
                unset($mayor_set[$i]);
            } else {
                // 2) Every one must know the mayor.
                // if i doens't know j, then j can't be the mayor
                unset($mayor_set[$j]);
            }
        }
    }

    return array_shift($mayor_set);
}

Reading the algorithmic complexity of this code can be tricky. Seeing a nested for loop usually means O(n^2), but in this case we are eliminating one element from the $mayor_set with every iteration so we only require one pass to find the mayor. This does require enough memory to hold the a list of identifiers for the entire town, so this algorithm may not work on very large data sets.

This question comes from the Cormin Book where 22.1-6 states:

When an adjacency-matrix representation is used, most graph algorithms require time O(V^2), but there are some exceptions. To show that determining whether a directed graph G contains a universal sink - vertex with in-degree |V|-1 and out-degree 0 - can be determined in O(V), given an adjacency matrix for G.

If you want to test the functions above, you can use this sample 'Knows' function:

function Knows($i, $j){
    $set = array();
    $set[] = array(1, 0, 1, 0, 1);
    $set[] = array(0, 1, 1, 1, 0);
    $set[] = array(0, 0, 1, 0, 0);
    $set[] = array(1, 1, 1, 1, 1);
    $set[] = array(0, 1, 1, 0, 1);
    return $set[$i][$j];
}

The data set was published by the Audubon Society Field Guide and describes the characteristics of mushrooms found in North America.  The version I’m using was compiled by the UCI Machine Learning Repository.  It contains 8124 records (one record per line) with its classification and each of the 22 mushroom characteristics represented by a character value in a comma separated list.  The first value describes the poisonous or edible classification.

p,x,s,n,t,p,f,c,n,k,e,e,s,s,w,w,p,w,o,p,k,s,u
e,x,s,y,t,a,f,c,b,k,e,c,s,s,w,w,p,w,o,p,n,n,g

JOONE requires semicolon separated numerical values for input, so I replaced each character value with its alphabetical position and changed the commas to semicolons.  Missing values were given the value 27.

16;24;19;14;20;16;6;3;14;11;5;5;19;19;23;23;16;23;15;16;11;19;21
5;24;19;25;20;1;6;3;2;11;5;3;19;19;23;23;16;23;15;16;14;14;7

The network has three layers: 22 input nodes, 10 hidden nodes, and a single output node.  If the output node is 16 (p), the mushroom is classified as poisonous.  If this node is 5 (e), it is classified as edible.  The hidden nodes and output node have a sigmoidal activation function.  The network is trained on the first 3000 elements of the data set using JOON’s built in back propagation functions and a Root Mean Squared Error (RMSE) function.  The remaining ~5124 nodes can be used in verifying the application.  Running in training batches of 10,000 iterations (epochs) and storing a serialized representation of the network to disk every 100 iterations allowed fine grained monitor the progress of the application and ensure net trained network could be recovered in the even of a crash.

Serialization is a mechanism where an object in memory is converted into a portable form (XML in this case) so it can be later retrieved and the object restored to memory exactly as it once was.  In this case, we are using the ‘serializeable’ java interface to store a neural network that contains the network diagram, weighted synapses, and trainer (error).

The error after the first 100 iterations was ~5%, and decreased to 4.25% after 50,000 iterations.  While this is rather slow, the error is still decreasing and could be within acceptable levels with a few million iterations.

Further research should be done on the design of the network and its training.  Adding another layer or changing the number of hidden nodes could converge more quickly.  The serialization mechanism could produce an easy way to distribute and parallelize the training.  If the current RMSE of the network were stored along with serialized net, a node could determine if its error is less than the current “best” for a group of nodes.  The node with the lowest error would write a new serialized net and global error file and nodes with greater error would use the least error net to continue training.

Here are the files required to continue developing this network:

MushroomFFNN.java

mushroom_numerical.data

After June 30, 2009, AvantGo will no longer be providing mobile Web content for sync or online access, and you will not be able to access or update your AvantGo content or account.  Your account information will continue to be protected by our privacy policy, and we plan to delete any personally identifiable information you’ve provided (e.g., your e-mail address) as soon as reasonably possible.

If you are an 8MB account subscriber you may be entitled to a refund for a portion of your subscription fees that are unused. To request a refund, please click here and submit the refund request form. You will need to reference your AvantGo User ID (included in this email).

To continue receiving news and information from your favorite content providers, you should visit that content provider’s channel before June 30 for details on how to obtain their content other than through AvantGo.  Also, AvantGo recommends the Snac mobile widget application – a new, fun way to get your favorite content on your mobile device. You can find out more about Snac at: http://www.mysnacs.com/landing?token=avantgo0609

Best wishes,
The AvantGo Team

The original AvantGo sync service was a pioneering effort.  It brought the web to your handheld device and allowed users to take any piece of the web they wished along for the ride.  Web applications were supported with an offline form submission system and advanced javascript hooks allowed mobile web developers to interact with their users in a useful way while they were offline.  The “HandheldFriendly” meta tag was originally developed by AvantGo to differentiate websites designed for desktop or mobile view and is now used by many browsers including Opera Mini, Opera Mobile and some BlackBerry devices.

AvantGo taught its approximately 10 million recorded users what to expect of the mobile web and defined a market for the mobile device browsers followed.  Aspects of the service’s design are sure to remain in the mobile computing field.  Many former AvantGo engineers went on to help develop Yahoo! Go, Google Gears, and other mobile content services.

If HTML 5 ever gets adopted, most of the offline browsing features will be available on a large number of browsers.  Safari 4 beta already supports the site manifest file aspects of HTML 5 for pre-downloading content (a feature that has caused some controversy over wasted bandwidth) which allows offline view of pages.

According to parent company Sybase, AvantGo will transition from a mobile web service, to an SMS advertising and content delivery system.