Wazstagram is a fun experiment with node.js on Windows Azure and the Instagram Realtime API. The project uses various services in Windows Azure to create a scalable window into Instagram traffic across multiple cities.

• View the demo on Windows Azure
• View the code on GitHub

The code I used to build WAZSTAGRAM is under an MIT license, so feel free to learn and re-use the code.

How does it work

The application is written in node.js, using cloud services in Windows Azure. A scalable set of backend nodes receive messages from the Instagram Realtime API. Those messages are sent to the front end nodes using Windows Azure Service Bus. The front end nodes are running node.js with express and socket.io.

Websites, and Virtual Machines, and Cloud Services, Oh My!

One of the first things you need to grok when using Windows Azure is the different options you have for your runtimes. Windows Azure supports three distinct models, which can be mixed and matched depending on what you're trying to accomplish:

Websites

Websites in Windows Azure match a traditional PaaS model, when compared to something like Heroku or AppHarbor. They work with node.js, asp.net, and php. There is a free tier. You can use git to deploy, and they offer various scaling options. For an example of a real time node.js site that works well in the Website model, check out my TwitterMap example. I chose not to use Websites for this project because a.) websockets are currently not supported in our Website model, and b.) I want to be able to scale my back end processes independently of the front end processes. If you don't have crazy enterprise architecture or scaling needs, Websites work great.

Virtual Machines

The Virtual Machine story in Windows Azure is pretty consistent with IaaS offerings in other clouds. You stand up a VM, you install an OS you like (yes, we support linux), and you take on the management of the host. This didn't sound like a lot of fun to me because I can't be trusted to install patches on my OS, and do other maintainency things.

Cloud Services

Cloud Services in Windows Azure are kind of a different animal. They provide a full Virtual Machine that is stateless – that means you never know when the VM is going to go away, and a new one will appear in it's place. It's interesting because it means you have to architect your app to not depend on stateful system resources pretty much from the start. It's great for new apps that you're writing to be scalable. The best part is that the OS is patched automagically, so there's no OS maintenance. I chose this model because a.) we have some large scale needs, b.) we want separation of conerns with our worker nodes and web nodes, and c.) I can't be bothered to maintain my own VMs.

Getting Started

After picking your runtime model, the next thing you'll need is some tools. Before we move ahead, you'll need to sign up for an account. Next, get the command line tools. Windows Azure is a little different because we support two types of command line tools:

• PowerShell Cmdlets: these are great if you're on Windows and dig the PowerShell thing.
• X-Platform CLI: this tool is interesting because it's written in node, and is available as a node module. You can actually just npm install -g azure-cli and start using this right away. It looks awesome, though I wish they had kept the flames that were in the first version.

For this project, I chose to use the PowerShell cmdlets. I went down this path because the Cloud Services stuff is not currently supported by the X-Platform CLI (I'm hoping this changes). If you're on MacOS and want to use Cloud Services, you should check out git-azure. To bootstrap the project, I pretty much followed the 'Build a Node.js Chat Application with Socket.IO on a Windows Azure Cloud Service' tutorial. This will get all of your scaffolding set up.

My node.js editor – WebMatrix 2

After using the PowerShell cmdlets to scaffold my site, I used Microsoft WebMatrix to do the majority of the work. I am very biased towards WebMatrix, as I helped build the node.js experience in it last year. In a nutshell, it's rad because it has a lot of good editors, and just works. Oh, and it has IntelliSense for everything:

Install the Windows Azure NPM module

The azure npm module provides the basis for all of the Windows Azure stuff we're going to do with node.js. It includes all of the support for using blobs, tables, service bus, and service management. It's even open source. To get it, you just need to cd into the directory you're using and run this command:

npm install azure

After you have the azure module, you're ready to rock.

The Backend

The backend part of this project is a worker role that accepts HTTP Post messages from the Instagram API. The idea is that their API batches messages, and sends them to an endpoint you define. Here's some details on how their API works. I chose to use express to build out the backend routes, because it's convenient. There are a few pieces to the backend that are interesting:

1. Use nconf to store secrets. Look at the .gitignore.

   If you're going to build a site like this, you are going to need to store a few secrets. The backend includes things like the Instagram API key, my Windows Azure Storage account key, and my Service Bus keys. I create a keys.json file to store this, though you could add it to the environment. I include an example of this file with the project. **DO NOT CHECK THIS FILE INTO GITHUB!** Seriously, don't do that. Also, pay **close attention** to my .gitignore file. You don't want to check in any *.cspkg or *.csx files, as they contain archived versions of your site that are generated while running the emulator and deploying. Those archives contain your keys.json file. That having been said – nconf does makes it really easy to read stuff from your config:

   // read in keys and secrets
   nconf.argv().env().file('keys.json');
   var sbNamespace = nconf.get('AZURE_SERVICEBUS_NAMESPACE');
   var sbKey = nconf.get('AZURE_SERVICEBUS_ACCESS_KEY');
   var stName = nconf.get('AZURE_STORAGE_NAME');
   var stKey = nconf.get('AZURE_STORAGE_KEY');
   

2. Use winston and winston-skywriter for logging.

   The cloud presents some challenges at times. Like *how do I get console output* when something goes wrong. Every node.js project I start these days, I just use winston from the get go. It's awesome because it lets you pick where your console output and logging gets stored. I like to just pipe the output to console at dev time, and write to Table Storage in production. Here's how you set it up:

   // set up a single instance of a winston logger, writing to azure table storage
   var logger = new (winston.Logger)({
       transports: [
           new (winston.transports.Console)(),
           new (winston.transports.Skywriter)({
               account: stName,
               key: stKey,
               partition: require('os').hostname() + ':' + process.pid
           })
       ]
   });
   
   logger.info('Started wazstagram backend');
   

3. Use Service Bus – it's pub/sub (+) a basket of kittens.

   Service Bus is Windows Azure’s swiss army knife of messaging. I usually use it in the places where I would otherwise use the PubSub features of Redis. It does all kinds of neat things like PubSub, Durable Queues, and more recently Notification Hubs. I use the topic subscription model to create a single channel for messages. Each worker node publishes messages to a single topic. Each web node creates a subscription to that topic, and polls for messages. There’s great support for Service Bus in the Windows Azure Node.js SDK.

   To get the basic implementation set up, just follow the Service Bus Node.js guide. The interesting part of my use of Service Bus is the subscription clean up. Each new front end node that connects to the topic creates it's own subscription. As we scale out and add a new front end node, it creates another subscription. This is a durable object in Service bus that hangs around after the connection from one end goes away (this is a feature). To make sure sure you don't leave random subscriptions lying around, you need to do a little cleanup:

   function cleanUpSubscriptions() {
       logger.info('cleaning up subscriptions...');
       serviceBusService.listSubscriptions(topicName, function (error, subs, response) {
           if (!error) {
               logger.info('found ' + subs.length + ' subscriptions');
               for (var i = 0; i < subs.length; i++) {
                   // if there are more than 100 messages on the subscription, assume the edge node is down
                   if (subs[i].MessageCount > 100) {
                       logger.info('deleting subscription ' + subs[i].SubscriptionName);
                       serviceBusService.deleteSubscription(topicName, subs[i].SubscriptionName, function (error, response) {
                           if (error) {
                               logger.error('error deleting subscription', error);
                           }
                       });
                   }
               }
           } else {
               logger.error('error getting topic subscriptions', error);
           }
           setTimeout(cleanUpSubscriptions, 60000);
       });
   }
   

4. The NewImage endpoint

   All of the stuff above is great, but it doesn’t cover what happens when the Instagram API actually hits our endpoint. The route that accepts this request gets metadata for each image, and pushes it through the Service Bus topic:

   serviceBusService.sendTopicMessage('wazages', message, function (error) {
   	if (error) {
           logger.error('error sending message to topic!', error);
       } else {
           logger.info('message sent!');
       }
   })
   

The Frontend

The frontend part of this project is (despite my 'web node' reference) a worker role that accepts accepts the incoming traffic from end users on the site. I chose to use worker roles because I wanted to take advantage of Web Sockets. At the moment, Cloud Services Web Roles do not provide that functionality. I could stand up a VM with Windows Server 8 and IIS 8, but see my aformentioned anxiety about managing my own VMs. The worker roles use socket.io and express to provide the web site experience. The front end uses the same NPM modules as the backend: express, winston, winston-skywriter, nconf, and azure. In addition to that, it uses socket.io and ejs to handle the client stuff. There are a few pieces to the frontend that are interesting:

1. Setting up socket.io

   Socket.io provides the web socket (or xhr) interface that we're going to use to stream images to the client. When a user initially visits the page, they are going to send a `setCity` call, that lets us know the city to which they want to subscribe (by default all cities in the system are returned). From there, the user will be sent an initial blast of images that are cached on the server. Otherwise, you wouldn't see images right away:

   // set up socket.io to establish a new connection with each client
   var io = require('socket.io').listen(server);
   io.sockets.on('connection', function (socket) {
       socket.on('setCity', function (data) {
           logger.info('new connection: ' + data.city);
           if (picCache[data.city]) {
               for (var i = 0; i < picCache[data.city].length; i++) {
                   socket.emit('newPic', picCache[data.city][i]);
               }
           }
           socket.join(data.city);
       });
   });
   

2. Creating a Service Bus Subscription

   To receive messages from the worker nodes, we need to create a single subscription for each front end node process. This is going to create subscription, and start listening for messages:

   // create the initial subscription to get events from service bus
   serviceBusService.createSubscription(topicName, subscriptionId,
       function (error) {
           if (error) {
               logger.error('error creating subscription', error);
               throw error;
           } else {
               getFromTheBus();
           }
   });
   

3. Moving data between Service Bus and Socket.IO

   As data comes in through the service bus subscription, you need to pipe it up to the appropriate connected clients. Pay special attention to `io.sockets.in(body.city)` – when the user joined the page, they selected a city. This call grabs all users subscribed to that city. The other **important thing to notice** here is the way `getFromTheBus` calls itself in a loop. There's currently no way to say "just raise an event when there's data" with the Service Bus Node.js implementation, so you need to use this model.

   function getFromTheBus() {
       try {
           serviceBusService.receiveSubscriptionMessage(topicName, subscriptionId, { timeoutIntervalInS: 5 }, function (error, message) {
               if (error) {
                   if (error == "No messages to receive") {
                       logger.info('no messages...');
                   } else {
                       logger.error('error receiving subscription message', error)
                   }
               } else {
                   var body = JSON.parse(message.body);
                   logger.info('new pic published from: ' + body.city);
                   cachePic(body.pic, body.city);
                   io.sockets. in (body.city).emit('newPic', body.pic);
                   io.sockets. in (universe).emit('newPic', body.pic);
               }
               getFromTheBus();
           });
       } catch (e) {
           // if something goes wrong, wait a little and reconnect
           logger.error('error getting data from service bus' + e);
           setTimeout(getFromTheBus, 1000);
       }
   }
   

Learning

The whole point of writing this code for me was to explore building performant apps that used a rate limited API for data. Hopefully this model can effectively be used to accept data from any API responsibly, and scale it out to a number of connected clients to a single service. If you have any ideas on how to make this app better, please let me know, or submit a PR!

Questions?

If you have any questions, feel free to submit an issue here, or find me @JustinBeckwith

I spend a lot of time at the command line. As someone who likes to code on OSX and Windows, I’ve always been annoyed by the Windows command line experience. Do I use cmd, or PowerShell? Where are my tabs? What about package management? What about little frivolous things like being able to resize the window. I’ve finally got my Windows command line experience running smoothly, and wanted to share my setup. Here are my 5 steps to a Windows command line that doesn’t suck.

1. Use Console2 or ConEmu

The first place to start is the actual console application. Scott Hanselman wrote an excellent blog post on setting up Console2, and I’ve been using it ever since. It adds tabs, a resizable window, transparency, and the ability to run multiple shells. I choose to run PowerShell (you should too, keep listening). There are other options out there, but I’ve really grown to love Console2.

2. Use PowerShell

I won’t spend a ton of time evangelizing PowerShell. There are a few good reasons to dump cmd.exe and move over:

• Most of the things you do in cmd will just work. There are obviously some exceptions, but for the better part all of the things I want to do in cmd are easily done in PowerShell.
• Tab Completion and Get-Help is awesome. PowerShell does a great job of making things discoverable as you learn.
• It’s a sane scripting tool. If you’ve ever tried to do anything significant in a batch script, I’m sorry. You can even create your own modules and cmdlets using managed code, if that’s your thing.
• Microsoft is releasing a lot of stuff built on PowerShell. Most of the new stuff we release is going to have great PowerShell support, including Windows Azure.
• It’s a growing community. Sites like PowerShell.org and PsGet provide a great place to ask questions and look at work others have done.

Now that I’ve sold you, there are a few things you’ll find through here that make using PowerShell a bit easier. To use this stuff, you’re going to want to set an execution policy in PowerShell that lets you run custom scripts. By default, the execution of PS scripts is disabled, but it’s kind of necessary to do anything interesting. I lead a wild and dangerous life, so I use an unrestricted policy. To set your policy, first run Console2 (or PowerShell) as an administrator:

Next, use the Set-ExecutionPolicy command. Note, this means any un-signed script can be run on your system, if you run it, and many people choose to use RemoteSigned. Here is the official doc on Set-ExecutionPolicy.

Set-ExecutionPolicy Unrestricted

Now you’re ready to start doing something interesting.

3. Use the Chocolatey package manager

Spending a lot of time in Ubuntu and OSX, I got really used to `sudo apt-get install ` and `brew install `. The closest I’ve found to that experience on Windows is the Chocolatey package manager. Chocolatey has all of the packages you would expect to find on a developer’s machine:

To install Chocolatey, just run cmd.exe and run the following command (minus the c:\> part):

C:\> @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://bit.ly/psChocInstall'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin

And you’re ready to rock. If you want to install something like 7zip, you can use the cinst command:

cinst 7zip

4. Use an alias for SublimeText

This seems kind of trivial, but one of the things I’ve really missed on Windows is the default shortcut to launch SublimeText, subl. I use my PowerShell profile to create an alias to SublimeText.exe, which allows me to `subl file.txt` or `subl .` just like I would from OSX. This article gives a basic overview on how to customize your PowerShell Profile; it’s really easy to follow, so I won’t go into re-creating the steps.

After you’ve got your PowerShell profile created, edit the script, and add this line:

Set-Alias subl 'C:\Program Files\Sublime Text 2\sublime_text.exe'

Save your profile, and spin up a new PowerShell tab in Console2 to reload the session. Go to a directory that contains some code, and try to open it:

subl .

This will load the current directory as a project in SublimeText from the command line. Small thing, but a nice thing.

5. Use PsGet and Posh-Git

One of the nice things about using PowerShell over cmd is the community that’s starting to emerge. There are a ton of really useful tools and cmdlets that others have already written, and the easiest way to get at most of these is to use PsGet. PsGet provides a super easy way to install PowerShell modules that extend the basic functionality of the shell, and provide other useful libraries. To install PsGet, run the following command from a PowerShell console:

(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex

If you get an error complaining about executing scripts, you need to go back to #2. Immediately, we can start using the `Install-Module` command to start adding functionality to our console.

The first module that led me to PsGet is a package that adds status and tab completion to git. Phil Haack did a great write up on setting up posh-git, and I’ve since discovered a few other cool things in the PsGet gallery. Installing Posh-Git is pretty straight forward:

The first nice thing here is that I now have command completion. As I type `git sta` and hit , it will be completed to `git status`. Some tools like posh-npm will even search the npm registry for packages using tab completion. The other cool thing you get with this module is the status of your repository right in the prompt:

Wrapping up

These are just the ways I know how to make the command line experience better. If any one else has some tips, I’d love to hear them!

A few months ago, we introduced the new node.js features we’ve added to WebMatrix 2. One of the missing pieces from that experience was a way to manage NPM (Node Package Manager) from within the IDE.

This week we shipped the final release of WebMatrix 2, and one of the fun things that comes with it is a new extension for managing NPM. For a more complete overview of the WebMatrix 2, check out Vishal Joshi’s blog post.

If you want to skip all of this and just download the bits, here you go:

Installing the Extension

The NPM extension can be installed using the extension gallery inside of WebMatrix. To get started, go ahead and create a new node site with express using the built in template:

After you create the site, click on the ‘Extensions’ button in the ribbon:

Search for ‘NPM’, and click through the wizard to finish installing the extension:

Now when you navigate to the files workspace, you should see the new NPM icon in the ribbon.

Managing Packages

While you’re working with node.js sites, the icon should always show up. To get started, click on the new icon in the ribbon:

This will load a window very similar to the other galleries in WebMatrix. From here you can search for packages, install, uninstall, update, any of the basic tasks you’re likely to do day to day with npm.

When you open up a new site, we also check your package.json to see if you’re missing any dependencies:

We’re just getting started with the node tools inside of WebMatrix, so if you have anything else you would like to see added please hit us up over at UserVoice.

More Information

If you would like some more information to help you get started, check out some of these links:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Open Source Applications
• Vishal Joshi’s blog post

Happy Coding!

After releasing WebMatrix 2 RC this week, I’m excited to head out to NYC for WordCamp 2012. While I get ready to present tomorrow, I figured I would share some of the amazing work the WebMatrix team has done to create a great experience for WordPress developers. For a more complete overview of the WebMatrix 2 RC, check out Vishal Joshi’s blog post.

If you want to skip all of this and just download the bits, here you go:

Welcome to WebMatrix

WebMatrix gives you a couple of ways to get started with your application. Anything we do is going to be focused on building web applications, with as few steps as possible. WebMatrix supports opening remote sites, opening local sites, creating new sites with PHP, or creating an application by starting with the Application Gallery.

The Application Gallery

We work with the community to maintain a list of open source applications that just work with WebMatrix on the Windows platform. This includes installing the application locally, and deploying to Windows Server or Windows Azure:

Install PHP and MySQL Automatically

When you pick the application you want to install, WebMatrix knows what dependencies need to be installed on your machine. This means you don’t need to set up a web server, install and configure MySQL, mess around with the MySQL command line – none of that. It all just happens auto-magically.

The Dashboard

After installing WordPress and all of it’s dependencies, WebMatrix provides you with a dashboard that’s been customized for WordPress. We open up an extensibility model that makes it easier for open source communities to plug into WebMatrix, and we’ve been working with several groups to make sure we provide this kind of experience:

Protected Files

When you move into the files work space, you’ll notice a lock file next to many of the files in the root. We worked with the WordPress community to define a list of files that are protected in WordPress. These are files that power the core of WordPress, and probably shouldn’t be changed:

We won’t stop you from editing the file, but hopefully this prevents people from making mistakes:

HTML5 & CSS3 Tools

The HTML editor in WebMatrix has code completion, validation, and formatting for HTML5. The editor is really, really good. The CSS editor includes code completion, validation, and formatting for CSS3, including the latest and greatest CSS3 modules. We also include support for CSS preprocessors like LESS and Sass.

I think my favorite part about the CSS editor is the way it makes dealing with color easier. If you start off a color property, WebMatrix will look at the current CSS file, and provide a palette built from the other colors used throughout your site. This prevents you from having 17 shades of the mostly same color blue:

If you want to add a new color, we also have a full color picker. This thing is awesome – my favorite part is the eye dropper that lets you choose colors in other applications.

PHP Code Completion

When you’re ready to start diving into PHP, we include a fancy new PHP editor. It provides code completion with documentation from php.net, and a lot of other little niceties that make writing PHP easier:

WordPress Code Completion

So you’ve written some PHP, but now you want to start using the built-in functions available in WordPress. We worked with the WordPress community to come up with a list of supported functions, along with documentation on how they work. Any open source application in the gallery can provide this kind of experience:

MySQL Database Editor

If you need to make changes directly to the database, WebMatrix has a full featured MySQL editor built right into the product. You can create tables, manage keys, or add data right through the UI. No command line needed.

Remote Editing

If you need to make edits to a live running site, we can do that to. Just enter your connection information (FTP or Web Deploy), and you can start editing your files without dealing with a FTP client:

After you make your changes, just save the file to automatically upload it to your server:

Easy Publishing

When you’re ready to publish your application, you have the choice of using FTP or Web Deploy. If you use Web Deploy, we can even publish your database automatically along with the files in your WordPress site. When you make subsequent publish calls, only the changed files are published:

More Information

If you would like some more information to help you get started, check out some of these links:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Open Source Applications
• Vishal Joshi’s blog post

Happy Coding!

After months of hard work by the WebMatrix team, it’s exciting to introduce the release candidate of WebMatrix 2. WebMatrix 2 includes tons of new features, but today I want to give an overview of the work we’ve done to enable building applications with Node.js.

If you want to skip all of this and just get a download link (it’s free!), here you go.

How far we have come

Less than a year ago, I was working at Carnegie Mellon University, trying to use Node.js with ASP.NET for real time components of our online learning environment. Running Linux inside of our customers’ data centers was a non-starter, and running a production system in cygwin was even less ideal. Developing node on Windows wasn’t exactly easy either – if you managed to get node running, getting NPM to work was near impossible. Using node in an environment favorable to Windows was more than an up hill battle.

In the last 12 months since I’ve joined Microsoft, we’ve seen various partnerships between Joyent and Microsoft, resulting in new releases of node and npm to support Windows, and a commitment to Node on Windows Azure. We’ve worked together to build a better experience for developers, IT administrators, and ultimately, the users who use our systems.

One of the results of that work is a vastly improved experience for building applications with Node.js on Windows Azure. Glenn Block on the SDK team has done a fabulous write up on the ways Microsoft is making Azure a great place for Node.js developers. As our favorite VP Scott Guthrie says on his blog, meet the new Windows Azure.

Enter WebMatrix 2

Today, getting started with node.js is a relatively simple task. You install node, npm (which is now bundled with the node installers), and get started with your favorite text editor. There are infinite possibilities, and limitless configurations for managing projects, compiling CoffeeScript & LESS, configuring your production settings, and deploying your apps. WebMatrix 2 sets out to provide another way to build node.js apps: everything you need to build great apps is one place.

WebMatrix 2 is first and foremost designed for building web applications. From the start screen, you can create applications using pre-built templates, or install common open source applications from the Web Gallery. The current set of templates support creating applications with Node.js, PHP, and (of course) ASP.NET Web Pages. Out of the box, WebMatrix 2 includes three templates for Node.js:

• Empty Node.js Site
• Express Site
• Express Starter Site

The empty site provides a very basic example of using an http server – the same sample that’s available on nodejs.org. The Express Site is a basic application generated using the scaffolding tool in the Node.js framework express. The Node Starter Site is where things start to get interesting. This boilerplate is hosted on GitHub, and shows how to implement sites that include parent/child layouts with jade, LESS css, logins with Twitter and Facebook, mobile layouts, and captcha. When you create a new application using any of these templates, WebMatrix 2 is going to ensure node, npm, and IISNode are installed on your system. If not, it will automatically install any missing dependencies. This feature is also particularly useful if you are building PHP/MySQL applications on Windows.

The end result of the Node Starter Site is a fully functional application that includes Express, Jade, LESS, chat with socket.io, logins with EveryAuth, and mobile support with jQuery Mobile:

IntelliSense for Node.js

One of the goals of WebMatrix 2 is reduce the barrier of entry for developers getting started with Node.js. One of the ways to do that is to provide IntelliSense for the core modules on which all applications are built. The documentation we use is actually built from the docs on the node.js docs site.

In addition to providing IntelliSense for core Node.js modules, WebMatrix 2 also provides code completion for your own JavaScript code, and third party modules installed through NPM. There are infinite ways to build your application, and the NPM gallery recently surpassed 10,000 entries. As developers start building more complex applications, it can be difficult (or even intimidating) to get started. WebMatrix 2 is making it easier to deal with open source packages:

Support for Jade & EJS

To build a truly useful tool for building Node.js web applications, we decided to provide first class editors for Jade and EJS. WebMatrix 2 provides syntax highlighting, HTML validation, code outlining, and auto-completion for Jade and EJS.

If you’re into the whole angle bracket thing, the experience in EJS even better, since it’s based off of our advanced HTML editor:

The best {LESS} editor on the planet

So I’ll admit it – I’m a bit of a CSS pre-processor geek. I don’t write CSS because I love it, but because I need to get stuff done, and I want to write as little of it as possible. Tools like LESS and Sass provide missing features for programmers in CSS like variables, mixins, nesting, and built in common functions.

The LESS editor in WebMatrix not only provides syntax highlighting, but also provides LESS specific validation, IntelliSense for variables and mixins, and LESS specific formatting. Most node developers are going to process their LESS on the server using the npm module, but if you want to compile LESS locally, you can use the Orange Bits compiler to compile your CSS at design time.

CoffeeScript Editor

In the same way LESS and Sass make it easier to write CSS, CoffeeScript simplifies the way you write JavaScript. WebMatrix 2 provides syntax highlighting, code outlining, and completion that simplifies the editing experience. If you want to use CoffeeScript without compiling it on the server, you can use the Orange Bits compiler to compile your CoffeeScript into JavaScript at design time.

Mobile Emulators

Designing applications for mobile can’t be an afterthought. WebMatrix 2 is trying to make this easier in a couple of ways. First – the visual templates (in this case the Node Starter Template) is designed taking advantage of responsive layouts in the main StyleSheet:

• styles.less

This is great if you don’t need to change the content of your site, but is lacking for more complex scenarios. To get around that, the node starter template uses a piece of connect middleware to detect if the user is coming from a mobile device, and sends them to a mobile layout based on jQuery Mobile (more on this in another post). For individual views, there is a convention based system that allows you to create {viewName}_mobile.jade views which are only loaded on mobile devices.

It gets even better. What if you need to see what your site will look like in various browsers and mobile devices? WebMatrix 2 provides an extensibility model that allows you to add mobile and desktop browsers to the run menu:

Today, we offer a Windows Phone emulator, and iPhone / iPad simulators. In the future we’re looking for people to build support for other emulators *coughs* android *coughs*, and even build bridges to online browser testing applications:

Extensions & Open Source

A code editing tool is only as valuable as the developers that commit to the platform. We want to achieve success with everyone, and grow together. As part of that goal, we’ve opened up an extensibility model that allows developers to build custom extensions and share them with other developers. The extension gallery is available online (more on this to come) at http://extensions.webmatrix.com. We’re planning to move a bunch of these extensions into GitHub, and the NodePowerTools extension is the first one to go open source:

• Node Power Tools
• OrangeBits Compiler

In the coming months you’ll start to see more extensions from Microsoft, and more open source.

Everyone worked together

I want to make sure I thank everyone who helped make this release happen, including the WebMatrix team, Glenn Block, Claudio Caldato, our Node Advisory board, Isaac Schlueter, and everyone at Joyent. For more information, please visit:

• WebMatrix on Microsoft.com
• WebMatrix on Twitter
• WebMatrix on GitHub
• WebMatrix on UserVoice
• WebMatrix and Node on Microsoft.com
• Windows Azure just got a lot friendlier to node.js developers
• Vishal Joshi’s blog post

Enjoy!

Building asynchronous real time apps with bidirectional communication has traditionally been a very difficult thing to do. HTTP was originally designed to speak in terms of requests and responses, long before concepts of rich media, social integration, and real time communication were considered staples of modern web development. Over the years, various solutions have been hacked together to solve this problem. You can use plugins like flash or silverlight to make a true socket connection on your behalf – but not all clients support plugins. You can use long polling to manage multiple connections via HTTP – but this can be tricky to implement, and can eat up system resources. The Web Socket standard promises to give web developers a first class socket connection, but browser support is spotty and inconsistent.

Various tools across multiple stacks have been release to solve this problem, but in this post I would like to talk about the first real asynchronous client/server package for ASP.NET: SignalR. SignalR allows .NET developers to change the way we think about client/server messaging: instead of worrying about implementation details of web sockets, we can focus on the way communication flows across the various components of our applications.

This sounds familiar: socket.io with node.js

Over the last year or so, node.js has burst onto the scene as a popular stack for building highly asynchronous applications. The event driven model of JavaScript, paired with a community of inventive developers, led to a platform well suited for these needs. The package socket.io provides what I have found to be the missing piece in the comet puzzle: a front and back end framework that just makes sockets over the web work. No more building flash applications to attempt opening connections over various ports. No more poorly implemented long polling solutions. Most importantly, socket.io made web sockets just plain easy to use:

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('http://localhost');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });
</script>

Node.js and socket.io paved the way for a series of new tools and frameworks across multiple stacks that enable developers to have a first class client/server messaging experience. Node.js and socket.io are wonderful tools – but let’s get back to focusing on SignalR.

Two ways to build apps with SignalR

There are two ways you can go about setting up the server for SignalR. If you want a low level experience, you can add a ‘PersistentConnection’ class along with a custom route. This will give you basic messaging capabilities, suitable for many apps. Straight from the SignalR github, here is an example:

using SignalR;

public class MyConnection : PersistentConnection {
    protected override Task OnReceivedAsync(string clientId, string data) {
        // Broadcast data to all clients
        return Connection.Broadcast(data);
    }
}

This works well if you’re dealing with simple messaging – the other model SignalR supports is the ‘hub’ model. This is where things start to get interesting. Using hubs, you can invoke client side functions from the server, and server side functions from the client. Here’s another example from the documentation:

Here is the server:

public class Chat : Hub {
    public void Send(string message) {
        // Call the addMessage method on all clients
        Clients.addMessage(message);
    }
}

And the client:

<script type="text/javascript">
$(function () {
    // Proxy created on the fly
    var chat = $.connection.chat;

    // Declare a function on the chat hub so the server can invoke it
    chat.addMessage = function(message) {
        $('#messages').append('<li>' + message + '</li>');
    };

    $("#broadcast").click(function () {
        // Call the chat method on the server
        chat.send($('#msg').val())
            .fail(function(e) { alert(e); }) // Supports jQuery deferred
    });

    // Start the connection
    $.connection.hub.start();
});
</script>

<input type="text" id="msg" />
<input type="button" id="broadcast" />

<ul id="messages">
</ul>

I chose the high level API, because well… it’s just cool. For a wonderful break down of the differences between these two methods, check out Scott Hanselman’s post on the topic.

Lets build something!

One of the common examples of using these frameworks is a chat room: it has all of the touch points that are otherwise difficult to implement. How do we know when someone joins the room? What about sending a message? What if I want to send a message to multiple people? This is a perfect example of how client/server messaging over the web can make our lives easier. The SignalR folks have a live sample of this application running on their demo site. With the chat idea done, I decided to combine two tools into one project: a user map. I want to maintain a map that uses a pushpin for every user on the page. As users come, a new pushpin will be added in their location in real time. As they leave, the pushpin will be removed. Before we dive into the code, check out the demo at http://signalrmap.apphb.com/. If no one is in the room, you can slightly randomize your position by using the “random flag” at http://signalrmap.apphb.com/?random=true. This will allow you to use multiple browser windows and watch the system add location push pins.

Building the client

The client of SignalRMap includes a Bing map, and some JavaScript to interact with the back end. I used ASP.NET MVC 3 for this example, but this will work just fine with a web form. To start, we need to include a few script files:

<script charset="UTF-8" type="text/javascript" src="http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0"></script>
<script src="@Url.Content("~/Scripts/jquery-1.6.4.min.js")" type="text/javascript"></script>
<script src="@Url.Content("~/Scripts/jquery.signalR.min.js")" type="text/javascript"></script>
<script type="text/javascript" src="@Url.Content("~/signalr/hubs")"></script>

The first thing we are including here is the Bing Maps JavaScript SDK – this will do all of the heavy lifting for our maps. The SignalR client is dependent upon JavaScript, so we need to include it along with our SignalR reference. Finally, we include the ‘hubs’ functionality into our application, linking our client and server side methods.

After including our scripts, connecting to a hub is crazy awesome easy:

// create the connection to our hub
var mapHub = $.connection.mapHub;

// define some javascript methods the server side hub can invoke

// add a new client to the map
mapHub.addClient = function (client) {
	addClient(client);
	centerMap();
	var pins = getPushPins();
	$("#userCount").html(pins.length)
};

// start the hub
$.connection.hub.start(function () {
	// after the hub has started, get the current location from the browser
	navigator.geolocation.getCurrentPosition(function (position) {

		// create the map element on the page
		mappit(position);

		// notify the server a new user has joined the party
		var coords = isRandom ? createRandomPosition(position) : position.coords;
		var message = { 'user': '', 'location': { latitude: coords.latitude, longitude: coords.longitude} };
		mapHub.join(message);
	});
});

There are a few things going on here. First, we reference our connection to the hub created on the server (note: the connection has not been established yet). Notice the mapHub.addClient method – this method will be exposed in a way such that it can be invoked from the server. *scratches head* – this is a neat concept. After defining methods which can be invoked from the server, we start the connection to the hub. Once the connection is established, we get the browser’s current location, and send that location back to the server. That’s about it. Remember how simple it was to use socket.io? Here we have the same experience. There’s a little more client script here to handle managing the map component. For the full client source for the application, check out my github.

Server side code

As mentioned above, I chose to take the ‘hubs’ route for my application. One of the nice things about using a hub is that it doesn’t require any custom routing – just create a class that extends ‘Hub’, and you’re set. In this example, I’m storing a persistent list of the clients connected to the application (obviously, this method will only work with a single web server). As users show up at the site, they send their current position to the server. The new MapClient is broadcasted to all of the connected clients, and the new client is given the master list of clients:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using SignalR.Hubs;

namespace SignalRMap
{
	public class MapHub : Hub, IDisconnect
	{
		private static readonly Dictionary<string, MapClient> _clients = new Dictionary<string, MapClient>();

		public void Join(MapClient message)
		{
			_clients.Add(this.Context.ClientId, message);
			Clients.addClient(message);
			this.Caller.addClients(_clients.ToArray());
		}

		public void Disconnect()
		{
			MapClient client = _clients[Context.ClientId];
			_clients.Remove(Context.ClientId);
			Clients.removeClient(client);
		}

		/// <summary>
		/// model class for the join message. I tried to use dynamic here, but it didn't work.
		/// </summary>
		public class MapClient
		{
			public string clientId { get; set; }
			public Location location { get; set; }

			public class Location
			{
				public float latitude { get; set; }
				public float longitude { get; set; }
			}
		}
	}
}

And that’s it! SignalR figured out what types of communication my browser supports, managed the tunnel, and just made the connection work. Enjoy!

• View the demo
• Download the source code
• SignalR on GitHub
• Bing Maps SDK

I wrote a blog post for the MSDN AppFabric Blog!

Using MSBuild to deploy your AppFabric Application

Frink! is a mobile client for the web site Reddit. It is designed specifically to be used with tablets, taking advantage of gestures in a unique user interface. Right now the app is available in the BlackBerry App World:

http://appworld.blackberry.com/webstore/content/38838?lang=en

After the code has a little time to settle, I plan on releasing the app to the Android Market as well. The entire project is open source, and available on my GitHub

For more information, here are a bunch of links that talk about the project:

• Frink! on the web: http://frinkapp.com
• Frink! on Reddit: http://www.reddit.com/r/frinkapp
• Frink! on Twitter: http://twitter.com/frinkapp
• Frink! on GitHub: https://github.com/JustinBeckwith/frink
• Frink! at BlackBerry App World: http://appworld.blackberry.com/webstore/content/38838

How do the internal workings of a browser that was released only two years ago have an enormous ripple effect on the future of streaming media on the internet? Last week Google announced on their chromium blog that they’re dropping support for the h.264 codec, in favor of the open source Ogg Theora and WebM/VP8 codecs. This is yet another snag in the messy attempt to unify the playback of video in HTML 5, as we now find the #2 and #3 most popular browsers lacking support for what currently is likely the most ubiquitous encoding format. So how did we get here?

The browser wars are back

After years of IE 6 and Firefox being the only real browsers around, the browser wars have exploded again. For the first time since Netscape 4.7 roamed the earth, Internet Explorer has dropped below 50% in the market share. That leaves a lot of space for the likes of Firefox, Chrome, Safari, and Opera. Well, maybe not Opera. The interesting thing in this graph is the dominance of Firefox, and the growth of Chrome. That leaves ~42% of the current desktop browser market with no H.264 native playback, and ~96% of the next desktop browser market that supports WebM (assuming everyone here upgrades to the latest version of course).

Source: StatCounter Global Stats – Browser Market Share

What’s scary about this is the proliferation of new browsers through mobile and embedded devices. As time goes on, iOS, Android, and RIM are going to eat more and more of those beautiful hits on our Google Analytics dashboards. While iOS is currently ahead in terms of volume, Android is catching up. Quickly.

As the homogeneity of the browser market continues to disappear, the likelihood that all browsers will support the same native HTML 5 playback goes down quickly. So why did Google do this?

Is YouTube making money yet? How about now? Now?

In 2006 Google acquired YouTube for $1.65 billion in stock. The costs of running a video delivery site are sky high, and the sale price certainly turned a lot of heads. While I’ve heard a lot of people question the acquisition, there are estimates that YouTube may be generating as much as 1 billion a year in revenue moving forward. Don’t be mistaken, YouTube is a vital piece to controlling advertising in the streaming media market. They will protect their investment, and continue to grow other revenue streams to support the costs of the platform. One of the ways I look for Google to do this is through providing a channel to charge for premium or protected content. MPEG LA makes providing content encoded using H.264 free until 2016 – given the content is available freely to the end user. The moment you charge for the delivery of that content, you are subject to a delivery fee of 2% revenue per title, up to a maximum $5 million cap. While I don’t think the $5 million is a huge deal to Google, it’s an enormous deal for smaller software shops, startups, integrators, and hardware companies that want to stream and decode video from YouTube on their sites or devices. This model will eventually have a stifling effect on innovation in the streaming media market, which directly affects Google’s YouTube line of business. This pretty easily explains why Google purchased the VP8 codec from On2 for $106.5 million.

In the short term, this decision isn’t going to cause a whole lot of impact. Even if most browsers did support HTML 5, which they don’t, most of the video out there today is in H.264 format. All of the hardware devices that support decoding are using H.264. This is a decision that pays dividends 5 years down the road. Estimates have YouTube receiving as much as 24 hours of content per minute, which is dizzying to think about. No matter how you store it, that’s a ton of storage space. More and more of the content being added to the system is in high definition, so that makes the problem even bigger. Now add the fact that you need to encode your content in two different formats over the long haul, and you have a huge problem. I want to know – will Google have the stones to yank H.264 support from YouTube all together?

Where does this leave everyone?

As mentioned above, I don’t think this changes a lot in the short term. Here is where I see the big players ending up with the change:

• Adobe – Adobe comes out of this situation in great shape. This pretty much just guarantees that flash isn’t going anywhere for a couple of years, and they’ve already announced support for VP8.
• Microsoft – Microsoft, who doesn’t have a horse in this race anymore, has already announced VP8 support for Internet Explorer 9.
• Google – Google gets to protect their YouTube and VP8 investments, while promoting innovation through an open standard.
• Mozilla – Firefox will remain relevant, given their VP8 support in version 4. I doubt the Mozilla Foundation had any intentions of paying MPEG LA $5 million.
• Apple – If Google drops H.264 support of YouTube (which won’t happen for a long time), Apple with have their hand forced into supporting WebM. Until that happens, this is a total mystery to me.

Overall, I think Google’s decision is a good thing for content developers and innovators. Not everyone agrees. For a few dissenting opinions on this, check out:

• Chrome’s love of WebM and hatred of H.264 has nothing to do with YouTube
• Google’s dropping H.264 from Chrome a step backward for openness

And of course, I want to know what you think. So lets start some discussions!

Over this holiday break I had the interesting opportunity to write a bookmarklet for a friend who runs a comic based website.   Instead of just manipulating the currently loaded page, the bookmarklet needed to send a list of images to another site.  Often when writing bookmarklets, we tend to only think of loading our code in the context of a HTML content page.  How often do you test your bookmarklets when the browser is viewing an image?  In this article I am going to go through the code I used to bootstrap my bookmarklet script, and discuss some of the interesting challenges I experienced along the way.

To get started with this code, I used a fantastic article by Tommy Saylor of Smashing Magazine.  It gave me a good start, but certainly left a lot of details out, and in my case, caused a lot of bugs.

Bookmarklet Architecture

That’s right:  we should talk about architecture before diving right into our JavaScript.  When writing a bookmarklet, it’s generally a good idea to keep as much code out of the actual bookmark as possible.  This is where ‘bootstrapping’ comes into play:  we will simply use our bookmark as a piece of code that actually loads the core bits of our JavaScript.  There are actually two reasons why this is a good idea:

• Different browsers have various max-lengths of bookmarks.  Keep in mind that a bookmarklet is kind of an accidental feature.  I think the average max length works out to around 2000 characters, but some browsers (like Internet Explorer 6) have limits as low as 508 characters.
• Users are unlikely to be bothered into refreshing your bookmarklet.  Once somebody bookmarks your code, how are they going to get updates?  It’s much easier if your bookmarklet simply loads a JavaScript file from a static URL.  This way we can update the code in the back whenever we want.

After our bootstrapper loads the script we created, any external libraries will be loaded.  For example, I used jQuery and jQuery UI for my most recent project.  After the dependencies are loaded, we will then execute our main code.

Another thing to keep in mind when you’re building your bookmarklet is how the site behaves after the function is disabled.  For example, if your bookmarklet gives all images on the site a red border, what happens when the user no longer wishes to use the bookmarklet?  For this reason, I tend to create a cleanup method that allows our bookmarklet changes to be undone, and leaves the script in a state that can later be used again.

The bootstrap code

For the purposes of this bookmarklet, I needed to write a piece of code that would interact with a standard HTML page and it’s images, or interact with a page that was a single loaded image. For that reason, the first thing we need to do is determine what type of page we’re dealing with.  If the page is HTML, we can insert a script.  If the page is an image, we need to behave differently.  While I found that Firefox and WebKit both generated a HTML container to render image pages, their behavior surrounding script events of these pages were too inconsistent to be depended upon.

Here is a formatted example of what my a href tag JavaScript looks like:

//
// <a> tag href javascript
//
javascript:(function() {
	if( (document.contentType && document.contentType.indexOf('image/')>-1)
		||/.png$/.test(location.href)
		||/.jpg$/.test(location.href)
		||/.jpeg$/.test(location.href)
		||/.gif$/.test(location.href))
		{
			location.href='http://jbeckwith.com/bookmarklet/';
		}
		else if (!window.main) {
			document.body.appendChild(document.createElement('script'))
				.src='http://jbeckwith.com/my-bookmarklet.js';
		}
		else {
			main();
		}
	})();

After tidying up our script, and adding the surrounding tag, here is a final rendered output of our code, I came up with the following:

<!--
    <a> tag example
-->
<a href="javascript:(function(){if((document.contentType&&document.contentType.indexOf('image/')>-1)||/.png$/.test(location.href)||/.jpg$/.test(location.href)||/.jpeg$/.test(location.href)||/.gif$/.test(location.href)){location.href='http://jbeckwith.com/bookmarklet/';}else if(!window.main){document.body.appendChild(document.createElement('script')).src='http://jbeckwith.com/my-bookmarklet.js';}else{main();}})();">It's a bookmarklet!</a>

Loading jQuery and jQueryUI

Now that the bootstrapper is created, I am going to focus the rest of the article on the external JavaScript file that contains the meat of the code. With the script I wrote, I needed to use a good deal of visual effects. I am already comfortable with JQuery, so I chose to use it as my JavaScript framework:

//
// create javascript libraries required for main
//
if (typeof jQuery == 'undefined') {
	// include jquery
	var jQ = document.createElement('script');
	jQ.type = 'text/javascript';
	jQ.onload=getDependencies;
	jQ.onreadystatechange=function() {
		if(this.readyState=='loaded' || this.readyState=='complete') {
			getDependencies();
		} // end if
	};
	jQ.src = 'http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js';
	document.body.appendChild(jQ);
} // end if
else {
	getDependencies();
} // end else

If you look at the example in the Smashing Magazine article, you will notice a couple of differences. We need to add an event for onreadystatechange to handle Internet Explorer. I found that IE inconsistently set the readyState of the script object to ‘loaded’ or ‘complete’ in various parts of the DOM, so as a rule I check for both. If you don’t make this change, IE will never notify the script that jQuery is finished loading.

Secondly, I have added the getDependencies() method to manage loading required scripts (in addition to jQuery). Since I am depending heavily on a few jQuery UI components, I needed to load both an external JavaScript file and an external CSS file:

//
// getDependencies
//
function getDependencies() {

	// make sure jqueryUI is loaded
	if (!jQuery.ui) {

		// get the link css tag
		var jQCSS = document.createElement('link');
		jQCSS.type = 'text/css';
		jQCSS.rel= 'stylesheet';
		jQCSS.href = 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/base/jquery-ui.css';
		document.body.appendChild(jQCSS);

		// grab jquery ui
		var jQUI = document.createElement('script');
		jQUI.type = 'text/javascript';
		jQUI.src = 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.7/jquery-ui.min.js';
		jQUI.onload=getDependencies;
		jQUI.onreadystatechange=function() {
			if(this.readyState=='loaded' || this.readyState=='complete') {
				getDependencies();
			} // end if
		};
		document.body.appendChild(jQUI);
	} // end if
	else {
		main();
	} // end else
} // end getDependencies function

In this case, I’m really only waiting on jQuery and jQuery UI to load. If there were more dependent scripts, I would likely create an array of scripts that need to be loaded, and check all of their completion every turn through the getDepenencies method.

Embedding Styles

With the supporting code written, we’re now ready to work on our main method. This is where bookmarklets really are different based on your task. In my case, I’m creating a visual element on the page, complete with styles to match the target site. This works pretty much as expected, with a single caveat: any style definitions you create must be at the very bottom of your appended script. Internet Explorer has a nasty habit of inconsistently handling styles and scripts when appended to the DOM. For some reason beyond my understanding, appended style definitions, whether via script or ajax calls, only work if they are at the very bottom of the appended code. This is fantastically fun to figure out on your own, so hopefully I’ve saved you some trouble.

//
// main
//
function main() {
	// only do this the first time the bar is loaded on the page
	if ($("#myBar").length == 0) {
		// append the styles and bar

		var barHtml =	"<div id='myBar'>\
							<div id='myBar-main' class='dragOff'>\
								<span id='myBar-thumbs'></span>\
								<span id='myBar-text'>drag images to the mainbar</span>\
								<span id='myBar-buttons'>\
									<a href='#' id='doneLink'>done</a>\
									<a href='#' id='cancelLink'>cancel</a>\
								</span>\
							</div>\
						</div>\
						<style type='text/css'>\
								#myBar {color: #FFFFFF; font-size: 130%; font-weight: bold; left: 0; position: fixed; text-align: center; top: 0; width: 100%; z-index: 99998; display: none; }\
								#myBar-main {border-bottom: 3px solid #000000; padding: 7px 0;}\
								#myBar-buttons { display: block; float: right; margin-right: 20px; }\
								#myBar-buttons a,\
								#myBar-buttons a:visited,\
								#myBar-buttons a:link,\
								#myBar-buttons a:active,\
								#myBar-buttons a:hover\
									{ padding: 4px; font-size: 0.7em; border: 2px solid #008600; background-color: #00cb00; color: #FFFFFF; text-decoration: none; }\
								#myBar-thumbs img { padding-left: 2px; padding-right: 2px; cursor: hand; }\
								.my-hover { border: 3px solid #4476b8 }\
								.dragOff { background-color: #4476b8; }\
								.dropHover{background-color: #FF0000; border: 1px dashed #e5a8a8;}\
								.dragActive {background-color: #759fd6}\
								.dropHighlight{border: 1px solid #000000;}\
								.dragHelper {z-index: 99999; border: 1px solid #000000;}\
							</style>";
		$("body").append(barHtml);

This code simply creates a formatted div and adds it to the top of the page.

Cleaning up the mess

If you look at the generated HTML above, you’ll notice that I include a cancel link. I like to give the user the option to cancel out of using the current bookmarklet, and even relaunch the bookmarklet without issue. So when you’re done, make sure to test closing and re-launching the code. I suggest keeping all of your elements on the page, and simply hiding them from the user:

		//
		// myBar close evnet
		//
		$("#cancelLink").click(function(e) {
			// hide the bar
			$("#myBar").fadeOut(750);

			// remove any img classes or handlers
			$("img").removeClass('my-hover').unbind().draggable("destroy");

			// reset the thumbnail span
			$("#myBar-thumbs").html('');

			// reset the text
			$("#myBar-text").html("drag images to the mybar");
		});

And for now, that’s it. For the source to this project, visit my GitHub.