Dependency management and Go

29 May 2015 Posted Under: Go [0] comments
 

I find dependency management and package managers interesting. Each language has its own package manager, and each one has characteristics that are specific to that community. NuGet for .NET has great tooling and Visual Studio support, since that’s important to the .NET developer audience. NPM has a super flexible model, and great command line tools.

In a lot of ways, golang is a little quirky. And that’s awesome. However - I’ve really struggled to wrap my head around dependency management in Go.

"Dependency management and golang"

When dealing with dependency management, I expect a few things:

1. Repeatable builds

Given the same source code, I expect to be able to reproduce the same set of binaries. Every. Time. Every bit of information needed to complete a build, whether it be on my local dev box or on a build server, should be explicitly called out in my source code. No surprises.

2. Isolated environments

I am likely to be working on multiple projects at a time. Each project may have a requirement on different compilers, and different versions of the same dependency. At no point should changing a dependency in one project have an effect on the dependencies on a completely separate project.

3. Consensus

Having a package management story is awesome. What’s even better is making sure everyone uses the same one :) As long as developers are inventive and curious, there will always be alternatives. But there needs to be consensus on the community accepted standard on how a package manager will work. If 5 projects use 5 different models of dependency management, we’re all out of luck.

How node.js does it

As I’ve talked about before, I like to use my experience with other languages as a way to learn about a new language (just like most people I’d assume). Let’s take a look at how NPM for node.js solves these problems.

Similar to the go get command, there is an npm install command. It looks like this:


npm install --save yelp

The big difference you’ll see is --save. This tells NPM to save the dependency, and the version I’m using into the package.json for my project:


{
  "name": "pollster",
  "version": "2.0.0",
  "private": true,
  "scripts": {
    "start": "node server"
  },
  "dependencies": {
    "express": "~3.1.0",
    ...
    "nconf": "~0.6.7",
    "socket.io": "~0.9.13"
  }
}

package.json is stored in the top level directory of my app. It provides my isolation. If I start another project - that means another project.json, another set of dependencies. The environments are entirely isolated. The list of dependencies and their versions provides my repeatability. Every time someone clones my repository and runs npm install, they will get the same list of dependencies from a centralized source. The fact that most people use NPM provides my consensus.

Version pinning is accomplished using semver. The ~ relaxes the rules on version matching, meaning I’m ok with bringing down a different version of my dependency, as long as it is only a PATCH - which means no API breaking changes, only bug fixes. If you’re being super picky (on production stuff I am), you can specify a specific version minus the ~. For downstream dependencies (dependencies of your dependencies) you can lock those in as well using npm-shrinkwrap. On one of my projects, I got bit by the lack of shrink-wrapping when a misbehaved package author used a wildcard import for a downstream dependency that actually broke us in production.

The typical workflow is to check in your package.json, and then .gitignore your node_modules directory that contains the actual source code of 3rd party packages.

It’s all pretty awesome.

Go out of the box

With the out of the box behavior, Go is less than ideal in repeatability, isolation, and consensus. If you follow the setup guide for golang, you’ll find yourself with a single directory where you’re supposed to keep all of your code. Inside of there, you create a /src directory, and a new directory for each project you’re going to work on. When you install a dependency using go get, it will essentially drop the source code from that repository into `$GOPATH/src’. In your source code, you just tell the compiler where it needs to go to grab the latest sources:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)
result, err := client.DoSimpleSearch("coffee", "seattle")

So this is really bad. The go-yelp library I’m importing from github is pulled down at compile time (if not already available from a go get command), and built into my project. That is pointing to the master branch of my github repository. Who’s to say I won’t change my API tomorrow, breaking everyone who has imported the library in this way? As a library author, I’m left with 3 options:

  1. Never make breaking changes.
  2. Make a completely new repository on GitHub for a new version of my API that has breaking changes.
  3. Make breaking changes, and assume / hope developers are using a dependency management tool.

Without using an external tool (or one of the methods I’ll talk about below), there is no concept of version pinning in go. You point towards a namespace, and that path is used to find your code during the build. For most open source projects - the out of the box behavior is broken.

My problem is that the default workflow on a go project leads you down a path of sadness. You start with a magical go get command that installs the latest and greatest version of a dependency - but doesn’t ask you which specific version or hash of that dependency you should be using. Most web developers have been conditioned to not check our dependencies into source control, if they’re managed by a package manager (see: gem, NuGet, NPM, bower, etc). The end result is that I could easily break someone else, and I can easily be broken.

Vendoring, import rewrites, and the GOPATH

There is currently no agreed upon package manager for Go. Recently the Go team kicked up a great thread asking the community for their thoughts on a package management system. There are a few high level concepts that are helpful to understand.

Vendoring

At Google, the source code for a dependency is copied into the source tree, and checked into source control. This provides repeatability. There is never a question on where the source is downloaded from, because it is always available in the source tree. Copying the source from a dependency into your own source is referred to as “vendoring”.

Import rewriting

After you copy the code into your source tree, you need to change your import path to not point at the original source, but rather to point at a path in your tree. This is called “Import rewriting”.

After copying a library into your tree, instead of this:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)

you would do this:

import "yourtree/third_party/github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)

.

GOPATH rewriting

Vendoring and import rewriting provide our repeatable builds. But what about isolation? If project (x) relies on go-yelp#v1.0, project (y) should be able to rely on go-yelp#v2.0. They should be isolated. If you follow How to write go code, you’re led down a path of a single workspace, which is driven by $GOPATH. $GOPATH is where libraries installed via go get will be installed. It controls where your own binaries are generated. It’s generally the defining variable for the root of your workspace. If you try to run multiple projects out of the same directory - it completely blows up isolation. If you want to be able to reference different versions of the same dependency, you need to change the $GOPATH variable for each current project. The act of changing the $GOPATH environment variable when switching projects is “GOPATH rewriting”.

Package managers & tools

Given the lack of prescriptive guidance and tools on how to deal with dependency management, just a few tools have popped up. In no particular order, here are a few I found:

Given my big 3 requirements above, I checked out the most popular of the repos above, and settled on godep. The alternatives all fell into at least one of these traps:

  • Forced rewriting the url, making it harder to manage dependency paths
  • Relied on a centralized service
  • Only works on a single platform
  • Doesn’t provide isolation in the $GOPATH

godep

Godep matched most of my requirements for a package manager, and is the most popular solution in the community. It solves the repeatability and isolation issues above. The workflow:

Run go get to install a dependency (nothing new here):


go get github.com/JustinBeckwith/go-yelp/yelp

When you’re done installing dependencies, use the godep save command. This will copy all of the referenced code imported into the project from the current $GOPATH into the ./Godeps directory in your project. Make sure to check this into source control.


godep save

It also will walk the graph of dependencies and create a ./Godeps/Godeps.json file:


{
	"ImportPath": "github.com/JustinBeckwith/coffee",
	"GoVersion": "go1.4.2",
	"Deps": [
		{
			"ImportPath": "github.com/JustinBeckwith/go-yelp/yelp",
			"Rev": "e0e1b550d545d9be0446ce324babcb16f09270f5"
		},
		{
			"ImportPath": "github.com/JustinBeckwith/oauth",
			"Rev": "a1577bd3870218dc30725a7cf4655e9917e3751b"
		},
    ....

When it’s time to build, use the godep tool instead of the standard go toolchain:


godep go build

The $GOPATH is automatically rewritten to use the local copy of dependencies, ensuring you have isolation for your project. This approach is great for a few reasons:

  1. Repeatable builds - When someone clones the repository and runs it, everything you need to build is present. There are no floating versions.
  2. No external repository needed for dependencies - with all dependencies checked into the local repository, there’s no need to worry about a centralized service. NPM will occasionally go down, as does NuGet.
  3. Isolated environment - With $GOPATH being rewritten at build time, you have complete isolation from one project to the next.
  4. No import rewriting - A few other tools operate by changing the import url from the origin repository to a rewritten local repository. This makes installing dependencies a little painful, and makes the import statement somewhat unsightly.

There are a few negatives though as well:

  1. Not checking in your dependencies is convenient. It’s a pain to check in thousands of source files I won’t really edit. Without a centralized repository, this is not likely to be solved.
  2. You need to use a wrapped toolchain with the godep commands. There is still no real consensus.

For an example of a project that uses godep, check out coffee.

Wrapping up

While using godep is great - I’d really love to see consensus. It’s way too easy for newcomers to fall into the trap of floating dependencies, and it’s hard without much official guidance to come to any sort of consensus on the right approach. At this stage - it’s really up to each team to pick what they value in their dependency management story and choose one of the (many) options out there. Until proven otherwise, I’m sticking with godep.

Great posts on this subject

There have been a lot of great posts by others on this subject, check these out as well:

 
 

Docker, Revel, and AppEngine

08 May 2015 Posted Under: Google Cloud [0] comments
 

"Revel running on Google AppEngine with Docker"

I’ve spent some time recently using go for my side web projects. The Go standard libraries are minimal by design - meaning it doesn’t come with a prescriptive web framework out of the box. The good news is that there are a ton of options:

Of course, you could decide to just not use a web framework at all. Comparing these is a topic of great debate - but that topic is for another post :) I decided to try out Revel first, as it was the closest to a full featured rails-esque framework at a glance. I’ll likely give all of these a shot at some point.

After building an app on Revel, I wanted to get a feel for deploying my app to see if it posed any unique challenges. I recently started a new gig working on Google Cloud, and decided to try out AppEngine. The default runtime environment for Go in AppEngine is sandboxed. This comes with some benefits, and a few challenges. You get a lot of stuff for free, but you also are restricted in terms of file system access, network access, and library usage. Given the restrictions, I decided to use the new managed VM service. Managed VMs let you deploy your application in a docker container, while still having access to the other AppEngine features like datastore, logging, caching, etc. The advantage of using docker here is that I don’t need to write any AppEngine specific code. I can write a standard Go/Revel app, and just deploy to docker.

Starting with Revel

There’s a pretty great getting started tutorial for Revel. After getting the libraries installed, scaffold a new app with the revel new command:

go get github.com/revel/revel
go get github.com/revel/cmd/revel
revel new myapp

Using Docker

Before touching managed VMs in AppEngine, the first step is to get it working with docker. It took a little time and effort, but once docker is completely set up on your machine, writing the docker file is straight forward.

Here’s the docker file I’m using right now:


# Use the official go docker image built on debian.
FROM golang:1.4.2

# Grab the source code and add it to the workspace.
ADD . /go/src/github.com/JustinBeckwith/revel-appengine

# Install revel and the revel CLI.
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel

# Use the revel CLI to start up our application.
ENTRYPOINT revel run github.com/JustinBeckwith/revel-appengine dev 8080

# Open up the port where the app is running.
EXPOSE 8080

There are a few things to call out with this Dockerfile:

  1. I chose to use the golang docker image as my base. You could replicate the steps needed to install and configure go with a base debian/ubuntu image, but I found this easier. I could have also used the pre-configured AppEngine golang image, but I did not need the additional service account support.

  2. The ENTRYPOINT command tells Docker (and AppEngine) which process to run when the container is started. I’m using the CLI included with revel.

  3. For the ENTRYPOINT and EXPOSE directives, make sure to use port 8080 - this is a hard coded port for AppEngine.

To start using docker with your existing revel app, you need to install docker and copy the dockerfile into the root of your app. Update the dockerfile to change the path in the ADD and ENTRYPOINT instructions to use the local path to your revel app instead of mine.

After you have docker setup, build your image and try running the app:


# make sure docker is running (I'm in OSX)
boot2docker up
$(boot2docker shellinit)

# build and run the image
docker build -t revel-appengine .
docker run -it -p 8080:8080 revel-appengine

This will run docker, build the image locally, and then run it. Try hitting http://localhost:8080 in your browser. You should see the revel startup page:

"Running revel in docker"

Now we’re running revel inside of docker.

AppEngine Managed VMs

The original version of AppEngine had a bit of a funny way of managing application runtimes. There are a limited set of stacks available, and you’re left using a locked down version an approved runtime. Managed VMs get rid of this restriction by letting you run pretty much anything inside of a container. You just need to define a little bit of extra config in a app.yaml file that tells AppEngine how to treat your container:


runtime: custom
vm: true
api_version: go1
health_check:
  enable_health_check: False

This config lets me use AppEngine, with a custom docker image as my runtime, running on a managed virtual machine. You can copy my app.yaml into your app directory, alongside the Dockerfile. Next, make sure you’ve signed up for a Google Cloud account, and download the Google Cloud SDK. After getting all of that setup, you’ll need to create a new project in the developer console.


# Install the Google Cloud SDK
curl https://sdk.cloud.google.com | bash

# Log into your account
gcloud auth login

# Install the preview components
gcloud components update app

# Set the project
gcloud config set project <project-id>

That covers the initial setup. After you have a project created, you can try running the app locally. This is essentially going to startup your app using the Dockerfile we defined earlier:


# Run the revel application locally
gcloud preview app run ./app.yaml

# Deploy the application
gcloud preview app deploy ./app.yaml

After deploying, you can visit your site here: http://revel-gae.appspot.com

Revel running on AppEngine

Wrapping up

So that’s it. I decided to use revel for this one, but the whole idea behind using docker for AppEngine is that you can bring pretty much any stack. If you have any questions, feel free to check out the source, or find me @JustinBeckwith.

 
 

Realtime services with io.js, redis and Azure

15 February 2015 Posted Under: azure [0] comments
 

"View the demo"

A few years ago, I put together a fun little app that used node.js, service bus, cloud services, and the Instagram realtime API to build a realtime visualization of images posted to Instagram. In 2 years time, a lot has changed on the Azure platform. I decided to go back into that code, and retool it to take advantage of some new technology and platform features. And for fun.

Let’s take a look through the updates!

Resource groups

I’m using resource groups to organize the various services. Resource groups provide a nice way to visualize and manage the services that make up an app. RBAC and aggregated monitoring are two of the biggest features that make this useful.

"Using a resource group makes it easier to organize services"

Websites & Websockets

In the original version of this app, I chose to use cloud services instead of Azure web sites. One of the biggest reasons for this choice was websocket support with socket.io. At the time, Azure websites did not support websockets. Well… now it does. There are a lot of reasons to choose websites over cloud services:

  • Fast continuous deployment via Github
  • Low concept count, no special tooling needed
  • Now supports deployment slots, ssl, enterprise features

When you create your site, make sure to turn on websockets:

"setting up websockets"

io.js

io.js is a fork of node.js that provides a faster release cycle and es6 support. It’s pretty easy to get it running on Azure, thanks to iojs-azure. Just to prove I’m running io.js instead of node.js, I added this little bit in my server.js:

logger.info(`Started wazstagram running on ${process.title} ${process.version}`);

The results:

"Console says it's io.js"

redis

In the previous version of this app, I used service bus for publishing messages from the back end process to the scaled out front end nodes. This worked great, but I’m more comfortable with redis. There are a lot of options for redis on Azure, but we recently rolled out a first class redis cache service, so I decided to give that a try. I’m really looking to use two features from redis:

  • Pub / Sub - Messages received by Instagram are published to the scaled out front end
  • Caching - I keep a cache of 100 messages around to auto-fill the page on the initial visit

You can create a new redis cache from the Gallery:

"Create a new redis cache"

After creating the cache, you have a good ol standard redis database. Nothing special/fancy/funky. You can connect to it using the standard redis-cli from the command line:

"I can connect using standard redis tools"

Note the password I’m using is actually one of the management keys provided in the portal. I also chose to disable SSL, as nothing I’m storing is sensitive data:

"Set up non-SSL connections"

I used node-redis to talk to the database, both for pub/sub and cache. First, create a new redis client:

function createRedisClient() {
    return redis.createClient(
        6379,
        nconf.get('redisHost'), 
        {
            auth_pass: nconf.get('redisKey'), 
            return_buffers: true
        }
    ).on("error", function (err) {
        logger.error("ERR:REDIS: " + err);
    });    
}

// create redis clients for the publisher and the subscriber
var redisSubClient = createRedisClient();
var redisPubClient = createRedisClient();

PROTIP: Use nconf to store secrets in json locally, and read from app settings in Azure.

When the Instagram API sends a new image, it’s published to a channel, and centrally cached:

logger.verbose('new pic published from: ' + message.city);
logger.verbose(message.pic);
redisPubClient.publish('pics', JSON.stringify(message));

// cache results to ensure users get an initial blast of (n) images per city
redisPubClient.lpush(message.city, message.pic);
redisPubClient.ltrim(message.city, 0, 100);
redisPubClient.lpush(universe, message.pic);
redisPubClient.ltrim(universe, 0, 100);

The centralized cache is great, since I don’t need to use up memory in each io.js process used in my site (keep scale out in mind). Each client also connects to the pub/sub channel, ensuring every instance gets new messages:

// listen to new images from redis pub/sub
redisSubClient.on('message', function(channel, message) {
    logger.verbose('channel: ' + channel + " ; message: " + message);
    var m = JSON.parse(message.toString());
    io.sockets.in (m.city).emit('newPic', m.pic);
    io.sockets.in (universe).emit('newPic', m.pic);
}).subscribe('pics');

After setting up the service, I was using the redis-cli to do a lot of debugging. There’s also some great monitoring/metrics/alerts available in the portal:

"monitoring and metrics"

Wrapping up

If you have any questions, feel free to check out the source, or find me @JustinBeckwith.

 
 

Please take this personally

01 February 2015 Posted Under: Product Management [0] comments
 

A few weeks ago I got pulled into a meeting. There’s another team at Microsoft that’s using our SDK to build their UI, and they had a few questions. Their devs had a chance to get their hands on our SDK, and like most product guys, I was interested in getting some unfiltered feedback from an internal team. Before getting into details, someone dropped the phrase “Please don’t take this personally, but…“

The feedback that comes after that sentence is really important. We write it down. We share it with the team. We stack it up against other priorities, compare it with feedback from teams who have similar pain points, and use it to find a way to make our product better. Customer feedback (from an internal team or external customer) is so incredibly critical, that any product team has a similar pattern/process for dealing with it. So what’s the problem?

Any feedback that starts with “Don’t take this personally” really pisses me off. When you say this to someone, you’re making one of two judgments about this person:

  1. They are not personally invested in their work. They go to their job, they do whatever work is put in front of them, and then they go home. If what they’ve made is not good, it doesn’t bother them.

  2. They are personally invested in their work. They want to create something amazing, and will go to great lengths to do so. Whatever you’re about to say - despite your warning - They’re going to take it personally.

For me, that expression elicits a sort of Marty Mcfly “nobody calls me chicken” response.

What’s more personal to me than my product!? I work at Microsoft because I genuinely believe it’s the best place for me to build stuff that has a real tangible impact. I went through years of school so I could do *this*. I moved my family moved across the country. I work 50-60 hours a week (probably more than I should) because I wanted to build *this*. My product is in many ways a reflection of me. What could possibly be more personal?

Does this mean I don’t want criticism? Of course I do! Objective criticism from an informed customer who has used your product is the greatest gift a product manager can receive. It’s how we get better. Just expect me to take it personally.

 
 

Comparing Go and .NET

04 January 2015 Posted Under: Go [0] comments
 

"The gopher image is Creative Commons Attributions 3.0 licensed. Credit Renee French."

2014 was a crazy year. I spent most of the year thinking about client side code while working on the new Azure Portal. I like to use the holiday break as a time to hang out with the family, disconnect from work, and learn about something new. I figured a programming language used for distributed systems was about as far from client side JavaScript as I could get for a few weeks, so I decided to check out golang.

Coming at this from 0 - I used a few resources to get started:

  • Tour of Go - this was a great getting started guide that walks step by step through the language.
  • Go by Example - I actually learned more from go by example than the tour. It was great.
  • The github API wrapper in Go - I figured I should start with some practical code samples written by folks at Google. When in doubt, I used this to make sure I was doing things ‘the go way’.

I’m a learn-by-doing kind of guy - so I decided to learn by building something I’ve built in the past - an API wrapper for the Yelp API. A few years ago, I was working with the ineffable Howard Dierking on a side project to compare RoR to ASP.NET. The project we picked needed to work with the Yelp API - and we noticed the lack of a NuGet package that fit the bill (for the record, there was a gem). To get that project kickstarted, I wrote a C# wrapper over the Yelp API - so I figure, why not do the same for Go? You can see the results of these projects here:

To get a feel for the differences, it’s useful to poke around the two repositories and compare apples to apples. This isn’t a “Why I’m rage quitting .NET and moving to Go” post, or a “Why Go sucks and is doing it wrong” post. Each stack has it’s strengths and weaknesses. Each is better suited for different teams, projects and development cultures. I found myself wanting to understand Go in terms of what I already know about .NET and nodejs, and wishing there was a guide to bridge those gaps. So I guess my goal is to make it easier for those coming from a .NET background to understand Go, and get a feel for how it relates to similar concepts in the .NET world. Here we go!

disclaimer: I’ve been writing C# for 14 years, and Go for 14 days. Please take all judgments of Go with a grain of salt.

Environment Setup

In the .NET world, after you install Visual Studio - you’re really free to set up a project wherever you like. By default, projects are created in ~/Documents, but that’s just a default. When we need to reference another project - you create a project reference in Visual Studio, or you can install a NuGet package. Either way - there really aren’t any restrictions on where the project lives.

Go takes a different approach. If you’re getting started, it’s really important to read/follow this guide:

How to write go code

All go code you write goes into a single root directory, which has a directory for each project / namespace. You tell the go tool chain where to find that directory via the $GOPATH environment variable. You can see on my box, I have all of the pieces I’ve written or played around with here:

$GOPATH

For a single $GOPATH, you have one set of dependencies, and you keep a single version of the go toolchain. It felt a little uncomfortable, and at times it made me think of the GAC. It’s also pretty similar to Ruby. Ruby has RVM to solve this problem, and Go has GVM. If you’re working on different Go projects that have different requirements for runtime version / dependencies - I’d imagine you want to use GVM. Today, I only have one project - so it is less of a concern.

Build & Runtime

In .NET land we use msbuild for compilation. On my team we use Visual Studio at dev time, and run msbuild via jenkins on our CI server. This is a pretty typical setup. At compile time, C# code is compiled down into IL, which is then just-in-time compiled at runtime to the native assembly language of the system’s host.

Go is a little different, as it compiles directly to native code on the current platform. If I compile my yelp library in OSX, it creates an executable that will run on my current machine. This binary is not interpreted, or IL - it is a good ol’ native executable. When linking occurs, the full native go runtime is embedded in your binary. It has much more of a c++ / gcc kind of feel. I’m sure this doesn’t hurt in the performance department - which is one of the reasons folks move from Python/Ruby/Node to Go.

Compilation of *.go files is done by running the go build command from within the directory that contains your go files. I haven’t really come across many projects using complex builds - its usually sufficient to just use go build. Outside of that - it seems like most folks are using makefiles to perform build automation. There’s really no equivalent of a *.csproj file, or *.sln file in go - so there’s no baked in file you would run through an msbuild equivalent. There are just *.go files in a directory, that you run the build tool against. At first I found all of this alarming. After a while - I realized that it mostly “just worked”. It feels very similar to the csproj-less build system of ASP.NET vNext.

Package management

In the .NET world, package management is pretty well known: it’s all about NuGet. You create a project it, add a nuspec, compile a nupkg with binaries, and publish them to nuget.org. There are a lot of NuGet packages that fall under the “must have” category - things like JSON.NET, ELMAH and even ASP.NET MVC. It’s not uncommon to have 30+ packages referenced in your project.

In .NET, we have a packages.config that contains a list of dependencies. This is nice, because it explicitly lays out what we depend upon, and the specific version we want to use:

<packages>
  <package id="Newtonsoft.Json" version="4.5.11" targetFramework="net40" />
  <package id="RestSharp" version="104.1" targetFramework="net40" />
</packages>

Go takes a bit of a different approach. The general philosophy of Go seems to trend towards avoiding external dependencies. I’ve found Blake Mizerany’s talk to be pretty standard for sentiment from the community:

In Go - there’s no equivalent of packages.config. It just doesn’t exist. Instead - dependency installation is driven from Git/Hg repositories or local paths. The dependency is installed into your go path with the go get command:

> go get github.com/JustinBeckwith/go-yelp/yelp

This command pulls down the relevant sources from the Git/Hg repository, and builds the binaries specific to your OS. To use the dependency, you don’t reference a namespace or dll - you just import the library using the same url used to acquire the library:

import "github.com/JustinBeckwith/go-yelp/yelp"
...
client := yelp.New(options)
result, err := client.DoSimpleSearch("coffee", "seattle")

When you go compile, the compiler walks through each *.go file, finds the list of external (non BCL) libraries, and implicitly does a go get if needed. This is both awesome and frightening at the same time. It’s great that go doesn’t require explicit dependencies. It’s great that I don’t need to think of the package and the namespace as different entities. It’s not cool that I cannot choose a specific version of a package. No wonder the go community is skeptical of external dependencies - I wouldn’t reference the tip of the master branch of any project and expect it to keep working for the long haul.

To get get around this limitation, a few package managers started to pop up in the community. There are a lot of them. Given the lack of a single winner in this space, I chose to write my package ‘the go way’ and not attempt using a package manager.

Tooling

In the .NET world - Visual Studio is king. I know a lot of folks that use things like JetBrains or SublimeText for code editing (I’m one of those SublimeText folks), but really it’s all about VS. Visual Studio gives us project templates, IntelliSense, builds, tests, refactoring, code outlines - you get it. It’s all in the box. A giant box.

With Go, most developers tend to use a more stripped down code editor. There are a lot of folks using vim, sublimetext, or notepad++. Here are some of the more popular options:

You can find a good conversation about the topic on this reddit thread. Personally - I’m comfortable with SublimeText, so I went with that + the GoSublime plugin. It gave me syntax highlighting, auto-format on save, and some lightweight IntelliSense for core packages. That having been said, LiteIDE feels a little more close to a full featured IDE:

LiteIDE

There are a lot of options out there - which is a good thing :) Go comes with a variety of other command line tools that make working with the framework easier:

  • go build - builds your code
  • go install - builds the code, and installs it in the $GOPATH
  • go test - runs all tests in the project
  • gofmt - formats your source code matching go coding standards
  • gocov - perform code coverage analysis

I used all of these while working on my library.

Testing

In YelpSharp, I have the typical unit test project included with my package. I have several test files created, each of which has several test functions. I can then run my test through Visual Studio or the test runner. A typical test would look like this:

[TestMethod]
public void VerifyGeneralOptions()
{
    var y = new Yelp(Config.Options);

    var searchOptions = new SearchOptions();
    searchOptions.GeneralOptions = new GeneralOptions()
    {
        term = "coffee"
    };

    searchOptions.LocationOptions = new LocationOptions()
    {
        location = "seattle"
    };

    var results = y.Search(searchOptions).Result;
    Assert.IsTrue(results.businesses != null);
    Assert.IsTrue(results.businesses.Count > 0);            
}

The accepted pattern in Go for tests is to write a corresponding <filename>_test.go for each Go file. Every method that starts with Test<RestOfFunctionName> in the name, is executed as part of the test suite. By running go test, you run every test in the current project. It’s pretty convenient, though I found myself wishing for something that auto-compiled my code and auto-ran tests (similar to the grunt/mocha/concurrent setup I like to use in node). A typical test function in go would look like this:

// TestGeneralOptions will verify search with location and search term.
func TestGeneralOptions(t *testing.T) {
	client := getClient(t)
	options := SearchOptions{
		GeneralOptions: &GeneralOptions{
			Term: "coffee",
		},
		LocationOptions: &LocationOptions{
			Location: "seattle",
		},
	}
	result, err := client.DoSearch(options)
	check(t, err)
	assert(t, len(result.Businesses) > 0, containsResults)
}

The assert I used here is not baked in - there are no asserts in Go. For code coverage reports, the gocov tool does a nice job. To automatically run test against my GitHub repository, and auto-generate code coverage reports - I’ve using Travis CI and Coveralls.io. I’m planning on writing up another post on the tools you can use to build an effective open source Go library - so more on that later :)

Programming language

Finally, let’s take a look at some code. C# is amazing. It’s been around now for 15 years or so, and it’s grown methodically (in a good way). In terms of basic syntax, it’s your standard C derivative language:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace YelpSharp.Data.Options
{
    /// <summary>
    /// options for locale
    /// </summary>
    public class LocaleOptions : BaseOptions
    {
      
        /// <summary>
        /// ISO 3166-1 alpha-2 country code. Default country to use when parsing the location field.
        /// United States = US, Canada = CA, United Kingdom = GB (not UK).
        /// </summary>
        public string cc { get; set; }

        /// <summary>
        /// ISO 639 language code (default=en). Reviews written in the specified language will be shown.
        /// </summary>
        public string lang { get; set; }

        /// <summary>
        /// format the properties for the querystring - bounds is a single querystring parameter
        /// </summary>
        /// <returns></returns>
        public override Dictionary<string, string> GetParameters()
        {
            var ps = new Dictionary<string, string>();
            if (!String.IsNullOrEmpty(cc)) ps.Add("cc", this.cc);
            if (!String.IsNullOrEmpty(lang)) ps.Add("lang", this.lang);
            return ps;
        }
    }
}

C# supports both static and dynamic typing, but generally trends towards a static type style. I’ve always appreciated the way C# can appeal to both new and experienced developers. The best part about the language in my opinion has been the steady, thoughtful introduction of new features. Some of the things that happened between C# 1.0 and C# 5.0 (the current version) include:

There’s more great stuff coming in C# 6.0.

With Go - I found most language features to be… well… missing. It’s a really basic language - you get interfaces (not the way we know them), maps, slices, arrays, and some primitives. It’s very minimal - and this is by design. I was surprised when I started using go and found a few things missing (in no order):

  • Generics
  • Exception handling
  • Method overloading
  • Optional parameters
  • Nullable types
  • Implementing an interface explicitly
  • foreach, while, yield, etc

To understand why Go doesn’t have these features - you have to understand the roots of the language. Rob Pike (one of the designers of Go) really explains the problems they were trying to solve, and the design decisions of the language well in his post “Less is exponentially more”. The idea is to provide a set of base primitive constructs: only the ones that are absolutely required.

Interfaces, methods, and not-classes

Having written a lot of C# - I got used to having all of these features at my disposal. I got used to traditional OOP features like classes, inheritance, method overloading, and explicit interfaces. I also write a lot of JavaScript - so I understand (and love) dynamic typing. What’s weird about Go is that is both statically typed - and doesn’t provide many of the OOP features I’ve grown to lean upon.

To see how this plays out - lets look at the same structure written in Go:

package yelp

// LocaleOptions provide additional search options that enable returning results
// based on a given country or locale.
type LocaleOptions struct {
	// ISO 3166-1 alpha-2 country code. Default country to use when parsing the location field. 
	// United States = US, Canada = CA, United Kingdom = GB (not UK).
	cc   string 
	
	// ISO 639 language code (default=en). Reviews written in the specified language will be shown.
	lang string
}

// getParameters will reflect over the values of the given struct, and provide a type appropriate 
// set of querystring parameters that match the defined values.
func (o *LocaleOptions) getParameters() (params map[string]string, err error) {
	params = make(map[string]string)
	if o.cc != "" {
		params["cc"] = o.cc
	}
	if o.lang != "" {
		params["lang"] = o.lang
	}
	return params, nil
}

There are a few interesting things to call out from these two samples:

  • The Go sample and C# are close to the same size - XMLDoc in this case really makes C# seem longer.
  • I’m not using a class - but rather an interface. Interfaces can support methods via the syntax above. If you define a func that takes a pointer to an object of the interface type - it now supports methods.
  • In my C# sample, this structure implements an interface. In Go, you write an interface, and then structures implement them ambiently - there is no implements keyword.
  • Pointers are an important concept in Go. I haven’t had to think about pointers since 2001 (the last time I wrote C++). It’s not a big deal, but not something I expected to run into.
  • Notice that the getParameters() function returns multiple results - that’s new (and kind of cool).
  • The getParameters() method returns an error as one of the potential return values. You need to do that since there is no concept of an exception in Go.

Error handling

Let that one sink for a moment. Go takes a strange (but effective) approach to error handling. Instead of tossing an exception and expecting the caller to catch and react, many (if not most) functions will return an error. It’s on the caller to check the value of that error, and choose how to react. You can learn more about error handling in Go here. The net result, is that I wrote a lot of code like this:

// DoSearch performs a complex search with full search options.
func (client *Client) DoSearch(options SearchOptions) (result SearchResult, err error) {

	// get the options from the search provider
	params, err := options.getParameters()
	if err != nil {
		return SearchResult{}, err
	}

	// perform the search request
	rawResult, _, err := client.makeRequest(searchArea, "", params)
	if err != nil {
		return SearchResult{}, err
	}

	// convert the result from json
	err = json.Unmarshal(rawResult, &result)
	if err != nil {
		return SearchResult{}, err
	}
	return result, nil
}

In this example, the DoSearch method returns multiple values (get used to this), one of which is an error. There are 3 different method calls made in this function - all of which may return an error. For each of them, you need to check the err value, and choose how to react - oftentimes, just bubbling the error back up through the callstack by hand. I haven’t quite learned to love this aspect of the language yet.

Writing async code

In the previous sample, you may have noticed something fishy. On the following line, I’m making an HTTP request, checking for an error, and them moving forward:

rawResult, _, err := client.makeRequest(searchArea, "", params)
if err != nil {
	return SearchResult{}, err
}
...

That code is synchronous. When I first wrote this code - I was fairly certain I was making a mistake. Years of callbacks or promises in node, and years of tasks and async/await in C# had taught me something really clear - synchronous methods that block the thread are bad. But here’s Go - just doing it’s thing. I thought I was making a mistake, until I started poking around and found a few people with the same misunderstanding. To make a call asynchronously in Go, it’s largely up to the caller, using a goroutine. Goroutines are kind of cool. You essentially point at a function and say ‘run this asynchronously’:

func Announce(message string, delay time.Duration) {
    go func() {
        time.Sleep(delay)
        fmt.Println(message)
    }()  // Note the parentheses - must call the function.
}

Running go <func> in this manner will run the function concurrently in the same process. This does not create a system thread or fork the process - it’s completely internal to the go runtime. Like most things with Go - I was confused and scared at first, as I tried to apply what I know about C# and JavaScript to their model. I haven’t written enough of this style of asynchronous code to have a great feel for the subject, but I plan to spend a lot of time here in the coming weeks.

What’s next

My experience so far with Go has been at times frustrating, but certainty not boring. The best advice I can give for those new to the language is to let go of your preconceived notion of how [insert concept or task] works - it’s almost like the designers of Go tried to do things differently for the sake of doing it differently at times. And that’s ok :) I was cursing far less at the end of the project than the beginning, and I’ve now started to move towards understanding why it’s different (except for the lack of generics - that’s just weird). It’s helping me question some of the design decisions I’ve made on my own APIs at work, and helped me better appreciate the niceties of C#.

I’ve really only scratched the surface of what’s out there for Go. Now that I’ve put together a library, I’m going to take the next step and start playing around with revel, which provides an ASP.NET style web framework on top of Go. From there, I’m going to keep on building, and see where this goes. Happy coding!

The gopher image is Creative Commons Attributions 3.0 licensed. Credit Renee French.