This is not an article about how you can work with JSON in Go: you can easily learn that from the articles and web pages in the bibliography. Rather, this post is about the concepts that you must understand clearly before you set yourself for the task. Don’t sweat, it’s just two concepts two, and I’ve tried to explain them here.

In the last few weeks I have worked together with a colleague to write some automation with Golang and the Atlassian Crowd API. With several separate user databases (and, at the current state, no hope to unify them in a smart way) it would be very handy to take advantage of the APIs offered by, say, G Suite to fetch all the email addresses related to a user and use that information to automatically deactivate that user from all systems.

Coming from a Perl 5 background, I was hoping that decoding and encoding JSON in Go was as simple as it is in Perl.  But it turns out that it wasn’t, and it’s obvious if you think about it: as Perl 5 is weakly typed, decoding any typed data into an “agnostic” data structure must be simple. Encoding a weakly typed data structure into a typed format may be a bit trickier, but as long as you don’t have too many fancy data (i.e., in this context: strings made of only digits or non-obvious boolean representations) this will also work well. But with strongly typed Go and struct field names having side effects depending on upper-/lowercase, that’s a different story.

As it often happens in cases like this, you will not find all the information you need in a single place. This is my attempt to collect it all and hand it to you, so that you won’t have to waste as much time as I did. You will still have to read through stuff though.

The problem at a high level

You are writing a program that will make a HTTP/HTTPS connection to a server exposing a REST API, and will be sending and receiving JSON objects. The first stop is clearly the documentation of Go’s json package. As soon as you read the second paragraph of the document you’ll start understanding that the task is not going to be a piece of cake:

See “JSON and Go” for an introduction to this package: https://golang.org/doc/articles/json_and_go.html

The documentation of the package, although fairly long, still requires an exegesis to be used fruitfully. Facts will prove that not even the exegesis will be enough. Anyway, the most important information sits in the documentation of the Marshal and Unmarshal functions. The documentation of Marshal tries to explain how you use struct tags to encode a struct into its JSON representation, but if you haven’t heard about struct tags that will mean nothing to you. It will get a meaning when you’ll have read the part of this post regarding struct tags. Until then, let’s focus on the fog that stems out of Unmarshal: from the docs we isolate three pieces of information:

• Unmarshal takes the JSON input in a slice of byte, decodes it into a variable of type interface{} and returns an error value;
• the doc provides a “map” of what JSON types are stored into what Go types; notice that we have []interface{} for JSON arrays and map[string]interface{} for JSON objects;
• when unmarshalling JSON into a struct, JSON object keys are mapped to struct field names or the corresponding tags, preferring a case-sensitive match or using a case-insensitive match otherwise; unmapped object fields are ignored.

The two things that made me scratch my head for a while here were type interface{} variables and struct tags. If you want to successfully work with JSON in Go you have to have both concepts clear, and once they are the documentation for Marshal will also make sense, so let’s go.

The type interface{}

The best explanation about Go interfaces I could find is in Go by Example. The article defines an interface called geometry with two methods, area and perim: they take no argument and return a float64 value. Any Go variable that implements the two methods as defined in the interface can be used wherever a geometry variable is required.

Now think, what type of variable implements an empty interface (an interface with no methods at all, aka interface{})? The answer is obviously “any”, and you’ll rush to the conclusion that you can assign anything to such a variable and just pull it back. But, no, or not quite.

If you try this, it will print “test”:

package main

import "fmt"

func main() {
	var v interface{}

	v = `test`
	fmt.Println(v)
}

and if you try this, it will print “3.14159”:

package main

import "fmt"

func main() {
	var v interface{}

	v = `test`
	v = 3.14159
	fmt.Println(v)
}

And this is interesting: you had a variable of type interface{} that you assigned as a string first, and it worked. Then you assigned it as a number, and the numeric value overrides the string value. You may think that you you have found something alike Perl 5 scalars, where this behaviour is commonplace without the need for forcing a fictitious variable type, right?

No, wrong. In fact you would expect that this program prints something like “map[boolean:true]”:

package main

import "fmt"

func main() {
	j := make(map[string]interface{})

	j["boolean"] = true

	if j["boolean"] {
		fmt.Println(j)
	}
}

but you actually get an error:

# command-line-arguments
./t3.go:10:2: non-bool j["boolean"] (type interface {}) used as if condition

This may be unexpected, but it’s interesting again: you would think that, since an if expects a boolean condition, it would have recognised j["boolean"] as valid. But it hasn’t. Variables of the interface{} type are not the same as Perl 5 scalars: there are cases where you need to pull out from a interface{} the underlying variable of a certain type.

How do you do that? Using Go’s type assertions. It’s here that the JSON and Go blog post comes handy, at the section Generic JSON with interface{}. If you go and check that blog post (and you should, if you are really interested in working with JSON in Go), you’ll see what a type assertion looks like and how you can change the program above to make it work:

package main

import "fmt"

func main() {
	j := make(map[string]interface{})

	j["boolean"] = true

	if j["boolean"].(bool) {
		fmt.Println(j)
	}
}

Where does this come handy? Any time you decode JSON into a interface{}, you’ll want to pull the right type of data out of it and that’s where type assertions (like the “.(bool)” above) and type switches will come into play. For a concise example about type assertions and type switches you can check the article Type assertions and type switches.

Struct tags

struct is one of the types supported in Go, where you define a template for a collection of data, usually of different type but not necessarily. If you have a JSON object defined, for example, like this:

{
    "name": "Marco",
    "country": "Italy"
}

and you have a struct of this type

type Citizen struct {
  name    string
  country string
}

the json package can unmarshal that JSON object into a variable of type Citizen just fine, except that the fields in the structure won’t be exported: you won’t be able to refer to those fields outside the package where the struct is defined. If you want those fields to be exported you’ll have to “capitalise” the first letter of the field name: json will still match the object keys with the fields and the fields will be exported:

type Citizen struct {
  Name    string
  Country string
}

However, we are getting in trouble quickly: first, the field names in the struct don’t match the JSON keys any more: if you use this struct as it is to marshal a Citizen variable into a JSON object, the keys will be wrong (Name instead of name and Country instead of country); second, you cannot expect a JSON object coming from wherever to be so simplistic: even a slight change in our JSON object is enough to put us into serious trouble:

{
    "first-name": "Marco",
    "last-name": "Marongiu",
    "country": "Italy"
}

Not only the keys are lowercase, now they also contain a character, the “-“, that is not valid in struct field names. That’s where we are supposed to use struct tags, but if you haven’t heard about them neither the package documentation nor the “JSON and Go” blog post are helpful. The latter points you to the Go language specification, however the specs are a bit obscure in their language, leaving you with the (false) first impression that you can write something like this and it will work:

type Citizen struct {
  FirstName string `first-name`
  LastName  string `last-name`
  Country   string
}

But it doesn’t. Worse, if you try to unmarshal our JSON object into a Citizen variable defined in this way, only the Country field will be decoded and neither Go nor the json package will report anything wrong with those tags.

There was a lot of banging my head on the wall, until I found two articles: the concise struct tags for encoding/decoding data and the more elaborate Custom struct field tags in Golang. In short: tags are actually key:value pairs, and the key will be something useful to the package that is supposed to use the value. Now the information in the documentation of json’s Marshal function makes sense: with the Citizen struct defined as

type Citizen struct {
  FirstName string `json:"first-name"`
  LastName  string `json:"last-name"`
  Country   string `json:"country"`
}

things will finally work and you will be able to decode and encode JSON objects in your program correctly.

Bibliography and conclusion

And that was all. Now you should be able to work on JSON with Go and get the results you expect. I hope you find this useful.

This post refers to information from the following sources.

• Go’s json package
• Go by Example: interfaces
• JSON and Go
• Type assertions and type switches
• struct tags for encoding/decoding data
• Custom struct field tags in Golang
• Go language specification: struct types