In the last few months I have been working, together with a colleague, on an API client for several well-known systems and cloud providers. When we started, I was a novice in the Go programming language, I had no experience in programming API clients, and I trusted the makers of the APIs enough to have great expectations at them.

Today, a few months later, several hours programming later and a bunch of lines of code later, I am a better novice Go programmer, I have some experience in interfacing with APIs, and my level of trust in the API makers is well beneath my feet.

This article will be a not-so-short summary of the reasons why we started this journey and all the unexpected bad surprises we got along the way. Unfortunately, I will be able to show only snippets of the code we have written because I didn’t get the authorisation to make it public. I will make the effort to ensure that the snippets are good enough to help you get a better understanding of the topics.

OK, enough preface. It’s time to tell the story.

The life of a cloud services administrator

We use several cloud services in Telenor Digital. We have G Suite. We have a Github Enterprise instance in house, but many repositories are still in github.com where we have several organisations. We have Slack. And we have our share of Atlassian products with their user database managed in Atlassian Crowd. And then some more, but these are enough for this post.

With all these systems, each one with their user database, keeping things in check is a real pain. It takes a systematic approach and discipline when creating accounts, and even more when off-boarding accounts, to ensure that no rogue access is kept by ex employees. A mistake in on-boarding an account can be easily fixed along the way, a mistake in off-boarding can go unnoticed, compromise the company’s intellectual property and, ultimately, the company’s business and must be taken seriously.

All these systems’ web interfaces are designed to make it easy to on-board and off-board single accounts (with the notable exception of G Suite, which allows you to add and change accounts in batches by uploading a CSV file). When the turnaround of accounts is large and you have several on-boardings and off-boardings every month, these web interfaces are not nice and practical any more and you find yourself buried into an orgy of clicks, text fields and sub-windows. You can have the best manual procedures and all the discipline in the world, but a mistake is just a blink away.

The problem with inconsistency

Another problem with multiple account directories is inconsistency. Separate systems lead to duplicated information, and keeping duplicated information in sync is hard. So you may have an account for John Smith in G Suite initially created as jo@example.com, and registered in Crowd as jo with the same address, and under the display name of “Jo”; no “John” nor “Smith” in there. Later on, Example.com becomes a big company and that jo@example.com email address sounds terribly unprofessional: an alias is added in G Suite as john.smith@example.com, John is mandated to use the new address for any communication and his business cards, and the old “jo” is progressively forgotten.

John gets increasingly fed up by how the nice start-up he once worked for is looking more and more like a classic dinosaur corporate, and one fine day he leaves the company to become an entrepreneur. Bob, the service administrator, gets an off-boarding request for John. Bob joined the company at a late stage, when John Smith was john.smith@example.com for everyone and the old alias jo@example.com was long forgotten. He quickly finds John’s account in G Suite and disables it, but he finds no match in Crowd for neither John’s full name nor the john.smith@example.com address.

Bob doesn’t think that he could search in Crowd for the aliases John had in G Suite and after some more research he concludes that this Johnny guy didn’t have an account in Crowd after all. And here we are, with a rogue account that could be exploited by John Smith himself (if he wasn’t the nice guy he is), or worse: by an external attacker who found John’s password in a password database shared by an Elbonian hacker.

The API client project

My colleague and me were discussing these problems when our then-boss suggested that we could used our Fun Fridays to develop an API client to automate most of these operations. The idea was interesting and challenging at the same time, as none of us had ever worked with APIs. But we decided to give it a try anyway.

We debated for some time about what services we wanted to interface with, which operations we wanted to automate, and which functionality we should implement first. Deciding which operations we wanted to automate was easy: on-boarding and off-boarding. Deciding what we wanted to tackle first and on which services was slightly more complicated and we took a systematic approach.

We made a list of the services we wanted to automate, and for each of the services we set the priority for the implementation of on-boarding and off-boarding. The score system was very simple, it was just three values: A (must be done now), B (must be done one day), and C (it would be nice to have).

Once we had set the priorities for each service and each operation, we started discussing what prerequisites we must have in place to make the ‘A’s happen for either on-boarding and off-boarding. It turned out that, in order to implement the on-boarding, there was a lot of simplification required in the account directories if we wanted to keep the client complexity at a sensible level, while the prerequisites for doing the off-boarding part were not that many. So we decided to focus on the off-boarding.

With the priorities set and the focus on off-boarding, the choice of the services was easy. We would start with our ‘A’s, that is: Google and Crowd, and then we would focus on the ‘B’s, that is: github.com and Slack. The rest would be done at a later stage, when the most urgent functionality is ready for A’s and B’s.

Given the services and the priorities, it was time to decide on what the actual functionality of the client should be. Of course, the client should suspend an account given a unique ID for it, but what else?

Here my precious co-worker Miha came with a great idea: given an email address, we query Google; if an account matches that email address, we fetch all of the addresses associated with the account and we suspend the user; we then use the list of email addresses we got from Google to search accounts registered in any other system that match one of those email addresses, and we disable those accounts, too.

And finally, what language should we use to write the client? Both me and Miha had a good Perl 5 background and we were also learning Go. In the end, it’s not about what language you master most, as much as how well that language supports the APIs you want to use. We would check Google API support for Perl and Go and then decide.

Finally, it was time to start coding.

Google

The home of Google’s Directory API has a “Getting Started” section  with examples in many different languages. But no Perl 5. No recent unofficial support for the Google API was available for Perl 5 either. That was a deal breaker and we decided to use Go.

Following the instructions in the quick start guide, we met the Google API console. The console got some improvements while we worked on our project, but overall it remained a ugly piece of a user-unfriendly interface, where the only safe way to find anything you are looking for in a reasonable time is to learn the path by heart: labels and menus won’t help much.

To work with the Google API in Go you need to install two packages: the Directory API Go client library and the OAuth2 package. The documentation of both is rather overwhelming for a novice Go programmer, and the underlying “philosophy” also needs some consideration. It took some time to understand which steps were required to get to the point where we could interact with the API. Here is a short summary:

• you read a configuration stored in a JSON file using the ConfigFromJSON function, and you get a *oauth.Config in return
• you use that Config to get a OAuth2 token (or to validate one you already have) that hopefully provides all the scopes (the “permissions”) you requested;
• with the right Config and the right token, you can finally get a Client object, that is: something you can use to interact with the API; but you are not quite there yet;
• you can use the Client to get a Service struct; the struct contains pointers to objects that give you access to all the services provided by the Google API;
• in our case, since we’ll be working on the users directory, we would “extract” the Users service and use just that one.

We factored out in separate packages the parts that we knew we would reuse for all other services, and this whole process is compactly written in our code as:

// Initialise the Google Directory API interface
// credential information will go in the google.config.json, or anything set through the -gc option
// the authentication token goes in google.token.json, or anything set through the -gt option
googleClient := goauth.GetClient(goauth.GetConfig(googleCredentials, googleAdmin.AdminDirectoryUserScope), googleToken)

// with this client, we can get a Service struct, and a User service out of it
googleServices, err := googleAdmin.New(googleClient)
if err != nil {
    log.Fatalf("Unable to retrieve directory Client %v", err)
}

googleUserSvc := googleServices.Users

When it comes to extracting information, the Go package uses a “chaining” style of calls that is very handy to use once you wrap your head around it, but a bit hard to understand if all you have is the package documentation (and, in addition, you are a Go novice). An example is worth a thousand words:

    r, err := srv.Users.List().Customer("my_customer").MaxResults(10).
            OrderBy("email").Do()

Basically, any method call returns an object that you can do more calls on, which in turn returns another object and so on, until you call Do(): that will put together all the parameters you have specified in the chain, query the API and, God willing, return the desired information.

These Go packages implement any feature we could wish for, including server-side filtering of results. Filtering is more important than it may seem at a first glance. A G Suite account contains a lot of information which results in fairly sized data structures if one pulls the whole thing.There is no point in pulling out so much information you don’t need or you don’t use: it will bloat memory usage and make your calls to the API slower. Let’s make an example.

Say the variable usersvc contains the Users service we got from the client: if addr was one of a user’s email addresses, you could fetch all the information by doing:

googleUser, err := usersvc.Get(addr).Do()

A user account contains, among many other things, a person’s first and last name, street address, phone number, other email addresses, the address of one’s boss… but you don’t need all that stuff if all you are trying to do is to select a user for off-boarding: you just want to know if the account is there, if it’s active and, maybe, a few more detail. Getting only the information you need is easy: you just add a call to Fields() in the chain before you call Do():

googleUser, err := usersvc.Get(addr).
    Fields(
        "name/fullName",
        "suspended",
        "primaryEmail",
        "aliases").Do()

Much better, easy to read, simple, faster.

We played a lot with the code in quickstart.go until we got a reasonable understanding of what we were doing. We didn’t however a clear understanding of OAuth2 yet. More on this later.

And, finally, the “suspend” functionality for G Suite accounts was ready and functional.

To summarise what we have seen so far: the worse part of the Google API is the console: the API is full of useful features and the Go libraries match the API well.

Atlassian Crowd

With the expectations set by the Google API there was quite some disappointment when we tried to do something sensible with Crowd. The API looked rather primitive; XML, and not JSON, is the default output format: JSON is supported but to get a JSON output you must ask for it explicitly, pretty please; there is no real Quick Start document and the documentation is fragmented; to search for information in Crowd, you must use the abstruse Crowd Query Language (CQL). To add some more, partial changes are not implemented: if, say, you have a user and you want to set it to inactive, you cannot simply make a request that changes that tiny boolean attribute: you have to fetch the whole set of information about the user, change the attribute and post the whole blob again. If you fail to do so, all values but the one you set (and a few fundamental others) will be nullified.

We couldn’t find a decent Go library for Crowd: those we found were either a bit “immature”, or they didn’t implement all the functionality we needed. In the end, we had to write something ourselves. We iterated several times, and failed a lot, before we built a library with enough functionality to allow us to deactivate users in Crowd. The library we wrote is not complete either, more parts will be added when required.

Armed with that library, we can create a “Server” object (which would actually be called a “Client” in all other API packages, we’ll have to fix this one day…)

// For this to work, you must have the crowd credentials in a JSON file from crowdCredentials (default:
// crowd.config.json in the directory where you launch this command from.)
crowdClient, err := crowd.NewServerFromFile(crowdCredentials)
if err != nil {
    log.Fatalf("Cannot instanciate a Crowd \"server\": %v", err)
}

… and then, for example:

    users, err := crowdClient.FetchUsersByEmail([]string{email})
    if err != nil {
        log.Printf("WARNING: while fetching %s from Crowd: %v\n", email, err)
    }

    ...

    if user.Active {
        user, err := crowdClient.DeactivateUser(user.Name)
        if err != nil {
                    log.Printf("WARNING: Cannot deactivate Crowd user %s: %v\n", user.Name, err)
        } else {
            fmt.Printf("Crowd user %s deactivated\n", user.Name)
        }
    }

With that functionality implemented, we could finally put the pieces together and use the information we get from Google. In fact, we started implementing the main features in our list:

• suspend a Google account given an email address
• deactivate a Crowd user given the ID
• deactivate a Crowd user given the email address
• get the list of email addresses associated to a Google account; suspend the Google account; suspend any Crowd account that matches one of the email addresses we got from Google

The first version of the client was not something we were really proud of, it even had authentication credentials in the code (not that we did ever commit credentials in our repository, but having pieces of configuration in the code itself is a horrible thing to do no matter what). But we got it to work, we factored out some code into separate libraries for reuse, and we even wrote the documentation for the code. It felt good, and gave us the boost to kick those credentials out of the code and in JSON files 🙂

As you will see further on, for all services we have tried to match the same “scheme” used in the Google library to create a client object. However, Crowd was different enough from G Suite that keeping the same approach didn’t make much sense. In fact, you don’t have OAUth2 in Crowd, you don’t use tokens but only a basic HTTP authentication. Instead of forcing ourselves into a GetClient(GetConfig) dance, we made the code much simpler:

// now that we have a functional user service, we are ready to query information from Google;
// let's get ready to query Crowd, as well.
// For this to work, you must have the crowd credentials in a JSON file from crowdCredentials (default:
// crowd.config.json in the directory where you launch this command from.)
crowdClient, err := crowd.NewServerFromFile(crowdCredentials)
if err != nil {
    log.Fatalf("Cannot instanciate a Crowd \"server\": %v", err)
}

Note that the call to create a client object is NewServerFromFile. In fact, our Crowd library represents the Crowd directory server, in a way, and we extract information from that directory by means of method calls. Still, it feels weird and an anomaly compared to the rest, and it may be that in the future we will introduce a backward-incompatible change and name the functions more consistently. But not now.

To summarise: we wrote a lot of code to be able to interact with Crowd, and being novices we had to rewrite it several times. When the code was functional and complete enough, it had an inconsistent interface. We’ll redo it once again, eventually.

Github

It happens sometimes that, when you though you could not be more disappointed, there comes another bad surprise, worse than any of the previous ones. Well, to us Github was a bag of unpleasant surprises. But let’s go in order.

Accounts

Our company has several organisations in Github enterprise, and our team is in charge for managing the membership in two of them. People often join the organisations with their pre-existing personal github account, and that’s an important fact. A github account is in no way “yours”, your company doesn’t own it. It’s not like G Suite, where your company has an instance of the services and only sees the accounts managed there. github.com is a flat space, and when you do a search for accounts you are actually searching the whole github. People may decide to keep some information private, and the fact that they are in one of your organisations doesn’t change anything in that respect: you cannot see or search what is not public: neither the address nor the real name. This was one of the bad surprises that we found, but not the first in chronological order.

No OAuth “token page”

When you are working with the Google API and your API client starts the OAuth flow for authorisation, you are first sent to a Google page where you must confirm that the application is authorised to do what it’s requesting. When you complete the authorisation, you are redirected to another Google page that shows you the token, which you can then use in your client configuration. For example, the client code shown in the quick start guide will show you the link to the authorisation page and wait for you to input the token; once you have followed the link, authorised the client and visualised the token, you must copy it and paste it back to the client: the client will start using it and also save it for later use.

Well, surprise. Github of course has the first page but doesn’t provide a second one: it expects you to provide a link to the second page (the “Redirection URL”). Of course we didn’t have one. In addition, we weren’t really sure how OAuth was supposed to work, so we stopped for a while and took an online training on OAuth. That didn’t make us super-experts in the field, but it gave us a much better idea of what we were looking at and how it was supposed to work. Summarising the whole training in a few lines is impossible, but if you set to work with APIs where OAuth is involved I kindly recommend that you get acquainted with the basics at the very least.

Let’s go back to the problem with the redirection URL, that is: the URL you are redirected to after the authorisation phase. In the beginning, we thought of having an HTTP server somewhere in Amazon to just take those calls and show the token, but that seemed to be a bit overkill, added a dependency and would add the burden on us to keep the server secure. Then we thought we could use a lambda function instead, and we tried looking into those for a while. Until we had an idea: the client could spawn a goroutine for a HTTP listener on the local host, which would then be contacted in the redirection phase; since the token is added as a query string to the redirection URL, the listener will decode the parameter and pass it back to the client, which would then eliminate the intermediate manual step of copy-pasting the token in the script. Miha worked on it and we made it work in the end. That also turned useful for Slack, as it shares the same s*ttiness as github.

API v3 versus v4

If you head to https://developer.github.com/ you’ll immediately notice that they push you very gently to use the GitHub API v4. This is the next generation of the GitHub API and it’s fundamentally different from v3. For one, queries are built using GraphQL, a query language expressed in JSON notation. If you want to use v4 you first have to grasp the query language itself; then, if you want to make queries through the API in Go, you must get into all the complications of mapping GraphQL’s JSON data with Go structs in a precise way. When you are through with all these ceremonies and boilerplate, you can finally query the data you were longing for.

According to GitHub, GraphQL gives you “The ability to define precisely the data you want—and only the data you want”. I am afraid it’s not the case. In fact, we found ourselves in a situation where we were not really getting what we wanted out of our calls to the APIs and we wrote to GitHub support. Our support request included, among others, this question:

Is there a way to make a single call with the Github API v4 that will return a list of users whose login matches a given string exactly and are in certain organizations?

and after a sizeable while they replied:

There isn’t a way to limit the search for users in certain organizations. However, it’s possible to use the GraphQL API to search for users.

Well, if there wasn’t a way to search for users through the API then they could well fold the whole company and grow vegetables instead, right? Anyway, v4 was clearly not covering our use cases. Besides, it turned out that the two APIs where not feature-equivalent..

But the funniest thing of all: the way that the developers web site, and the home page in particular, is structured made us think that v4 is a stable API (and if it wasn’t, maybe it should be called “preview”, or “beta”, or something along those lines, shouldn’t it?). But no, it’s not. If you look at the changelog the specs are changing all the time, and often in a backward-incompatible way.

So, guess what? After having invested quite some effort in using v4 we switched to v3. v3 is a bit primitive and inefficient, e.g.: if you want the details for each of the users in a certain organisation, it’s not enough to fetch the user list for the organisation because the user objects returned will only have the login attribute set: you must then iterate over the user objects and fetch the details for each, one by one. OK for small organisations, but not so much for bigger ones since the amount of queries that you can run on the API is limited.

Calling github with v3

Once we set for v3, the rest of the plan unrolled as gracefully as possible. We created an OAuth app in github with all the necessary scopes and access and then we started to code. We tried to be consistent and created GetClient and GetConfig calls that would match the ones we made for the other services as much as possible. So in the code we have:

// Get the Github API client config (to be used immediately after in a couple of places, otherwise we could just
// chain GetClient/GetConfig like we did elsewhere)
githubClientConfig := ghoauth.GetConfig(githubCredentials, "admin:org read:user user:email")

// Initialise the Github API interface
githubClient := ghoauth.GetClient(githubClientConfig, githubToken)

What was different from Google’s case however is that our github client configuration is now a struct of structs, so that we can have access to both the information we got from the configuration file and the OAuth configuration object that we need to create a client:

  // Type Config is a representation of the whole configuration of a Github API client. It holds both the OAuth2 part (the OAuth2
  // field, which maps to a *oauth2.Config), and the client specific information (the Client field, which holds a
  // *ghoauth.ClientInfo). This type is used in GetConfig and GetClient, which makes it pretty much fundamental for anything in
  // this package.
  type Config struct {
      Client *ClientInfo
      OAuth2 *oauth2.Config
  }

So, when we need the organisations service we can do something similar to what we do with a Google API client:

// get to the OrganizationsService in the client
githubOrgSvc := githubClient.Organizations

and if we need to have the list of the names of organisations we manage, we can still access that (without parsing the configuration file a second time):

// organizations in Github that we manage
githubOrgs := githubClientConfig.Client.Organizations

Now we can happily use these two pieces of information to do, for example, this:

// Remove any given github accounts from our organizations
for _, githubID := range suspendFromGithub {
    for _, org := range githubOrgs {
        err := RemoveGithubUserIdFromOrg(githubOrgSvc, githubID, org, apply)
        errorutil.LogWarning("%v", err)
    }
}

In summary: we had to invest a lot of time before we could actually write the code to remove users from github organizations; the prerequisites included learning about OAuth and find out about the shortcomings of using the API v4 instead of v3.

Slack

Of these four APIs that we worked with, the Slack API was the most disappointing. A proper user management functionality in the API seems to be only available through SCIM and only for the most expensive plans.

With a more “normal” paid plan like the one we have, the user management functionality is ridiculously narrow. In particular, there is no API endpoint to create a new user or to reactivate a suspended user. There is no official API endopoint to suspend a user either, but there is an unofficial and undocumented endpoint. We needed this functionality dearly, so we started to search for go packages that implemented the Slack API and see what functionality they could offer.

According to godoc.org, the Slack API implementation more widely adopted are nlopes’ and lestrrat-go’s . The lestrrat package is reportedly based on nlopes, is nicely done and quite similar to Google’s in the structure of the methods but, alas, it doesn’t implement setInactive. I didn’t fancy the idea of forking the software and then have to maintain a fork, so I took a look at the other package.

The nlopes package implements the setInactive call. Other than that, it’s well behind the lestrrat package: the methods are not as nicely structured as its competitor and the documentation is ridiculous. I gave it a try for some time, and then decided I couldn’t take it any more. I went back to the lestrrat package and got an hint from the documentation about how I could implement an additional endpoint for setInactive. Then I forked the repo and started hammering it until I got it working — in fact, it was not as plain simple as the documentation hinted, but still possible.

Once I got my fork of the package working, we could finally get to implementing account suspension in our client. And that wasn’t a clean path either because… Slack is stupid.

The first thing that we needed was to create an application for our client on the Slack side and assign it the proper scopes. Slack has a nice web interface to do that, so far so good. Now, in order to manipulate certain attributes of a user, you need a certain set of scopes, namely: users:read, users:read.email, users.profile:read, users.profile:write. However, in order to call the setInactive endopoint, you need the client scope. Funny fact, you just can’t select client together with the other scopes.

But, as I just said. Slack is stupid. So if you first request some scopes for an application from a client, and then make a request for a different scope from the same client, you’ll be asked to confirm that it’s OK and the new scope will be added to the existing. And there comes the ugly code:

// The Slack API is rather stupid, and so is the implementation of OAuth2, so we have to work around it.
// To make the lookups work, we need the following scopes:
// users:read users:read.email users.profile:read users.profile:write
// To make setInactive work, we need the client scope
// These two sets of scopes cannot be requested together, so we have to request them separately and
// authorise the client twice, so that the associated user gets all the scopes that he needs.
// This is crazy, but... that's how it is.
slackClient := slackoauth.GetClient(slackoauth.GetConfig(slackCredentials, `users:read users:read.email users.profile:read users.profile:write`), slackToken)
slackClient = slackoauth.GetClient(slackoauth.GetConfig(slackCredentials, `client`), slackToken)

Notice how we create slackClient the first time only to authorise the first set of scopes, and then we just overwrite it one line later in order to authorise the second set of scopes. Isn’t that great?

You may have also noticed that we implemented GetConfig and GetClient here, too, and that their arguments are as consistent as they can be with their counterparts in Google and github.

Once we have the clients, we can extract the handle for the Users service (used for lookups) and the handle for the UsersAdmin service (used to set accounts to inactive):

// // get to the Slack users and usersadmin services
slackUserSvc := slackClient.Users()
slackUsersAdminSvc := slackClient.UsersAdmin()

And then with those handles you can do things like, e.g.:

// Deactivate Slack accounts given by user ID
for _, slackId := range suspendFromSlack {
    // Try to fetch a user with the requested ID, see if we get an error
    user, err := slackUserSvc.Info(slackId).Do(slackCtx)

    if err != nil {
        // notify a user with the given ID doesn't exist, and iterate
        fmt.Printf("Cannot lookup a Slack user with ID %s: %v\n", slackId, err)
        continue
    }

    err = DeactivateSlackUser(slackUsersAdminSvc, user, slackCtx, apply)
    errorutil.LogWarning("%v", err)
}

This seems less simple than it looks and deserves some explanation. When you think of a Slack ID you may think of the username, a handle like, e.g., @alice or @bob. Well, it’s not: that handle has no value to Slack. The user ID however, is a string that you have to dig for and I am pretty sure that none of the readers do know theirs: it’s in your profile, in the ‘…’ menu, and it’s something like UXXXXXXXX (an 8-characters, ASCII alphanumeric string prefixed with a “U”). All API calls that can manipulate users require the user ID, and since you rarely know it you have to resort to some other information that you can map back to the ID. So, rather than use code like the one above, you’ll more often do:

// Deactivate any Slack account that matches the given email addresses
for _, slackEmail := range suspendEmailFromSlack {
    err = DeactivateSlackEmail(slackUserSvc, slackUsersAdminSvc, slackEmail, slackCtx, apply)
    errorutil.LogWarning("%v", err)
}

and in that function you will first look up the user by email:

user, err := usersvc.LookupByEmail(email).Do(ctx)

If you get a user object, that will have the user ID in the ID attribute, so you can call the DeactivateSlackUser we mentioned earlier. Here it would really take some more code to show, sorry I can’t show more.

In summary: Slack’s “standard” APIs sucks when it comes to user management; the web interface sucks when it comes to define client authorisations; it has the same problem as github when it comes to the redirection URL for OAuth; the lestrrat-go slack library was our choice although incomplete, but still preferable to the messy and badly documented nlopes implementation.

Inventory of an experience

At the end of this long ride we got:

• a nice API client to suspend users from Google G Suite, Crowd, Slack, and to remove them from organizations in github.com
• a set of go packages to manage OAuth2 authorization to G Suite, Slack and Github
• a embryonic go package for the Crowd API, that we’ll grow over time;
• some additional tooling that we reused in the packages mentioned above for managing OAuth tokens, errors and command-line flags;
• more knowledge about OAuth
• more Go skills
• less trust in API providers in general
• an experience to tell

I hope you enjoyed this. We did… to a point 🙂