I have my opinions on API design

So I’m going to write about them.

In this context I’m really talking about REST APIs: those wonderful HTTP requests between you and some application that allow you to do all sorts of great things with your data.  Most projects that I have the most fun with involve working with an API; reading about it, testing against it, building the solution with it and coming up with other crazy sh*t to do with it.

Quickly, about REST

REST APIs provide a simple interface to an application over – normally – HTTP.  It follows the familiar standards of other architectures: create: POST, read: GET, update: PUT/PATCH and delete: DELETE.  You use these methods to interact with ‘resources’ at different ‘endpoints’ of the service.  These endpoints will return and/or accept data to achieve a desired result.

That’s the high level overview.

From there you will start encountering a wide range of variations and differences.  Some APIs will allow XML with your requests, but not JSON.  Some work with JSON, but not XML.  APIs may require you to explicitly declare the media type you’re going to interact with.  You might have one API that accepts HTTP basic authentication while others have a token based authentication workflow (like OAuth/OAuth2).  There is a lot of variance in the designs from service to service.

Which brings us to the opinions on design

The APIs I do the most work with currently are for the JSS (of course), JIRA and HipChat, but I’ve also poked around in CrashPlan and Box on the side.  There are a lot of things that I like about all of these APIs and, frankly, some things that really irk me.  And, I mean, really irk me.  Those experiences started me in the direction of learning what it was like to create my own.

If you know me at all you know that I have a real passion about Python.  My current obsession has been the Flask, a microframework for Python that allows you to write web applications.  I’ve been using it for HipChat add-ons that I’m developing, but I was really excited to get into Flask because I could start building my own REST APIs and dig into how they are designed.

Between working with established APIs and the reading and experimenting as I work on my own, I’ve determined there are a number of design choices I would want implemented in any API I worked with.

But it’s in the interface…

Two years ago I had the opportunity to attend Dreamforce.  That year was a biggie as Salesforce was transitioning their development platform and announced their intention to be mobile first and API first.  It was a pretty “phenomenal” keynote.  There were tons of sessions during the mega-conference devoted to the plethora of new and revamped APIs that now made up the Salesforce platform.  My big take away was a slide that provided an extremely high overview of the new stack.  All of Salesforce’s apps and services sat above a unified API layer.

I can’t say why that stuck with me so much at the time since I didn’t even know how to write a simple Python script, but it did.  This was the first big idea that I held onto about API design: implement your features at the API level first, document the implementation and then use that to build onto the end-user solution.

There are plenty of examples out there of services that segregate their user interface from their API and I’ve seen forums with a lot of developers or IT professionals asking why something was implemented in the GUI but inaccessible through their API which prevented an app/integration/automation from advancing.  So, as Salesforce put it, API first.

Documented without the docs

I’ve seen a lot of great examples of API documentation out there.  CrashPlan, JIRA and HIpChat are at the top of my “how to do it right” examples in that they provides representations of data for each supported request method for an endpoint, returned HTTP status codes and potential error messages with their causes.  This is invaluable information for anyone who is writing a script or application against an API, but they all share the same weakness: they’re docs that exist outside the API.

A robust API can provide all the information a developer requires through through the same HTTP methods that – allowing for automated discovery of the API’s capabilities without scrolling around web pages and then flipping back to your console.

There’s an HTTP method I’ve read about, but not one I’ve seen in any of the docs for these APIs as supported.  That would be the OPTIONS method.  It’s a great idea!  Want to know what you can do to a resource?  Pass OPTIONS as the method and in the response there will be a header “Allow” that will list them.

This could be extended to be a contextual method based upon the level of access the provided credentials have.  Say a resource supports GET, POST, PUT, PATCH and DELETE but our user account only supports creating and updating resources.  An admin would return all five in the response header, but our user would only have GET, PUT and PATCH as valid options.

So ok, there’s an HTTP method in the REST standard that allows us to discovery how we can interact with our resources.  Now how do we determine what the valid format of our data in our requests is supposed to be?  JIRA actually implements a solution this for ‘Issues.’  Check out the following endpoints:

/rest/api/2/issue/createmeta
/rest/api/2/issue/{issueIdOrKey}/editmeta

Text The ‘createmeta’ endpoint will return a wealth of data including available projects, issues types, fields and what is required when creating a new issue.  That’s a goldmine of data that’s specific to my JIRA instance!  Then it gets even better when parameters are passed to filter it down even further to better identify what you need to do.  Like this:

/rest/api/2/issue/createmeta?projectIds=10201&issuetypeIds=3

That will return all of the required fields I require to create a new ‘Task’ within the ‘Information Technology’ project in my JIRA board.  If I create a task and then want to update it I can call the second endpoint to reveal all of the fields relevant to this issue, which are required and acceptable values for input.

Despite how great the above is, that’s about all we get for the discovery through JIRA’s API.  We still need to go back to the online docs to reference the other endpoints.

Something I read on RESTful API Design struck a note on this topic.  The idea pitched here is to use forms to provide back to the client a representation of a valid request for the endpoint by passing the appropriate MIME type (for example: ‘application/x-form+json’).  This isn’t something you could expect to have a uniform definition of, but that wouldn’t matter!  You could still programmatically obtain information about any API endpoint by passing the the MIME type for the desired format.

Here’s an example of what a response might look like to such a request:

curl http://my.api.com/users -H "content-type: application/x-form+json" -X POST

{
    "method": "POST",
    "type": "user",
    "fields": {
        "name": {
            "type": "string",
            "required": true
        },
        "email": {
            "type": "string",
            "required": false
        },
        "is_admin": {
            "type": "bool",
            "required": false
        }
    }
}

They can do a lot more work for you

Usually if you’re making a request to an object there will be references, links, within the data to other objects that you can make calls to.  Sometimes this is as simple as an ID or other unique value that can be used to build another request to retrieve that resource.  Seems like an unnecessary amount of code to handle this on the part of the client.

There are two ways of improving this.  The first is to include the full URL to the linked resource as a part of the parent.

curl http://my.api.com/users -H "content-type: application/json"

{
    'name': 'Bryson',
    'email': 'bryson.tyrrell@gmail.com,
    'computers': [
        {
            'id': 1,
            'name': 'USS-Enterprise',
            'url': 'https://my.api.com/computers/1'
        }
    ]
}

The second can build upon this by allowing parameters to be passed that tell the API to return linked objects that are expanded to include all of the data in one request.  JIRA’s API does this for nearly every endpoint.

curl http://my.api.com/users?expand=computers -H "content-type: application/json"

{
    'name': 'Bryson',
    'email': 'bryson.tyrrell@gmail.com,
    'is_admin': true,
    'computers': [
        {
            'id': 1,
            'name': 'USS-Enterprise',
            'url': 'https://my.api.com/computers/1'
            'uuid': 'FBFF2117-B5A2-41D7-9BDF-D46866FB9A54',
            'serial': 'AA1701B11A2B',
            'mac_address': '12:A3:45:B6:7C:DE',
            'model': '13-inch Retina MacBook Pro',
            'os_version': '10.10.2'
        }
    ]
}

Versions are a good thing

All APIs change over time.  Under the hood bug fixes that don’t affect how the client interacts with the service aren’t much to advertise, but additions or changes to endpoints need to be handled in a way that can (potentially) preserve compatibility.

The most common kind of versioning I interact with has it directly in the URL.  I’m going to reference HipChat on this one:

api.hipchat.com/v1
api.hipchat.com/v2

The v1 API was deprecated some time ago as HipChat migrated to their newer and more robust v2 API.  While the v1 API is still accessible it has limitations compared to v2, is lacking many of the endpoints and is no longer supported which means that a lot of integrations that were written using v1 are steadily being phased out.

The differences between the two versions of the API are huge, especially when it comes to authentication, but even after its release the v2 API has had a number of changes and additions made to it.  Unless you’re watching for them they would be easy to miss.

Going the route of maintaining the version of the API in the URL, I found this example:

my.api.com/ < Points to the latest version of the API
my.api.com/2/ < Points to latest version of the v2 API
my.api.com/2.0/ < Points to a specific version of the v2 API

On the backend the objects would need to track which version a field or endpoint was added (or even removed) and handle the response to a request based upon the version passed in the URL.  Anything requested that falls outside of the version would prompt the appropriate 4XX response.

Another method of versioning is used with GitHub’s API.  By default your API requests are made against the latest version of the API, but you you can specify a previous version by having it passed as a part of the ‘Accept’ header:

curl https://api.github.com/users/brysontyrrell -H "Accept: application/vnd.github.v3.full+json"

I’ve read about pros and cons for both approaches, but they serve the purpose of identifying changes in an API as it evolves while providing a means for compatibility with existing clients.

Multiple formats isn’t a sin

My personal preference for any REST API I work with is JSON.  JSON is easy to me, it makes sense, it just works.  I can think of one glaring example off the top of my head of an API I frequently work with that lets me read back objects in JSON but only accepts XML for POST/PUT requests.  Frustrating.

Still, JSON is my preference.  Plenty of people prefer XML.  In some cases XML may be easier to work with than JSON (such as parsing in shell scripts) or be the better data set for an application.  Structurally XML and JSON can be very interchangeable depending upon the data that is being accessed.

If the object can be converted to multiple formats then it may be a good idea to support it.  By passing the appropriate MIME type the API can return data in the requested format.  If no MIME type is passed there should be a default type that is always returned or accepted.

Wrap-up

It’s late now and I’ve dumped a lot of words onto the page.  There’s a PyCharm window open with the shell of my sample API project that attempts to implement all of the design ideas I describe above.  Once I finish it I’ll throw it up on GitHub and see about incorporating some of the requests/responses to it into the article.

Advertisements

Managed VPP via Self Service

A goal I have had for some time was to get away from users simply “requesting” their VPP apps through Self Service and being able to grant them the ability to self-assign those apps (as long as they were eligible).  After doing some work on a little HTML scraping of the JSS I finally have working code to achieve this goal.

HTML Scraping?

If we’re going to allow a user to self-assign some App Store apps we need the ability to ensure there are enough available seats for them to do so.  As of version 9.62, Content as it relates to managed VPP seats is not accessible using the REST API.  There is no quick call we can make to view unassigned VPP content.

What I spent a little time working on was the method by which I could load an Advanced Content Search and parse the results.  This is different that just making an REST API request to the JSS using HTTP basic authentication (this is what you see in pretty much all examples of interacting with the JSS REST API, among others).  The web app uses session cookies for this.

Enter Python (as always, with me).

Using some code I already wrote for getting a cookie for the JIRA REST API on another project (Nessus2JIRA – you’ll hear more about that later), I wrote a new mini JSS class that incorporated both HTTP basic authentication for some the API calls that would need to be made for User Extension Attributes as well as obtaining a session cookie for when we needed to pull up an advanced content search (the web interface and the REST API do not share authentication methods!).

If you’re looking for that already, don’t worry.  There’s a GitHub link at the bottom (where I am now keeping all my code for you guys to grab) that contains the entire sample script for Self Service.

I’ve seen some impressive examples of HTML scraping in shell scripts using SED and AWK.  For what I’m extracting from the HTML of the JSS page I found the solution to be fairly simple.  Let me break it down:

The Advanced Content Search

We need the ability to pull in the information on all of the OS X apps that we are distributing via managed VPP so we can parse out the app in question and if it has unassigned seats for the user.  In my environment I had two content searches created for us to reference this at a glance for both iOS and OS X.  They report the Content Name, Total Content, the Assigned Content and Unassigned Content for a single criteria: Content Type: is: Mac app (or iOS app for the former).

For the script we only care about Unassigned Content so we really only need that and the Content Name, but the other fields are nice if you are pulling up the search in the GUI to view and don’t conflict with how we’re going to perform the HTML scraping.

Of course, there’s still the problem with generating the search.  Going to the URL to view the search requires us to click the View button to get our results.  As it so happens, Zach Halmstad recently dropped some knowledge on a thread for a feature request related to sharing search result links:

https://jamfnation.jamfsoftware.com/featureRequest.html?id=3011

In Zach’s words: “…If you look at the URL of the group or search, it will look like this:  smartMobileDeviceGroups.html?id=2&o=r  If you change the value of the “o” parameter to “v” so it looks like this:  smartMobileDeviceGroups.html?id=2&o=v  It should forward the URL onto the search results.”

Boom.  We can perform an HTTP request using that parameter value and get back a search result!

Now, how do we extract that data? I took a look through the HTML and found the data which is populated into a table by some JavaScript.

...
 <script>
 $(document).ready(function(){

	var data = [

	['Keynote',"15","13","2",],
	['Numbers',"12","12","0",],
	['OS X Server',"20","17","3",],
	['Pages',"9","8","1",],];

	var sortable = new Array;
 ...

It’s just an array which means I could convert it into a native Python list type and iterate over the values with ease.  As I’m being very specific about what data I need I came up with a solution for finding and extracting these lines:

  1. I took the response from my HTTP request, the HTML of the page, and then converted it into a Python list at every newline character.
  2. I began a loop through this HTML list looking for the index value matching “\tvar data = [“ which denotes the beginning of the array.
  3. I restarted my loop at the above index +1 and started concatenating the lines of the array together into a single string (skipping the blank lines). Once I reached the line matching “\tvar sortable = new Array;” I killed the loop.
  4. I evaluate my string and out comes the list containing each entry of my VPP content with the values.

Here’s what that code looks like in action:

# Breaking up the returned HTML into a list
html = response.read().splitlines()

# The applist string starts with the open bracket for our list
applist = "["

# Here is the loop through the html list pulling 
for item in html:
    if item == "\tvar data = [":
        for line in html[html.index(item) + 1:]:
            if line == "\tvar sortable = new Array;":
                break
            elif line.rstrip():
                applist += line.strip(';')[1:]
        break

# We need the 'ast' module to perform the eval into a list
import ast
applist = ast.literal_eval(applist)

Parsing through this new list is now super easy:

for x in applist:
    if int(x[-1]) > 0:
        print(x[0] + " has " + x[-1] + " seats available.")

Keynote has 2 seats available.
OS X Server has 3 seats available.
Pages has 1 seats available.

The Golden Triangle: Extension Attribute to Smart Group to VPP Assignment

All of the VPP assignments in my environment are handled via User Extension Attribute.  This was done for a number of reasons including the easy of dropping a user into scope for one of these apps, but also to future-proof us for when we would start leveraging the API to handle those assignments.

The setup is very simple.  Each App Store app that we distribute through managed VPP has its own extension attribute.  Let’s take Baldur’s Gate as an example (if you don’t have this available to your org, look deep inside and ask yourself, “why not?”).  For every VPP extension attribute there are two available values from a pop-up menu: Assigned and Unassigned.

(Note: since you can set a pop-up menu back to a blank value, ‘Unassigned’ is actually unnecessary from a technical standpoint, but if you have other IT staff working in a JSS it makes more visual sense to set value to ‘Unassigned’ instead of nothing in my opinion) 

Once the user extension attribute is in place create a matching Smart User Group with the sole criteria being the value is set to ‘Assigned.’  Now you make this smart group the scope for a VPP Assignment that only assigns that app.  That’s it!  You now have an App Store app that you can dynamically assign via the JSS REST API (or with ease by flipping values directly on a user’s record).

Updating a User Extension Attribute

The last piece of this is using the REST API to flip the User Extension Attribute for the logged in user to ‘Assigned’.  If you want to get in deeper with the API you can check out my two earlier blog posts “The JSS REST API for Everyone” which give a general introduction and overview.

The two pieces of information you need to update a User Extension Attribute are the user’s username or ID and the ID of the extension attribute that will be updated.  Perform a PUT on either of these resources with the following XML to change the value (be sure to update the ID values!):

../JSSResource/users/id/1
../JSSResource/users/name/bryson.tyrrell

<user>
    <extension_attributes>
        <extension_attribute>
            <id>1</id>
            <value>Assigned</value>
        </extension_attribute>
    </extension_attributes>
</user>

(Note: this is pretty much what you would do for almost ANY extension attribute in the JSS)

Check Out the Full Script on GitHub

As promised, here is a full working example of the script for you to grab:

https://github.com/brysontyrrell/Self-Service-VPP-Assignment

View the README for a breakdown of how to setup the policy (and be sure to do this in a test environment).  The one omission in this code is inside the function that is triggered when there are no available seats of the app to assign:

def create_ticket():
    """This function would generate a ticket with information on the app and user
    IT staff would purchase additional seats of the app and then to assign it

    This function would be called where the number of available seats was not greater than 0
    Customize to suit your environment"""
    print("Creating ticket.")

In my case I would have code in here to take the different values that were passed to the script and generate a Zendesk ticket on the behalf of the user informing IT that more seats needed to be purchased and that the user should be assigned this app once the purchase process is complete.  That takes the onus off of the user to perform yet another step when they are informed the app isn’t immediately available.

If you’re also a Zendesk user you can review a Python script I have here that creates a request ticket for a user from Self Service:

https://github.com/brysontyrrell/Zendesk-Self-Service-Request

Otherwise, you should feel free to add your own code for whatever remediation action you would want to take should there not be any available seats for the user.  If you have questions about that please feel free to reach out to me and we can discuss it.

Take it Further

The entire setup described above allows for apps to be assigned and unassigned easily.  You can take the existing code and modify it to allows users to voluntarily return managed VPP seats if they are no longer using them.  The script produces dialog prompts in the event of an error, unavailable seats or success in assigning the app.  You’ll notice these are embedded AppleScripts (a little PyObjC fun that gets around needing to use Tkinter) so you can work with those as well to further customize the feedback to your users.

And as I already said, feel free to hit me up if you have questions.

Happy New Year!