I have my opinions on API design

So I’m going to write about them.

In this context I’m really talking about REST APIs: those wonderful HTTP requests between you and some application that allow you to do all sorts of great things with your data.  Most projects that I have the most fun with involve working with an API; reading about it, testing against it, building the solution with it and coming up with other crazy sh*t to do with it.

Quickly, about REST

REST APIs provide a simple interface to an application over – normally – HTTP.  It follows the familiar standards of other architectures: create: POST, read: GET, update: PUT/PATCH and delete: DELETE.  You use these methods to interact with ‘resources’ at different ‘endpoints’ of the service.  These endpoints will return and/or accept data to achieve a desired result.

That’s the high level overview.

From there you will start encountering a wide range of variations and differences.  Some APIs will allow XML with your requests, but not JSON.  Some work with JSON, but not XML.  APIs may require you to explicitly declare the media type you’re going to interact with.  You might have one API that accepts HTTP basic authentication while others have a token based authentication workflow (like OAuth/OAuth2).  There is a lot of variance in the designs from service to service.

Which brings us to the opinions on design

The APIs I do the most work with currently are for the JSS (of course), JIRA and HipChat, but I’ve also poked around in CrashPlan and Box on the side.  There are a lot of things that I like about all of these APIs and, frankly, some things that really irk me.  And, I mean, really irk me.  Those experiences started me in the direction of learning what it was like to create my own.

If you know me at all you know that I have a real passion about Python.  My current obsession has been the Flask, a microframework for Python that allows you to write web applications.  I’ve been using it for HipChat add-ons that I’m developing, but I was really excited to get into Flask because I could start building my own REST APIs and dig into how they are designed.

Between working with established APIs and the reading and experimenting as I work on my own, I’ve determined there are a number of design choices I would want implemented in any API I worked with.

But it’s in the interface…

Two years ago I had the opportunity to attend Dreamforce.  That year was a biggie as Salesforce was transitioning their development platform and announced their intention to be mobile first and API first.  It was a pretty “phenomenal” keynote.  There were tons of sessions during the mega-conference devoted to the plethora of new and revamped APIs that now made up the Salesforce platform.  My big take away was a slide that provided an extremely high overview of the new stack.  All of Salesforce’s apps and services sat above a unified API layer.

I can’t say why that stuck with me so much at the time since I didn’t even know how to write a simple Python script, but it did.  This was the first big idea that I held onto about API design: implement your features at the API level first, document the implementation and then use that to build onto the end-user solution.

There are plenty of examples out there of services that segregate their user interface from their API and I’ve seen forums with a lot of developers or IT professionals asking why something was implemented in the GUI but inaccessible through their API which prevented an app/integration/automation from advancing.  So, as Salesforce put it, API first.

Documented without the docs

I’ve seen a lot of great examples of API documentation out there.  CrashPlan, JIRA and HIpChat are at the top of my “how to do it right” examples in that they provides representations of data for each supported request method for an endpoint, returned HTTP status codes and potential error messages with their causes.  This is invaluable information for anyone who is writing a script or application against an API, but they all share the same weakness: they’re docs that exist outside the API.

A robust API can provide all the information a developer requires through through the same HTTP methods that – allowing for automated discovery of the API’s capabilities without scrolling around web pages and then flipping back to your console.

There’s an HTTP method I’ve read about, but not one I’ve seen in any of the docs for these APIs as supported.  That would be the OPTIONS method.  It’s a great idea!  Want to know what you can do to a resource?  Pass OPTIONS as the method and in the response there will be a header “Allow” that will list them.

This could be extended to be a contextual method based upon the level of access the provided credentials have.  Say a resource supports GET, POST, PUT, PATCH and DELETE but our user account only supports creating and updating resources.  An admin would return all five in the response header, but our user would only have GET, PUT and PATCH as valid options.

So ok, there’s an HTTP method in the REST standard that allows us to discovery how we can interact with our resources.  Now how do we determine what the valid format of our data in our requests is supposed to be?  JIRA actually implements a solution this for ‘Issues.’  Check out the following endpoints:

/rest/api/2/issue/createmeta
/rest/api/2/issue/{issueIdOrKey}/editmeta

Text The ‘createmeta’ endpoint will return a wealth of data including available projects, issues types, fields and what is required when creating a new issue.  That’s a goldmine of data that’s specific to my JIRA instance!  Then it gets even better when parameters are passed to filter it down even further to better identify what you need to do.  Like this:

/rest/api/2/issue/createmeta?projectIds=10201&issuetypeIds=3

That will return all of the required fields I require to create a new ‘Task’ within the ‘Information Technology’ project in my JIRA board.  If I create a task and then want to update it I can call the second endpoint to reveal all of the fields relevant to this issue, which are required and acceptable values for input.

Despite how great the above is, that’s about all we get for the discovery through JIRA’s API.  We still need to go back to the online docs to reference the other endpoints.

Something I read on RESTful API Design struck a note on this topic.  The idea pitched here is to use forms to provide back to the client a representation of a valid request for the endpoint by passing the appropriate MIME type (for example: ‘application/x-form+json’).  This isn’t something you could expect to have a uniform definition of, but that wouldn’t matter!  You could still programmatically obtain information about any API endpoint by passing the the MIME type for the desired format.

Here’s an example of what a response might look like to such a request:

curl http://my.api.com/users -H "content-type: application/x-form+json" -X POST

{
    "method": "POST",
    "type": "user",
    "fields": {
        "name": {
            "type": "string",
            "required": true
        },
        "email": {
            "type": "string",
            "required": false
        },
        "is_admin": {
            "type": "bool",
            "required": false
        }
    }
}

They can do a lot more work for you

Usually if you’re making a request to an object there will be references, links, within the data to other objects that you can make calls to.  Sometimes this is as simple as an ID or other unique value that can be used to build another request to retrieve that resource.  Seems like an unnecessary amount of code to handle this on the part of the client.

There are two ways of improving this.  The first is to include the full URL to the linked resource as a part of the parent.

curl http://my.api.com/users -H "content-type: application/json"

{
    'name': 'Bryson',
    'email': 'bryson.tyrrell@gmail.com,
    'computers': [
        {
            'id': 1,
            'name': 'USS-Enterprise',
            'url': 'https://my.api.com/computers/1'
        }
    ]
}

The second can build upon this by allowing parameters to be passed that tell the API to return linked objects that are expanded to include all of the data in one request.  JIRA’s API does this for nearly every endpoint.

curl http://my.api.com/users?expand=computers -H "content-type: application/json"

{
    'name': 'Bryson',
    'email': 'bryson.tyrrell@gmail.com,
    'is_admin': true,
    'computers': [
        {
            'id': 1,
            'name': 'USS-Enterprise',
            'url': 'https://my.api.com/computers/1'
            'uuid': 'FBFF2117-B5A2-41D7-9BDF-D46866FB9A54',
            'serial': 'AA1701B11A2B',
            'mac_address': '12:A3:45:B6:7C:DE',
            'model': '13-inch Retina MacBook Pro',
            'os_version': '10.10.2'
        }
    ]
}

Versions are a good thing

All APIs change over time.  Under the hood bug fixes that don’t affect how the client interacts with the service aren’t much to advertise, but additions or changes to endpoints need to be handled in a way that can (potentially) preserve compatibility.

The most common kind of versioning I interact with has it directly in the URL.  I’m going to reference HipChat on this one:

api.hipchat.com/v1
api.hipchat.com/v2

The v1 API was deprecated some time ago as HipChat migrated to their newer and more robust v2 API.  While the v1 API is still accessible it has limitations compared to v2, is lacking many of the endpoints and is no longer supported which means that a lot of integrations that were written using v1 are steadily being phased out.

The differences between the two versions of the API are huge, especially when it comes to authentication, but even after its release the v2 API has had a number of changes and additions made to it.  Unless you’re watching for them they would be easy to miss.

Going the route of maintaining the version of the API in the URL, I found this example:

my.api.com/ < Points to the latest version of the API
my.api.com/2/ < Points to latest version of the v2 API
my.api.com/2.0/ < Points to a specific version of the v2 API

On the backend the objects would need to track which version a field or endpoint was added (or even removed) and handle the response to a request based upon the version passed in the URL.  Anything requested that falls outside of the version would prompt the appropriate 4XX response.

Another method of versioning is used with GitHub’s API.  By default your API requests are made against the latest version of the API, but you you can specify a previous version by having it passed as a part of the ‘Accept’ header:

curl https://api.github.com/users/brysontyrrell -H "Accept: application/vnd.github.v3.full+json"

I’ve read about pros and cons for both approaches, but they serve the purpose of identifying changes in an API as it evolves while providing a means for compatibility with existing clients.

Multiple formats isn’t a sin

My personal preference for any REST API I work with is JSON.  JSON is easy to me, it makes sense, it just works.  I can think of one glaring example off the top of my head of an API I frequently work with that lets me read back objects in JSON but only accepts XML for POST/PUT requests.  Frustrating.

Still, JSON is my preference.  Plenty of people prefer XML.  In some cases XML may be easier to work with than JSON (such as parsing in shell scripts) or be the better data set for an application.  Structurally XML and JSON can be very interchangeable depending upon the data that is being accessed.

If the object can be converted to multiple formats then it may be a good idea to support it.  By passing the appropriate MIME type the API can return data in the requested format.  If no MIME type is passed there should be a default type that is always returned or accepted.

Wrap-up

It’s late now and I’ve dumped a lot of words onto the page.  There’s a PyCharm window open with the shell of my sample API project that attempts to implement all of the design ideas I describe above.  Once I finish it I’ll throw it up on GitHub and see about incorporating some of the requests/responses to it into the article.

Advertisements

Update App Info – New Script on GitHub

As usually happens with us, I went digging around in some old folders I had been stashing a bunch of old (frankly, quite ugly) code and came across something I had done as a proof of concept.

The issue this tries to address is that the JSS never updates a posted app after you have created it.  So “Microsoft Word for iPad” became “Microsoft Word” several versions later, and your users will see this in the App Store, but in Self Service it still has the old name, version number, description and icon.

The original script only dealt with version numbers to address the problem with the Self Service web clip sending false positives for app updates (for admins who chose to show them).  What happened is the version installed on a device wouldn’t match the one in the JSS  (they would, in fact, be at a higher version usually) and the end-user would see an update for that app that didn’t exist.

I don’t know if many admins do that any more, or use the Self Service web clip for that matter, but the problem of inaccurate app names and description text still remained for Self Service.

That is what this takes care of.

https://github.com/brysontyrrell/Update-App-Info

Currently the script doesn’t handle the icon portion due to an issue with the API I encountered.  I’ll be adding that functionality in once I’m sure it will work without error.  It will, however, run on Mac/Linux/Windows so you have flexibility in how you go about using it.  Check out the README for more details.

Quick note: If you are using anything I’ve posted to GitHub and run into problems please use the ‘Issues’ feature to let me know and I’ll see about making updates to fix it.

Managed VPP via Self Service

A goal I have had for some time was to get away from users simply “requesting” their VPP apps through Self Service and being able to grant them the ability to self-assign those apps (as long as they were eligible).  After doing some work on a little HTML scraping of the JSS I finally have working code to achieve this goal.

HTML Scraping?

If we’re going to allow a user to self-assign some App Store apps we need the ability to ensure there are enough available seats for them to do so.  As of version 9.62, Content as it relates to managed VPP seats is not accessible using the REST API.  There is no quick call we can make to view unassigned VPP content.

What I spent a little time working on was the method by which I could load an Advanced Content Search and parse the results.  This is different that just making an REST API request to the JSS using HTTP basic authentication (this is what you see in pretty much all examples of interacting with the JSS REST API, among others).  The web app uses session cookies for this.

Enter Python (as always, with me).

Using some code I already wrote for getting a cookie for the JIRA REST API on another project (Nessus2JIRA – you’ll hear more about that later), I wrote a new mini JSS class that incorporated both HTTP basic authentication for some the API calls that would need to be made for User Extension Attributes as well as obtaining a session cookie for when we needed to pull up an advanced content search (the web interface and the REST API do not share authentication methods!).

If you’re looking for that already, don’t worry.  There’s a GitHub link at the bottom (where I am now keeping all my code for you guys to grab) that contains the entire sample script for Self Service.

I’ve seen some impressive examples of HTML scraping in shell scripts using SED and AWK.  For what I’m extracting from the HTML of the JSS page I found the solution to be fairly simple.  Let me break it down:

The Advanced Content Search

We need the ability to pull in the information on all of the OS X apps that we are distributing via managed VPP so we can parse out the app in question and if it has unassigned seats for the user.  In my environment I had two content searches created for us to reference this at a glance for both iOS and OS X.  They report the Content Name, Total Content, the Assigned Content and Unassigned Content for a single criteria: Content Type: is: Mac app (or iOS app for the former).

For the script we only care about Unassigned Content so we really only need that and the Content Name, but the other fields are nice if you are pulling up the search in the GUI to view and don’t conflict with how we’re going to perform the HTML scraping.

Of course, there’s still the problem with generating the search.  Going to the URL to view the search requires us to click the View button to get our results.  As it so happens, Zach Halmstad recently dropped some knowledge on a thread for a feature request related to sharing search result links:

https://jamfnation.jamfsoftware.com/featureRequest.html?id=3011

In Zach’s words: “…If you look at the URL of the group or search, it will look like this:  smartMobileDeviceGroups.html?id=2&o=r  If you change the value of the “o” parameter to “v” so it looks like this:  smartMobileDeviceGroups.html?id=2&o=v  It should forward the URL onto the search results.”

Boom.  We can perform an HTTP request using that parameter value and get back a search result!

Now, how do we extract that data? I took a look through the HTML and found the data which is populated into a table by some JavaScript.

...
 <script>
 $(document).ready(function(){

	var data = [

	['Keynote',"15","13","2",],
	['Numbers',"12","12","0",],
	['OS X Server',"20","17","3",],
	['Pages',"9","8","1",],];

	var sortable = new Array;
 ...

It’s just an array which means I could convert it into a native Python list type and iterate over the values with ease.  As I’m being very specific about what data I need I came up with a solution for finding and extracting these lines:

  1. I took the response from my HTTP request, the HTML of the page, and then converted it into a Python list at every newline character.
  2. I began a loop through this HTML list looking for the index value matching “\tvar data = [“ which denotes the beginning of the array.
  3. I restarted my loop at the above index +1 and started concatenating the lines of the array together into a single string (skipping the blank lines). Once I reached the line matching “\tvar sortable = new Array;” I killed the loop.
  4. I evaluate my string and out comes the list containing each entry of my VPP content with the values.

Here’s what that code looks like in action:

# Breaking up the returned HTML into a list
html = response.read().splitlines()

# The applist string starts with the open bracket for our list
applist = "["

# Here is the loop through the html list pulling 
for item in html:
    if item == "\tvar data = [":
        for line in html[html.index(item) + 1:]:
            if line == "\tvar sortable = new Array;":
                break
            elif line.rstrip():
                applist += line.strip(';')[1:]
        break

# We need the 'ast' module to perform the eval into a list
import ast
applist = ast.literal_eval(applist)

Parsing through this new list is now super easy:

for x in applist:
    if int(x[-1]) > 0:
        print(x[0] + " has " + x[-1] + " seats available.")

Keynote has 2 seats available.
OS X Server has 3 seats available.
Pages has 1 seats available.

The Golden Triangle: Extension Attribute to Smart Group to VPP Assignment

All of the VPP assignments in my environment are handled via User Extension Attribute.  This was done for a number of reasons including the easy of dropping a user into scope for one of these apps, but also to future-proof us for when we would start leveraging the API to handle those assignments.

The setup is very simple.  Each App Store app that we distribute through managed VPP has its own extension attribute.  Let’s take Baldur’s Gate as an example (if you don’t have this available to your org, look deep inside and ask yourself, “why not?”).  For every VPP extension attribute there are two available values from a pop-up menu: Assigned and Unassigned.

(Note: since you can set a pop-up menu back to a blank value, ‘Unassigned’ is actually unnecessary from a technical standpoint, but if you have other IT staff working in a JSS it makes more visual sense to set value to ‘Unassigned’ instead of nothing in my opinion) 

Once the user extension attribute is in place create a matching Smart User Group with the sole criteria being the value is set to ‘Assigned.’  Now you make this smart group the scope for a VPP Assignment that only assigns that app.  That’s it!  You now have an App Store app that you can dynamically assign via the JSS REST API (or with ease by flipping values directly on a user’s record).

Updating a User Extension Attribute

The last piece of this is using the REST API to flip the User Extension Attribute for the logged in user to ‘Assigned’.  If you want to get in deeper with the API you can check out my two earlier blog posts “The JSS REST API for Everyone” which give a general introduction and overview.

The two pieces of information you need to update a User Extension Attribute are the user’s username or ID and the ID of the extension attribute that will be updated.  Perform a PUT on either of these resources with the following XML to change the value (be sure to update the ID values!):

../JSSResource/users/id/1
../JSSResource/users/name/bryson.tyrrell

<user>
    <extension_attributes>
        <extension_attribute>
            <id>1</id>
            <value>Assigned</value>
        </extension_attribute>
    </extension_attributes>
</user>

(Note: this is pretty much what you would do for almost ANY extension attribute in the JSS)

Check Out the Full Script on GitHub

As promised, here is a full working example of the script for you to grab:

https://github.com/brysontyrrell/Self-Service-VPP-Assignment

View the README for a breakdown of how to setup the policy (and be sure to do this in a test environment).  The one omission in this code is inside the function that is triggered when there are no available seats of the app to assign:

def create_ticket():
    """This function would generate a ticket with information on the app and user
    IT staff would purchase additional seats of the app and then to assign it

    This function would be called where the number of available seats was not greater than 0
    Customize to suit your environment"""
    print("Creating ticket.")

In my case I would have code in here to take the different values that were passed to the script and generate a Zendesk ticket on the behalf of the user informing IT that more seats needed to be purchased and that the user should be assigned this app once the purchase process is complete.  That takes the onus off of the user to perform yet another step when they are informed the app isn’t immediately available.

If you’re also a Zendesk user you can review a Python script I have here that creates a request ticket for a user from Self Service:

https://github.com/brysontyrrell/Zendesk-Self-Service-Request

Otherwise, you should feel free to add your own code for whatever remediation action you would want to take should there not be any available seats for the user.  If you have questions about that please feel free to reach out to me and we can discuss it.

Take it Further

The entire setup described above allows for apps to be assigned and unassigned easily.  You can take the existing code and modify it to allows users to voluntarily return managed VPP seats if they are no longer using them.  The script produces dialog prompts in the event of an error, unavailable seats or success in assigning the app.  You’ll notice these are embedded AppleScripts (a little PyObjC fun that gets around needing to use Tkinter) so you can work with those as well to further customize the feedback to your users.

And as I already said, feel free to hit me up if you have questions.

Happy New Year!

Using Box as a Distribution Point – The 9.x Version

I never updated my original post on using Box.com as a Casper Share after the Casper Suite 9.0 was released  The process is still essentially the same as in 8.x, but instead of only posting updated screenshots I thought I would add in a couple of alternatives to relying upon Box Sync and Casper Admin as I had described before.

But first, something pretty important I’ve learned for anyone considering this workflow…

Single Sign On Considerations

WebDAV doesn’t work with SSO credentials.  If you check out Box’s article here you can follow the instructions for setting up an ‘external password’ that will use the account’s email address and that password for authentication.  If you choose to go this route consider setting password requirements that force the password to match or exceed what is used for SSO.

Now for putting it all together.

Setup the Casper Share Folder in Box

Before putting all of the settings into your JSS, make sure the Box folder you want to use is in place with the correct permissions.

Create a user account in Box that is just for read-only (viewer) access and invite it to that folder.  By inviting the user you can have the ownership of the Casper Share rest with any other user in your Box account, located anywhere in their folder hierarchy, and it will still be a root-level folder for the read-only user.

Remember, you can invite as many other users are you need to for managing your packages, or you can get real crafty and do some fun stuff using Box’s automation tools they’ve introduced.  Example:

  1. You have a folder in Box called “Package Staging”.
  2. Your IT staff upload a package here and it triggers a task to the party responsible for reviewing packages prior to distribution.
  3. If the task is marked as completed the package is then automatically moved to the Casper Share.

Nifty stuff.  I won’t really dive too much into it beyond that, but you get the gist.

Now, inside the Casper Share create a new “Packages” folder.  Because this is being treated as a traditional File Share Distribution Point the JSS will take the address and path we input below and then look for a “Packages” folder for the files to download.

The Box based Casper Share is now ready for use.

File Share Distribution Point Settings

In 9.x there was a change where the File Sharing tab for your File Share Distribution Point could not have empty values.  In 8.x as I had previously described you could get around the File Sharing settings by never clicking on it.  Even so, these settings are meaningless as we will never interact with this distribution point using AFP/SMB, so inputting junk values is acceptable.

The following screenshots detail the settings to use.  Here is also a quick summary:

  • General > Server Address: dav.box.com
  • HTTP/HTTPS > “Use HTTP downloads” is checked
  • HTTP/HTTPS > “Use SSL” is checked
  • HTTP/HTTPS > Context: /dav/CasperShare — This can also be any folder you want instead of labeling it “CasperShare”
  • HTTP/HTTPS > Authentication Type: Username and Password

 

 

Configure Your Network Segment

You may be in a position where you want to have specific network segments being directed to this distribution point, but generally you would want to use Box as the external catch-all for anyone who is not inside one of your offices.  You Network Segment settings will look something like this to direct all external clients to download from Box:

  • Starting IP Address: 1.1.1.1
  • Ending IP Address: 255.255.255.255
  • Default Distribution Point: Specific file share distribution point
  • Default File Share Distribution Point: Your Box.com distribution point from above

Skip Casper Admin, Use a Browser and the API

In the previous article I had details how to setup Box Sync on a Mac, make the directory mountable allowing you to continue to leverage Casper Admin for package management.

You don’t really need to do that.  Using the Box web interface works great for uploading even large files that are multiple gigabytes in size.

The one thing that was nice about Casper Admin was it created the package object for you in the JSS after you dragged it in and the app copied the file out to your master distribution point.  You can easily do this yourself through the API and build out a workflow that works best for your staff.  If you’re not familiar with the JSS REST API you can read my introduction post here.  There are code example for how to interact with it.

The minimal XML you would use to post your package’s information into the JSS would look like this:

<package>
    <name>Twitter</name>
    <filename>Twitter.pkg</filename>
</package>

That’s it.  POST that to ../JSSResource/packages/id/0 and you’ll have a package you can assign to your policies.  Of course, there are a lot of options you can set in the XML.  You only need to include the elements that you want to specify in the JSS.  Otherwise, everything not included in the XML will be set at their defaults (disabled checkboxes, no category, priority 10).

<package>
    <name>Twitter.pkg</name>
    <category>Social</category>
    <filename>Twitter.pkg</filename>
    <info>Repackaged on 2014-09-12</info>
    <notes>Packaged by Bryson</notes>
    <priority>15</priority>
    <boot_volume_required>false</boot_volume_required>
    <os_requirements>10.7, 10.8, 10.9, 10.10</os_requirements>
</package>

There are also elements that are very specific to the type of package you’re working with.

If the installer requires a reboot, set that flag with this element:

<reboot_required>false</reboot_required>

If you’re using a DMG instead of a PKG you can set the “Fill User Template” options:

<fill_user_template>true</fill_user_template>
<fill_existing_users>true</fill_existing_users>

If you are still working in an environment with PowerPC Macs in the mix you can set the restrictions for the architecture and an alternative package to run:

<required_processor>x86</required_processor>
<switch_with_package>Twitter_PPC.pkg</switch_with_package>

Lastly, if you’re dealing with distributing software updates and want them installable only if they are listed in the Mac’s available software updates, you can enable that:

<install_if_reported_available>true</install_if_reported_available>

 

Using OS X VMs 
for FileVault 2 Testing

This post details how to create a FileVault 2-ready base OS X virtual machine using Parallels Desktop 10 and then create either a virtual machine template or make use of the new ‘linked clones’ feature to quickly create new virtual machines to test FileVault 2 workflows.

Using a virtual machine for FileVault 2 testing has a number of benefits.

  • A virtual machine is much smaller than most Macs (without repartitioning) and it takes less time to fully encrypt.
  • Instead of having to decrypt a Mac to run a test again, or wipe it to reinstall OS X, you can clone virtual machines endlessly and delete the encrypted ones when you are finished with them.
  • You can also snapshot virtual machines and them revert to previous states.

Create a New Base OS X 10.9 VM

You can easily create a new virtual machine by dragging the “Install OS X Mavericks.app” onto the Parallels icon in your dock.

Choose the default options for CPU (2), RAM (2 GB) and Disk space (64 GB).

Click Continue to create the virtual machine.

Repartition the VM to Conserve Disk Space

When a virtual machine is FileVault 2 encrypted it will encrypt all of the disk space that it is assigned.  If left at the default of 64 GB the virtual disk file will expand to the full size and take up unneeded space on your Mac.

To prevent this, repartition the disk drive so the boot volume only takes up what is required to install OS X.

Once you are in the OS X Installer, go to the Utilities menu and open Disk Utility.

Split the drive into two partitions with the first/boot partition being set to 16 GB.

Click Apply to create the new partitions and then quit Disk Utility.

Install OS X to the 16 GB boot partition.

Reinstall OS X to Create a Recovery Partition

Once the install of OS X has finished and you have created your account open Terminal and run ‘diskutil list’.  You will note there is no Recovery HD partition present.

Per Parallel’s KB article, copying the “Install OS X Mavericks.app” to the virtual machine and running it will correctly create the Recovery HD partition and allow FileVault 2 encryption.

http://kb.parallels.com/en/122661

Mount your host Mac in the virtual machine and copy the “Install OS X Mavericks.app” to your second partition.  Then run it and re-install OS X Mavericks.

Upon completion of the second install, run ‘diskutil list’ again and you will find the Recovery HD is present.

The final virtual machine takes up less than ~15 GB of disk space on your Mac.

DO NOT FILEVAULT 2 ENCRYPT THIS!

Close all open applications and shut down the virtual machine for the next steps.

Create a New VM for FileVault 2 Testing

Option 1) Template the Base OS X 10.9 VM

You can convert your base OS X virtual machine into a VM Template.  Select the virtual machine and then from the File menu select “Clone to Template…” to create a copy that will be the template, or select “Convert to Template” to convert the virtual machine and not create a copy.

Once you have your template, double-click on it and you will receive a prompt to create a new virtual machine from it.  This will create a new virtual machine

You can share a virtual machine template with other Parallels users or move it to other Macs.  Each virtual machine created from a template will have unique identifiers to avoid conflicts.

Option 2) Create a Linked Clone of the Base OS X 10.9 VM

“Linked Clones” are a new feature of Parallels Desktop 10.  A linked clone is a virtual machine created from a snapshot of a “parent” virtual machine.  This method is faster than the above virtual machine templates for spinning up instances.

http://kb.parallels.com/en/122669

By selecting “New Linked Clone…” from the virtual machine’s context menu a snapshot will be  automatically created from the parent virtual machine and the cloned virtual machine window will immediately appear and be ready to launch.

Enable FileVault 2 on the Cloned VM

With your clone of the base virtual machine you can now enroll it to a JSS (or whatever) to test whatever FileVault 2 workflows are required.

In the below screenshots you can see this virtual machine was enrolled, a configuration profile was deployed requiring FileVault 2 and included key redirection to the JSS.  Once logged out, FileVault 2 was triggered.

Done Migrating

Very please with the performance boost. I’m vastly under-utilizing the total available capacity of the Pool, but that’s OK.  I can increase the size of any of these Spaces at any time.

Once I had removed all of the drives for the old Pool it no longer appeared when I booted up.  I can now safely store those drives and reattach them in the future should I ever need to recover data from one of the old Spaces.

Migrating Storage Spaces

A while back I posted this shaky-cam YouTube video demonstrating the Storage Spaces feature of Windows 8.  It’s essentially a software-level RAID that allows you to group a bunch of physical disks together and then thin provision logical volumes with different levels of resiliency.  Very cool tech.

When I originally set this up I was using three aging HDDs from different vendors and of different sizes (400 GB, 250 GB and 250 GB respectively) that were brought over from my previous system.  Storage Spaces enabled me to consolidate them into simpler logical volumes and I ended up with two spaces: a Media Space with two-way redundancy (equivalent to RAID1) and a Programs Space with no resiliency (equivalent to RAID0 for better read/write performance).  Both were NTFS formatted as I had made them under Windows 8.0.

I now have four higher performance 4 TB disks that will take their places in the tower.  My original plan for migrating over the existing Spaces was to add these new disks to the Storage Pool and then begin removing the older three disks one-by-one and allowing the data to be redistributed.  Storage Spaces does support this, but only if the disk contains data for Spaces with resilience.  My Programs Space, being non-resilient, prevents me from doing this.

(If I had intended on keeping the original three disks this would not be an issue, I would add the new drives into the existing Storage Pool and they would be utilized without any further action on my part.)

The backup plan is to build a new Storage Pool, recreate the two existing Spaces and then copy the contents over.  I still plan on keeping the Spaces just as they are (the performance for the Programs space should be much greater in a four disk pool), but the initial allocation to both spaces will be set to 1 TB each (Storage Spaces allows me to increase this value at any time).  I will also  be creating a File History Space this time around to enable Windows’ File History feature for local backups of my user files.

All four 4 TB disks are attached to the PC via a USB 3.0 enclosure.  Storage Spaces doesn’t care how a disk is connected to my PC.  The disks themselves contain all of the configuration information so no matter how I choose to attach them (USB, eSATA, internal SATA, etc.) Windows will correctly recognize what Storage Pool a disk belongs to.

With all that said, my step-by-step plan is:

  1. Create the new Storage Pool with the matching Storage Spaces
  2. Copy all content of the old Spaces to the new Spaces
  3. Reassign the corresponding drive letters of the original Storage Spaces to the new ones (i.e. drives D: and E:)
  4. Delete the original Storage Pool (optional: I could leave it and Windows would alert me the disks are missing once I remove them, but if I had to reattach them the Pool’s data would all be intact – CORRECTION: Removing all of the disks associate to a Pool removes the Pool from Windows. If at any time any one of the disks are reattached the Pool will appear with an error message about missing disks, but attaching all of the disks for a pool will restore it just as it was before)
  5. Power off the PC, remove the old physical disks and install the four new disks

 

Original Storage Space Setup:

(Both of the Media and File History spaces will be using the new ReFS format included in Windows 8.1.  Among many of the cool new things this file system sports, it is capable of self-healing and does not require maintenance via chkdsk.  Neither Simple or Parity storage space types support ReFS; only mirrors.)

Creating the New Storage Space:

The Two Storage Pools:

From here I’ll be doing some simple copy operations from the command line to copy everything over:

C:\> robocopy E:\ G:\ /E

I’ll probably set both copy operations to run overnight the day before I have some time to do the physical swaps.  I’ll leave an update here once that’s done just so you can know how it went.