Patch Server for Jamf Pro

(TL;DR, gimme the link: https://github.com/brysontyrrell/PatchServer)

After several months of not getting around to it, my PatchServer project on GitHub is finally nearing a true 1.0 state.

I am openly asking for those who have been following this project, and those who are interested in this project, to download, use, and provide feedback on what should be finished before the release of Jamf Pro 10.2.

Please create issues on GitHub for bugs and feature requests that you would want to make the cut for 1.0.

Some time late last year (and I say some time because it’s all becoming a blur), I was brought into a meeting where I was shown our (Jamf’s) progress on providing a framework for customers to be able to create their own patch definitions. This framework would allow customers to setup their own patch servers and add them to their JSS.

A day or so later, I wrote the first rough version of my own implementation.

Backing up a sec:

What’s a patch definition?

In Jamf Pro v10 we introduced a feature called Patch Management. With this, you could subscribe to a number of software titles that Jamf curates and maintains. Once subscribed, your JSS will, on a schedule, read in the patch definitions of those software titles to stay updated.

For more about Patch Management, see the Jamf Pro Admin Guide (10.1):

These patch definitions (which are JSON data) contain historical information about a software title’s version history and requirements for determining if the software is installed on a managed computer. This allows admins to use the Patch Management feature to create reports and update policies to automatically patch those software titles on computers.

Of course, when these features came out there was one resounding question from nearly everyone:

“Why can’t we make our own patch definitions?”

External Patch Sources

The framework I mentioned above is the answer to this. In Jamf Pro 10.2+ you will have the option of adding External Patch Sources to your JSS. Then, in addition to the official Jamf software titles, you will be able to subscribe to your own and use the same reporting and policy features.

 

The external patch source must be a server your JSS is able to reach via HTTP/HTTPS. This patch server must expose the following endpoints:

  • /software
    This returns a JSON array of all the software titles that are available on this server. For example:

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfImaging", 
        "lastModified": "2018-02-03T03:34:36Z", 
        "name": "Jamf Imaging", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /software/TitleId,TitleId
    This returns the same JSON as above, but limited to the comma separated list of software titles. For example (passing JamfAdmin,JamfRemote):

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /patch/TitleId
    This returns the full patch definition JSON of the software title. Here is an abbreviated example:

    {
      "id": "JamfAdmin",
      "name": "Jamf Admin",
      "publisher": "Jamf", 
      "appName": "Jamf Admin.app", 
      "bundleId": "com.jamfsoftware.JamfAdmin", 
      "currentVersion": "10.1.1", 
      "lastModified": "2018-02-03T03:34:34Z", 
      "extensionAttributes": [
        {"ExtensionAttributeObjects"}
      ],
      "patches": [
        {"PatchObjects"}
      ], 
      "requirements": [
        {"RequirementsObjects"}
      ]
    }

If you had a patch server located at http://patch.my.org, the full URLs would be:

At this time, there is no product that Jamf is providing for customers to install and have a ready to use patch server. The focus has been on opening up the framework that the official patch source uses and allow customers to extend their environments through a little engineering work.

Not all of us are engineers, of course. Thus…

Enter: Patch Server

gui_01_index.png

I wanted to have a working patch server ready for the Jamf community in time for 10.2’s release. My initial patch server implementation (I call it an implementation because it’s one way of providing a patch source) achieved serving the proper JSON data for each of the endpoints described above using a database (SQLite) for the backend.

While my original goals were much grander, including the ability to fully manage a patch definition in a GUI instead of writing out JSON, I had to pare it back in order to get the project into a deliverable state.

In the past week I went through the code and ripped out everything that I felt was not needed, or doable. Then, I went through and added in new features (ported from another project) and streamlined the UI elements that were left.

This patch server features:

  • All required Jamf Pro endpoints to serve as an External Patch Source
  • An API for programmatic management of patch definitions and versions.
    • Create/delete patch definitions.
    • Add versions to existing patch definitions.
    • Create backup archives of all patch definitions.
  • UI for management of patch definitions.
  • Validation of uploaded patch definitions.
    gui_05_validation.png
  • Full user documentation at http://patchserver.readthedocs.io/
    patchserver_docs.png

    • UI Overview
    • Setup Instructions
    • API Documentation

Bring the Requests

Until Jamf Pro 10.2 is released, I’m not going to tag the project at a 1.0 version. If you are in Jamf’s beta program and testing 10.2, I invite you to give this a try and let me know what you think. Specifically, I’m asking for you do open up issues on GitHub for:

  • Bugs you find
  • Features you want, such as:
    • Connect to an actual database like MySQL (?)
  • Documentation you want, such as:
    • Instructions for installing on X

Not everything that is reported might get worked on, but the good news is I released the patch server under the MIT license. If you have some Python chops you can fork it and do whatever you want with the codebase to suit your needs!

But, I don’t wanna setup a server…

If you had that reaction to the idea of setting up your own external patch source, ask yourself if you match any of these descriptions:

  1. My JSS can talk to pretty much anything if I want it to,
  2. I want a patch server; I don’t want to host a patch server,
  3. It doesn’t matter where my patches live as long as I can get and manage them,
  4. Can’t this be a cloud thing?

If so… stayed tuned for a future blog post.

Advertisements

Building AWS Serverless Applications – Part 2

In my previous Serverless post I covered creating a basic “Hello World” app with a SAM template and a Lambda function written in Python. In this post we will extend the SAM template using a Swagger definition and enable a custom authorizer on our endpoint.

Extending with Swagger

Here is the SAM template from the previous post:

serverless_02

Here, our API Gateway resource is implicitly defined by way of the GetApi event on our Lambda function. This tells CloudFormation that an API Gateway with a path of “/” and the GET method must be created to execute the Lambda function.

The API Gateway supports a lot of features and functionality, but much of it is not a part of the SAM spec. In order to extend our API Gateway beyond basic Lambda functions executed upon requests to endpoints, we will need to define an AWS::Serverless::Api resource and provide a Swagger definition. Within that Swagger definition we will be able to extend our API.

Here is our application above with the API resource added:

serverless_07.png

Let’s focus in on the new API resource.

serverless_08.png

Before, our API Gateway was being created with the default Stage and Prod stages. Now that we are defining the resource we must provide the stage name. Otherwise, the rest of this object is the Swagger definition.

There are two ways of setting the Swagger definition. Inline Swagger definitions use the DefinitionBody key as shown above. To point to the location of a Swagger definition file you can use the DefinitionUri key. This key works just like the CodeUri key for our Lambda function.

The info:title key is being set to the name of the CloudFormation stack we deploy our function to. The keys under path should all be the path of the API Gateway endpoint that is being defined, and each key for the path should be an HTTP method. We only have one path, “/”, and it only supports one method, GET.

Here we encounter our first of AWS’s Swagger Extensions for API Gateway. x-amazon-apigateway-integration allows API Gateway to integrate with different backends including HTTP services, SNS, SQS, and Lambda. The type: aws_proxy is specifically for Lambda functions. You may be confused about the httpMethod: post here as our endpoint is set to use the GET method. This is specifically for API Gateway invoking the Lambda function and does not involve the HTTP request.

Then we have the Uri to the backend integration:

arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorld.Arn}/invocations

The Fn::Sub function is a CloudFront template function for replacing values within a string. This ARN (an AWS resource identifier) populates the Region our stack is running in, and the ARN of our Lambda function that is also defined in our template.

Which, we have only made one change to our Lambda resource:

serverless_09.png

Under the event, we have added a reference (Ref) to our API Gateway resource. Ref is another CloudFormation function that returns the value of an item (in this case, our API Gateway resource returns it’s ID when referenced).

At this point, you may notice that we are now doubling up on how we define our API. Not only do we need to define all of our endpoints in the Swagger definition, but we must still write our Lambda functions in full. While not ideal, it is necessary as we move into advanced integrations between API Gateway and Lambda.

Custom Authorizers

A Custom Authorizer is a Lambda function attached to an API Gateway endpoint. This Lambda function is invoked when a request to that endpoint is made. Based upon authorization tokens, headers, query strings, stage variables, or context parameters. Once the request is evaluated, it is either denied or the Lambda function will return an IAM policy granting the request. It is then allowed through as normal.

You can only define a custom authorizer in your SAM template by using an AWS Swagger extension. Here is our updated SAM template with the custom authorizer included:

serverless_10.png

The Swagger definition has been expanded and we have added another Lambda function, but this time with a Lambda Permission resource.

serverless_11.png

The securityDefinition key is where we define our authentication methods for our API. We are using two additional AWS Swagger extensions to define the custom authorizer: x-amazon-apigateway-authtype and x-amazon-apigateway-authorizer. The former is always custom for a custom authorizer.

The the example authorizer in my example has a static token that it validates. so we specify the type as token. The authorizerUri is the same style ARN as above with the x-amazon-apigateway-integration, but using the ARN of our authorizer’s Lambda function. The authorizerResultTtlInSeconds if the amount of time the resulting IAM policy is cached. More on that in a bit.

serverless_12.png

Our definition of the authorizer’s Lambda function is bare bones. There are no events that trigger it as a part of the SAM spec – that is defined in our Swagger definition. Because we have no events, we must create the permissions resource that allows API Gateway to invoke this function.

Here is the Python code for the authorizer function:

serverless_13.png

The Custom Authorizer only needs to perform two tasks: 1) validate the request by the chosen method (in our example: the Authorization header’s value which must match ‘Bearer a.b.c’), and 2) responds with an object containing a principalId and an IAM policy.

The principalId comes into play with the authorizerResultTtlInSeconds value above. API Gateway will cache the result of an authorization for an amount of time equal to this value. If the origin repeats a request, and they have already authenticated, they will be allowed through without invoking the custom authorizer again! In my example I am passing the Authorization header’s value as the principal ID.

The IAM policy is a document that defines permissions for the request. You can specify actions to ALLOW and actions to DENY for a given Resource (in this case, granting execute-api:Invoke for the HTTP endpoint that was requested).

In my example, a successful authentication generates the following response to API Gateway:

serverless_15.png

Our project directory now has the following structure:

serverless_14.png

We can update the application stack deployed in the previous post by re-running our package and deploy commands on the same stack name. Alternatively, you can deploy to an entirely new stack using the same commands.

serverless_04

serverless_05

Now if we attempt to call our API we will receive an error message telling us we are unauthorized. Reattempt with the Authorization header in place and we get a successful response!

serverless_16.png

 

Up Next

Now that we have covered how to extend our serverless application using a Swagger definition and secured it with a Custom Authorizer, we can expand into having an S3 bucket provide back-end storage and use API Gateway’s binary data support to POST and GET files!

You can find the templates and code shown in this post here on my GitHub:

https://github.com/brysontyrrell/Serverless-Hello-World

Building AWS Serverless Applications – Part 1

The AWS Serverless Application Model (SAM) is an exciting technology stack to start building applications upon. You can write and deploy large web and/or event-based apps and services without worrying about up-front costs, scalability, and infrastructure.

SAM apps are defined by a YAML template that you deploy using AWS CloudFormation. While basic applications are easy to define, things become more complex as you begin to add in more advanced features like Authorizers and Binary data support (more on those later).

In this post, I’m going to walk through a simple Serverless Hello World app.

In a later post, we will then extend this to include token authentication and serving image files.

You can learn more about SAM here at the  project’s GitHub page.

A Basic SAM App

My approach to serverless applications is to contain all of the templates and function code within a single repository. The SAM template remains at the top level with each Lambda function within it’s own sub-directory.

Here is the beginning structure of our Hello World app’s repository:

serverless_01.png

First, let’s look at the contents of our template.yaml file:

serverless_02.png

The SAM template does a lot of behind-the-scenes magic for us when we go to deploy with CloudFormation. Note the Transform key at the top with the AWS::Serverless-2016-10-31 value. This is an instruction that tells CloudFormation to take the template content and translate it into native CloudFormation. Our templates would be much, much larger and more complex if we had to write them in native CloudFormation objects.

Moving on to Resources, this is where we define the Lambda functions that will perform the work for our app. There one function for our app, HelloWorldFunc (the key is just a label for the resource), with the Type is AWS::Serverless::Function.

Under the Properties of our function we further define it and how it is invoked. The CodeUri key is the path to the code that makes up the function. In this case it is a relative path to our /HelloWorldFunc/ directory (it must be a directory and not a file).

The Handler is two parts: filename.function. Within /HelloWorldFunc/ we have a single Python file: lambda_function.py. In that file is the function that will be executed when the file is run in Lambda: lambda_handler().

You don’t need to name your files and functions this way. Your handler value just needs to point to the right file and function within that file.

Set the Runtime value to the language you are using for the Lambda function. In this case it is python3.6, but Lambda supports a number of languages, and because they are all independent you can mix-and-match across multiple functions (e.g. one can be Python and the other nodeJS).

Events define how the function is invoked. There are a lot of different types of events you can use as a part of the SAM spec, but here we have defined an API operation. GetApi is the label for this event (you can define as many events as you want), the Type is Api which will be through API Gateway, and for the API’s properties the Path is “/” (or the root of the domain) and the Method is get (standard HTTP methods).

Nothing else needs to be defined. I’ll explain why in the “Package & Deploy” section.

Now, let’s look at the code in /HelloWorldFunc/lambda_function.py:

serverless_03.png

It’s not a straightforward print(“Hello World”) as you might have been expecting. The integration between API Gateway and Lambda uses what’s called Lambda Proxy. When the API Gateway invokes the Lambda function it expects back a dictionary object that provides what the HTTP response to the initiating client should be.

  • isBase64Encoded is a boolean flag to tell API Gateway if your response body is a bas64 encoded binary blob. You’ll set this to True if you’re returning a file that will need to be base64 encoded into the body.
  • statusCode is the HTTP code to responde with. 2xx for success, 4xx for an error due to the client request.
  • body is the content you are returning. It must be a string and not any other data type. If you’re returning binary content (e.g. a file), you will need to base63 encode it, put that string into this body key and set isBase64Encoded to True.
  • headers must be set here with at least the Content-Type of the response body. You can also set whatever other headers you want.

Package and Deploy

Using the AWS CLI, we can easily package our Lambda functions, upload them to S3, update the template, and deploy using CloudFormation in two steps.

First, we will need to run the package command from the repository’s directory:

serverless_04.png

  • –template-file is the filename of the SAM template.
  • –s3-bucket is the name of an S3 bucket you have access and can upload files to.
  • –output-template-file is a filename you provide for an updated SAM template that will be generated by this command. If you don’t specify this key, the template data will be sent to STDOUT.

Packaging converts the code for the Lambda functions into zip files that CloudFormation will use when creating the AWS objects. In the template example above, the CodeUri key is a directory path. Everything within that directory will be put into the zipped artifact for the function and uploaded to the specified S3 bucket.

The CodeUri must be either a directory path or an S3 location. In a later post, I’ll talk about deployment workflows where you may need to handle the packaging on your own and then update your template.

Our newly created deployment.yaml file will contain the S3 locations of the zipped artifacts for each function in your SAM template in their CodeUri key – replacing the directory paths in the source template.

We can now deploy our packaged serverless application:

serverless_05.png

  • –template-file is the filename of our deployment template.
  • –stack-name is the name we are going to give to the CloudFormation stack that will be created.
  • –capabilities CAPABILITY_IAM is granting CloudFormation permissions to generate IAM roles and permissions.

Whenever you execute a CloudFormation template (SAM is an extension to CloudFormation) you are creating a stack. The stack is the deployed application according to the template for the given name.

The first time you run this command, CloudFormation is going to create a brand new stack and all of the resources you’ve defined. The next time you run your template and specify this stack name it will update all of the stack’s resources according to whatever changes you have defined from the last run to the current run. This include the creation, updating, and deletion of resources. CloudFormation handles it all for you.

We can give a different name for the stack and run the template again to create a second running instance of our serverless application. This is an important feature of using CloudFormation: you can re-use the same template multiple times for completely different stacks. This means you can deploy to an entire test/staging environment before deploying to your production environment using the same template.

You should see the following output as the stack is created:

serverless_06.png

You can log into your AWS console and navigate to the CloudFormation page (be sure you’re in the right region!) to see everything that has been created for this stack:

serverless_07.png

Click on Resources tab and you will be able to navigate to the page for each resources that was created as a part of the stack. Go to the API Gateway, click on Stages and the Prod stage. You’ll be presented with the default generated URL.

Go there and you should see:

serverless_09.png

And there we have the Serverless Hello World app.

Up Next

There is, of course, so much more that can be done through the Serverless Application Model, and many more AWS services to tie into!

The next article I write will cover extending our Hello World app with a Swagger definition to lock it behind basic token authentication. From there, we will continue to build upon that swagger definition to enable binary data support and serve images from an S3 bucket!

You can find the templates and code shown in this post here on my GitHub:

https://github.com/brysontyrrell/Serverless-Hello-World

Stop everything! Start using Pipenv!

Seven months ago I tweeted about Pipenv:

That was about it.

The project- which was at v3.6.1 at the time – still felt new. At Jamf, we were working on our strategy for handling project requirements and dependencies going forward for Jamf Cloud’s automation apps and Pipenv, while a cool idea, didn’t have maturity and traction that.

As @elios pointed out on the MacAdmins Slack: Pipenv was announced in January of this year as an experimental project.

Today it is the officially recommended Python packaging tool from Python.org.

In his words, “That’s awfully fast…”

The GitHub repository for Pipenv has 5400+ stars and over 100+ contributors to the project. It is proceeding at breakneck development speeds and adding all kinds of juicy features every time you turn around. It’s on v8.3.2.

Within ten minutes of finally, FINALLY, giving Pipenv a try today I made the conscious decision to chuck the tooling we had made and convert all of our projects over.

(Oh, and the person who wrote that tooling wholly agreed)

Let’s talk about that.

Part 1: Why requirements.txt sucks

There is was this standard workflow to your Python projects. You needed two key tools: pip and virtualenv. When developing, you would create a Python virtual environment using virtualenv in order to isolate your project from the system Python or other virtual environments.

You would then use pip to install all the packages your project needs. Once you had all of those packages installed you would then run a command to generate a requirements.txt file that listed every package at the exact installed version.

The requirements.txt file stays with your project. When someone else gets a copy of the repository to work on it or deploy it they would follow the same process of creating a virtual environment and then using pip to install all of the required packages by feeding that requirements.txt file into it.

Now, that doesn’t seem so bad, right? The drawbacks become apparent as you begin to craft your build pipelines around your projects.

Maintenance is a Manual Chore

The first thing to call out is how manual this process is. You need to create the same version virtual environment for the project once you’ve cloned it and then install the packages. If you want to begin upgrading packages you need to be sure to manually export a new requirements.txt file each time.

Removing packages is far more problematic. Uninstalling with pip will not remove dependencies for a package! Take Flask, for example. It’s not just one package. It has four sub-packages that get installed with it, and one of those (Jinja2) has its own sub-package.

So, maybe the answer is to manually maintain only the top-level packages in your requirements.txt file? That’s a hard no. You don’t want to do that because it makes your environment no longer deterministic. What do I mean? I mean, those sub-packages we just talked about are NOT fixed versions. When you’re installing Python packages from the Python Package Index (PyPI) the aren’t using requirements.txt files with fixed versions for their dependencies. They instead are specifying ranges allowing for installs at a minimum and/or maximum version for that sub-package.

This makes a lot of sense when you think about it. When installing a package for the first time as a part of a new project, or updating a package in an existing project, you are going to want the latest allowed versions for all the sub-packages for any patches that have been made.

Not so much when it comes to a project/application. You developed and tested your code at specific package versions that are a known working state. This is past the development phase when bugs from updates to those packages can be identified and addressed. When it comes to deployment, your project’s requirements.txt file must be deterministic and list every package at the exact version.

Hence, maintenance of this file becomes a manual chore.

But Wait, There’s Testing!

There are a lot of packages that you might need to install that have nothing to do with running your Python application, but they have everything to do with running tests and building documentation. Common package requirements for this would by Pytest, Tox, and Sphinx with maybe a theme. Important, but not needed in a deployment.

We want to be able to specify a set of packages to install that will be used during the testing and build process. The answer unfortunately, is a second requirements.txt file. This one would have a special name like requirements-dev.txt which is only used during a build. This file could only contain the specific packages and be installed with pip after the standard requirements.txt, or it could contain all of those plus the build packages. In either case, the maintenance problem continues to grow.

So We Came Up With Something…

Our process at Jamf ended up settling on three types of requirements.txt files in an effort to address all the shortcomings described.

  • requirements.txt
    This file contained the full package list for the project at fixed versions. It would be used for spinning up new development environments or during deployment.
  • core-requirements.txt
    This file contained the top-level packages without fixed versions. Development environments would not be built from this. Instead, this file would be used for recreating the standard requirements.txt file at updated package versions and eliminating orphaned packages that were no longer used or removed from the project at some point.
  • build-requirements.txt
    This is a manually maintained file per-project that only contained the additional packages needed during the build process.

This approach isn’t too dissimilar to what others have implemented for many of the same reasons we came up with.

Part 2: Enter Pipfile

Not Pipenv? We’re getting there.

Pipfile was introduced almost exactly a year ago at the time of this post (first commit on November 18th, 2016). The goal of Pipfile is to replace requirements.txt and address the pain points that we covered above.

Here is what a Pipfile looks like:

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]



[requires]

python_version = "2.7"

PyPI is the default source in all new Pipfiles. Using the scheme shown, you can add additional [source] sections with unique names to specify your internal Python package indexes (we have our own at Jamf for shared packages across projects).

There are two groups for listing packages: dev-packages and packages. In a Pipfile these lists follow how we were handling this where only the build packages go into the dev-packages list and all project packages go under the standard packages list.

Packages in the Pipfile can have fixed versions or be set to install whatever the latest version is.

[packages]

flask = "*"
requests = "==2.18.4"

The [requires] section dictates the version of Python that the project is meant to run under.

From the Pipfile you would create what is called a Pipfile.lock which contains all of the environment information, the installed packages, their installed versions, and their SHA256 hashes. The hashes are a recent security feature of pip to validate packages that are installed when deployed. If there is a hash mismatch you can abort. A powerful security tool in preventing malicious from entering your environments.

Note that you can specify the SHA256 hashes of packages in a normal requirements.txt file. This is a feature of pip and not of Pipfile.

It is this Pipfile.lock that will be used to generate environments on other systems whether for development or deployment. The Pipfile will be used for maintaining packages and dependencies and regenerating your Pipfile.lock.

All of this is just a specification. Pip as of yet still does not support Pipfiles, but there is another..

Part 3: Pipenv Cometh

Pipenv is the implementation of the Pipfile standard. It is built on top of pip and virtualenv and manages both your environment and package dependencies, and it does so like a boss.

Get Started

Install Pipenv with pip (or homebrew if that’s your jam).

$ pip install pipenv

Then in a project directory create a virtual environment and install some packages!

$ pipenv --two
Creating a virtualenv for this project…
<...install output...>
Virtualenv location: /Users/me/.local/share/virtualenvs/test-zslr3BOw
Creating a Pipfile for this project…

$ pipenv install flask
Installing flask…
<...install output...>
Adding flask to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (36eec0)!

$ pipenv install requests==2.18.4
Installing requests==2.18.4…
<...install output...>
Adding requests==2.18.4 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (572f23)!

$

Notice how we see the Locking messages after every install? Pipenv automatically regenerated the Pipfile.lock each time the Pipfile is modified. Your fixed environment is being automatically maintained!

Graph All The Things

Let’s look inside the Pipfile itself.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]

flask = "*"
requests = "==2.18.4"


[requires]

python_version = "2.7"

No sub-packages? Nope. It doesn’t need to track those (they end up in the Pipfile.lock, remember?). But, if you’re curious, you can use the handy graph feature to view a full dependency tree of your project!

$ pipenv graph
Flask==0.12.2
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
   - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
 
$

Check that out! Notice how you can see the requirement that was specified for the sub-package in addition to the actual installed version?

Environment Management Magic

Now let’s uninstall Flask.

$ pipenv uninstall flask
Un-installing flask…
<...uninstall output...>
Removing flask from Pipfile…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (4ddcaf)!

$

And re-run the graph command.

$ pipenv graph
click==6.7
itsdangerous==0.24
Jinja2==2.9.6
 - MarkupSafe [required: >=0.23, installed: 1.0]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
Werkzeug==0.12.2

 

Yes, the sub-packages have now been orphaned within the existing virtual environment, but that’s not the real story. If we look inside Pipfile we’ll see that requests is the only package listed, and if we look inside our Pipfile.lock we will see that only requests and it’s sub-packages are present.

We can regenerate our virtual environment cleanly with only a few commands!

$ pipenv uninstall --all
Un-installing all packages from virtualenv…
Found 20 installed package(s), purging…
<...uninstall output...>
Environment now purged and fresh!

$ pipenv install
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell

$ pipenv graph
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

$

Mindblowing!!!

Installing the dev-packages for our builds uses an additional flag with the install command.

$ pipenv install sphinx --dev
Installing sphinx…
<...install output...>
Adding sphinx to Pipfile's [dev-packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (d7ccf2)!

The appropriate locations in our Pipfile and Pipefile.lock have been updated! To install the dev environment perform the same steps for regenerating above but add the –dev flag.

$ pipenv install --dev
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 18/18 — 00:00:05
To activate this project's virtualenv, run the following:
 $ pipenv shell

Part 4: Deploy Stuff!

The first project I decided to apply Pipenv to in order to learn the tool is ODST. While there is a nice feature in Pipenv where it will automatically import a requirements.txt file if detected, I opted to start clean and install all my top-level packages directly. This gave me a proper Pipfile and Pipfile.lock.

Here’s the resulting Pipfile.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]

pytest = "*"
sphinx = "*"
sphinx-rtd-theme = "*"


[packages]

flask = "*"
cryptography = "*"
celery = "*"
psutil = "*"
flask-sqlalchemy = "*"
pymysql = "*"
requests = "*"
dicttoxml = "*"
pyjwt = "*"
flask-login = "*"
redis = "*"


[requires]

python_version = "2.7"

Here’s the graph of the installed packages (without the dev-packages).

$ pipenv graph
celery==4.1.0
 - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.3]
 - kombu [required: <5.0,>=4.0.2, installed: 4.1.0]
 - amqp [required: >=2.1.4,<3.0, installed: 2.2.2]
 - vine [required: >=1.1.3, installed: 1.1.4]
 - pytz [required: >dev, installed: 2017.3]
cryptography==2.1.3
 - asn1crypto [required: >=0.21.0, installed: 0.23.0]
 - cffi [required: >=1.7, installed: 1.11.2]
 - pycparser [required: Any, installed: 2.18]
 - enum34 [required: Any, installed: 1.1.6]
 - idna [required: >=2.1, installed: 2.6]
 - ipaddress [required: Any, installed: 1.0.18]
 - six [required: >=1.4.1, installed: 1.11.0]
dicttoxml==1.7.4
Flask-Login==0.4.0
 - Flask [required: Any, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
Flask-SQLAlchemy==2.3.2
 - Flask [required: >=0.10, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
 - SQLAlchemy [required: >=0.8.0, installed: 1.1.15]
psutil==5.4.0
PyJWT==1.5.3
PyMySQL==0.7.11
redis==2.10.6
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

In my Dockerfiles for the project I swapped out using requirements.txt and switched it to install my packages using the Pipefile.lock into the system Python (in a containerized app there’s no real need to create a virtual environment).

RUN /usr/bin/apt-get update -q && \
    /usr/bin/apt-get install -qqy build-essential git && \
    /usr/bin/apt-get install -qqy python-pip python-dev && \
    /usr/bin/pip install pipenv && \
    /usr/bin/apt-get install -qqy libssl-dev libffi-dev && \
    /usr/bin/apt-get install -qqy uwsgi uwsgi-plugin-python && \
    /usr/bin/apt-get clean && \
    /bin/rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY /docker/webapp/web-app.ini /etc/uwsgi/apps-enabled/

COPY /Pipfile* /opt/odst/
WORKDIR /opt/odst/
RUN /usr/local/bin/pipenv install --system --deploy

COPY /ods/ /opt/odst/ods/
COPY /application.py /opt/odst/
COPY /docker/ods_conf.cfg /opt/odst/

RUN /bin/chown -R www-data:www-data /opt/odst/ods/static

CMD ["/usr/bin/uwsgi", "--ini", "/etc/uwsgi/apps-enabled/web-app.ini"]

The –system flag tells Pipenv to install to the system Python. The –deploy flag will abort the installation if the Pipfile.lock is out of date or the Python version is wrong. Out of date? Pipenv knows if your Pipfile.lock matches up with the Pipfile by comparing a SHA256 hash of the Pipfile that it has saved. If there’s a mismatch then there is an issue and it aborts.

If I didn’t want to rely on this hash match feature, I could instead pass a –ignore-pipfile flag that will tell it to proceed using only the Pipfile.lock.

I should mention at this point that Pipenv dramatically speeds up build times. When using pip your packages will install sequentially. Pipenv will install packages in parallel. The difference is immediately noticeable, especially when regenerating your development environments.

Still, The World is Catching Up

Python development workflows area clearly moving to this tool, but Pipenv is less than a year old at this point and not everything will support generating environments from the provided Pipfiles.

The two I am currently using that fall in that category are AWS Elastic Beanstalk and ReadTheDocs.org. Both need a requirements.txt file in order to build the Python environments.

That is thankfully a simple task. You can generate a requirements.txt file from Pipenv by using the lock option.

$ pipenv lock --requirements > requirements.txt

 

For Elastic Beanstalk applications, I can have this command run as a part of my build pipeline so the file is included with the ZIP archive before it goes to S3.

In the case of Read the Docs there is an open issue for adding Pipfile support, but until then I will need to generate the requirements.txt file as I make changes to my environment and save it with the repository.

For Read the Docs I’ll want the extra dev-packages. This is done by using the pipenv run command. This will execute the succeeding string as if it were run in the environment.

$ pipenv run pip freeze > requirements.txt

The bright side is that I am no longer actually managing this file. It is straight output from my Pipenv managed environment!

Part 5: Profit

I hope you enjoyed my overview and impressions of Pipenv. It’s a tool that is going to have a huge impact on development and deployment workflows – for the better!

A huge shout out to Kenneth Reitz for creating yet another invaluable package for the Python community!

* Edit Nov 9: Changed the command for exporting a requirements.txt file. Using pipenv lock -r -d only outputs the packages under the dev-packages section.

Farewell to the Unofficial JSS API Docs

Hey everyone.

With the launch of the Jamf Developer Portal I think it’s time I took down my Unofficial JSS API Docs site on Confluence.

I launched it as a community resource for a lack of API documentation, but now that Jamf has something out there I feel it’s time I save my $10 a month. If you found these resources helpful in the past, great! That was the whole point.

The site will come down after November 17th. For those Google searching and coming across this post, click on the dev portal link I provided above to reach the official documentation provided by Jamf.

Open Distribution Server Technology (w/JNUC Recap)

ODST @JNUC

At JNUC 2017, I was given the opportunity to do a session detailing the progress I’ve made and the vision I have for a new file distribution server that can serve to replace the now discontinued JDS (Jamf Distribution Server).

This was a last minute addition to the conference schedule and we were unable to record it, but the Mac admin community took notes which can be found here. I’ve also uploaded the presentation’s slide deck on SlideShare.

The source code for ODST is available on GitHub. It is currently in an early Alpha state with some of the core functionality complete.

Project Goals

ODST came about with the sunsetting of the JDS. I set out to design my own implementation of an automated file distribution server but with additional features to make it a more powerful component of an administrator’s environment.

The goal of ODST is to provide an on-premise file syncing and distribution server solution that puts automation and integration features first.

The ODS (Open Distribution Server) application itself is modular and being designed to fit into as many deployment models as possible. This ranges from a simple single-server installation on Linux, Windows, or macOS to containerized deployments in Docker or Kubernetes.

While there will be initial support for the ODS to integrate with Jamf Pro it is not a requirement for using the application. This will allow administrators using other management tools to take advantage of the solution and submit feature requests for integrations with them as well.

Planned Features

  • A full web interface (built on top of the Admin API)
  • The Admin API for integrating your ODS instances with existing automations and workflows.
  • Many-to-many registration and syncing which will allow package uploads to any ODS and still replicate throughout your network.
  • Package and ODS staging tags to restrict how certain levels of packages replicate through the network.
  • Webhooks and email to send notifications to other services alerting them to events that are occurring on your ODS instances.
  • LDAP integration for better control and accountability when granting other administrators and techs access to your ODS instances.
  • And more to come…

Package Syncing

Where the JDS synced by running an every five minute loops task to poll another server, the ODS application uses a private ODS API for communicating between instances.

When two ODS instances are registered to each other they will have each others’ keys saved to their databases and use those keys to sign API requests.

The standard order of operations during a package upload would be:

  1. The admin uploads a package to ODS1.
  2. ODS1 generates the SHA1 hash of the package and also generates SHA1 hashes for every 1 megabyte chunk of that package. This information is saved to the database.
  3. ODS1 sends a notification to every registered ODS instance that a new package is available.
  4. ODS2 receives this notification and makes a return API request for the full details of the package.
  5. ODS2 saves the pending package to the database and a download task is sent to the queue.
  6. The ODS2 worker takes the download task off the queue and begins downloading the package in 1 megabyte chunks, comparing hashes for every chunk, and saving them to a temporary location.
  7. Once the ODS2 worker has downloaded all chunks it recombines them to the single file, performs a final SHA1 check, and moves the package to the public download directory.
  8. ODS2 then performs step #3 to propagate the package to other ODS instances it is registered with.

If the download process seems familiar, it is borrowed from how Apple performs MDM initiated application installs.

Application Architecture

The ODS application is more complex than the JDS in order to facilitate the additional features that are being built on top of the file syncing. In addition to the application server, a production deployment would also include a front-end web server (Nginx or Apache), a Redis server for the queuing system, a database server (ODST falls back to a local SQLite database file if there is not a database service to connect to), and workers that process queued actions.

Single Server

ODS_Single_Server.png

Multi-Server or Containerized

ODS_Multi_or_Containerized.png

The queuing system is an important element as it backgrounds many of the processes that the server will need to perform in reaction to notifications or requests (such as queuing notifications, API requests to other ODS instances, file downloads, and file hashing operations). This frees up the application to continue accepting requests by removes long process blocks.

How the Community Can Help

When I gave the JNUC presentation I only took up half of the allotted time to discuss what was completed with the project and what was planned. The second half was spent in open discussion to take in feedback and guidance from the target audience on what was needed on the road to a 1.0 release.

Adding LDAP support was the first item to come out of this and is my next planned feature to write in after the file syncing framework is finished. I encouraged participants to open GitHub issues on the repo as we discussed their questions and asks. I want to continue to encourage this. The ODST project is meant for the community and should continue to be community driven in its roadmap.

When it comes to contributing to the project I am not asking for code help at this time. Don’t feel that you need to know Python or web development with Flask in order to contribute. There are many other areas that I am in need of help:

  • Testing! As I make new commits to the repository and add in more features you can help ensure everything is working by running the latest version and trying them out. Submit issues, provide logs, provide details on how you’re deploying the application (the provided Docker Compose file is the quickest and easiest way), and by doing so you will help verify features work as expected and solidify the quality of the application.
  • Determine optimal configurations. There are quite a few components to the ODS application and I am learning as I go for how to configure the web server. More experienced administrators who are familiar with these technologies, especially in production environments, can help work towards a baseline for…
  • Installers! The ODS application can be custom setup for almost any kind of deployment, but we still want an easy option where an admin can grab an installer for load it onto a single Linux or Windows server. If you have experience building installers on those platforms please reach out! I’ve also mentioned containerization a few times, and having an official Docker images for the ODS application and worker components should be a part of this initiative.
  • Documentation. Much Documentation. There will be official docs available at odst.readthedocs.io which will be generated from the main repository on GitHub. You can help maintain and improve that documentation with pull requests as you find errors or inaccurate instructions/details as the project iterates. The documentation will be especially invaluable when it comes to the aforementioned installers, custom installations, and the administrator user guide portion that will walk user through how to perform actions.

If you haven’t yet, please join the #odst channel in the Mac Admins Slack where you can discuss the project with me directly as well as other admins who are using, testing, and contributing as they can.

I hope to build something that will provide great value to our community and fill the gap the JDS left in a lot of environments. I hope to see you on GitHub and Slack soon!

Build Your Own Jamf Pro Integrations: Part I An Intro to Webhooks and Flask

Welcome back to the BYO Jamf Pro Integrations tutorial! In Part I we will be giving an introduction to both the webhooks feature of Jamf Pro and the Flask microframework that you installed into your virtual environment in the introduction post.

Webhooks in Jamf Pro

Webhooks are a framework introduced in Jamf Pro v9.93. A webhook itself is an HTTP callback: an HTTP POST that occurs when something happens.

A webhook itself is an HTTP request made by a server to a destination, with a payload, in response to an event. Jamf Pro’s webhooks are built directly on top of a pre-existing Java API called the Events API.

The Events API allowed for Java plugins as .jar files to be installed on the Jamf Pro server. While not an option for Jamf Cloud customers, self-hosted users can take advantage of this. Learn more here: https://github.com/jamf/JSSEventsAPI

While not 100%, the available events for Jamf Pro Webhooks closely matches the list for the Events API.

  • ComputerAdded
  • ComputerCheckIn
  • ComputerInventoryCompleted
  • ComputerPolicyFinished
  • ComputerPushCapabilityChanged
  • JSSShutdown
  • JSSStartup
  • MobileDeviceCheckIn
  • MobileDeviceCommandCompleted
  • MobileDeviceEnrolled
  • MobileDevicePushSent
  • MobileDeviceUnEnrolled
  • PatchSoftwareTitleUpdated
  • PushSent
  • RestAPIOperation
  • SCEPChallenge
  • SmartGroupComputerMembershipChange
  • SmartGroupMobileDeviceMembershipChange

To setup a webhook for one of these events, log into your Jamf Pro Server and navigate to Settings -> Global Management -> Webhooks. Click the + New button and you will be taken to a screen to set and select the following:

  • Name
    A description.
  • URL
    The address that you want Jamf Pro to send the event data to.
  • Content Type
    Choose whether that data in in XML or JSON format.
  • Event
    The event from the list above that you want to send on.

When you setup your webhook, Jamf Pro will send an HTTP POST with a payload of the content type you selected containing contextual data on the event. These payloads are broken into two parts: the webhook and eventObject/event keys.

Here is an example in XML:

<JSSEvent>
    <webhook>
        <id>1</id>
        <name></name>
        <webhookEvent>JSSShutdown</webhookEvent>
    </webhook>
    <eventObject>
        <institution></institution>
        <hostAddress></hostAddress>
        <webApplicationPath></webApplicationPath>
        <isClusterMaster>false</isClusterMaster>
        <jssUrl></jssUrl>
    </eventObject>
</JSSEvent>

Here is that same example as JSON:

{
    "webhook": {
        "id": 1,
        "name": "",
        "webhookEvent": "JSSShutdown"
    },
    "event": {
        "institution": "",
        "hostAddress": "",
        "webApplicationPath": "",
        "isClusterMaster": false,
        "jssUrl": ""
    }
}

The webhook key is about the Jamf Pro Webhook itself. The database ID, name you set and the type of event are contained here. This is data that, later in the tutorials, can be used to identify which events you are receiving from Jamf Pro.

The eventObject (XML) or event (JSON) key contains the contextual data of what triggered the event, or contextual data about the event depending upon which event was triggered. Many of the events all send the exact same data under this key with the difference will be what is contained under the webhook key. This is, for examples, true for Computer* and MobileDevice* events.

You can dive into full examples of every webhook event in XML and JSON formats at the Unofficial JSS API Docs site: https://unofficial-jss-api-docs.atlassian.net/wiki/spaces/JRA/pages/14450694/Webhooks+API

Flask

There needs to be something on the receiving end, the destination URL of the Jamf Pro webhook, to receive the payload and process it / take action on. This is where web development enters the picture. The type of integration that works with webhooks is a web app with endpoints that can accept POST requests.

There are many web technologies out there for all kinds of programming languages. The tutorial series will focus on Python (very popular in the Mac admin community for scripting alongside Ruby) and a microframework called Flask.

The “micro” portion means that Flask does not contains many elements of larger frameworks like a pre-defined database interface. Instead, Flask relies on extensions that plug into the framework and extend the functionality of your code. Flask also does not dictate design choices. Flask apps can be hundreds of files in size, structured in nearly any way, or just one single file. The size and complexity of the project is determined by the scope of your work.

Here is the absolute smallest Flask app that you could write (and you can use this as the boilerplate code to start any of your projects from):

# my-jamf-app.py
import flask

app = flask.Flask(__name__)


@app.route('/')
def root():
    return "Hello Penn State MacAdmins!"

if __name__ == '__main__':
    app.run()

Seven lines.

You should have Flask installed and available for your project in the virtual environment created in the first post. At the top of our file we are importing the package.

Then we create an app object that is an instance of the flask.Flask() class:

app = flask.Flask(__name__)

This object will represent the web app throughout our code. To add endpoints, or routes, to the web app we will use the route() decorator.

Decorators are a special kind of Python syntax that “wrap” a function around another function.

In this case, the route() decorator will register an endpoint based upon the path we give as it’s first argument and then execute the wrapped function below it whenever that endpoint is requested! You’ll be able to see clearly this in a moment.

@app.route('/')
def root():
    return "Hello Penn State MacAdmins!"

The“/” path means the root of the web server. Once this app is running you will be able to reach it in your web browser by navigating to http://localhost:5000. The “/” path resolves to that address.

Flask has a built in development server you can start by calling the run() method on the app object. When you call a Python file as a script from the command line the __name__ dunder becomes set to a value of __main__.

If you are a little confused by the word “dunder” at this point don’t worry, you can continue on without understanding some of these concepts, but you may want to brush up on your Python with some online resources.

By checking if the __name__ dunder is __main__ you can control what your Python scripts do based on whether they have been called from the command line or, later on, imported into other Python code. When imported, the __name__ dunder takes on the name of the file!

So, the last two lines of this single file Flask app mean it will only run the app using the development server if it has been called as a script from the command line:

if __name__ == '__main__':
    app.run()
(byojamf) ~$ python /path/to/my-jamf-app.py
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Your web app is now available when you navigate to http://localhost:5000 in your web browser. Give it a try and you should see the text message in the code! See how when you request the “/” or root of the web app the root() function is executing?

Decorators in action.

Next up…

Accepting Webhooks and Testing with Jook

I’ll see you in Part II tomorrow!