CommunityPatch.com (beta)

In previous posts, I talked about two projects I had been working on for the Jamf community to better help admins get started using the new “External Patch Sources” feature in Jamf Pro 10.2+. While working on Patch Server and the companion Patch-Starter-Script, I also wrote a quick proof of concept for a serverless version that would run in an AWS account.

The Stupid Simple Patch Server uses API Gateway and Lambda functions to serve patch definitions you stored in an S3 bucket. I even included the same API endpoints from the Patch Server so workflows between the two could be shared. I even took it a step further and added a subscription API so it would sync with a remote patch definition via URL.

That side project (of a side project) made me think about how I could take the basic design and build upon it into something that could be used by multiple admins. At first, I wrote a lot of code to transform the Stupid Simple Patch Server into a multi-tenant application. At a point, I considered the limitations of what could be done in a manner that could be considered secure and scrapped much of it.

But not everything. The work I had done was retooled into a new concept: a single, public, community managed patch source for Jamf Pro. A service where anyone could contribute patch definitions, and be able to manage and update them. Five minutes after having this idea I bought the communitypatch.com domain and setup a beta instance of my work-in-progress:

https://beta.communitypatch.com

CommunityPatchBeta.png

New API

The big green “Read the docs” button on the main page will take you to… the documentation! There you will find those APIs in much greater detail.

The community managed patch source mirrors a number of features from my Patch Server project. The /jamf endpoints are here to integrate with Jamf Pro and the service can be used as an external patch source.

The /api endpoints are slightly different from the Patch Server, but allow for creating definitions by providing the full JSON or a URL to an external source (creating a synced definition) and updating the versions afterwards.

From the docs, here’s the example for creating a new patch definition using the Patch-Starter-Script:

curl https://beta.communitypatch.com/api/v1/title \
   -X POST \
   -d "{\"author_name\": \"<NAME>\", \"author_email\": \"<EMAIL>\", \"definition\": $(python patchstarter.py /Applications/<APP> -p "<PUBLISHER>")}" \
   -H 'Content-Type: application/json'

Here, there are required author_name and author_email keys you need to provide when creating a definition. The author_name you choose will be injected into the ID and name keys of the definition you’re providing.

For example, if I provide “Bryson” for my name, and I’m creating the “Xcode.app” definition, it’s ID will become “Xcode_Bryson” and the display name “Xcode (Bryson)”. These changes make it possible to differentiate titles when browsing in Jamf Pro, and for members of the community to better identify who is managing what (as well as sharing with each other).

After you create a patch definition, you will be emailed an API token to the address you provided in author_email. This token is specifically for managing that title, and is the only way to update the title after. Your email address is not saved with CommunityPatch. A hash of it is stored with the title so you can reset the token should you lose it or need the previous one invalidated (this feature is not implemented yet).

Updating works similarly to Patch Server (but without the items key):

curl http://beta.communitypatch.com/api/v1/title/<ID>/version \
   -X POST \
   -d "$(python patchstarter.py /Applications/<APP> --patch-only)" \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer <TOKEN>'

 

Try It Out

I had a number of admins on Slack giving me feedback and testing the API for a few weeks. While I have work left to do to ensure the production version of CommunityPatch is performant, and still some more features to finish writing, I am at a stage where I would like those interesting in contributing to and using CommunityPatch to join in, and try the documented features (in your test environments).

You can jump right in by joining the #communitypatch channel on the MacAdmins Slack, hitting the CommunityPatch documentation, play around with the API, test definitions you create in your Jamf Pro test environments, and discuss what you find.

CommunityPatch is being written out in the open. You can go to GitHub and see the code for yourself. You can even contribute at a code/docs level if you like! For the immediate, having admins test it out and report back will provide me a lot of value as I work towards completing the application and deploying it to production.

Links

Advertisements

Possum – A packaging tool for Python AWS Serverless Applications

The applications I build on AWS are all written in Python using the Serverless Application Model (SAM). Building my applications using a template and Lambda functions, I quickly ran into a limitation of the aws command line tools: external dependencies.

If your Lambda functions have no dependencies (not including the AWS SDKs), or you pre-download and embed them alongside your code, the standard package command works:

serverless_04

However, if you want to install dependencies at the time of packaging the application, you are left in a position where you need to roll your own build system. Amazon provides instructions on creating a Python deployment package, but it would be nice if running the aws cloudformation command did this for us.

Possum

I wrote a packaging tool to fill in the gap left by Amazon’s. Possum (an amalgamation of “Python AWS SAM”) processes a SAM template file just as aws cloudformation package but creates per-function Lambda deployment packages if it detects a requirements file within the function’s directory (Pipfile or requirements.txt).

Possum can be installed from the Python Package Index:

possum_01

Once installed, Possum becomes available as a command line tool (it is loaded into your Python installation’s /bin directory):

possum_02.png

What Possum does is iterate over the Resources section of your SAM template and find all the objects of the AWS:Serverless:Function type, determine the location of their code using the Properties:CodeUri value, and through the magic of Pipenv create individual virtual environments to download the external dependencies, if any, and zip the files together into a Lambda package. Once the package and upload process is complete, Possum will either print your updated deployment template on the screen or write it out to a filename that you specified.

possum_03.png

In the above example, my HelloWorld function didn’t have any defined dependencies within it’s directory so the contents were zipped up as they were. For the Authorizer, there was a Pipfile present which triggered the build process. The approach to Lambda function dependencies with Possum is to handle them on a per-function basis. This creates artifacts that only include the required packages for that function (or none).

Pipenv is not installed with Possum. Instead, Possum will shell-out to run the Pipenv commands (so you will need to have Pipenv installed separately).

After Possum has finished, I can take the deployment.yaml file and deploy the application using aws cloudformation deploy or the AWS console.

Try It Out

If you’re working with Python Lambda functions, please give Possum a try! If you encounter an issue, or have a feature request, you can open an issue on the GitHub page.

Possum’s GitHub Page
https://github.com/brysontyrrell/Possum

Possum on the Python Package Index
https://pypi.org/project/possum/

 

Patch Server for Jamf Pro

(TL;DR, gimme the link: https://github.com/brysontyrrell/PatchServer)

After several months of not getting around to it, my PatchServer project on GitHub is finally nearing a true 1.0 state.

I am openly asking for those who have been following this project, and those who are interested in this project, to download, use, and provide feedback on what should be finished before the release of Jamf Pro 10.2.

Please create issues on GitHub for bugs and feature requests that you would want to make the cut for 1.0.

Some time late last year (and I say some time because it’s all becoming a blur), I was brought into a meeting where I was shown our (Jamf’s) progress on providing a framework for customers to be able to create their own patch definitions. This framework would allow customers to setup their own patch servers and add them to their JSS.

A day or so later, I wrote the first rough version of my own implementation.

Backing up a sec:

What’s a patch definition?

In Jamf Pro v10 we introduced a feature called Patch Management. With this, you could subscribe to a number of software titles that Jamf curates and maintains. Once subscribed, your JSS will, on a schedule, read in the patch definitions of those software titles to stay updated.

For more about Patch Management, see the Jamf Pro Admin Guide (10.1):

These patch definitions (which are JSON data) contain historical information about a software title’s version history and requirements for determining if the software is installed on a managed computer. This allows admins to use the Patch Management feature to create reports and update policies to automatically patch those software titles on computers.

Of course, when these features came out there was one resounding question from nearly everyone:

“Why can’t we make our own patch definitions?”

External Patch Sources

The framework I mentioned above is the answer to this. In Jamf Pro 10.2+ you will have the option of adding External Patch Sources to your JSS. Then, in addition to the official Jamf software titles, you will be able to subscribe to your own and use the same reporting and policy features.

 

The external patch source must be a server your JSS is able to reach via HTTP/HTTPS. This patch server must expose the following endpoints:

  • /software
    This returns a JSON array of all the software titles that are available on this server. For example:

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfImaging", 
        "lastModified": "2018-02-03T03:34:36Z", 
        "name": "Jamf Imaging", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /software/TitleId,TitleId
    This returns the same JSON as above, but limited to the comma separated list of software titles. For example (passing JamfAdmin,JamfRemote):

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /patch/TitleId
    This returns the full patch definition JSON of the software title. Here is an abbreviated example:

    {
      "id": "JamfAdmin",
      "name": "Jamf Admin",
      "publisher": "Jamf", 
      "appName": "Jamf Admin.app", 
      "bundleId": "com.jamfsoftware.JamfAdmin", 
      "currentVersion": "10.1.1", 
      "lastModified": "2018-02-03T03:34:34Z", 
      "extensionAttributes": [
        {"ExtensionAttributeObjects"}
      ],
      "patches": [
        {"PatchObjects"}
      ], 
      "requirements": [
        {"RequirementsObjects"}
      ]
    }

If you had a patch server located at http://patch.my.org, the full URLs would be:

At this time, there is no product that Jamf is providing for customers to install and have a ready to use patch server. The focus has been on opening up the framework that the official patch source uses and allow customers to extend their environments through a little engineering work.

Not all of us are engineers, of course. Thus…

Enter: Patch Server

gui_01_index.png

I wanted to have a working patch server ready for the Jamf community in time for 10.2’s release. My initial patch server implementation (I call it an implementation because it’s one way of providing a patch source) achieved serving the proper JSON data for each of the endpoints described above using a database (SQLite) for the backend.

While my original goals were much grander, including the ability to fully manage a patch definition in a GUI instead of writing out JSON, I had to pare it back in order to get the project into a deliverable state.

In the past week I went through the code and ripped out everything that I felt was not needed, or doable. Then, I went through and added in new features (ported from another project) and streamlined the UI elements that were left.

This patch server features:

  • All required Jamf Pro endpoints to serve as an External Patch Source
  • An API for programmatic management of patch definitions and versions.
    • Create/delete patch definitions.
    • Add versions to existing patch definitions.
    • Create backup archives of all patch definitions.
  • UI for management of patch definitions.
  • Validation of uploaded patch definitions.
    gui_05_validation.png
  • Full user documentation at http://patchserver.readthedocs.io/
    patchserver_docs.png

    • UI Overview
    • Setup Instructions
    • API Documentation

Bring the Requests

Until Jamf Pro 10.2 is released, I’m not going to tag the project at a 1.0 version. If you are in Jamf’s beta program and testing 10.2, I invite you to give this a try and let me know what you think. Specifically, I’m asking for you do open up issues on GitHub for:

  • Bugs you find
  • Features you want, such as:
    • Connect to an actual database like MySQL (?)
  • Documentation you want, such as:
    • Instructions for installing on X

Not everything that is reported might get worked on, but the good news is I released the patch server under the MIT license. If you have some Python chops you can fork it and do whatever you want with the codebase to suit your needs!

But, I don’t wanna setup a server…

If you had that reaction to the idea of setting up your own external patch source, ask yourself if you match any of these descriptions:

  1. My JSS can talk to pretty much anything if I want it to,
  2. I want a patch server; I don’t want to host a patch server,
  3. It doesn’t matter where my patches live as long as I can get and manage them,
  4. Can’t this be a cloud thing?

If so… stayed tuned for a future blog post.

Building AWS Serverless Applications – Part 2

In my previous Serverless post I covered creating a basic “Hello World” app with a SAM template and a Lambda function written in Python. In this post we will extend the SAM template using a Swagger definition and enable a custom authorizer on our endpoint.

Extending with Swagger

Here is the SAM template from the previous post:

serverless_02

Here, our API Gateway resource is implicitly defined by way of the GetApi event on our Lambda function. This tells CloudFormation that an API Gateway with a path of “/” and the GET method must be created to execute the Lambda function.

The API Gateway supports a lot of features and functionality, but much of it is not a part of the SAM spec. In order to extend our API Gateway beyond basic Lambda functions executed upon requests to endpoints, we will need to define an AWS::Serverless::Api resource and provide a Swagger definition. Within that Swagger definition we will be able to extend our API.

Here is our application above with the API resource added:

serverless_07.png

Let’s focus in on the new API resource.

serverless_08.png

Before, our API Gateway was being created with the default Stage and Prod stages. Now that we are defining the resource we must provide the stage name. Otherwise, the rest of this object is the Swagger definition.

There are two ways of setting the Swagger definition. Inline Swagger definitions use the DefinitionBody key as shown above. To point to the location of a Swagger definition file you can use the DefinitionUri key. This key works just like the CodeUri key for our Lambda function.

The info:title key is being set to the name of the CloudFormation stack we deploy our function to. The keys under path should all be the path of the API Gateway endpoint that is being defined, and each key for the path should be an HTTP method. We only have one path, “/”, and it only supports one method, GET.

Here we encounter our first of AWS’s Swagger Extensions for API Gateway. x-amazon-apigateway-integration allows API Gateway to integrate with different backends including HTTP services, SNS, SQS, and Lambda. The type: aws_proxy is specifically for Lambda functions. You may be confused about the httpMethod: post here as our endpoint is set to use the GET method. This is specifically for API Gateway invoking the Lambda function and does not involve the HTTP request.

Then we have the Uri to the backend integration:

arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorld.Arn}/invocations

The Fn::Sub function is a CloudFront template function for replacing values within a string. This ARN (an AWS resource identifier) populates the Region our stack is running in, and the ARN of our Lambda function that is also defined in our template.

Which, we have only made one change to our Lambda resource:

serverless_09.png

Under the event, we have added a reference (Ref) to our API Gateway resource. Ref is another CloudFormation function that returns the value of an item (in this case, our API Gateway resource returns it’s ID when referenced).

At this point, you may notice that we are now doubling up on how we define our API. Not only do we need to define all of our endpoints in the Swagger definition, but we must still write our Lambda functions in full. While not ideal, it is necessary as we move into advanced integrations between API Gateway and Lambda.

Custom Authorizers

A Custom Authorizer is a Lambda function attached to an API Gateway endpoint. This Lambda function is invoked when a request to that endpoint is made. Based upon authorization tokens, headers, query strings, stage variables, or context parameters. Once the request is evaluated, it is either denied or the Lambda function will return an IAM policy granting the request. It is then allowed through as normal.

You can only define a custom authorizer in your SAM template by using an AWS Swagger extension. Here is our updated SAM template with the custom authorizer included:

serverless_10.png

The Swagger definition has been expanded and we have added another Lambda function, but this time with a Lambda Permission resource.

serverless_11.png

The securityDefinition key is where we define our authentication methods for our API. We are using two additional AWS Swagger extensions to define the custom authorizer: x-amazon-apigateway-authtype and x-amazon-apigateway-authorizer. The former is always custom for a custom authorizer.

The the example authorizer in my example has a static token that it validates. so we specify the type as token. The authorizerUri is the same style ARN as above with the x-amazon-apigateway-integration, but using the ARN of our authorizer’s Lambda function. The authorizerResultTtlInSeconds if the amount of time the resulting IAM policy is cached. More on that in a bit.

serverless_12.png

Our definition of the authorizer’s Lambda function is bare bones. There are no events that trigger it as a part of the SAM spec – that is defined in our Swagger definition. Because we have no events, we must create the permissions resource that allows API Gateway to invoke this function.

Here is the Python code for the authorizer function:

serverless_13.png

The Custom Authorizer only needs to perform two tasks: 1) validate the request by the chosen method (in our example: the Authorization header’s value which must match ‘Bearer a.b.c’), and 2) responds with an object containing a principalId and an IAM policy.

The principalId comes into play with the authorizerResultTtlInSeconds value above. API Gateway will cache the result of an authorization for an amount of time equal to this value. If the origin repeats a request, and they have already authenticated, they will be allowed through without invoking the custom authorizer again! In my example I am passing the Authorization header’s value as the principal ID.

The IAM policy is a document that defines permissions for the request. You can specify actions to ALLOW and actions to DENY for a given Resource (in this case, granting execute-api:Invoke for the HTTP endpoint that was requested).

In my example, a successful authentication generates the following response to API Gateway:

serverless_15.png

Our project directory now has the following structure:

serverless_14.png

We can update the application stack deployed in the previous post by re-running our package and deploy commands on the same stack name. Alternatively, you can deploy to an entirely new stack using the same commands.

serverless_04

serverless_05

Now if we attempt to call our API we will receive an error message telling us we are unauthorized. Reattempt with the Authorization header in place and we get a successful response!

serverless_16.png

 

Up Next

Now that we have covered how to extend our serverless application using a Swagger definition and secured it with a Custom Authorizer, we can expand into having an S3 bucket provide back-end storage and use API Gateway’s binary data support to POST and GET files!

You can find the templates and code shown in this post here on my GitHub:

https://github.com/brysontyrrell/Serverless-Hello-World

Building AWS Serverless Applications – Part 1

The AWS Serverless Application Model (SAM) is an exciting technology stack to start building applications upon. You can write and deploy large web and/or event-based apps and services without worrying about up-front costs, scalability, and infrastructure.

SAM apps are defined by a YAML template that you deploy using AWS CloudFormation. While basic applications are easy to define, things become more complex as you begin to add in more advanced features like Authorizers and Binary data support (more on those later).

In this post, I’m going to walk through a simple Serverless Hello World app.

In a later post, we will then extend this to include token authentication and serving image files.

You can learn more about SAM here at the  project’s GitHub page.

A Basic SAM App

My approach to serverless applications is to contain all of the templates and function code within a single repository. The SAM template remains at the top level with each Lambda function within it’s own sub-directory.

Here is the beginning structure of our Hello World app’s repository:

serverless_01.png

First, let’s look at the contents of our template.yaml file:

serverless_02.png

The SAM template does a lot of behind-the-scenes magic for us when we go to deploy with CloudFormation. Note the Transform key at the top with the AWS::Serverless-2016-10-31 value. This is an instruction that tells CloudFormation to take the template content and translate it into native CloudFormation. Our templates would be much, much larger and more complex if we had to write them in native CloudFormation objects.

Moving on to Resources, this is where we define the Lambda functions that will perform the work for our app. There one function for our app, HelloWorldFunc (the key is just a label for the resource), with the Type is AWS::Serverless::Function.

Under the Properties of our function we further define it and how it is invoked. The CodeUri key is the path to the code that makes up the function. In this case it is a relative path to our /HelloWorldFunc/ directory (it must be a directory and not a file).

The Handler is two parts: filename.function. Within /HelloWorldFunc/ we have a single Python file: lambda_function.py. In that file is the function that will be executed when the file is run in Lambda: lambda_handler().

You don’t need to name your files and functions this way. Your handler value just needs to point to the right file and function within that file.

Set the Runtime value to the language you are using for the Lambda function. In this case it is python3.6, but Lambda supports a number of languages, and because they are all independent you can mix-and-match across multiple functions (e.g. one can be Python and the other nodeJS).

Events define how the function is invoked. There are a lot of different types of events you can use as a part of the SAM spec, but here we have defined an API operation. GetApi is the label for this event (you can define as many events as you want), the Type is Api which will be through API Gateway, and for the API’s properties the Path is “/” (or the root of the domain) and the Method is get (standard HTTP methods).

Nothing else needs to be defined. I’ll explain why in the “Package & Deploy” section.

Now, let’s look at the code in /HelloWorldFunc/lambda_function.py:

serverless_03.png

It’s not a straightforward print(“Hello World”) as you might have been expecting. The integration between API Gateway and Lambda uses what’s called Lambda Proxy. When the API Gateway invokes the Lambda function it expects back a dictionary object that provides what the HTTP response to the initiating client should be.

  • isBase64Encoded is a boolean flag to tell API Gateway if your response body is a bas64 encoded binary blob. You’ll set this to True if you’re returning a file that will need to be base64 encoded into the body.
  • statusCode is the HTTP code to responde with. 2xx for success, 4xx for an error due to the client request.
  • body is the content you are returning. It must be a string and not any other data type. If you’re returning binary content (e.g. a file), you will need to base63 encode it, put that string into this body key and set isBase64Encoded to True.
  • headers must be set here with at least the Content-Type of the response body. You can also set whatever other headers you want.

Package and Deploy

Using the AWS CLI, we can easily package our Lambda functions, upload them to S3, update the template, and deploy using CloudFormation in two steps.

First, we will need to run the package command from the repository’s directory:

serverless_04.png

  • –template-file is the filename of the SAM template.
  • –s3-bucket is the name of an S3 bucket you have access and can upload files to.
  • –output-template-file is a filename you provide for an updated SAM template that will be generated by this command. If you don’t specify this key, the template data will be sent to STDOUT.

Packaging converts the code for the Lambda functions into zip files that CloudFormation will use when creating the AWS objects. In the template example above, the CodeUri key is a directory path. Everything within that directory will be put into the zipped artifact for the function and uploaded to the specified S3 bucket.

The CodeUri must be either a directory path or an S3 location. In a later post, I’ll talk about deployment workflows where you may need to handle the packaging on your own and then update your template.

Our newly created deployment.yaml file will contain the S3 locations of the zipped artifacts for each function in your SAM template in their CodeUri key – replacing the directory paths in the source template.

We can now deploy our packaged serverless application:

serverless_05.png

  • –template-file is the filename of our deployment template.
  • –stack-name is the name we are going to give to the CloudFormation stack that will be created.
  • –capabilities CAPABILITY_IAM is granting CloudFormation permissions to generate IAM roles and permissions.

Whenever you execute a CloudFormation template (SAM is an extension to CloudFormation) you are creating a stack. The stack is the deployed application according to the template for the given name.

The first time you run this command, CloudFormation is going to create a brand new stack and all of the resources you’ve defined. The next time you run your template and specify this stack name it will update all of the stack’s resources according to whatever changes you have defined from the last run to the current run. This include the creation, updating, and deletion of resources. CloudFormation handles it all for you.

We can give a different name for the stack and run the template again to create a second running instance of our serverless application. This is an important feature of using CloudFormation: you can re-use the same template multiple times for completely different stacks. This means you can deploy to an entire test/staging environment before deploying to your production environment using the same template.

You should see the following output as the stack is created:

serverless_06.png

You can log into your AWS console and navigate to the CloudFormation page (be sure you’re in the right region!) to see everything that has been created for this stack:

serverless_07.png

Click on Resources tab and you will be able to navigate to the page for each resources that was created as a part of the stack. Go to the API Gateway, click on Stages and the Prod stage. You’ll be presented with the default generated URL.

Go there and you should see:

serverless_09.png

And there we have the Serverless Hello World app.

Up Next

There is, of course, so much more that can be done through the Serverless Application Model, and many more AWS services to tie into!

The next article I write will cover extending our Hello World app with a Swagger definition to lock it behind basic token authentication. From there, we will continue to build upon that swagger definition to enable binary data support and serve images from an S3 bucket!

You can find the templates and code shown in this post here on my GitHub:

https://github.com/brysontyrrell/Serverless-Hello-World

Stop everything! Start using Pipenv!

Seven months ago I tweeted about Pipenv:

That was about it.

The project- which was at v3.6.1 at the time – still felt new. At Jamf, we were working on our strategy for handling project requirements and dependencies going forward for Jamf Cloud’s automation apps and Pipenv, while a cool idea, didn’t have maturity and traction that.

As @elios pointed out on the MacAdmins Slack: Pipenv was announced in January of this year as an experimental project.

Today it is the officially recommended Python packaging tool from Python.org.

In his words, “That’s awfully fast…”

The GitHub repository for Pipenv has 5400+ stars and over 100+ contributors to the project. It is proceeding at breakneck development speeds and adding all kinds of juicy features every time you turn around. It’s on v8.3.2.

Within ten minutes of finally, FINALLY, giving Pipenv a try today I made the conscious decision to chuck the tooling we had made and convert all of our projects over.

(Oh, and the person who wrote that tooling wholly agreed)

Let’s talk about that.

Part 1: Why requirements.txt sucks

There is was this standard workflow to your Python projects. You needed two key tools: pip and virtualenv. When developing, you would create a Python virtual environment using virtualenv in order to isolate your project from the system Python or other virtual environments.

You would then use pip to install all the packages your project needs. Once you had all of those packages installed you would then run a command to generate a requirements.txt file that listed every package at the exact installed version.

The requirements.txt file stays with your project. When someone else gets a copy of the repository to work on it or deploy it they would follow the same process of creating a virtual environment and then using pip to install all of the required packages by feeding that requirements.txt file into it.

Now, that doesn’t seem so bad, right? The drawbacks become apparent as you begin to craft your build pipelines around your projects.

Maintenance is a Manual Chore

The first thing to call out is how manual this process is. You need to create the same version virtual environment for the project once you’ve cloned it and then install the packages. If you want to begin upgrading packages you need to be sure to manually export a new requirements.txt file each time.

Removing packages is far more problematic. Uninstalling with pip will not remove dependencies for a package! Take Flask, for example. It’s not just one package. It has four sub-packages that get installed with it, and one of those (Jinja2) has its own sub-package.

So, maybe the answer is to manually maintain only the top-level packages in your requirements.txt file? That’s a hard no. You don’t want to do that because it makes your environment no longer deterministic. What do I mean? I mean, those sub-packages we just talked about are NOT fixed versions. When you’re installing Python packages from the Python Package Index (PyPI) the aren’t using requirements.txt files with fixed versions for their dependencies. They instead are specifying ranges allowing for installs at a minimum and/or maximum version for that sub-package.

This makes a lot of sense when you think about it. When installing a package for the first time as a part of a new project, or updating a package in an existing project, you are going to want the latest allowed versions for all the sub-packages for any patches that have been made.

Not so much when it comes to a project/application. You developed and tested your code at specific package versions that are a known working state. This is past the development phase when bugs from updates to those packages can be identified and addressed. When it comes to deployment, your project’s requirements.txt file must be deterministic and list every package at the exact version.

Hence, maintenance of this file becomes a manual chore.

But Wait, There’s Testing!

There are a lot of packages that you might need to install that have nothing to do with running your Python application, but they have everything to do with running tests and building documentation. Common package requirements for this would by Pytest, Tox, and Sphinx with maybe a theme. Important, but not needed in a deployment.

We want to be able to specify a set of packages to install that will be used during the testing and build process. The answer unfortunately, is a second requirements.txt file. This one would have a special name like requirements-dev.txt which is only used during a build. This file could only contain the specific packages and be installed with pip after the standard requirements.txt, or it could contain all of those plus the build packages. In either case, the maintenance problem continues to grow.

So We Came Up With Something…

Our process at Jamf ended up settling on three types of requirements.txt files in an effort to address all the shortcomings described.

  • requirements.txt
    This file contained the full package list for the project at fixed versions. It would be used for spinning up new development environments or during deployment.
  • core-requirements.txt
    This file contained the top-level packages without fixed versions. Development environments would not be built from this. Instead, this file would be used for recreating the standard requirements.txt file at updated package versions and eliminating orphaned packages that were no longer used or removed from the project at some point.
  • build-requirements.txt
    This is a manually maintained file per-project that only contained the additional packages needed during the build process.

This approach isn’t too dissimilar to what others have implemented for many of the same reasons we came up with.

Part 2: Enter Pipfile

Not Pipenv? We’re getting there.

Pipfile was introduced almost exactly a year ago at the time of this post (first commit on November 18th, 2016). The goal of Pipfile is to replace requirements.txt and address the pain points that we covered above.

Here is what a Pipfile looks like:

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]



[requires]

python_version = "2.7"

PyPI is the default source in all new Pipfiles. Using the scheme shown, you can add additional [source] sections with unique names to specify your internal Python package indexes (we have our own at Jamf for shared packages across projects).

There are two groups for listing packages: dev-packages and packages. In a Pipfile these lists follow how we were handling this where only the build packages go into the dev-packages list and all project packages go under the standard packages list.

Packages in the Pipfile can have fixed versions or be set to install whatever the latest version is.

[packages]

flask = "*"
requests = "==2.18.4"

The [requires] section dictates the version of Python that the project is meant to run under.

From the Pipfile you would create what is called a Pipfile.lock which contains all of the environment information, the installed packages, their installed versions, and their SHA256 hashes. The hashes are a recent security feature of pip to validate packages that are installed when deployed. If there is a hash mismatch you can abort. A powerful security tool in preventing malicious from entering your environments.

Note that you can specify the SHA256 hashes of packages in a normal requirements.txt file. This is a feature of pip and not of Pipfile.

It is this Pipfile.lock that will be used to generate environments on other systems whether for development or deployment. The Pipfile will be used for maintaining packages and dependencies and regenerating your Pipfile.lock.

All of this is just a specification. Pip as of yet still does not support Pipfiles, but there is another..

Part 3: Pipenv Cometh

Pipenv is the implementation of the Pipfile standard. It is built on top of pip and virtualenv and manages both your environment and package dependencies, and it does so like a boss.

Get Started

Install Pipenv with pip (or homebrew if that’s your jam).

$ pip install pipenv

Then in a project directory create a virtual environment and install some packages!

$ pipenv --two
Creating a virtualenv for this project…
<...install output...>
Virtualenv location: /Users/me/.local/share/virtualenvs/test-zslr3BOw
Creating a Pipfile for this project…

$ pipenv install flask
Installing flask…
<...install output...>
Adding flask to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (36eec0)!

$ pipenv install requests==2.18.4
Installing requests==2.18.4…
<...install output...>
Adding requests==2.18.4 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (572f23)!

$

Notice how we see the Locking messages after every install? Pipenv automatically regenerated the Pipfile.lock each time the Pipfile is modified. Your fixed environment is being automatically maintained!

Graph All The Things

Let’s look inside the Pipfile itself.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]

flask = "*"
requests = "==2.18.4"


[requires]

python_version = "2.7"

No sub-packages? Nope. It doesn’t need to track those (they end up in the Pipfile.lock, remember?). But, if you’re curious, you can use the handy graph feature to view a full dependency tree of your project!

$ pipenv graph
Flask==0.12.2
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
   - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
 
$

Check that out! Notice how you can see the requirement that was specified for the sub-package in addition to the actual installed version?

Environment Management Magic

Now let’s uninstall Flask.

$ pipenv uninstall flask
Un-installing flask…
<...uninstall output...>
Removing flask from Pipfile…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (4ddcaf)!

$

And re-run the graph command.

$ pipenv graph
click==6.7
itsdangerous==0.24
Jinja2==2.9.6
 - MarkupSafe [required: >=0.23, installed: 1.0]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
Werkzeug==0.12.2

 

Yes, the sub-packages have now been orphaned within the existing virtual environment, but that’s not the real story. If we look inside Pipfile we’ll see that requests is the only package listed, and if we look inside our Pipfile.lock we will see that only requests and it’s sub-packages are present.

We can regenerate our virtual environment cleanly with only a few commands!

$ pipenv uninstall --all
Un-installing all packages from virtualenv…
Found 20 installed package(s), purging…
<...uninstall output...>
Environment now purged and fresh!

$ pipenv install
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell

$ pipenv graph
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

$

Mindblowing!!!

Installing the dev-packages for our builds uses an additional flag with the install command.

$ pipenv install sphinx --dev
Installing sphinx…
<...install output...>
Adding sphinx to Pipfile's [dev-packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (d7ccf2)!

The appropriate locations in our Pipfile and Pipefile.lock have been updated! To install the dev environment perform the same steps for regenerating above but add the –dev flag.

$ pipenv install --dev
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 18/18 — 00:00:05
To activate this project's virtualenv, run the following:
 $ pipenv shell

Part 4: Deploy Stuff!

The first project I decided to apply Pipenv to in order to learn the tool is ODST. While there is a nice feature in Pipenv where it will automatically import a requirements.txt file if detected, I opted to start clean and install all my top-level packages directly. This gave me a proper Pipfile and Pipfile.lock.

Here’s the resulting Pipfile.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]

pytest = "*"
sphinx = "*"
sphinx-rtd-theme = "*"


[packages]

flask = "*"
cryptography = "*"
celery = "*"
psutil = "*"
flask-sqlalchemy = "*"
pymysql = "*"
requests = "*"
dicttoxml = "*"
pyjwt = "*"
flask-login = "*"
redis = "*"


[requires]

python_version = "2.7"

Here’s the graph of the installed packages (without the dev-packages).

$ pipenv graph
celery==4.1.0
 - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.3]
 - kombu [required: <5.0,>=4.0.2, installed: 4.1.0]
 - amqp [required: >=2.1.4,<3.0, installed: 2.2.2]
 - vine [required: >=1.1.3, installed: 1.1.4]
 - pytz [required: >dev, installed: 2017.3]
cryptography==2.1.3
 - asn1crypto [required: >=0.21.0, installed: 0.23.0]
 - cffi [required: >=1.7, installed: 1.11.2]
 - pycparser [required: Any, installed: 2.18]
 - enum34 [required: Any, installed: 1.1.6]
 - idna [required: >=2.1, installed: 2.6]
 - ipaddress [required: Any, installed: 1.0.18]
 - six [required: >=1.4.1, installed: 1.11.0]
dicttoxml==1.7.4
Flask-Login==0.4.0
 - Flask [required: Any, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
Flask-SQLAlchemy==2.3.2
 - Flask [required: >=0.10, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
 - SQLAlchemy [required: >=0.8.0, installed: 1.1.15]
psutil==5.4.0
PyJWT==1.5.3
PyMySQL==0.7.11
redis==2.10.6
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

In my Dockerfiles for the project I swapped out using requirements.txt and switched it to install my packages using the Pipefile.lock into the system Python (in a containerized app there’s no real need to create a virtual environment).

RUN /usr/bin/apt-get update -q && \
    /usr/bin/apt-get install -qqy build-essential git && \
    /usr/bin/apt-get install -qqy python-pip python-dev && \
    /usr/bin/pip install pipenv && \
    /usr/bin/apt-get install -qqy libssl-dev libffi-dev && \
    /usr/bin/apt-get install -qqy uwsgi uwsgi-plugin-python && \
    /usr/bin/apt-get clean && \
    /bin/rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY /docker/webapp/web-app.ini /etc/uwsgi/apps-enabled/

COPY /Pipfile* /opt/odst/
WORKDIR /opt/odst/
RUN /usr/local/bin/pipenv install --system --deploy

COPY /ods/ /opt/odst/ods/
COPY /application.py /opt/odst/
COPY /docker/ods_conf.cfg /opt/odst/

RUN /bin/chown -R www-data:www-data /opt/odst/ods/static

CMD ["/usr/bin/uwsgi", "--ini", "/etc/uwsgi/apps-enabled/web-app.ini"]

The –system flag tells Pipenv to install to the system Python. The –deploy flag will abort the installation if the Pipfile.lock is out of date or the Python version is wrong. Out of date? Pipenv knows if your Pipfile.lock matches up with the Pipfile by comparing a SHA256 hash of the Pipfile that it has saved. If there’s a mismatch then there is an issue and it aborts.

If I didn’t want to rely on this hash match feature, I could instead pass a –ignore-pipfile flag that will tell it to proceed using only the Pipfile.lock.

I should mention at this point that Pipenv dramatically speeds up build times. When using pip your packages will install sequentially. Pipenv will install packages in parallel. The difference is immediately noticeable, especially when regenerating your development environments.

Still, The World is Catching Up

Python development workflows area clearly moving to this tool, but Pipenv is less than a year old at this point and not everything will support generating environments from the provided Pipfiles.

The two I am currently using that fall in that category are AWS Elastic Beanstalk and ReadTheDocs.org. Both need a requirements.txt file in order to build the Python environments.

That is thankfully a simple task. You can generate a requirements.txt file from Pipenv by using the lock option.

$ pipenv lock --requirements > requirements.txt

 

For Elastic Beanstalk applications, I can have this command run as a part of my build pipeline so the file is included with the ZIP archive before it goes to S3.

In the case of Read the Docs there is an open issue for adding Pipfile support, but until then I will need to generate the requirements.txt file as I make changes to my environment and save it with the repository.

For Read the Docs I’ll want the extra dev-packages. This is done by using the pipenv run command. This will execute the succeeding string as if it were run in the environment.

$ pipenv run pip freeze > requirements.txt

The bright side is that I am no longer actually managing this file. It is straight output from my Pipenv managed environment!

Part 5: Profit

I hope you enjoyed my overview and impressions of Pipenv. It’s a tool that is going to have a huge impact on development and deployment workflows – for the better!

A huge shout out to Kenneth Reitz for creating yet another invaluable package for the Python community!

* Edit Nov 9: Changed the command for exporting a requirements.txt file. Using pipenv lock -r -d only outputs the packages under the dev-packages section.

Farewell to the Unofficial JSS API Docs

Hey everyone.

With the launch of the Jamf Developer Portal I think it’s time I took down my Unofficial JSS API Docs site on Confluence.

I launched it as a community resource for a lack of API documentation, but now that Jamf has something out there I feel it’s time I save my $10 a month. If you found these resources helpful in the past, great! That was the whole point.

The site will come down after November 17th. For those Google searching and coming across this post, click on the dev portal link I provided above to reach the official documentation provided by Jamf.