Blog switch up

I haven’t been posting to the blog with as much frequency as I used to. Partially, I think this is due to my ideas around larger, more in depth posts that require a lot more time sitting down and crafting.

I’m going to try something different post Penn State MacAdmins. I want to start forcing myself to write more technical content, but in smaller bite-sized pieces that focus on fundamentals rather than all encompassing solutions. Topics that are generic, but provide examples which can be used as building blocks. As these are smaller, and more example driven, I’m going to set a goal to post every week on Monday morning, and maybe Wednesday too if I have enough posts in the pipeline.

On Fridays, I want to start posting “Dev Updates”. I have a number of projects that I have open sourced and am committed to updating for Mac admin community. These projects do have channels in the MacAdmins Slack, and their GitHub repos are linked there, but keeping up for followers would involve a lot of back scrolling to find out what has been discussed. I plan to include in these weekly updates: new issues raised on GitHub, recapping discussions from the Slack channels, and describing any work that has been done during that week on features/bugs. This should not only provide a digestible status updates for those who want them, but help keep me focused.

The ultimate goal here is that I’m writing more again. When it comes to learning and bettering myself professionally, there are two ways I go about it: post about it publicly, or present on it publicly. Both of which force you to cover all your bases in the face of public scrutiny.

We’ll see how this goes.

Advertisements

MacAdmins 2018

Four Years of MacAdmins

Back in February of this year I was able to present at MacAD.UK in London (I attended in 2017; had a blast both times). This marked my eight appearance at a conference as a speaker since joining Jamf in 2012 as the second member of their fledgling IT department. To be fair, four of those appearances were at JNUC. ¯\_(ツ)_/¯

In about month, I’ll be making my fourth appearance, third speaking, at the MacAdmins Conference at Penn State. I have loved this conference every year I’ve attended, and credit is due to the organizers who accumulate a great roster of speakers with a range of content subjects. You’re never without something to listen to.

My first time speaking her, in 2016, I gave what would end up being my most widely viewed presentation to date: Craft Your Own GUIs with Python and Tkinter. The video on YouTube has garnered an insane 82K+ views. I’ll attribute much of that to the subject’s appeal outside of Mac admin circles.

On the second round in 2017 I went a bit further. I attempted, to mixed results, a half day workshop on building Jamf Pro Integrations along with another presentation: How Docker Compose Changed My Life. The workshop had a number of challenges that were all lessons I took to heart for the future: I had drastically underestimated the time needed for my content (we didn’t finish), the notice about prerequisite experience was lost from the sched.com listing, and I had no helpers to assist with questions cause us to pause frequently as I went around the room.


This year I’ll be doing another double feature, but no workshop. Two presentations at the 2018 conference!

Bryson’s doing a Jamf preso?

It’s true. Not counting JNUC, I will be delivering my first official Jamf presentation at a conference. Our gracious Marketing department offered our sponsor slot to me and even allowed me to pick whatever I wanted for the subject!

My choice is something near and dear to me: the recently announced Jamf Marketplace. Why is this near and dear? Creating integrations with Jamf Pro has been a passion of mine, and the Marketplace is a step towards a beautiful future where admins and developers can publish their work for all to share in. I’m very excited for this one.

Session Link: Get Your Tools in Front of Thousands with the Jamf Marketplace

Talking Serverless and AWS

My personal session (not affiliated with Jamf) is all about the new focus in my professional life: serverless application architectures in AWS. That alone can be a pretty broad subject. My presentation will focus on Lambda: the AWS service for running code without servers.

There is a lot of potential for Lambda within your org if you have an AWS account, or would be allowed to create one (you’d be shocked at what you can achieve within the free tier – which I’ll touch on). Beyond the tried and true cron job, you can implement all sorts of crazy even driven workflows with custom processing handled by Lambda functions you’ve written in your preferred language (which is Python, right?).

I’ll be doing a deep dive into subject. We’ll cover the basics of Lambda, how IAM permissions work and how to apply them, the best practices of defining and deploying using CloudFormation (what I call template-first development), and hopefully more if time allows. It’s an area I’ve become very passionate about and I’m looking so forward to being able to present on this to the Mac admin community.

Session Link: Diving into AWS Lambda: An Intro to Serverless for Admins


I hope to see you next month! If you don’t find me wandering the halls between sessions, please reach out on Slack, or peek into Legends. It’s a favorite.

If you’re interested in the presentations I’ve done over the years at various conferences, you can find that list with YouTube links here.

CommunityPatch.com (beta)

In previous posts, I talked about two projects I had been working on for the Jamf community to better help admins get started using the new “External Patch Sources” feature in Jamf Pro 10.2+. While working on Patch Server and the companion Patch-Starter-Script, I also wrote a quick proof of concept for a serverless version that would run in an AWS account.

The Stupid Simple Patch Server uses API Gateway and Lambda functions to serve patch definitions you stored in an S3 bucket. I even included the same API endpoints from the Patch Server so workflows between the two could be shared. I even took it a step further and added a subscription API so it would sync with a remote patch definition via URL.

That side project (of a side project) made me think about how I could take the basic design and build upon it into something that could be used by multiple admins. At first, I wrote a lot of code to transform the Stupid Simple Patch Server into a multi-tenant application. At a point, I considered the limitations of what could be done in a manner that could be considered secure and scrapped much of it.

But not everything. The work I had done was retooled into a new concept: a single, public, community managed patch source for Jamf Pro. A service where anyone could contribute patch definitions, and be able to manage and update them. Five minutes after having this idea I bought the communitypatch.com domain and setup a beta instance of my work-in-progress:

https://beta.communitypatch.com

CommunityPatchBeta.png

New API

The big green “Read the docs” button on the main page will take you to… the documentation! There you will find those APIs in much greater detail.

The community managed patch source mirrors a number of features from my Patch Server project. The /jamf endpoints are here to integrate with Jamf Pro and the service can be used as an external patch source.

The /api endpoints are slightly different from the Patch Server, but allow for creating definitions by providing the full JSON or a URL to an external source (creating a synced definition) and updating the versions afterwards.

From the docs, here’s the example for creating a new patch definition using the Patch-Starter-Script:

curl https://beta.communitypatch.com/api/v1/title \
   -X POST \
   -d "{\"author_name\": \"<NAME>\", \"author_email\": \"<EMAIL>\", \"definition\": $(python patchstarter.py /Applications/<APP> -p "<PUBLISHER>")}" \
   -H 'Content-Type: application/json'

Here, there are required author_name and author_email keys you need to provide when creating a definition. The author_name you choose will be injected into the ID and name keys of the definition you’re providing.

For example, if I provide “Bryson” for my name, and I’m creating the “Xcode.app” definition, it’s ID will become “Xcode_Bryson” and the display name “Xcode (Bryson)”. These changes make it possible to differentiate titles when browsing in Jamf Pro, and for members of the community to better identify who is managing what (as well as sharing with each other).

After you create a patch definition, you will be emailed an API token to the address you provided in author_email. This token is specifically for managing that title, and is the only way to update the title after. Your email address is not saved with CommunityPatch. A hash of it is stored with the title so you can reset the token should you lose it or need the previous one invalidated (this feature is not implemented yet).

Updating works similarly to Patch Server (but without the items key):

curl http://beta.communitypatch.com/api/v1/title/<ID>/version \
   -X POST \
   -d "$(python patchstarter.py /Applications/<APP> --patch-only)" \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer <TOKEN>'

 

Try It Out

I had a number of admins on Slack giving me feedback and testing the API for a few weeks. While I have work left to do to ensure the production version of CommunityPatch is performant, and still some more features to finish writing, I am at a stage where I would like those interesting in contributing to and using CommunityPatch to join in, and try the documented features (in your test environments).

You can jump right in by joining the #communitypatch channel on the MacAdmins Slack, hitting the CommunityPatch documentation, play around with the API, test definitions you create in your Jamf Pro test environments, and discuss what you find.

CommunityPatch is being written out in the open. You can go to GitHub and see the code for yourself. You can even contribute at a code/docs level if you like! For the immediate, having admins test it out and report back will provide me a lot of value as I work towards completing the application and deploying it to production.

Links

Possum – A packaging tool for Python AWS Serverless Applications

The applications I build on AWS are all written in Python using the Serverless Application Model (SAM). Building my applications using a template and Lambda functions, I quickly ran into a limitation of the aws command line tools: external dependencies.

If your Lambda functions have no dependencies (not including the AWS SDKs), or you pre-download and embed them alongside your code, the standard package command works:

serverless_04

However, if you want to install dependencies at the time of packaging the application, you are left in a position where you need to roll your own build system. Amazon provides instructions on creating a Python deployment package, but it would be nice if running the aws cloudformation command did this for us.

Possum

I wrote a packaging tool to fill in the gap left by Amazon’s. Possum (an amalgamation of “Python AWS SAM”) processes a SAM template file just as aws cloudformation package but creates per-function Lambda deployment packages if it detects a requirements file within the function’s directory (Pipfile or requirements.txt).

Possum can be installed from the Python Package Index:

possum_01

Once installed, Possum becomes available as a command line tool (it is loaded into your Python installation’s /bin directory):

possum_02.png

What Possum does is iterate over the Resources section of your SAM template and find all the objects of the AWS:Serverless:Function type, determine the location of their code using the Properties:CodeUri value, and through the magic of Pipenv create individual virtual environments to download the external dependencies, if any, and zip the files together into a Lambda package. Once the package and upload process is complete, Possum will either print your updated deployment template on the screen or write it out to a filename that you specified.

possum_03.png

In the above example, my HelloWorld function didn’t have any defined dependencies within it’s directory so the contents were zipped up as they were. For the Authorizer, there was a Pipfile present which triggered the build process. The approach to Lambda function dependencies with Possum is to handle them on a per-function basis. This creates artifacts that only include the required packages for that function (or none).

Pipenv is not installed with Possum. Instead, Possum will shell-out to run the Pipenv commands (so you will need to have Pipenv installed separately).

After Possum has finished, I can take the deployment.yaml file and deploy the application using aws cloudformation deploy or the AWS console.

Try It Out

If you’re working with Python Lambda functions, please give Possum a try! If you encounter an issue, or have a feature request, you can open an issue on the GitHub page.

Possum’s GitHub Page
https://github.com/brysontyrrell/Possum

Possum on the Python Package Index
https://pypi.org/project/possum/

 

Patch Starter Script

Jamf Pro 10.2 is not far. I recently released my Patch Server project for admins in the beta (and after the release) to host their own custom patch definitions from. The server hosts the definitions and provides an API for maintaining them afterwards with automation. A missing piece is a tool to create your initial definitions to upload.

This morning I posted a script that aims to address that gap:

https://github.com/brysontyrrell/Patch-Starter-Script

This command line utility will take an existing application on your Mac and generate a basic (we’ll call it default) patch definition. This is primarily done using the Info.plist file within the app bundle.

Creating a Patch Definition

Using GitHub Desktop.app as an example, the script can output the JSON to stdout:

$ python patchstarter.py /Applications/GitHub\ Desktop.app -p "Github"

Or it can write a JSON file to a directory of your choice (in this example the current working directory):

$ python patchstarter.py /Applications/GitHub\ Desktop.app -p "Github" -o .

The -p or –publisher argument allows you to give the name of the application’s publisher. This was included as this information is not (normally) found in Info,plist.

Below, I’ve shown the GitHub Desktop.app example’s output (the keys print out of order in Python 2, but keep in mind that key order doesn’t matter). Next to each line I’ve included where the value maps from:

{
    "id": "GitHubDesktop",                          <-- CFBundleName without spaces
    "name": "GitHub Desktop",                       <-- CFBundleName
    "appName": "GitHub Desktop.app",                <-- Application filename
    "bundleId": "com.github.GitHub",                <-- CFBundleIdentifier
    "publisher": "GitHub",                          <-- Optional command line argument (see above)
    "currentVersion": "Hasty Things Done Hastily",  <-- CFBundleShortVersionString
    "lastModified": "2018-02-12T18:33:02Z",         <-- UTC Timestamp of when the script ran
    "requirements": [
        {
            "name": "Application Bundle ID", 
            "operator": "is", 
            "value": "com.github.GitHub",  <-- CFBundleIdentifier
            "type": "recon", 
            "and": true
        }
    ], 
    "patches": [
        {
            "version": "Hasty Things Done Hastily",  <-- CFBundleShortVersionString
            "releaseDate": "2017-05-22T20:24:33Z",   <-- Application last modified timestamp
            "standalone": true, 
            "minimumOperatingSystem": "10.9",        <-- LSMinimumSystemVersion
            "reboot": false, 
            "killApps": [
                {
                    "appName": "GitHub Desktop.app",  <-- Application filename
                    "bundleId": "com.github.GitHub"  <-- CFBundleIdentifier
                }
            ], 
            "components": [
                {
                    "version": "Hasty Things Done Hastily",  <-- CFBundleShortVersionString
                    "name": "GitHub Desktop",                <-- CFBundleName
                    "criteria": [
                        {
                            "name": "Application Bundle ID", 
                            "operator": "is", 
                            "value": "com.github.GitHub",  <-- CFBundleIdentifier
                            "type": "recon", 
                            "and": true
                        }, 
                        {
                            "name": "Application Version", 
                            "operator": "is", 
                            "value": "Hasty Things Done Hastily\",  <-- CFBundleShortVersionString
                            "type": "recon" 
                        }
                    ]
                }
            ], 
            "capabilities": [
                {
                    "name": "Operating System Version", 
                    "operator": "greater than or equal", 
                    "value": "10.9",  <-- LSMinimumSystemVersion
                    "type": "recon"
                }
            ],
            "dependencies": []
        }
    ],
    "extensionAttributes": []
}

It is important to understand how the above values map in the event that you create a definition, but it has used incorrect values because the developer assigned them differently than what is considered standard (especially true for version strings).

In the event the CFBundleShortVersionString or LSMinimumSystemVersion keys are missing from Info.plist, the script will prompt you for an alternative.

Add Extension Attributes

You can also include extension attributes that need to be a part of your definition. For example, if you have the following bash script as an extension attribute for GitHub Desktop.app saved as github-ea.sh:

#!/bin/bash

outputVersion="Not Installed"

if [ -d /Applications/GitHub\ Desktop.app ]; then
    outputVersion=$(defaults read /Applications/GitHub\ Desktop.app/Contents/Info.plist CFBundleShortVersionString)
fi

echo "<result>$outputVersion</result>"

You can pass it to the -e or –extension-attribute argument:

$ python patchstarter.py /Applications/GitHub\ Desktop.app -p "Github" -e github-ea.sh

The extension attribute will be appended to the extensionAttributes key in the definition (the key here is the appName in lowercase with spaces replacing dashes):

{
    "...",
    "extensionAttributes": [
        {
            "key": "github-desktop",
            "value": "IyEvYmluL2Jhc2gKCm91dHB1dFZlcnNpb249Ik5vdCBJbnN0YWxsZWQiCgppZiBbIC1kIC9BcHBsaWNhdGlvbnMvR2l0SHViXCBEZXNrdG9wLmFwcCBdOyB0aGVuCiAgICBvdXRwdXRWZXJzaW9uPSQoZGVmYXVsdHMgcmVhZCAvQXBwbGljYXRpb25zL0dpdEh1YlwgRGVza3RvcC5hcHAvQ29udGVudHMvSW5mby5wbGlzdCBDRkJ1bmRsZVNob3J0VmVyc2lvblN0cmluZykKZmkKCmVjaG8gIjxyZXN1bHQ+JG91dHB1dFZlcnNpb248L3Jlc3VsdD4iCg==",
            "displayName": "GitHub Desktop"
        }
    ]
}

This will not update any of the capabilities or components/criteria in the generated definition! It will be up to you to make the edits to reference the extension attribute.

Create Patch Data Only

This script is only meant to create a starter definition for an application. It will only have the one version so there will be no historical data for reporting. However, if you upload it to your own running Patch Server you can maintain it by updating the version through the API.

The –patch-only argument will take an application and only generate that portion of the definition:

python patchstarter.py /Applications/GitHub\ Desktop.app -p "GitHub" --patch-only

The JSON output can be sent to the Patch Server’s API to update the stored definition. An example curl command is in the GitHub readme:

$ curl -X POST http://localhost:5000/api/v1/title/GitHubDesktop/version -d "{\"items\": [$(python patchstarter.py /Applications/GitHub\ Desktop.app -p "GitHub" --patch-only)]}" -H 'Content-Type: application/json'

Try It Out

This script should ease the process of getting started with your own custom patch definitions. If you have access to multiple versions of an application, you can use the –patch-only argument to generate the data for each and place them into your starter definition.

Note that versions must be in descending order! The latest version must always be at the top of your patches array and the older version at the bottom.

If you run into problems or have feature requests, open an issue on the GitHub repository!

Patch Server for Jamf Pro

(TL;DR, gimme the link: https://github.com/brysontyrrell/PatchServer)

After several months of not getting around to it, my PatchServer project on GitHub is finally nearing a true 1.0 state.

I am openly asking for those who have been following this project, and those who are interested in this project, to download, use, and provide feedback on what should be finished before the release of Jamf Pro 10.2.

Please create issues on GitHub for bugs and feature requests that you would want to make the cut for 1.0.

Some time late last year (and I say some time because it’s all becoming a blur), I was brought into a meeting where I was shown our (Jamf’s) progress on providing a framework for customers to be able to create their own patch definitions. This framework would allow customers to setup their own patch servers and add them to their JSS.

A day or so later, I wrote the first rough version of my own implementation.

Backing up a sec:

What’s a patch definition?

In Jamf Pro v10 we introduced a feature called Patch Management. With this, you could subscribe to a number of software titles that Jamf curates and maintains. Once subscribed, your JSS will, on a schedule, read in the patch definitions of those software titles to stay updated.

For more about Patch Management, see the Jamf Pro Admin Guide (10.1):

These patch definitions (which are JSON data) contain historical information about a software title’s version history and requirements for determining if the software is installed on a managed computer. This allows admins to use the Patch Management feature to create reports and update policies to automatically patch those software titles on computers.

Of course, when these features came out there was one resounding question from nearly everyone:

“Why can’t we make our own patch definitions?”

External Patch Sources

The framework I mentioned above is the answer to this. In Jamf Pro 10.2+ you will have the option of adding External Patch Sources to your JSS. Then, in addition to the official Jamf software titles, you will be able to subscribe to your own and use the same reporting and policy features.

 

The external patch source must be a server your JSS is able to reach via HTTP/HTTPS. This patch server must expose the following endpoints:

  • /software
    This returns a JSON array of all the software titles that are available on this server. For example:

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfImaging", 
        "lastModified": "2018-02-03T03:34:36Z", 
        "name": "Jamf Imaging", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /software/TitleId,TitleId
    This returns the same JSON as above, but limited to the comma separated list of software titles. For example (passing JamfAdmin,JamfRemote):

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /patch/TitleId
    This returns the full patch definition JSON of the software title. Here is an abbreviated example:

    {
      "id": "JamfAdmin",
      "name": "Jamf Admin",
      "publisher": "Jamf", 
      "appName": "Jamf Admin.app", 
      "bundleId": "com.jamfsoftware.JamfAdmin", 
      "currentVersion": "10.1.1", 
      "lastModified": "2018-02-03T03:34:34Z", 
      "extensionAttributes": [
        {"ExtensionAttributeObjects"}
      ],
      "patches": [
        {"PatchObjects"}
      ], 
      "requirements": [
        {"RequirementsObjects"}
      ]
    }

If you had a patch server located at http://patch.my.org, the full URLs would be:

At this time, there is no product that Jamf is providing for customers to install and have a ready to use patch server. The focus has been on opening up the framework that the official patch source uses and allow customers to extend their environments through a little engineering work.

Not all of us are engineers, of course. Thus…

Enter: Patch Server

gui_01_index.png

I wanted to have a working patch server ready for the Jamf community in time for 10.2’s release. My initial patch server implementation (I call it an implementation because it’s one way of providing a patch source) achieved serving the proper JSON data for each of the endpoints described above using a database (SQLite) for the backend.

While my original goals were much grander, including the ability to fully manage a patch definition in a GUI instead of writing out JSON, I had to pare it back in order to get the project into a deliverable state.

In the past week I went through the code and ripped out everything that I felt was not needed, or doable. Then, I went through and added in new features (ported from another project) and streamlined the UI elements that were left.

This patch server features:

  • All required Jamf Pro endpoints to serve as an External Patch Source
  • An API for programmatic management of patch definitions and versions.
    • Create/delete patch definitions.
    • Add versions to existing patch definitions.
    • Create backup archives of all patch definitions.
  • UI for management of patch definitions.
  • Validation of uploaded patch definitions.
    gui_05_validation.png
  • Full user documentation at http://patchserver.readthedocs.io/
    patchserver_docs.png

    • UI Overview
    • Setup Instructions
    • API Documentation

Bring the Requests

Until Jamf Pro 10.2 is released, I’m not going to tag the project at a 1.0 version. If you are in Jamf’s beta program and testing 10.2, I invite you to give this a try and let me know what you think. Specifically, I’m asking for you do open up issues on GitHub for:

  • Bugs you find
  • Features you want, such as:
    • Connect to an actual database like MySQL (?)
  • Documentation you want, such as:
    • Instructions for installing on X

Not everything that is reported might get worked on, but the good news is I released the patch server under the MIT license. If you have some Python chops you can fork it and do whatever you want with the codebase to suit your needs!

But, I don’t wanna setup a server…

If you had that reaction to the idea of setting up your own external patch source, ask yourself if you match any of these descriptions:

  1. My JSS can talk to pretty much anything if I want it to,
  2. I want a patch server; I don’t want to host a patch server,
  3. It doesn’t matter where my patches live as long as I can get and manage them,
  4. Can’t this be a cloud thing?

If so… stayed tuned for a future blog post.

Stop everything! Start using Pipenv!

Seven months ago I tweeted about Pipenv:

That was about it.

The project- which was at v3.6.1 at the time – still felt new. At Jamf, we were working on our strategy for handling project requirements and dependencies going forward for Jamf Cloud’s automation apps and Pipenv, while a cool idea, didn’t have maturity and traction that.

As @elios pointed out on the MacAdmins Slack: Pipenv was announced in January of this year as an experimental project.

Today it is the officially recommended Python packaging tool from Python.org.

In his words, “That’s awfully fast…”

The GitHub repository for Pipenv has 5400+ stars and over 100+ contributors to the project. It is proceeding at breakneck development speeds and adding all kinds of juicy features every time you turn around. It’s on v8.3.2.

Within ten minutes of finally, FINALLY, giving Pipenv a try today I made the conscious decision to chuck the tooling we had made and convert all of our projects over.

(Oh, and the person who wrote that tooling wholly agreed)

Let’s talk about that.

Part 1: Why requirements.txt sucks

There is was this standard workflow to your Python projects. You needed two key tools: pip and virtualenv. When developing, you would create a Python virtual environment using virtualenv in order to isolate your project from the system Python or other virtual environments.

You would then use pip to install all the packages your project needs. Once you had all of those packages installed you would then run a command to generate a requirements.txt file that listed every package at the exact installed version.

The requirements.txt file stays with your project. When someone else gets a copy of the repository to work on it or deploy it they would follow the same process of creating a virtual environment and then using pip to install all of the required packages by feeding that requirements.txt file into it.

Now, that doesn’t seem so bad, right? The drawbacks become apparent as you begin to craft your build pipelines around your projects.

Maintenance is a Manual Chore

The first thing to call out is how manual this process is. You need to create the same version virtual environment for the project once you’ve cloned it and then install the packages. If you want to begin upgrading packages you need to be sure to manually export a new requirements.txt file each time.

Removing packages is far more problematic. Uninstalling with pip will not remove dependencies for a package! Take Flask, for example. It’s not just one package. It has four sub-packages that get installed with it, and one of those (Jinja2) has its own sub-package.

So, maybe the answer is to manually maintain only the top-level packages in your requirements.txt file? That’s a hard no. You don’t want to do that because it makes your environment no longer deterministic. What do I mean? I mean, those sub-packages we just talked about are NOT fixed versions. When you’re installing Python packages from the Python Package Index (PyPI) the aren’t using requirements.txt files with fixed versions for their dependencies. They instead are specifying ranges allowing for installs at a minimum and/or maximum version for that sub-package.

This makes a lot of sense when you think about it. When installing a package for the first time as a part of a new project, or updating a package in an existing project, you are going to want the latest allowed versions for all the sub-packages for any patches that have been made.

Not so much when it comes to a project/application. You developed and tested your code at specific package versions that are a known working state. This is past the development phase when bugs from updates to those packages can be identified and addressed. When it comes to deployment, your project’s requirements.txt file must be deterministic and list every package at the exact version.

Hence, maintenance of this file becomes a manual chore.

But Wait, There’s Testing!

There are a lot of packages that you might need to install that have nothing to do with running your Python application, but they have everything to do with running tests and building documentation. Common package requirements for this would by Pytest, Tox, and Sphinx with maybe a theme. Important, but not needed in a deployment.

We want to be able to specify a set of packages to install that will be used during the testing and build process. The answer unfortunately, is a second requirements.txt file. This one would have a special name like requirements-dev.txt which is only used during a build. This file could only contain the specific packages and be installed with pip after the standard requirements.txt, or it could contain all of those plus the build packages. In either case, the maintenance problem continues to grow.

So We Came Up With Something…

Our process at Jamf ended up settling on three types of requirements.txt files in an effort to address all the shortcomings described.

  • requirements.txt
    This file contained the full package list for the project at fixed versions. It would be used for spinning up new development environments or during deployment.
  • core-requirements.txt
    This file contained the top-level packages without fixed versions. Development environments would not be built from this. Instead, this file would be used for recreating the standard requirements.txt file at updated package versions and eliminating orphaned packages that were no longer used or removed from the project at some point.
  • build-requirements.txt
    This is a manually maintained file per-project that only contained the additional packages needed during the build process.

This approach isn’t too dissimilar to what others have implemented for many of the same reasons we came up with.

Part 2: Enter Pipfile

Not Pipenv? We’re getting there.

Pipfile was introduced almost exactly a year ago at the time of this post (first commit on November 18th, 2016). The goal of Pipfile is to replace requirements.txt and address the pain points that we covered above.

Here is what a Pipfile looks like:

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]



[requires]

python_version = "2.7"

PyPI is the default source in all new Pipfiles. Using the scheme shown, you can add additional [source] sections with unique names to specify your internal Python package indexes (we have our own at Jamf for shared packages across projects).

There are two groups for listing packages: dev-packages and packages. In a Pipfile these lists follow how we were handling this where only the build packages go into the dev-packages list and all project packages go under the standard packages list.

Packages in the Pipfile can have fixed versions or be set to install whatever the latest version is.

[packages]

flask = "*"
requests = "==2.18.4"

The [requires] section dictates the version of Python that the project is meant to run under.

From the Pipfile you would create what is called a Pipfile.lock which contains all of the environment information, the installed packages, their installed versions, and their SHA256 hashes. The hashes are a recent security feature of pip to validate packages that are installed when deployed. If there is a hash mismatch you can abort. A powerful security tool in preventing malicious from entering your environments.

Note that you can specify the SHA256 hashes of packages in a normal requirements.txt file. This is a feature of pip and not of Pipfile.

It is this Pipfile.lock that will be used to generate environments on other systems whether for development or deployment. The Pipfile will be used for maintaining packages and dependencies and regenerating your Pipfile.lock.

All of this is just a specification. Pip as of yet still does not support Pipfiles, but there is another..

Part 3: Pipenv Cometh

Pipenv is the implementation of the Pipfile standard. It is built on top of pip and virtualenv and manages both your environment and package dependencies, and it does so like a boss.

Get Started

Install Pipenv with pip (or homebrew if that’s your jam).

$ pip install pipenv

Then in a project directory create a virtual environment and install some packages!

$ pipenv --two
Creating a virtualenv for this project…
<...install output...>
Virtualenv location: /Users/me/.local/share/virtualenvs/test-zslr3BOw
Creating a Pipfile for this project…

$ pipenv install flask
Installing flask…
<...install output...>
Adding flask to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (36eec0)!

$ pipenv install requests==2.18.4
Installing requests==2.18.4…
<...install output...>
Adding requests==2.18.4 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (572f23)!

$

Notice how we see the Locking messages after every install? Pipenv automatically regenerated the Pipfile.lock each time the Pipfile is modified. Your fixed environment is being automatically maintained!

Graph All The Things

Let’s look inside the Pipfile itself.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]



[packages]

flask = "*"
requests = "==2.18.4"


[requires]

python_version = "2.7"

No sub-packages? Nope. It doesn’t need to track those (they end up in the Pipfile.lock, remember?). But, if you’re curious, you can use the handy graph feature to view a full dependency tree of your project!

$ pipenv graph
Flask==0.12.2
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
   - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
 
$

Check that out! Notice how you can see the requirement that was specified for the sub-package in addition to the actual installed version?

Environment Management Magic

Now let’s uninstall Flask.

$ pipenv uninstall flask
Un-installing flask…
<...uninstall output...>
Removing flask from Pipfile…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (4ddcaf)!

$

And re-run the graph command.

$ pipenv graph
click==6.7
itsdangerous==0.24
Jinja2==2.9.6
 - MarkupSafe [required: >=0.23, installed: 1.0]
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
Werkzeug==0.12.2

 

Yes, the sub-packages have now been orphaned within the existing virtual environment, but that’s not the real story. If we look inside Pipfile we’ll see that requests is the only package listed, and if we look inside our Pipfile.lock we will see that only requests and it’s sub-packages are present.

We can regenerate our virtual environment cleanly with only a few commands!

$ pipenv uninstall --all
Un-installing all packages from virtualenv…
Found 20 installed package(s), purging…
<...uninstall output...>
Environment now purged and fresh!

$ pipenv install
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell

$ pipenv graph
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

$

Mindblowing!!!

Installing the dev-packages for our builds uses an additional flag with the install command.

$ pipenv install sphinx --dev
Installing sphinx…
<...install output...>
Adding sphinx to Pipfile's [dev-packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (d7ccf2)!

The appropriate locations in our Pipfile and Pipefile.lock have been updated! To install the dev environment perform the same steps for regenerating above but add the –dev flag.

$ pipenv install --dev
Installing dependencies from Pipfile.lock (f58d9f)…
 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 18/18 — 00:00:05
To activate this project's virtualenv, run the following:
 $ pipenv shell

Part 4: Deploy Stuff!

The first project I decided to apply Pipenv to in order to learn the tool is ODST. While there is a nice feature in Pipenv where it will automatically import a requirements.txt file if detected, I opted to start clean and install all my top-level packages directly. This gave me a proper Pipfile and Pipfile.lock.

Here’s the resulting Pipfile.

[source]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]

pytest = "*"
sphinx = "*"
sphinx-rtd-theme = "*"


[packages]

flask = "*"
cryptography = "*"
celery = "*"
psutil = "*"
flask-sqlalchemy = "*"
pymysql = "*"
requests = "*"
dicttoxml = "*"
pyjwt = "*"
flask-login = "*"
redis = "*"


[requires]

python_version = "2.7"

Here’s the graph of the installed packages (without the dev-packages).

$ pipenv graph
celery==4.1.0
 - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.3]
 - kombu [required: <5.0,>=4.0.2, installed: 4.1.0]
 - amqp [required: >=2.1.4,<3.0, installed: 2.2.2]
 - vine [required: >=1.1.3, installed: 1.1.4]
 - pytz [required: >dev, installed: 2017.3]
cryptography==2.1.3
 - asn1crypto [required: >=0.21.0, installed: 0.23.0]
 - cffi [required: >=1.7, installed: 1.11.2]
 - pycparser [required: Any, installed: 2.18]
 - enum34 [required: Any, installed: 1.1.6]
 - idna [required: >=2.1, installed: 2.6]
 - ipaddress [required: Any, installed: 1.0.18]
 - six [required: >=1.4.1, installed: 1.11.0]
dicttoxml==1.7.4
Flask-Login==0.4.0
 - Flask [required: Any, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
Flask-SQLAlchemy==2.3.2
 - Flask [required: >=0.10, installed: 0.12.2]
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.9.6]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]
 - SQLAlchemy [required: >=0.8.0, installed: 1.1.15]
psutil==5.4.0
PyJWT==1.5.3
PyMySQL==0.7.11
redis==2.10.6
requests==2.18.4
 - certifi [required: >=2017.4.17, installed: 2017.11.5]
 - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
 - idna [required: >=2.5,<2.7, installed: 2.6]
 - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

In my Dockerfiles for the project I swapped out using requirements.txt and switched it to install my packages using the Pipefile.lock into the system Python (in a containerized app there’s no real need to create a virtual environment).

RUN /usr/bin/apt-get update -q && \
    /usr/bin/apt-get install -qqy build-essential git && \
    /usr/bin/apt-get install -qqy python-pip python-dev && \
    /usr/bin/pip install pipenv && \
    /usr/bin/apt-get install -qqy libssl-dev libffi-dev && \
    /usr/bin/apt-get install -qqy uwsgi uwsgi-plugin-python && \
    /usr/bin/apt-get clean && \
    /bin/rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY /docker/webapp/web-app.ini /etc/uwsgi/apps-enabled/

COPY /Pipfile* /opt/odst/
WORKDIR /opt/odst/
RUN /usr/local/bin/pipenv install --system --deploy

COPY /ods/ /opt/odst/ods/
COPY /application.py /opt/odst/
COPY /docker/ods_conf.cfg /opt/odst/

RUN /bin/chown -R www-data:www-data /opt/odst/ods/static

CMD ["/usr/bin/uwsgi", "--ini", "/etc/uwsgi/apps-enabled/web-app.ini"]

The –system flag tells Pipenv to install to the system Python. The –deploy flag will abort the installation if the Pipfile.lock is out of date or the Python version is wrong. Out of date? Pipenv knows if your Pipfile.lock matches up with the Pipfile by comparing a SHA256 hash of the Pipfile that it has saved. If there’s a mismatch then there is an issue and it aborts.

If I didn’t want to rely on this hash match feature, I could instead pass a –ignore-pipfile flag that will tell it to proceed using only the Pipfile.lock.

I should mention at this point that Pipenv dramatically speeds up build times. When using pip your packages will install sequentially. Pipenv will install packages in parallel. The difference is immediately noticeable, especially when regenerating your development environments.

Still, The World is Catching Up

Python development workflows area clearly moving to this tool, but Pipenv is less than a year old at this point and not everything will support generating environments from the provided Pipfiles.

The two I am currently using that fall in that category are AWS Elastic Beanstalk and ReadTheDocs.org. Both need a requirements.txt file in order to build the Python environments.

That is thankfully a simple task. You can generate a requirements.txt file from Pipenv by using the lock option.

$ pipenv lock --requirements > requirements.txt

 

For Elastic Beanstalk applications, I can have this command run as a part of my build pipeline so the file is included with the ZIP archive before it goes to S3.

In the case of Read the Docs there is an open issue for adding Pipfile support, but until then I will need to generate the requirements.txt file as I make changes to my environment and save it with the repository.

For Read the Docs I’ll want the extra dev-packages. This is done by using the pipenv run command. This will execute the succeeding string as if it were run in the environment.

$ pipenv run pip freeze > requirements.txt

The bright side is that I am no longer actually managing this file. It is straight output from my Pipenv managed environment!

Part 5: Profit

I hope you enjoyed my overview and impressions of Pipenv. It’s a tool that is going to have a huge impact on development and deployment workflows – for the better!

A huge shout out to Kenneth Reitz for creating yet another invaluable package for the Python community!

* Edit Nov 9: Changed the command for exporting a requirements.txt file. Using pipenv lock -r -d only outputs the packages under the dev-packages section.