Trick Sam into building your Lambda Layers

Right now, the SAM CLI doesn’t support building Lambda Layers; those magical additions to Lambda that allow you to defined shared dependencies and modules. If you’re unfamiliar, you can read more about them here:

New for AWS Lambda – Use Any Programming Language and Share Common Components

If you read my last article on using sam build, you might think to yourself, “Hey, I can add Layers into my template to share common code across all my Lambdas!”, but hold on! At the moment sam build does not support building Layers the way it builds Lambda packages.

But, there is a hacky way around that. Here’s our repository:

carbon (7)

Now here’s the contents of template.yaml:

carbon (6).png

We’ve defined an AWS::Serverless::Function resource, but with no events, or any other attributes for that matter. We have also defined a AWS::Serverless::LayerVersion resource for our Lambda Layer, but the ContentUri path points to the build directory for the Lambda function.

See where this is going?

sam build will install all the dependencies for our Layer and copy its code into the build directory, and then when we call sam package the Layer will use that output! Spiffy. This does result in an orphan Lambda function that will never be used, but it won’t hurt anything just sitting out there.

Now, we aren’t done quite yet. According to AWS’s documentation, you need to place Python resources within a python directory inside your Layer. The zip file that sam build creates will be extracted into /opt, but the runtimes will only look, by default in a matching directory within /opt (so in the case of Python, that would be /opt/python).

See AWS Lambda Layers documentation for more details.

We can’t tell sam build to do that, but we can still get around this inside our Lambda functions that use the new Layer by adding /opt into sys.path (import searches all of the locations listed here when you call it). Here’s an example Python Lambda function that does this:

carbon (5)

Performing a test execution gives us the following output:

START RequestId: fd5a0bf2-f9af-11e8-bff4-ab8ada75cf17 Version: $LATEST
['/var/task', '/opt/python/lib/python3.6/site-packages', '/opt/python', '/var/runtime', '/var/runtime/awslambda', '/var/lang/lib/python36.zip', '/var/lang/lib/python3.6', '/var/lang/lib/python3.6/lib-dynload', '/var/lang/lib/python3.6/site-packages', '/opt/python/lib/python3.6/site-packages', '/opt/python', '/opt']
<module 'pymysql' from '/opt/pymysql/__init__.py'>
<module 'sqlalchemy' from '/opt/sqlalchemy/__init__.py'>
<module 'stored_procedures' from '/opt/stored_procedures.py'>
END RequestId: fd5a0bf2-f9af-11e8-bff4-ab8ada75cf17
REPORT RequestId: fd5a0bf2-f9af-11e8-bff4-ab8ada75cf17	Duration: 0.82 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 34 MB

Voila! We can see the inclusion of /opt into our path (and the expected path of /opt/python before it) and that our dependencies and custom module were all successfully imported.

It breaks PEP8 a little, but it gets the job done and we have now successfully automated the building and deployment of our Lambda Layer using AWS’s provided tooling.

 

 

Advertisements

Possum is dead; long live the squirrel.

Sam Build

A part of me is sorry to say that the title is not clickbait. Just before re:Invent, the AWS SAM developers made a pretty big announcement:

SAM CLI Introduces sam build Command

You can now use the sam build command to compile deployment packages for AWS Lambda functions written in Python using the AWS Serverless Application Model (AWS SAM) Command Line Interface (CLI).

All you need to do is*:

carbon

This command will iterate over your SAM template and output ready to package versions of your template and Python Lambdas to a .aws-sam/build directory. This lines up exactly with work I was preparing to do for possum, but AWS has gone ahead and done all the work.

* With other required arguments depending on your environment.

In fact, you’ll find that sam build nearly has feature parity with possum with a few exceptions which I’ll go into. Let’s take a look at what one of my serverless projects looks like as an example:

MyApp/
├── src/
|   └── functions/
│       └── MyLambda/
│           ├── my_lambda.py
│           └── requirements.txt
├── Pipfile
├── Pipfile.lock
└── template.yaml

I use pipenv for managing my development environments. The project’s overall dependencies are defined in my Pipfile while the pinned versions of those dependencies are in the Pipfile.lock. Individual dependencies for my Lambdas are defined in their own requirements.txt files within their directories.

I use PyCharm for all of my Python development. Using pipenv to manage the individual virtual environment for a given project allows me to take advantage of the autocompletion features of PyCharm across all the Lambda functions I’m working on. I maintain the individual requirements.txt files for each of my Lambdas and have their listed packages match the version in my Pipfile.lock (I have scripting in possum 1.5.0 that manages syncing the package versions in the requirements.txt files for me).

Now, when I run sam build it will perform all the same actions as possum, but instead of creating the zipped archive and uploading straight to S3 the built Lambdas will be available within the project’s directory.

Possum was originally written as a replacement for sam package that would include dependencies. It would upload the Lambda packages directly to an S3 bucket.

MyApp/
├── .aws-sam/
|   └── build/
|       ├── MyLambda/
│       |   ├── installed_depencency/
│       |   |   └── {dependency files}
│       |   └── my_lambda.py
|       └── template.yaml
├── src/
|   └── functions/
│       └── MyLambda/
│           ├── my_lambda.py
│           └── requirements.txt
├── Pipfile
├── Pipfile.lock
└── template.yaml

The new template located at .aws-sam/build/template.yaml has had the CodeUri keys updated to reference the relative paths within the .aws-sam/build directory. You will see that these copies of the Lambda code now contain all the dependencies that were defined within the requirements.txt file.

The example above generalizes this. Just to show you, the ApiContributorRegistration Lambda for CommunityPatch installs the cryptography and jsonschema packages. This is what the output looks like for a built Lambda:

CommunityPatch/
├── .aws-sam/
    └── build/
        └── ApiContributorRegistration/
            ├── asn1crypto/
            ├── asn1crypto-0.24.0.dist-info/
            ├── cffi/
            ├── cffi-1.11.5.dist-info/
            ├── cryptography/
            ├── cryptography-2.4.1.dist-info/
            ├── idna/
            ├── idna-2.7.dist-info/
            ├── jsonschema/
            ├── jsonschema-2.6.0.dist-info/
            ├── pycparser/
            ├── pycparser-2.19.dist-info/
            ├── schemas/
            ├── six-1.11.0.dist-info/
            ├── _cffi_backend.cpython-36m-x86_64-linux-gnu.so
            ├── api_contributor_registration.py
            ├── requirements.txt
            └── six.py

Dependencies usually have dependencies of their own (those two packages became seven!). And that’s just one Lambda.

Sam Invoke

Now, at this point you could take the output from sam build and perform sam package to get everything loaded into S3 and have a deployment template to run in CloudFormation. However, now that we have a build template we can take advantage of the SAM CLI’s powerful test features which possum was working towards adopting:

carbon (1).png

We can unit test our Lambdas using generated AWS events from the SAM CLI! I’ll cover my workflow for this in more detail at a later time, but before deploying the entire app out to AWS we can now perform some sanity checks that the Lambdas should execute successfully when given a proper payload. Ideally, you would want to generate multiple event payloads to cover a variety of potential invocations.

Sam Package/Deploy

From here the standard package and deploy steps follow (using either the sam or aws CLI tools) which I won’t cover here as I’ve done so in other posts. The full process referencing the new .aws-sam/build directory looks like this:

carbon (4).png

sam package knows to use the template output from sam build without having to specify the path to it!

Gotchas

While all of this is great, let’s cover the exceptions I alluded to earlier.

sam build will perform the build every single time. Even if you don’t make changes between builds it will still rebuild all your functions. This is agonizingly slow. Preventing unneeded builds was one of the first features that went into possum to speed up my personal development. The AWS devs have been listening to some of my feedback on how I implemented this and are looking into adopting a similar solution for sam build.

Every Lambda must have a requirements.txt file even if they don’t have any external dependencies. I ran into this one right away. At the moment, sam build expects there to always be a requirements.txt file within a Lambda function’s code directory. Use a blank file for simple Lambdas as a workaround. The AWS devs are aware of this and will be fixing it.

python must resolve to a Python environment of the same version as your serverless app. If python resolves to a different version (like on a Mac where it resolves to the 2.7 system executable) activate a virtual environment of the correct version as a workaround. You should be able to easily do this if you’re using pipenv by running pipenv shell. The reason this isn’t an issue for possum is because possum relies on pipenv for generating the correct Python build environment based upon the runtime version defined in the template. The AWS devs have been taking my feedback and are looking into this.

Edit: The below wheel issue is fixed in sam 0.8.1!

You may run into The error message “Error: PythonPipBuilder:ResolveDependencies – {pycparser==2.19(sdist)}”. This happens if you’re using a version of Python that didn’t include the wheel package. This will be fixed in a future release, but you can pip install wheel in the Python environment that sam was installed to as a workaround.

You’re also going to run into that error when you try to us the –use-container option because the Docker image pulled for the build environment is also missed that package.

The workaround is to build intermediary image based on lambci/lambda:build-python3.6, install the wheel package, and then tag it using the same tag (yes, you’re tagging an image to override the existing tag with your own custom one) . This will also be fixed in a future release.

CommunityPatch.com (beta)

In previous posts, I talked about two projects I had been working on for the Jamf community to better help admins get started using the new “External Patch Sources” feature in Jamf Pro 10.2+. While working on Patch Server and the companion Patch-Starter-Script, I also wrote a quick proof of concept for a serverless version that would run in an AWS account.

The Stupid Simple Patch Server uses API Gateway and Lambda functions to serve patch definitions you stored in an S3 bucket. I even included the same API endpoints from the Patch Server so workflows between the two could be shared. I even took it a step further and added a subscription API so it would sync with a remote patch definition via URL.

That side project (of a side project) made me think about how I could take the basic design and build upon it into something that could be used by multiple admins. At first, I wrote a lot of code to transform the Stupid Simple Patch Server into a multi-tenant application. At a point, I considered the limitations of what could be done in a manner that could be considered secure and scrapped much of it.

But not everything. The work I had done was retooled into a new concept: a single, public, community managed patch source for Jamf Pro. A service where anyone could contribute patch definitions, and be able to manage and update them. Five minutes after having this idea I bought the communitypatch.com domain and setup a beta instance of my work-in-progress:

https://beta.communitypatch.com

CommunityPatchBeta.png

New API

The big green “Read the docs” button on the main page will take you to… the documentation! There you will find those APIs in much greater detail.

The community managed patch source mirrors a number of features from my Patch Server project. The /jamf endpoints are here to integrate with Jamf Pro and the service can be used as an external patch source.

The /api endpoints are slightly different from the Patch Server, but allow for creating definitions by providing the full JSON or a URL to an external source (creating a synced definition) and updating the versions afterwards.

From the docs, here’s the example for creating a new patch definition using the Patch-Starter-Script:

curl https://beta.communitypatch.com/api/v1/title \
   -X POST \
   -d "{\"author_name\": \"<NAME>\", \"author_email\": \"<EMAIL>\", \"definition\": $(python patchstarter.py /Applications/<APP> -p "<PUBLISHER>")}" \
   -H 'Content-Type: application/json'

Here, there are required author_name and author_email keys you need to provide when creating a definition. The author_name you choose will be injected into the ID and name keys of the definition you’re providing.

For example, if I provide “Bryson” for my name, and I’m creating the “Xcode.app” definition, it’s ID will become “Xcode_Bryson” and the display name “Xcode (Bryson)”. These changes make it possible to differentiate titles when browsing in Jamf Pro, and for members of the community to better identify who is managing what (as well as sharing with each other).

After you create a patch definition, you will be emailed an API token to the address you provided in author_email. This token is specifically for managing that title, and is the only way to update the title after. Your email address is not saved with CommunityPatch. A hash of it is stored with the title so you can reset the token should you lose it or need the previous one invalidated (this feature is not implemented yet).

Updating works similarly to Patch Server (but without the items key):

curl http://beta.communitypatch.com/api/v1/title/<ID>/version \
   -X POST \
   -d "$(python patchstarter.py /Applications/<APP> --patch-only)" \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer <TOKEN>'

 

Try It Out

I had a number of admins on Slack giving me feedback and testing the API for a few weeks. While I have work left to do to ensure the production version of CommunityPatch is performant, and still some more features to finish writing, I am at a stage where I would like those interesting in contributing to and using CommunityPatch to join in, and try the documented features (in your test environments).

You can jump right in by joining the #communitypatch channel on the MacAdmins Slack, hitting the CommunityPatch documentation, play around with the API, test definitions you create in your Jamf Pro test environments, and discuss what you find.

CommunityPatch is being written out in the open. You can go to GitHub and see the code for yourself. You can even contribute at a code/docs level if you like! For the immediate, having admins test it out and report back will provide me a lot of value as I work towards completing the application and deploying it to production.

Links

Possum – A packaging tool for Python AWS Serverless Applications

The applications I build on AWS are all written in Python using the Serverless Application Model (SAM). Building my applications using a template and Lambda functions, I quickly ran into a limitation of the aws command line tools: external dependencies.

If your Lambda functions have no dependencies (not including the AWS SDKs), or you pre-download and embed them alongside your code, the standard package command works:

serverless_04

However, if you want to install dependencies at the time of packaging the application, you are left in a position where you need to roll your own build system. Amazon provides instructions on creating a Python deployment package, but it would be nice if running the aws cloudformation command did this for us.

Possum

I wrote a packaging tool to fill in the gap left by Amazon’s. Possum (an amalgamation of “Python AWS SAM”) processes a SAM template file just as aws cloudformation package but creates per-function Lambda deployment packages if it detects a requirements file within the function’s directory (Pipfile or requirements.txt).

Possum can be installed from the Python Package Index:

possum_01

Once installed, Possum becomes available as a command line tool (it is loaded into your Python installation’s /bin directory):

possum_02.png

What Possum does is iterate over the Resources section of your SAM template and find all the objects of the AWS:Serverless:Function type, determine the location of their code using the Properties:CodeUri value, and through the magic of Pipenv create individual virtual environments to download the external dependencies, if any, and zip the files together into a Lambda package. Once the package and upload process is complete, Possum will either print your updated deployment template on the screen or write it out to a filename that you specified.

possum_03.png

In the above example, my HelloWorld function didn’t have any defined dependencies within it’s directory so the contents were zipped up as they were. For the Authorizer, there was a Pipfile present which triggered the build process. The approach to Lambda function dependencies with Possum is to handle them on a per-function basis. This creates artifacts that only include the required packages for that function (or none).

Pipenv is not installed with Possum. Instead, Possum will shell-out to run the Pipenv commands (so you will need to have Pipenv installed separately).

After Possum has finished, I can take the deployment.yaml file and deploy the application using aws cloudformation deploy or the AWS console.

Try It Out

If you’re working with Python Lambda functions, please give Possum a try! If you encounter an issue, or have a feature request, you can open an issue on the GitHub page.

Possum’s GitHub Page
https://github.com/brysontyrrell/Possum

Possum on the Python Package Index
https://pypi.org/project/possum/

 

Patch Server for Jamf Pro

(TL;DR, gimme the link: https://github.com/brysontyrrell/PatchServer)

After several months of not getting around to it, my PatchServer project on GitHub is finally nearing a true 1.0 state.

I am openly asking for those who have been following this project, and those who are interested in this project, to download, use, and provide feedback on what should be finished before the release of Jamf Pro 10.2.

Please create issues on GitHub for bugs and feature requests that you would want to make the cut for 1.0.

Some time late last year (and I say some time because it’s all becoming a blur), I was brought into a meeting where I was shown our (Jamf’s) progress on providing a framework for customers to be able to create their own patch definitions. This framework would allow customers to setup their own patch servers and add them to their JSS.

A day or so later, I wrote the first rough version of my own implementation.

Backing up a sec:

What’s a patch definition?

In Jamf Pro v10 we introduced a feature called Patch Management. With this, you could subscribe to a number of software titles that Jamf curates and maintains. Once subscribed, your JSS will, on a schedule, read in the patch definitions of those software titles to stay updated.

For more about Patch Management, see the Jamf Pro Admin Guide (10.1):

These patch definitions (which are JSON data) contain historical information about a software title’s version history and requirements for determining if the software is installed on a managed computer. This allows admins to use the Patch Management feature to create reports and update policies to automatically patch those software titles on computers.

Of course, when these features came out there was one resounding question from nearly everyone:

“Why can’t we make our own patch definitions?”

External Patch Sources

The framework I mentioned above is the answer to this. In Jamf Pro 10.2+ you will have the option of adding External Patch Sources to your JSS. Then, in addition to the official Jamf software titles, you will be able to subscribe to your own and use the same reporting and policy features.

 

The external patch source must be a server your JSS is able to reach via HTTP/HTTPS. This patch server must expose the following endpoints:

  • /software
    This returns a JSON array of all the software titles that are available on this server. For example:

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfImaging", 
        "lastModified": "2018-02-03T03:34:36Z", 
        "name": "Jamf Imaging", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /software/TitleId,TitleId
    This returns the same JSON as above, but limited to the comma separated list of software titles. For example (passing JamfAdmin,JamfRemote):

    [
      {
        "currentVersion": "10.1.1", 
        "id": "JamfAdmin", 
        "lastModified": "2018-02-03T03:34:34Z", 
        "name": "Jamf Admin", 
        "publisher": "Jamf"
      }, 
      {
        "currentVersion": "10.1.1", 
        "id": "JamfRemote", 
        "lastModified": "2018-02-03T03:34:40Z", 
        "name": "Jamf Remote", 
        "publisher": "Jamf"
      }
    ]
  • /patch/TitleId
    This returns the full patch definition JSON of the software title. Here is an abbreviated example:

    {
      "id": "JamfAdmin",
      "name": "Jamf Admin",
      "publisher": "Jamf", 
      "appName": "Jamf Admin.app", 
      "bundleId": "com.jamfsoftware.JamfAdmin", 
      "currentVersion": "10.1.1", 
      "lastModified": "2018-02-03T03:34:34Z", 
      "extensionAttributes": [
        {"ExtensionAttributeObjects"}
      ],
      "patches": [
        {"PatchObjects"}
      ], 
      "requirements": [
        {"RequirementsObjects"}
      ]
    }

If you had a patch server located at http://patch.my.org, the full URLs would be:

At this time, there is no product that Jamf is providing for customers to install and have a ready to use patch server. The focus has been on opening up the framework that the official patch source uses and allow customers to extend their environments through a little engineering work.

Not all of us are engineers, of course. Thus…

Enter: Patch Server

gui_01_index.png

I wanted to have a working patch server ready for the Jamf community in time for 10.2’s release. My initial patch server implementation (I call it an implementation because it’s one way of providing a patch source) achieved serving the proper JSON data for each of the endpoints described above using a database (SQLite) for the backend.

While my original goals were much grander, including the ability to fully manage a patch definition in a GUI instead of writing out JSON, I had to pare it back in order to get the project into a deliverable state.

In the past week I went through the code and ripped out everything that I felt was not needed, or doable. Then, I went through and added in new features (ported from another project) and streamlined the UI elements that were left.

This patch server features:

  • All required Jamf Pro endpoints to serve as an External Patch Source
  • An API for programmatic management of patch definitions and versions.
    • Create/delete patch definitions.
    • Add versions to existing patch definitions.
    • Create backup archives of all patch definitions.
  • UI for management of patch definitions.
  • Validation of uploaded patch definitions.
    gui_05_validation.png
  • Full user documentation at http://patchserver.readthedocs.io/
    patchserver_docs.png

    • UI Overview
    • Setup Instructions
    • API Documentation

Bring the Requests

Until Jamf Pro 10.2 is released, I’m not going to tag the project at a 1.0 version. If you are in Jamf’s beta program and testing 10.2, I invite you to give this a try and let me know what you think. Specifically, I’m asking for you do open up issues on GitHub for:

  • Bugs you find
  • Features you want, such as:
    • Connect to an actual database like MySQL (?)
  • Documentation you want, such as:
    • Instructions for installing on X

Not everything that is reported might get worked on, but the good news is I released the patch server under the MIT license. If you have some Python chops you can fork it and do whatever you want with the codebase to suit your needs!

But, I don’t wanna setup a server…

If you had that reaction to the idea of setting up your own external patch source, ask yourself if you match any of these descriptions:

  1. My JSS can talk to pretty much anything if I want it to,
  2. I want a patch server; I don’t want to host a patch server,
  3. It doesn’t matter where my patches live as long as I can get and manage them,
  4. Can’t this be a cloud thing?

If so… stayed tuned for a future blog post.