Jamf The Gathering: A Slack Bot’s Story

JNUC happened recently. You might heard about it. Some stuff came up. There was a really awesome burn during the opening keynote.

It was a really nice burn.

Continue reading “Jamf The Gathering: A Slack Bot’s Story”

Open Distribution Server Technology (w/JNUC Recap)

ODST @JNUC

At JNUC 2017, I was given the opportunity to do a session detailing the progress I’ve made and the vision I have for a new file distribution server that can serve to replace the now discontinued JDS (Jamf Distribution Server).

This was a last minute addition to the conference schedule and we were unable to record it, but the Mac admin community took notes which can be found here. I’ve also uploaded the presentation’s slide deck on SlideShare.

The source code for ODST is available on GitHub. It is currently in an early Alpha state with some of the core functionality complete.

Project Goals

ODST came about with the sunsetting of the JDS. I set out to design my own implementation of an automated file distribution server but with additional features to make it a more powerful component of an administrator’s environment.

The goal of ODST is to provide an on-premise file syncing and distribution server solution that puts automation and integration features first.

The ODS (Open Distribution Server) application itself is modular and being designed to fit into as many deployment models as possible. This ranges from a simple single-server installation on Linux, Windows, or macOS to containerized deployments in Docker or Kubernetes.

While there will be initial support for the ODS to integrate with Jamf Pro it is not a requirement for using the application. This will allow administrators using other management tools to take advantage of the solution and submit feature requests for integrations with them as well.

Planned Features

  • A full web interface (built on top of the Admin API)
  • The Admin API for integrating your ODS instances with existing automations and workflows.
  • Many-to-many registration and syncing which will allow package uploads to any ODS and still replicate throughout your network.
  • Package and ODS staging tags to restrict how certain levels of packages replicate through the network.
  • Webhooks and email to send notifications to other services alerting them to events that are occurring on your ODS instances.
  • LDAP integration for better control and accountability when granting other administrators and techs access to your ODS instances.
  • And more to come…

Package Syncing

Where the JDS synced by running an every five minute loops task to poll another server, the ODS application uses a private ODS API for communicating between instances.

When two ODS instances are registered to each other they will have each others’ keys saved to their databases and use those keys to sign API requests.

The standard order of operations during a package upload would be:

  1. The admin uploads a package to ODS1.
  2. ODS1 generates the SHA1 hash of the package and also generates SHA1 hashes for every 1 megabyte chunk of that package. This information is saved to the database.
  3. ODS1 sends a notification to every registered ODS instance that a new package is available.
  4. ODS2 receives this notification and makes a return API request for the full details of the package.
  5. ODS2 saves the pending package to the database and a download task is sent to the queue.
  6. The ODS2 worker takes the download task off the queue and begins downloading the package in 1 megabyte chunks, comparing hashes for every chunk, and saving them to a temporary location.
  7. Once the ODS2 worker has downloaded all chunks it recombines them to the single file, performs a final SHA1 check, and moves the package to the public download directory.
  8. ODS2 then performs step #3 to propagate the package to other ODS instances it is registered with.

If the download process seems familiar, it is borrowed from how Apple performs MDM initiated application installs.

Application Architecture

The ODS application is more complex than the JDS in order to facilitate the additional features that are being built on top of the file syncing. In addition to the application server, a production deployment would also include a front-end web server (Nginx or Apache), a Redis server for the queuing system, a database server (ODST falls back to a local SQLite database file if there is not a database service to connect to), and workers that process queued actions.

Single Server

ODS_Single_Server.png

Multi-Server or Containerized

ODS_Multi_or_Containerized.png

The queuing system is an important element as it backgrounds many of the processes that the server will need to perform in reaction to notifications or requests (such as queuing notifications, API requests to other ODS instances, file downloads, and file hashing operations). This frees up the application to continue accepting requests by removes long process blocks.

How the Community Can Help

When I gave the JNUC presentation I only took up half of the allotted time to discuss what was completed with the project and what was planned. The second half was spent in open discussion to take in feedback and guidance from the target audience on what was needed on the road to a 1.0 release.

Adding LDAP support was the first item to come out of this and is my next planned feature to write in after the file syncing framework is finished. I encouraged participants to open GitHub issues on the repo as we discussed their questions and asks. I want to continue to encourage this. The ODST project is meant for the community and should continue to be community driven in its roadmap.

When it comes to contributing to the project I am not asking for code help at this time. Don’t feel that you need to know Python or web development with Flask in order to contribute. There are many other areas that I am in need of help:

  • Testing! As I make new commits to the repository and add in more features you can help ensure everything is working by running the latest version and trying them out. Submit issues, provide logs, provide details on how you’re deploying the application (the provided Docker Compose file is the quickest and easiest way), and by doing so you will help verify features work as expected and solidify the quality of the application.
  • Determine optimal configurations. There are quite a few components to the ODS application and I am learning as I go for how to configure the web server. More experienced administrators who are familiar with these technologies, especially in production environments, can help work towards a baseline for…
  • Installers! The ODS application can be custom setup for almost any kind of deployment, but we still want an easy option where an admin can grab an installer for load it onto a single Linux or Windows server. If you have experience building installers on those platforms please reach out! I’ve also mentioned containerization a few times, and having an official Docker images for the ODS application and worker components should be a part of this initiative.
  • Documentation. Much Documentation. There will be official docs available at odst.readthedocs.io which will be generated from the main repository on GitHub. You can help maintain and improve that documentation with pull requests as you find errors or inaccurate instructions/details as the project iterates. The documentation will be especially invaluable when it comes to the aforementioned installers, custom installations, and the administrator user guide portion that will walk user through how to perform actions.

If you haven’t yet, please join the #odst channel in the Mac Admins Slack where you can discuss the project with me directly as well as other admins who are using, testing, and contributing as they can.

I hope to build something that will provide great value to our community and fill the gap the JDS left in a lot of environments. I hope to see you on GitHub and Slack soon!

Open Distribution Server and JNUC 2017

Two posts in one day! I wanted to do a quick JNUC update and promote a session that I’m really excited for.

This year, as with years past, I will be pretty involved with the conference. Aside from finding me roaming the halls of the Hyatt; I am on the committee for the first ever JNUC Hachathon, participating in the API Scripting and Webhooks labs, and delivering the Webhooks Part Deux! presentation with Oliver Lindsey from our Pro Services team.

But the session I am most excited about is a very late addition that was put onto the JNUC App’s schedule this morning.

The Open Distribution Server

Around July (Penn State), I began work on an alternative distribution server to the JDS. As the community recently learned, the JDS has been discontinued and will no longer be supported by Jamf as cloud-centric options are being focused on. Prior to that announcement, I was involved in some talks with Product Management at Jamf about the JDS, and I took the opportunity to show them what I was working on.

Joe Bloom, our Jamf Pro Product Manager who you will hear talk at several product sessions this year, was very excited about this and urged me to continue working on my distribution server and release it as a free, open source solution.

Joe has secured an additional session slot on Tuesday at 4:00 PM dedicated to the Open Distribution Server. You can find it at the link or in the JNUC App (it is not listed on the website).

During this session I’m going to talk about the goals of this project, what it aims to solve, what features I have implemented and plan to implement, but then turn the rest of the time over to you so we can talk about the key things that will make this a successful solution:

  • What features don’t work as described or need changed to fit your workflows?
  • What features are missing that you need?
  • How can the community contribute to this project?

The current code base for this project was posted to GitHub a couple weeks ago:

https://github.com/brysontyrrell/ODST/tree/develop

The Open Distribution Server (ODS) is an open-source package distribution and syncing solution for IT administrators to serve as a potential alternative for the Jamf Distribution Server.

For those looking for an on-premise, automated distribution point solution, and those who are in need of a replacement for their JDS infrastructure, please attend and be a part of the discussion.

I hope to see you there!