Open Distribution Server Technology (w/JNUC Recap)

ODST @JNUC

At JNUC 2017, I was given the opportunity to do a session detailing the progress I’ve made and the vision I have for a new file distribution server that can serve to replace the now discontinued JDS (Jamf Distribution Server).

This was a last minute addition to the conference schedule and we were unable to record it, but the Mac admin community took notes which can be found here. I’ve also uploaded the presentation’s slide deck on SlideShare.

The source code for ODST is available on GitHub. It is currently in an early Alpha state with some of the core functionality complete.

Project Goals

ODST came about with the sunsetting of the JDS. I set out to design my own implementation of an automated file distribution server but with additional features to make it a more powerful component of an administrator’s environment.

The goal of ODST is to provide an on-premise file syncing and distribution server solution that puts automation and integration features first.

The ODS (Open Distribution Server) application itself is modular and being designed to fit into as many deployment models as possible. This ranges from a simple single-server installation on Linux, Windows, or macOS to containerized deployments in Docker or Kubernetes.

While there will be initial support for the ODS to integrate with Jamf Pro it is not a requirement for using the application. This will allow administrators using other management tools to take advantage of the solution and submit feature requests for integrations with them as well.

Planned Features

• A full web interface (built on top of the Admin API)
• The Admin API for integrating your ODS instances with existing automations and workflows.
• Many-to-many registration and syncing which will allow package uploads to any ODS and still replicate throughout your network.
• Package and ODS staging tags to restrict how certain levels of packages replicate through the network.
• Webhooks and email to send notifications to other services alerting them to events that are occurring on your ODS instances.
• LDAP integration for better control and accountability when granting other administrators and techs access to your ODS instances.
• And more to come…

Package Syncing

Where the JDS synced by running an every five minute loops task to poll another server, the ODS application uses a private ODS API for communicating between instances.

When two ODS instances are registered to each other they will have each others’ keys saved to their databases and use those keys to sign API requests.

The standard order of operations during a package upload would be:

1. The admin uploads a package to ODS1.
2. ODS1 generates the SHA1 hash of the package and also generates SHA1 hashes for every 1 megabyte chunk of that package. This information is saved to the database.
3. ODS1 sends a notification to every registered ODS instance that a new package is available.
4. ODS2 receives this notification and makes a return API request for the full details of the package.
5. ODS2 saves the pending package to the database and a download task is sent to the queue.
6. The ODS2 worker takes the download task off the queue and begins downloading the package in 1 megabyte chunks, comparing hashes for every chunk, and saving them to a temporary location.
7. Once the ODS2 worker has downloaded all chunks it recombines them to the single file, performs a final SHA1 check, and moves the package to the public download directory.
8. ODS2 then performs step #3 to propagate the package to other ODS instances it is registered with.

If the download process seems familiar, it is borrowed from how Apple performs MDM initiated application installs.

Application Architecture

The ODS application is more complex than the JDS in order to facilitate the additional features that are being built on top of the file syncing. In addition to the application server, a production deployment would also include a front-end web server (Nginx or Apache), a Redis server for the queuing system, a database server (ODST falls back to a local SQLite database file if there is not a database service to connect to), and workers that process queued actions.

Single Server

Multi-Server or Containerized

The queuing system is an important element as it backgrounds many of the processes that the server will need to perform in reaction to notifications or requests (such as queuing notifications, API requests to other ODS instances, file downloads, and file hashing operations). This frees up the application to continue accepting requests by removes long process blocks.

How the Community Can Help

When I gave the JNUC presentation I only took up half of the allotted time to discuss what was completed with the project and what was planned. The second half was spent in open discussion to take in feedback and guidance from the target audience on what was needed on the road to a 1.0 release.

Adding LDAP support was the first item to come out of this and is my next planned feature to write in after the file syncing framework is finished. I encouraged participants to open GitHub issues on the repo as we discussed their questions and asks. I want to continue to encourage this. The ODST project is meant for the community and should continue to be community driven in its roadmap.

When it comes to contributing to the project I am not asking for code help at this time. Don’t feel that you need to know Python or web development with Flask in order to contribute. There are many other areas that I am in need of help:

• Testing! As I make new commits to the repository and add in more features you can help ensure everything is working by running the latest version and trying them out. Submit issues, provide logs, provide details on how you’re deploying the application (the provided Docker Compose file is the quickest and easiest way), and by doing so you will help verify features work as expected and solidify the quality of the application.
• Determine optimal configurations. There are quite a few components to the ODS application and I am learning as I go for how to configure the web server. More experienced administrators who are familiar with these technologies, especially in production environments, can help work towards a baseline for…
• Installers! The ODS application can be custom setup for almost any kind of deployment, but we still want an easy option where an admin can grab an installer for load it onto a single Linux or Windows server. If you have experience building installers on those platforms please reach out! I’ve also mentioned containerization a few times, and having an official Docker images for the ODS application and worker components should be a part of this initiative.
• Documentation. Much Documentation. There will be official docs available at odst.readthedocs.io which will be generated from the main repository on GitHub. You can help maintain and improve that documentation with pull requests as you find errors or inaccurate instructions/details as the project iterates. The documentation will be especially invaluable when it comes to the aforementioned installers, custom installations, and the administrator user guide portion that will walk user through how to perform actions.

If you haven’t yet, please join the #odst channel in the Mac Admins Slack where you can discuss the project with me directly as well as other admins who are using, testing, and contributing as they can.

I hope to build something that will provide great value to our community and fill the gap the JDS left in a lot of environments. I hope to see you on GitHub and Slack soon!