JNUC happened recently. You might heard about it. Some stuff came up. There was a really awesome burn during the opening keynote.
It was a really nice burn.
But other stuff happened to! Joel had a really awesome demo NoJamf Connect ( 🤔 ), everyone got to learn about Jamf Heroes, and we had another Hackathon! Kinda. Really, Joel and I borrowed a page from what we did at Penn State MacAdmins this year and did two nights of introduction and education around the two languages we love most:
Python and Python! …ok, maybe Swift too.
PSA: We’re going to have a lot more to talk about concerning the 2018 JNUC Hackathon in the near future. That’s a teaser. Stay tuned; keep an ear open.
No projects this year, but that doesn’t mean ideas were not abundant. Amongst the attendee swag were first-time-ever-edition Jamf trading card packs! Everyone got two and you had to find others to trade with and collect the full set of 18 cards!
Some of the cards were of Jamfs. If you found us (yes, I was one), we may have even signed them…
The social trading aspect of the cards actually took off really well, but being that this is a conference full of people who like to be techy (techie? tech-ay? teché?), someone did casually ask this in the #jnuc channel of the MacAdmins Slack:
Those who know me know not to say things like this around me. Despite this year’s Hackathon not being about projects, I immediately began to design and code a Slack bot that would achieve this goal.
Whiteboard access is not to be underestimated!
Like everything else I build nowadays, Jamf The Gathering would be a serverless architecture. A couple of API Gateway endpoints and Lambda functions to facilitate the OAuth2/install process for adding a bot to a Slack team, and one to receive message events at and process them asynchronously.
Go get yourself the code!
Let’s dive into it!
There are only three endpoints that make up this rather simple bot:
/slack/install /slack/oauth/redirect /slack/events
The /slack/install endpoint is a simple redirect. It exists so, were this bot to be posted on the Slack App Directory (which it won’t be), I could provide one-click installation on the product page. Going here will send you to:
Which will in turn render a webpage for your currently active Slack Team prompting you to allow the chat bot access!
Clicking the “Authorize” button will perform a redirect back to /slack/oauth/redirect containing a code in the URL that will be used for the bot to obtain access credentials that will be stored in the database for that Slack Team.
And that’s it! In a more polished product I would then redirect the installing user to another page to display a success message, but I was going for fast and janky and there was no time for such niceties.
Making the Magic Happen (swidt?)
Once Jamf The Gathering is installed the helpful bot is available to interact with via direct messages or within any channel in the Team. The commands were all very basic, but it got the job done!
Any direct message or mention to the bot will trigger a webhook from Slack to the /slack/events endpoint. Now, if you paid attention to my whiteboard photo above you might have noticed the Lambda function that accepts the event is pushing it off to an SNS topic (a notification service in AWS). What’s happening there is that the receiver is not doing any of the actual processing.
Why would I do that? Check out this text from Slack’s Events API documentation:
We consider any of these scenarios a single failure condition:
- we are unable to negotiate or validate your server’s SSL certificate
- we wait longer than 3 seconds to receive a valid response from your server
- we encounter more than 2 HTTP redirects to follow
- we receive any other response than a HTTP 200-series response (besides allowed redirects mentioned above)
While we limit the number of failure conditions we’ll tolerate over time, we also gracefully retry sending your events according to an exponential backoff strategy.
Maintain a successful response rate of 5% or above to avoid automatic event delivery disabling. Apps receiving less than 1,000 events per hour will not be automatically disabled.
Oh-HO! If I don’t respond within 3 seconds WITH a 200 status the event is considered not successful, and Slack will cease sending events completely if I don’t maintain that!
Granted, 5% is super generous of them, but let’s not be completely janky here. Let’s be performant. Let’s go a-sync! My event endpoint can fire off the message to an asynchronous processor and immediately return a 200 error.
The SNS-to-Lambda design pattern is perfect for this. Why not have the receiving Lambda be the one to start the processing directly, you ask? There are clear benefits. First, posting a message to an SNS topic is going to be faster than invoking another function. Second, I don’t want to wait for that function to complete, so I would be performing a single call and assuming it went. SNS will attempt to execute a Lambda function three times before giving up.
Event latency problem: solved.
How the Sausage Gets Made
Now all the heavy lifting is being done by our back-end User Events Lambda on the other side of that SNS topic. This function performs a lot of interaction with the database and handles sending nicely formatted messages back into Slack to either the channel or user that initiated the event. The order of operations is:
- Get triggered (heh).
- Initiate a unique database session at the start.
- Load the initiating Slack user from the database, or create them if they aren’t there.
This is a just-in-time (JIT) record creation. The user’s team ID is looked up, and then a lookup is performed for the user. If a user of the given user ID doesn’t exist for that Team a new record is immediately created and returned.
- The command that was sent in the message is processed using some Regex-fu.
The available commands are:
- I have … (report the card numbers in your possession to trade away)
- I need … (report the card numbers you need)
- I traded … for … (tell the bot if you performed a trade)
- Show trades (find users who match up to trade with)
- Show mine (show what you reported to the bot)
- Based on the command, some database queries are made and response message text is generated. If the show trades command was requested the matched users will be included in the message text mention links!
- Database session is closed (let’s be clean).
- The response message is sent to Slack! If the response is going into a channel, and not a direct message, a mention of the initiating Slack user will be included.
All of that happens fairly quickly with the exception of the Lambda cold start. Lambda functions are not always ready to go: their containers are not kept running continuously. If your Lambda isn’t called for a set amount of time, that container is torn down until such time as the function is invoked (and that happens anyway, but that’s a different subject).
This was the first time I had written a Lambda with enough significant resources that the cold start time caused noticeable, painful delays one first invocation. This is one particular function where I would, in a less janky build, have a Lambda warmer deployed with the project.
A what? A Lambda warmer is, in fact, a Lambda function running on a cron-like schedule that invokes other Lambda functions with dummy events to make sure there is always an active container so your changes of hitting a cold start delay are significantly lower. Sound dumb? Well, maybe it is, but it’s a thing, and this is the first time I would have wanted to deploy one.
In fact, I realize that I got lucky with my OAuth Redirect Lambda as it too used all the same database resources and would suffer similar cold start delays. I only installed the bot into two Teams so I never realized. 😅
Not too many people in the #jnuc channel had a chance to use this, sadly. I had about ~20 user records populated by the end. The idea was presented Monday night, but the bot was not finished and deployed until Wednesday lunchtime. While 36-ish hours for a fully functioning chat bot isn’t bad, it was too late to get a lot of engagement for a three day conference.
So, What was the point?
Then why did I go through all of that? Let me walk you through some of the meaning behind my madness.
Firstly: I was presented with challenge, a task, a quandary, that I could apply to something I’ve been wanting to do anyway: Slack bot integrations. You will always learn more and learn faster when you are working on a project instead of just poking around examples.
Secondly: I piloted new tech that I haven’t done before. Fun thing about every single project that I start – both for the Mac Admin community in open source, and also for internal services I develop for Jamf – I introduce something new that I haven’t used before (maybe don’t let Jamf know that…).
What was the tech here? Autora Serverless. The MySQL database backend for this chat bot uses the new serverless database offering from AWS. Plus, let’s add on to this that I had not yet written Lambda that interacted with SQL backends and had to learn a bit about how I wanted to manage database sessions in such an environment (and likely have much, much more to learn to be optimal).
Thirstily: I didn’t write a trading card bot.
This wasn’t a trading card bot.
It was never a trading card bot.
You only thought it was a trading card bot.
I wrote the new prototype for Jackalope.
You can all look forward to that real soon. 😉