Trial by Fire

JAMF's NetSUS Appliance - Netboot in a Box

Today, JAMF released a new appliance VM based on Ubuntu 10.04 LTS (Lucid) that, for once, provides an ‘out of the box’ implementation of Netboot and Software Update Service WITHOUT requiring OS X hardware (based on Reposado and other open-source technologies.)

For those people who are relatively new to Linux, I thought I’d provide an easy walkthrough of how to get started using this VM on your Laptop. I’m going to demo this using VMware Fusion instead of VirtualBox mainly because I prefer how Fusion handles Networking. I’ve used many VMs, and Fusion always tends to be much more solid and stable for me. If you don’t have Fusion or don’t WANT to use Fusion, feel free to use VirtualBox.

Download and Convert the OVA File

The appliance can be downloaded directly from JAMF and comes as a VirtualBox OVA file. Rich Trouton has provided a great walkthrough on converting the OVA file to a VMX file, so I’ll link you to that post (since he did it so well).

Start up the VM

Once you have your VMX file, you can open it up with Fusion and import it into your Virtual Machine Library. There is one change we’ll want to make, so click on the VM and open its Settings panel (this is different from Fusion 3.0 and 4.0 – I’ll be describing the 4.0 method). In the Settings panel, you’ll want to select the Network Adapter icon and make sure the radio button next to “Share the Mac’s network connection (NAT)” is selected. This will create a local IP address so we can contact it directly from our Laptop. Once you’ve done that, start the VM and get to the login screen (you’ll need to click the Ok button through the message about logging into the NetSUS instance). Feel free to login with the username ‘shelluser’ and the password ‘shelluser’. You should see a Message of the Day splash with one of the sections being “IP address for eth0:”. Note the IP address, we’ll need it in the next section (if you don’t see this, just run ifconfig from the command line and look for the IP address for the eth0 adapter – it should be in the format of 192.168.x.x).

Prepare to Break Free!

It really sucks to work with VMs from WITHIN the VM window for various reasons (lack of Copy/Paste, loss of mouse, etc…), so we’re going to setup a host entry on your local laptop so we can SSH into the VM as we need. I’m going to give my VM a hostname of ‘netsus.puppetlabs.vm’ but you can name it whatever you want. Let’s first edit the /etc/hosts file on your laptop (NOT within the VM, this is ON your laptop) by running the following from the command line:

sudo vim /etc/hosts

Feel free to use pico/nano to edit /etc/hosts in case you don’t have any experience with vim. We want to add the following line in the /etc/hosts file:

192.168.217.154 netsus.puppetlabs.vm

NOTE SUBSTITUTE 192.168.217.154 WITH THE IP ADDRESS YOU GOT FROM THE PREVIOUS STEP! That’s the IP address that was assigned on MY laptop, and it will definitely be different from the IP address you got on YOUR computer. Also, you don’t HAVE to use ‘netsus.puppetlabs.vm’ – you can assign it any name you want. Save and close /etc/hosts and then let’s test connectivity from the command line by doing a ping:

ping 192.168.217.154

As long as we have connection, we’re fine (remember, substitute your IP Address for mine). If you DON’T have connectivity, make sure your VM is running and review your /etc/hosts file to make sure the changes were saved (also, check the IP address on your VM again).

Snapshot Time

The beauty of Fusion is that you can take snapshots and revert to them should you mess things up. Let’s take a snapshot now in case we screw things up in the following steps. I’m using VMware Fusion 4.0, so I take a snapshot by clicking on the VM window, clicking on the Virtual Machine menu at the top, and then down to Snapshots. From there, you can click the ‘Take Snapshot’ button and let it do its thing. When it’s done, hit the close button and return to the VM

Enable SSH and Login

By default, the VM doesn’t have the SSHD process running, so we can’t yet SSH in from our laptop. Enable this by doing the following from within the VM:

sudo apt-get install openssh-server

Hit “Y” when it asks if it should download and install. When it completes, switch back to your LAPTOP (i.e. NOT inside the VM), open up Terminal, and do the following:

ssh shelluser@netsus.puppetlabs.vm

Type yes to add the SSH key to your known_hosts file and use the password ‘shelluser’ to login. Tada! Now you can work directly from Terminal and enable copy/pasting directly from your Terminal window. You can always type exit to close the ssh connection when you’re done.

Install Puppet

So, it wouldn’t be one of my write-ups if I DIDN’T get Puppet installed. Actually, if you’ve only managed Macs, then you might not have seen the benefit of Puppet before. Now that we’re on Linux, you’ll definitely see the benefit of Puppet (especially if you’re not familiar with the service/package manipulation commands). Puppet can help abstract that for you, so we’re going to enable it.

The version of Puppet that’s in the main package repository is quite old (0.25 as opposed to the current 2.7.10 version we support). Let’s add Puppet Labs' Apt Repository to the list of repositories queried so we can get a recent version of Puppet. Do that by opening up the /etc/apt/sources.list file for editing with the following:

sudo vim /etc/apt/sources.list

If it prompts you for your password, go ahead and enter it. Also, if you’re not familiar with vi or vim, I’ll try and help you with the basics. Arrow down to the line JUST above the line that begins with “deb http://” and press the i button on your keyboard to insert the following lines:

deb http://apt.puppetlabs.com/ubuntu lucid main
deb-src http://apt.puppetlabs.com/ubuntu lucid main

When you’re done, hit the escape key and press ZZ (hold shift and hit Z twice, you want two capital-Z’s) to save and close the file. This will add Puppet Labs' repository to the list of repos queried.

Next, we’ll need to add Puppet Labs' GPG key to the system so we can validate the packages that are downloaded. From the terminal, execute the following:

gpg --recv-key --keyserver keys.gnupg.net 4BD6EC30

Great, we should have the key, now we want to load it. Do that with this command:

gpg -a --export 4BD6EC30 | sudo apt-key add -

Finally, let’s update apt so that it knows about the repository. Do that with this command:

sudo apt-get update

When it finishes running, you can finally install Puppet with the following command:

sudo apt-get install puppet

Type “Y” when it asks you if it should download and install all the dependencies. That’s it! Puppet should be installed! Run the following to see what version we’ve installed:

puppet --version

As of this writing, 2.7.10 is the current version. As long as you don’t have version 0.25 you should be good!

Snapshots away

It would be wise to take another snapshot at this point. Don’t worry, I’ll wait.

Inspect the System

Puppet lets us see what’s been installed on the VM. Take a look at all the packages on the system by doing the following:

sudo puppet resource package

Type your password when prompted and Puppet will return you with a list of all packages on the system and the versions that are running. Similarly, you can see what services are running with the following:

sudo puppet resource service

Notice that sshd is running and other services like netatalk (that enables AFP on Linux) should also be running. Sweet. We’ll come back to Puppet later.

Login to the Web GUI

If your VM is running, you should be able to open a web browser on your Laptop and navigate to the following address:

https://netsus.puppetlabs.vm

Note the httpS as we’ll need to use SSL to get into the Web GUI. When it asks if you want to accept the self-signed cert, make sure to choose Yes. You can login with the user ‘webadmin’ and the password ‘webadmin’

Once you’re logged in, feel free to read through JAMF’s instructions for enabling their services. They do a much better job than I about walking you through that.

Access to the Raw Files

Back in the Terminal window, you can access all the PHP/Shell magic that comprises the Web GUI in the /var/www/webadmin directory. You can poke around through all the files (and use sudo to open some of the key bits) at your leisure.

Note that /etc/dhcpd.conf is the main configuration file for the DHCPD service and it’s got some special bits to accommodate Apple Hardware.

Also, /var/appliance has some magic (and the /var/appliance/configurefornetboot script has some sed statements for fixing a stock dhcpd.conf file and enabling those Apple Hardware bits).

Most of the NetBoot and SUS items are located in /srv, so you could redirect that path to somewhere OFF the VM if you like (or symlink to your heart’s content). NFS mounts are your friend (and beyond the scope of this writeup).

Reposado’s source is located in /var/lib/reposado

Locate ALL the Things!

Linux has a command called ‘locate’ which will allow you to search for things on-disk pretty quickly. If you’ve used mdfind from the command line on OS X, it’s reasonably similar. To enable it you need to do the following

sudo touch /var/lib/mlocate/mlocate.db
sudo updatedb

Great, now any time you want to search for something (like where the Reposado source is located) you can do something like this:

locate reposado

Just substitute anything for ‘reposado’ and it should help locate it for you on disk. It’s great if you’re new to Linux and forget the path to certain configuration files.

Accomodate Deploystudio Booter Images

Rich Trouton describes how to do this WITHOUT using vi or vim (and even includes pictures) so check out the link for his method. To use vi or vim, proceed with the next couple of paragraphs ;)

As @bruienne reported today, it looks like you need to make some changes to /etc/dhcpd.conf to accomodate for the .sparseimage that Deploy Studio creates for its booter images. Fortunately, there’s an adminHelper.sh script that will allow us access to these changes. Let’s open that file for editing by doing the following:

sudo vim /var/www/webadmin/scripts/adminHelper.sh

Once you’re in vim, you can go to the line we need to change by typing :98 and hitting enter. This will jump us to line 98 that begins like ‘dmgfile=’. Change the line to look like this (hit the i button to enter insert mode and make the following changes)

dmgfile=`echo $files | egrep dmg\|sparseimage`

When you’re done, hit the escape key on your keyboard, type :wq! and then hit the return button on your keyboard. That should save the file and return you to the command line.

From here, you can use Deploystudio to generate a Netboot booter image and upload it through the GUI. The changes you made to the adminHelper.sh script should get propagated to /etc/dhcpd.conf when you use the GUI to setup a Netboot image. (remember, this is a work in progress so further changes may be necessary. Thanks to Pepijn Bruienne for the help!).

More Later?

That’s it for now – I just wanted to get something online to show people how to setup and poke around in the VM. Later I can show you how to manipulate and manage the VM with Puppet, which is pretty fun. Your VM should get you started and let you play around locally, but if you want to test out Netboot and the like then you’ll need an externally-accessible IP address (as apposed to the 192.168.x.x IP, which is only accessible from your Laptop). You can change the settings in VMware Fusion to enable that (like we did in step 2).

Hope this helped you out!

Using Git, for Mac Sysadmins - Part 1

As a former Mac Sysadmin, I frequently felt like I had one toe in the world in which Linux Sysadmins frequently found themselves, but was surrounded by a culture of GUI-Clickery that either feared, scorned, or flat-out avoided tools that were command-line based (or, as many will point out, used the GUI (Graphic User Interface) tools because ‘they worked’ and that was all they needed). This sucked because Linux Sysadmins have some tools that, as a current colleague would say, ‘are kind-of awesome.’

One of those tools is Git, which is a Distributed Version Control System (or, DVCS). Git, in a TOTAL nutshell, is a tool that allows you to create multiple ‘what-if scenarios’ with very little cost and the ability to save yourself from…well, yourself.

Yes – You’re Going to Use the Command Line

Like I mentioned before, the GUI is ingrained in the culture of the Mac. It wasn’t until OS X that Mac Sysadmins finally had a proper command-line (which I will refer to as the CLI, which is short for Command Line Interface. If you’re not familiar with that terminology, I’m speaking about the interface you get when you run /Applications/Utilities/Terminal.app), so we’re a bit ‘new’ to the whole idea. I can help you if you fear or avoid the CLI; there’s always room for learning and I’ll try to hold your hand as much as possible. For those that downright refuse to use the CLI…well, I’m sorry?

What IS Git?

Git is relatively new in the DVCS world (being initially released in 2005). Developers have been using DVCS for MANY years with tools like CVS, Subversion, Git, Mercurial, and others. For a team of developers who ALL need to make contributions to a SINGLE project or code base, it’s vital that they have a way to modify a file without breaking something that someone else is doing. That’s what DVCS tools do – allow you to take something as large as a code base like OS X 10.7 (or something as small as a directory with a couple of files) and make changes without affecting everyone else who’s working on the SAME code base (or files in the directory).

So why am I choosing to talk about Git instead of, say, Subversion? That’s easy: lightweight branching and merging. Git allows you to easily create a ‘topic branch’ (or, using the metaphor above, a ‘what-if scenario’), make changes, and see if your changes are favorable or if they downright stink. If your changes are desirable, you can then ‘merge’ your topic branch back in with your master branch (or, the original state the files were in when you created the topic branch in the first place). If the changes don’t work well, you can just delete the topic branch and ‘no-harm, no-foul’; your original files remain unchanged and in the pristine condition that they existed when you first created the topic branch. True, Subversion could do the same thing, but Git allows MANY people to do the same thing and then EASILY merge in EVERYONE’S changes without having to fight Subversion (just trust me on this if you’ve never used Subversion).

And why am I choosing to talk about Git instead of Mercurial? Easy – Git is what I learned :) As I understand it Mercurial functions similarly, and I truly don’t have the experience with Mercurial to tell you the differences. I’m talking about Git because Git is what I use and Git is what my company uses.

So what does this have to do with me as a Mac Sysadmin?

Good question. Mac Sysadmins are more frequently finding themselves having to use the CLI to tune their clients and servers. Also, many Mac Sysadmins are finding that certain CLI tools are working better for their teams (in terms of simplicity, responsibility, and visibility). As they dip their toes into this CLI world, they’re finding that they want the same features they had in the GUI: the ability to do a ‘save-as’, to create ‘versions’, and to have the ability to have a rollback function similar to Time Machine. Git gives you this through the CLI (or, yes, even through the GUI as I will mention later in the article).

If you’re ready to take that step then follow me…

A note on Git and Github

You might have heard of Github. Github is a free (they have a free tier and plenty of subscription tiers of service) service that allows you to ‘push’ your git repositories up to them for storage in ‘the cloud’. Github is basically centralized-storage for your git repositories and is NOT NECESSARY for you to use git as a tool. In reality, you CAN use Github or setup something like Gitolite on a local server in your environment to get most of the benefits of Github without having to push your repositories to Github’s servers.

The bottom line: Github and git (the tool) are two different things. Github didn’t create git, and git doesn’t need Github to work properly. Keep that in mind as you read through.

Installing Git

Git can be installed in a variety of ways. First and foremost, you can download the latest package from Git’s Googlecode Page. You can also choose to install git through Macports or Homebrew as long as you have them installed (both of which require the Developer Tools…but so does The Luggage, the tool we’ll be using later). Simply download the package from Git’s Googlecode Page, install it, and you should be good to go!

If you’re using Macports, make sure the Developer Tools are installed, Macports is installed, and then do the following from the command line (PLEASE NOTE: The dollar sign is a prompt – you SHOULDN’T type it. Lines BEGINNING with a dollar sign, or a prompt, should be typed by you. Lines that DON’T begin with a prompt represent the output you receive from typing the command. If you don’t see a line that DOESN’T begin with a prompt, then I’m not listing the output. Got it? Good!):

$ sudo port install git-core

If you’re using Homebrew, make sure the Developer Tools are installed, Homebrew is installed, and then do the following:

$ sudo brew install git

To check to see where and what version of git is installed, execute the following commands :

$ which git
$ git --version

You should receive the path to where git is installed after you run the first command, and you should receive the version of git that’s currently installed on your system. Any version of Git greater-than or equal to 1.7 is probably ideal, though most of the commands we’ll be using will probably work just fine with previous versions.

Your first Git repo

I find it’s easier to understand Git if I describe an actual scenario instead of talking in terms of ‘git can do…’. I’m going to talk about using git with The Luggage. I’ve written about The Luggage in several articles on this site, so feel free to familiarize yourself with it if you haven’t used the tool before. As a quick review, The Luggage is a tool that lets you create plain text Makefiles that can be used to build packages to install files, deploy scripts, and basically distribute ‘things’ to the users in your organization. Auditing HOW a package is made can be incredibly difficult for a team of sysadmins, but The Luggage helps you with this by using plain text Makefiles. Git will give you the ability to track WHO made changes, WHEN they were made, and ROLLBACK the changes if you notice something undesirable happening. Git and The Luggage are a perfect combo if you’re a sysadmin.

Incidentally, The Luggage exists as a Github repository, so we have to CLONE it (or download it) from Github first before we can start playing with it. I recommend creating a directory in your home directory (/Users/-username-) and cloning your repositories there – that way you retain ownership of the files and can work on them at will. I put all my git repositories in ~/src, but if you’re using something like Mobile Home Directories or even Network Home Directories, then you’ll want to make sure that you’re not syncing these directories back to some central location and burning up your quota in some fashion. Let’s create our repositories directory and clone The Luggage source code with these steps:

$ mkdir -p ~/src
$ cd ~/src
$ git clone git://github.com/unixorn/luggage.git

Doing this will create the ~/src directory where we’re putting all our repositories and download The Luggage source code to ~/src/luggage. As you might be able to infer, git clone will clone an existing git repository from its location on another server (we call that the remote location, or just remote for short) down to a local directory. This is usually how many sysadmins first encounter git – by needing to download something from Github or a remote server. Let’s look and see what git knows about this repo by doing the following:

$ git remote -v
origin  git://github.com/unixorn/luggage.git (fetch)
    origin  git://github.com/unixorn/luggage.git (push)

This command will list all the remote repositories that we’re tracking. By default, you will get a remote by the name of ‘origin’ that lists the location where you originally cloned the repository (in our case, this is Github).

Why is it listed twice? Well, with git, you can pull down, or fetch, changes from a remote repository as well as push up changes from your LOCAL repository to the remote repository. Frequently these paths will be the same, but in our case these links are to the READ-ONLY version of this repository, so trying to push changes will always fail because we don’t have access. We’ll talk later about how you can push up changes, but for now let’s create our OWN repository instead of working with someone ELSE’S.

Creating your OWN git repo

The Luggage has a directory called Examples, but it’s absent of any actual examples. We want to create our own directory where we can store our Luggage Examples. For the purposes of this demo, I’m going to create a directory called luggage_examples that will contain our…well… Luggage examples:

$ cd ~/src
$ mkdir luggage_examples

Now that we have a directory, we need to make sure that git is tracking our changes so that we can create those ‘what if’ scenarios and save our work.

$ cd ~/src/luggage_examples
$ git init
Initialized empty Git repository in /Users/gary/src/luggage_examples/.git/

Git created a new empty repository at ~/src/luggage_examples and created a .git directory inside luggage_examples. This .git directory is where git stores all its dynamic data that’s pertinent to your repository, so MAKE SURE not to delete it or modify any files in there (without knowing what you’re doing).

That’s it! We’ve created a repository! Now let’s actually see how git works locally on your machine.

Working with Files

As I mentioned before, you don’t NEED Github to use git. Git functions as a mechanism to ‘version’ your files so you can rollback to a previous version or create our ‘what-if’ scenarios and then accept (merge) or deny (destroy) them depending on our needs. You can easily create a git repository on your local computer ONLY, but of course you lose the ability to restore your work if your hard drive malfunctions. This is what Github gives us, and we’ll look at that functionality later.

The example I’m going to use is based on the previous blog post I wrote on Using the Google Mac Sysops Crankd Code Let’s create a directory for this example and create a starter Makefile:

$ mkdir -p ~/src/luggage_examples/crankd_google
$ cd ~/src/luggage_examples/crankd_google
$ vim Makefile

I use vi or vim to edit files from the Terminal. Feel free to use any text editor you like (nano, emacs, Textmate, TextWrangler, etc…), but CREATE a file called ‘Makefile’ (note the capital M) and put it in ~/src/luggage_examples/crankd_google. Here’s what I’ll put into this file:

# Title:       Crankd-Google Example
# Author:      Gary Larizza
# Description: Something so Marnin doesn't kill me w/ postinstalls :)

include /Users/gary/src/luggage/luggage.make

TITLE=Crankd-Google
REVERSE_DOMAIN=com.googlecode
PAYLOAD=pack-crankd

Great, so we have our skeleton of a file! The file is saved in our folder, but we haven’t actually told git to track the file (and so git is AWARE that the file EXISTS, but it isn’t tracking its changes…yet). This is a good time to explain the three states in which files can exist in a git repository.

The Three Stages of Files: Working Tree, Index, and Repository

Git has three ‘locations’ by which a file can exist (even though it technically exists, in our case, in ~/src/luggage_examples/crankd_google). These three locations are called the working tree, the index, and the repository

The Working Tree

When you put a file in a directory that’s being managed by git, but you haven’t told git to track the files changes, then this file is said to be in the working tree. Our file we just created, Makefile, exists in the working tree currently. Git knows that the file EXISTS, but it hasn’t saved the file’s ‘state’ so we can rollback the file if need be.

The Index

The index is also known as the cache or the staging area, and it’s a place to put files before you’re ready to make a ‘commit’ (which is something of a ‘milestone’ or a version). A commit is a representation of a point in time in the history of your git repository. Think of it as a snapshot of your repository – ‘this is how our files looked at this specific time’. With commits, you can always rollback to a specific ‘commit’ (or even advance to a future commit if you had rolled-back to an earlier commit). In order to rollback to a specific version of a file, though, you need to make a commit, and in order to make a commit you need to tell git what FILES will comprise our commit. As I mentioned, a commit is just a point in time…but it doesn’t HAVE to be a point in time for a SINGLE file. A commit can represent a specific point in time for a NUMBER of files all at once. To make life easier for everyone, though, if you make a commit comprised of multiple files, then you want to make sure that the changes to the multiple files all represent a single functional change (for example, say we wanted to change a file that existed in our crankd_google folder that was being copied into a package, but we ALSO had to change the Makefile to change WHERE this file was to be put in the final package. These two changes are related and comprise a single functional change, so you would want to make a commit with the changes to both files at once.).

Our Makefile exists in the working tree currently, but we want to put it in the index because we’re ready to make a commit (no, our Makefile functionally doesn’t DO anything yet, but we want to make a commit so we can rollback to this point in time should we mess up anything). Understand that the locations of the ‘working tree’ and ‘index’ are relative to git ONLY. Yes, this file lives in ~/src/luggage_examples, and it will CONTINUE to exist in that directory (as far as the OS X filesystem is concerned), but to git it currently exists in the working tree. How do we know that the file is in the working tree and NOT in the index? Use the git status command to show you that info:

$ git status

# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   Makefile
nothing added to commit but untracked files present (use "git add" to track)

Anything listed under ‘Untracked files:’ is in the working tree. You can add a file to the index by doing the following:

$ git add Makefile
$ git status

# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#   new file:   Makefile
#

Great, the file is in the index! Notice that the file is listed under ‘Changes to be committed:’ meaning that it’s in the index but NOT YET committed (or, ‘in the repository’).

The Repository

The Repository is where your files exist once they’ve been made part of a commit. As I mentioned before, files must be committed before we can roll back their states (and their states can only be rolled back to individual commits). You can never rollback a file’s state UNLESS it’s tied to a commit. If you made a commit to a file last week and haven’t made another since, but wanted to revert a file to how it existed three days ago, you would be out of luck. Git ONLY rolls back to specific commits (this is why it’s important to make frequent commits). Before we make a commit, however, we need to make sure to identify ourself to git (if you haven’t previously set this up). The user.email and user.name settings are the most important settings as they will be used to trace back what code you committed. If you’re not sure of how git is setup, use the following command to list the git config settings:

$ git config -l
user.name=Gary Larizza
user.email=gary@puppetlabs.com

Should the user.name and user.email settings NOT be configured, you can configure them with the following commands:

$ git config user.name 'Your name'
$ git config user.email your@email.com

Now that you’ve been identified (so we can lay the blame on you in the future), let’s actually commit some code. The git commit command allows git to commit all staged files to the repository in one fell swoop.

Every commit has three things: the list of files and CONTENT of the files that should be committed, a SHA1 hash value representing the commit itself (essentially, a unique identifier for the commit so it can be tracked), and the commit message describing the what and why of the commit.

Creating a commit message seems insignificant but should NOT be taken lightly. Great commit messages usually consist of a title line containing no more than 60 characters followed by a paragraph describing the changes. Because many people may potentially need to trace your code and commits, it’s important to list how your code functioned previously, the reasoning behind the change, and what specifically was changed. Don’t skimp with your commit messages! Always assume that the next person to maintain your code owns many dangerous weapons and knows where you sleep, so do that person a favor and be as kind to them (in the form of great commit messages) as possible!

Let’s begin the process of making a commit by issuing the following command:

git commit

Executing git commit will drop you into the default text editor (which is vim by default – you can change this by running git config core.editor /path/to/your/editor before doing git commit) and show the following:

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
# new file:   Makefile
#

Note that everything that begins with a hash ( # ) is solely for YOUR benefit and will be stripped out of the commit message, so don’t worry about it being included. Here’s an example of a commit message I would use in this instance:

Initial commit for crankd_google code

Previously we didn't have an example for creating a package of the 
Google Mac Sysops code.  This commit adds the shell for the Makefile and
includes the luggage.make file in ~/src/luggage.

Great! When you save and close your editor, you should see something that looks like this:

[master (root-commit) f423ac6] Initial commit for crankd_google code
 1 files changed, 9 insertions(+), 0 deletions(-)
 create mode 100644 Makefile

This tells us that we made a commit on the master branch with a SHA1 hash ID beginning in f423ac6 being attributed to this commit. In most cases, the first 7 digits of the hash are all you need for uniqueness when you refer to this commit, and that’s usually all that git will provide you in THIS instance. YOUR SHA1 hash won’t match the hash provided above (hence a unique ID), so don’t panic if it doesn’t match up for you. And that’s it! We’ve done it! Now what?

Examining your Work

Git tracks the ENTIRE history of a repository, and thus will track any and every commit made. To display this history, run the following command:

$ git log
            commit f423ac6e9548d93f53b15d1154723c5abcd69738
    Author: Gary Larizza 
    Date:   Sun Jan 22 23:27:31 2012 -0500

        Initial commit for crankd_google code

        Previously we didn't have an example for creating a package of the
        Google Mac Sysops code.  This commit adds the shell for the Makefile and
        includes the luggage.make file in ~/src/luggage.

By default, git log will show ALL information about ALL of your commits (including the full commit message). There’s a LARGE number of arguments that git log will accept that allows you to format the output for ANY purpose. One of the common lists of arguments that I like to employ is the following (long) command:

$ git log --pretty=format:'%C(yellow)%h%C(reset) %s %C(cyan)%cr%C(reset) %C(blue)%an%C(reset) %C(green)%d%C(reset)' --graph --date-order
* f423ac6 Initial commit for crankd_google code 10 minutes ago Gary Larizza  (HEAD, master)

This format both colorizes the output for readability and also shows a nice graph of commits as you do things like branching and merging. Here’s an example of this command (with line length chopped for brevity) being run on a repository with many commits:

* 72a2fe0 (#5445) Create /var/lib/puppet in OS X Package 26 hours ago Gary Larizza  (HEAD, gary/bug/2.7.x/5445_crea
    *   c6667c5 Merge pull request #324 from glarizza/bug/2.7.x/2273_launchd_dashw 12 days ago Daniel Pittman  (origin/
    |\  
    * \   bc6c642 Merge pull request #318 from glarizza/bug/2.7.x/11202_apple_rake 12 days ago Michael Stahnke 
    |\ \  
    | * | 6c14a28 Build a Rake task for building Apple Packages 12 days ago Gary Larizza  (gary/bug/2.7.x/11202_apple_r
    * | |   5ec5657 Merge pull request #127 from adrienthebo/ticket/2.7.x/8341-prevent_duplicate_loading_of_facts 12 da
    |\ \ \  
    * \ \ \   450da6a Merge pull request #92 from duritong/tickets/2.7.x/1886 12 days ago Daniel Pittman 
    |\ \ \ \  
    | | | | * d5bef5e (#2773) Use launchctl load -w in launchd provider 12 days ago Gary Larizza  (gary/bug/2.7.x/2273_
    * | | | |   1cb2b6a Merge branch 'ticket/2.7.x/11714-envpuppet' into 2.7.x 12 days ago Josh Cooper 
    |\ \ \ \ \  
    | |_|_|_|/  
    |/| | | |   
    | * | | | 5accc69 (#11714) Use **%~dp0** to resolve bat file's install directory 12 days ago Josh Cooper 
    * | | | |   fe79537 Merge pull request #323 from glarizza/bug/2.7.x/fix_launchd_spec 12 days ago Jeff McCune 
    |\ \ \ \ \  
    | * | | | | c865a80 Clean up launchd spec tests 12 days ago Gary Larizza  (gary/bug/2.7.x/fix_launchd_spec, bug/2.7
    |/ / / / /

As you can see, git log is powerful and handy to list out what’s happened to your repository. You’ll also notice that it lists WHO made the changes, which is handy if you have MANY people contributing to the same code base.

That’s it for now

Hey , what gives?! We haven’t really DONE anything! Git has a tremendous amount of power, and it’s important to know WHAT’S happening before I can progress and show you the power of git. This article is ALREADY quite long, so I’ve broken it up for ‘easier’ reading. Rest assured, in parts 2 and beyond we will modify files, create topic branches, merge topic branches, use Github (for backup AND to allow others to contribute to your project), and much more. Stay tuned for the following article(s) and as always, put any questions in the comments or email me directly!

Using the Google MacOps Crankd and Facter Code

On December 21st, the MacOps team at Google released a Googlecode repository containing a couple of scripts that they use for managing Macs. The scripts provided interface with crankd (available in the Pymacadmin code suite) and Facter, which are a couple of tools with which you MIGHT not be familiar.

I saw a couple of questions on the ##osx-server IRC channel about how these scripts work, and I decided to do a quick writeup on how you would implement the scripts in your environment. This is meant to be a walkthrough to getting their scripts up-and-running, but please READ EVERYTHING FIRST before implementing.

Prepare Crankd:

Crankd is a vital part of the Google MacOps puzzle. Crankd essentially executes Python code in response to System/Network events on your system. I have MUCH MORE information on crankd in a guide I posted a while back, but I’m going to give you the ‘quick version’ for those looking to get started quickly:

• Clone or download the Pymacadmin Github repo. If you have git installed you can accomplish that by doing the following from the command line:

  git clone https://github.com/acdha/pymacadmin.git

If you DON’T have git on your machine just visit https://github.com/acdha/pymacadmin, click on the Downloads tab, and download the .tar.gz or .zip version of the repo. Next, double-click on the resultant file to expand it into a folder.

• Change to the pymacadmin folder and run the install-crankd.sh script which will install crankd.py into /usr/local/sbin and create /Library/Application Support/crankd Once you’ve changed to the pymacadmin source directory, run the following from the command line:

  sudo ./install-crankd.sh

OR…

If you already have a Puppet environment configured, I’ve written a module that will install crankd for you. The upside is that this module should work out of the box, but the downside is that I’ve copied the source to the module’s files/ directory. Since Pymacadmin hasn’t been changed since 2009, though, this shouldn’t pose a big issue. Simply copy this module to your modulepath and declare it with the following line in your node declarations:

include crankd

Download the Google-macops source code

You can download the source by using subversion with the following command:

svn checkout http://google-macops.googlecode.com/svn/trunk/ google-macops-read-only

This will create a folder called ‘google-macops-read-only’ in the current directory.

Copy the Python crankd files

Change to the directory where you downloaded the Google-macops source code and you will notice two folders: crankd and facter. Change to the crankd directory and copy all of the python files to /Library/Application Support/crankd with the following command from the command line:

sudo cp *.py /Library/Application\ Support/crankd/

This copies two files into /Library/Application Support/crankd: ApplicationUsage.py and NSWorkspaceHandler.py. The ApplicationUsage.py file contains most of the magic. In a nutshell, it will create a SQLite database file called /var/db/application_usage.sqlite containing information on applications that are launched and quit (we will inspect this information later), and it will update the database any time an application is (you guessed it) launched or quit. Crankd provides the mechanism by which the code is run any time an application is launched/quit and these Python files actually update the database with the pertinent information.

Copy the provided crankd plist to /Library/Preferences

Changing to the crankd directory in the google-macops-read-only directory you should find a file called ‘com.googlecode.pymacadmin.crankd.plist’ which is a sample plist for utilizing Google’s code that tracks application usage via crankd. It’s up to you to merge these Python methods into your existing crankd setup, or utilize the plist they’ve provided. The plist essentially says that any time an application is LAUNCHED, you should call the ‘OnApplicationLaunch’ method in the NSWorkspaceHandler class, and any time an application is QUIT, you should call the ‘OnApplicationQuit’ method in the NSWorkspaceHandler class. Let’s accept these conditions and copy their plist into /Library/Preferences (assuming that we don’t have an existing crankd setup) with the following command:

cp com.googlecode.pymacadmin.crankd.plist /Library/Preferences

Test out crankd from the command line

With crankd.py installed, the Python methods copied to /Library/Application Support/crankd, and the com.googlecode.pymacadmin.crankd.plist plist copied to /Library/Preferences, we can finally test out the code to see how it works. Start crankd with the following command:

sudo /usr/local/sbin/crankd.py --config=/Library/Preferences/com.googlecode.pymacadmin.crankd.plist

After entering your password, you should see the following in your terminal:

INFO: Loading configuration from /Library/Preferences/com.googlecode.pymacadmin.crankd.plist
INFO: Listening for these NSWorkspace notifications:  NSWorkspaceWillLaunchApplicationNotification, NSWorkspaceDidTerminateApplicationNotification

Now, let’s launch a couple of applications and see how we track their usage through the sqlite database. Do this by starting Safari and TextEdit, and then closing them (note that you MUST start them fresh – if they’re already opened make sure to close them, then open them, and THEN close them again). You should see something that looks like the following in the terminal:

INFO: Application Launched: bundle_id: com.apple.Safari version: 6534.52.7 path: /Applications/Safari.app
INFO: Application Launched: bundle_id: com.apple.TextEdit version: 264 path: /Applications/TextEdit.app
INFO: Application Quit: bundle_id: com.apple.TextEdit version: 264 path: /Applications/TextEdit.app
INFO: Application Quit: bundle_id: com.apple.Safari version: 6534.52.7 path: /Applications/Safari.app

Finally, you can quit crankd from the terminal by holding the control button down and pressing the ‘c’ key on your keyboard.

Inspect the application_usage.sqlite database

If you’re proficient in SQL, feel free to browse through the contents of the database. To keep things simple, let’s download an app called SQLite Database Browser from the following link. Launch SQLite Database Browser, go to File –> Open Database, hold the Shift and Command buttons down on your keyboard and then press the ‘g’ button to bring up the ‘Go to the folder:’ dialog box, type in /var/db/application_usage.sqlite in the box and press return, and then finally click the open button to open the database. If you click the ‘Browse Data’ tab near the top of the window, you should be able to see the raw data in the database. It should list the last time (in epoch time), bundle id, version, path, and number of times that the application has been launched (note that you can use an online utility to convert from epoch time to regular time).

Inspect the Facter fact

If you’re not familiar with Facter, you should click on the link and check it out. Facter is an awesome binary designed to gather information about your system and report it back in the form of a ‘fact’, which is essentially a key => value pair (for example, the ‘operatingsystem’ fact will return a value of ‘Darwin’ on Macs running OS X or ‘RedHat’ on machines running RedHat Linux). Google has provided a custom Facter fact that will report a fact NAME that corresponds with an application you’ve launched on your system (such as ‘app_lastrun_com.apple.safari’ or ‘app_lastrun_com.apple.textedit’) and a VALUE with the epoch time or the last time that application was run. The output will ultimately look like this:

app_lastrun_com.apple.safari => 1325203354
app_lastrun_com.apple.textedit => 1325203357

NOTE: This Facter fact requires the SQLite database that is created by the crankd code from above, so make sure you’ve followed the previous steps or you might have problems getting fact values.

We’ll need to install Facter. I’ve provided a package of Facter version 1.6.4 suitable for download here. Simply download the DMG, expand it, and run the package installer inside.

To test out the fact, we will need the full path to the folder containing the google-macops code we downloaded in step two above. I downloaded the source to a path of /Users/gary/Desktop/google-macops-read-only and so that’s the path I’m going to use in the next step.

Facter is designed to work with Puppet, but you can also run it independently. Facter is ALSO designed to be plug-able with custom facts, and that’s what Google has shipped us. In order to TEST these custom facts, we need to set the RUBYLIB environment variable (as Facter will look for a custom facts inside a ‘facter’ directory that’s located inside the path returned by the RUBYLIB environment variable). RUBYLIB should be set to the path of the downloaded google-macops source, so in my case I would run the following from the command line:

export RUBYLIB=/Users/gary/Desktop/google-macops-read-only

NOTE: make sure RUBYLIB is in capital letters. If you downloaded the google-macops source to a different path, substitute your path for ‘/Users/gary/Desktop/google-macops-read-only’.

Next, we need to make sure we have the ‘sqlite3’ ruby library installed on our system as the custom fact requires this library for correct operation. One way to install sqlite3 is by using Rubygems with the following command from the command line:

sudo gem install sqlite3

NOTE: If you have troubles using Rubygems to install sqlite3, you can either build it from source (which is beyond the scope of this article), or you can use Macports (if you have it installed) to install it with the following command:

sudo port install sqlite3

The last step is to actually run Facter to test out the fact. You may have a problem TESTING this fact if you installed sqlite3 via Rubygems – the reason is because the custom fact doesn’t actually load Rubygems before it tries to load sqlite3 (this isn’t a problem if you use the fact with Puppet as Puppet loads Rubygems. It’s ALSO not a problem if you’re running version 1.9 or greater of ruby, as you no longer need to load Rubygems BEFORE trying to include a ruby library with ruby 1.9 or greater). To remedy this problem, let’s add a single line to the custom fact. Notice the line that says the following:

require 'sqlite3'

This line is where the custom fact loads the sqlite3 library. Now, the line BEFORE this line we need to add a single line:

require 'rubygems'

This line will load Rubygems so that it knows where to access the sqlite3 library. Note that this line is unnecessary if you use this custom fact with Puppet as, like I mentioned before, Puppet already loads Rubygems. For the purpose of our testing, though, we will need this line.

Finally, let’s test the fact by running Facter from the command line:

facter

You SHOULD get a list of ALL your Facter facts, but up at the top you should have a couple of facts that begin with ‘app_lastrun_’ and then end with the bundle_id of the application that was launched. These are the custom facts that have been created.

You’ll note that the time reported is in epoch time. If you’d prefer a date/time string that’s more legible, you’ll need to modify the apps.rb file in the facter directory. Line 39 in the file (or, line 40 if you added the require ‘rubygems’ line previously) should say ‘appdate’ but change it to say the following:

Time.at(appdate.to_i)

Running facter again (like above) will give you a more legible date:

app_lastrun_com.apple.safari => Thu Dec 29 19:02:34 -0500 2011
app_lastrun_com.apple.textedit => Thu Dec 29 19:02:37 -0500 2011

???

Profit!

Now that you know how the code works, you’re free to modify, extend, and improve it as you see fit! The one thing that’s missing here is a LaunchDaemon that keeps crankd running in the background. Since you’re maintaining your environment, I’ll leave that up to you (though I walk you through this step in ‘Step 5. A LaunchDaemon to keep crankd running’ in my previous crankd writeup). Feel free to check out my other writeups on Facter, Puppet, and Crankd to see how I used to manage hundreds of Macs in an educational environment!

IRC Alerts with irssi and Boxcar

Pushing alerts to Boxcar

I’ve recently found the need to remain persistently connected to IRC but to ONLY get notifications if someone sends a message directed at my nickname. The solution I found is relatively simple to setup, and works great for pushing alerts to my iPhone (or ANY iOS device, really).

irssi + screen = ideal

irssi is a popular command line IRC client that’s pretty scriptable and ripe for customization. One of the biggest problems I have with IRC is that it’s not been very…portable. Context switching is annoying both for the user and everyone else in the room (if you’ve seen someone using Colloquy on the iPad/iPhone and logging on/off constantly, you realize how annoying it can be). One of the more popular solutions is to setup a screen session on a machine with a stable network connection, and simply ssh into it whenever you need to interact with an IRC channel. This works great for me because my employer maintains an easily-accessible linux box with ssh access. Here are the commands I use:

1. ssh gary@linuxmachine.myemployer.com
2. screen (sets up a screen session)
3. irssi
4. ( connect to IRC channels )
5. Ctrl + a, D (detaches the screen session)

When you detach the screen session, it still stays live. To resume the session, just ssh back into the machine and enter “screen -r” to resume the session. You’ll notice you pick right up where you left off. Better yet, you can use any ssh client to reconnect to the screen session.

Notifications FTW

While a persistently accessible IRC connection is rather awesome, it’s still quite hard to follow. This is where the fnotify.pl script comes in. This script (included below) will generate a file called “fnotify” in your ~/.irssi directory whenever someone directs a message your way (such as “glarizza: What’s up?”). Every time you receive a message directed towards your nickname, it logs it to that file. While this consolidates all of the messages that are directed towards your nickname, it doesn’t ALERT you that they’ve occurred. For that, we need another script (or two).

Below you will find a one-line script called event_loop.sh that tails the last line in the ~/.irssi/fnotify file and pipes it to ANOTHER script that sends it to my iOS devices. Because it uses tail -f, it will continue to monitor the ~/.irssi/fnotify anytime it’s changed.

Finally, the push_to_boxcar.rb script sends the alert to Boxcar (a push notification service for iOS devices). With Boxcar, you can create a custom Provider dedicated specifically for these alerts (create an account, login, and visit http://boxcar.io/site/providers ). When you setup this provider, Boxcar gives you an API key and an API secret for authentication (that way you don’t have to embed your username/password in the script). Because these alerts are piped through a specific provider, you can login to Boxcar from anywhere to see every alert that was pushed. How awesome is that?

Show me the scripts

Below are the scripts in their entirety (sanitized for my protection). Hopefully they work as well for you as they have for me!

fnotify.pl

# todo: grap topic changes

use strict;
use vars qw($VERSION %IRSSI);

use Irssi;
$VERSION = '0.0.3';
%IRSSI = (
    authors     => 'Thorsten Leemhuis',
    contact     => 'fedora@leemhuis.info',
    name        => 'fnotify',
    description => 'Write a notification to a file that shows who is talking to you in which channel.',
    url         => 'http://www.leemhuis.info/files/fnotify/',
    license     => 'GNU General Public License',
    changed     => '$Date: 2007-01-13 12:00:00 +0100 (Sat, 13 Jan 2007) $'
);

#--------------------------------------------------------------------
# In parts based on knotify.pl 0.1.1 by Hugo Haas
# http://larve.net/people/hugo/2005/01/knotify.pl
# which is based on osd.pl 0.3.3 by Jeroen Coekaerts, Koenraad Heijlen
# http://www.irssi.org/scripts/scripts/osd.pl
#
# Other parts based on notify.pl from Luke Macken
# http://fedora.feedjack.org/user/918/
#
#--------------------------------------------------------------------

#--------------------------------------------------------------------
# Private message parsing
#--------------------------------------------------------------------

sub priv_msg {
    my ($server,$msg,$nick,$address,$target) = @_;
    filewrite($nick." " .$msg );
}

#--------------------------------------------------------------------
# Printing hilight's
#--------------------------------------------------------------------

sub hilight {
    my ($dest, $text, $stripped) = @_;
    if ($dest->{level} & MSGLEVEL_HILIGHT) {
    filewrite($dest->{target}. " " .$stripped );
    }
}

#--------------------------------------------------------------------
# The actual printing
#--------------------------------------------------------------------

sub filewrite {
    my ($text) = @_;
    # FIXME: there is probably a better way to get the irssi-dir...
        open(FILE,">>$ENV{HOME}/.irssi/fnotify");
    print FILE $text . "\n";
        close (FILE);
}

#--------------------------------------------------------------------
# Irssi::signal_add_last / Irssi::command_bind
#--------------------------------------------------------------------

Irssi::signal_add_last("message private", "priv_msg");
Irssi::signal_add_last("print text", "hilight");

#- end

push_to_boxcar.rb

#!/usr/bin/env ruby

require 'rubygems'
require 'boxcar_api'

SETTINGS = {
  :provider_key => 'myproviderkey',
  :provider_secret => 'mysuperserialsecret',
  :email => 'gary@mycooldomain.com'
}

Signal.trap(:TERM) { exit; }
bp = BoxcarAPI::Provider.new(SETTINGS[:provider_key], SETTINGS[:provider_secret])

$stdin.each_line do |line|
  bp.broadcast(line)  
end

event_loop.sh

#!/bin/bash
#
tail -n1 -f ~/.irssi/fnotify | ~/bin/irc_notify_loop.rb

Retiring at 30?

Come mothers and fathers throughout the land, don’t criticize what you can’t understand, your sons and your daughters are beyond your command, your old road is rapidly agin'. -Bob Dylan

As I sit in a Portland hotel at 9:30am, awaiting a 12:10pm flight, I look around to see what will become the next great chapter of my life. It’s exciting and slightly foreign, but I can’t wait for it to arrive.

I don’t think it surprises anyone who knows me (or follows what I write) to hear that I’m resigning my position as Director of Technology with the Huron City School District (in Ohio), moving my belongings 7 states to the west, and accepting a position with Puppet Labs in Portland, Oregon as of May, 2011. If you scan through the articles I’ve written, I would venture to say that 8 out of 10 of them are Puppet-related in some manner. The product excites me, I’m jazzed to see its usage spreading, and I absolutely cannot wait to train users who have never worked with it before.

As a company Puppet Labs employs some incredibly smart cookies, the environment in their office is conducive to developing great solutions to common problems, and Portland is…well…the place where young people go to retire. Let’s just say that while me in my flat-cap was about as unique as it came in my previous home, it’s about as vanilla whitebread as you’ll find around there.

My comments and opinions will still ring out over Twitter, I’ll still be posting any cool tips, tricks, and hacks, and you’ll most likely still see me at the Mac Conferences to which I’ve become accustomed…it’s just that now you’ll also have the opportunity to see me doing trainings in a city near you.

So wish me well as I grab my car, my dog, and my stuff and schlep it to that pearl in the Northwest known as Portland(ia).

Using Run Stages with Puppet

As of version 2.6, Puppet introduced a feature called “Run Stages” that will allow you to better control the order that Puppet configures your system. The problem with Run Stages (as of right now) is that there’s not that much good Documentation out there. Hopefully this document helps someone else out there who wants to setup Run Stages in their own environment.

A NOTE ON SYNTAX!!!

One of THE MOST DIFFICULT concepts to understand for puppet newbies is that these two things are identical:

include foo
class { 'foo': }

Both of these lines INVOKE the ‘foo’ class for use on a particular node. The problem is that the second method (the parameterized class method) looks nearly IDENTICAL to the method you would use to DECLARE the foo class in foo.pp. The only difference is that the class name is now INSIDE the curly braces (versus being OUTSIDE the braces when you DECLARE the class in your foo.pp file). It is VERY important that you distinguish the difference between DECLARING a class and using a parameterized class to INCLUDE the class in your site.pp file. I will point this out later too…

A Linux Example – Repositories

I would bet that the resource with the most “require” and “before” dependencies in the Linux world (Well, in the RHEL world anyways) would be the yumrepo. I’m sure most of us have many declared yumrepo resources in our manifests that each have their own tangled web of dependencies. My goal was to create a “repo” class that would have all of my repositories and use a Run Stage to ensure that my repo class was installed prior to any other package installs. Let’s look at some code:

class general::repo {
  yumrepo {"huronrepo":
    descr       => "Huron Repository",
    enabled     => 1,
    gpgcheck    => 0,
    baseurl     => "http://10.13.0.6/huronrepo",
  }
}

This is my repo subclass based on my general base class. I have a single repository, but could easily have another 5 or 10 as need arises (I’m keeping things simple for demo purposes). Let’s look at another class:

class general::centos {
  include general::repo

  package { "mcollective":
    name        => "mcollective",
    ensure      => present,
    require     => Package["stomp"],
  }
}

Here’s a centos subclass off of the general base class. We’re including our repo class and declaring a single package that requires our “huronrepo” yumrepo. If we only needed to install a single package, we could use the “require” parameter; but if you assume that we will eventually need to install MULTIPLE packages, then this logic doesn’t scale. Just to keep things consistent, here’s my general base class:

class general {
  case $operatingsystem {
    CentOS: { include general::centos }
    Darwin: { include general::osx }
  }
}

You can see that we include the general::centos class as long as Puppet is running on a CentOS box. There’s one final piece to this puzzle – it’s the actual declaration and assignment of the Run Stages. I’m doing this in my site.pp file. Here’s a snippit of what’s in my site.pp file:

# /etc/puppet/manifests/site.pp

Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }

# Run Stages
stage {"pre": before => Stage["main"]}

# Node Declaration
node 'server.whomever.com' {
  class {'general::repos': stage => 'pre'}
}

Here’s where we’ve declared a “pre” stage that needs to run before the “main” stage. We don’t have to declare the “main” stage because Puppet uses that stage by default. Next, we’re including the general::repos class inside our ‘server.whomever.com’ node declaration and assigning it to the “pre” stage (which means that everything in that class will get configured prior to our “main” stage). Remember that you can declare as many stages as you need and chain them all to setup the order that you want, but that can become very unmanageable very quickly. I find it ideal to setup a ‘pre’ stage for repo setup, and then setting up dependencies within class files.

The World is Your Stage

This might not be the “best” way to use Run Stages, but it works for me. Hopefully this little writeup cements the concept for you too.

Join the new Luggage Mailing List

For all of those people who are interested in The Luggage but have questions that haven’t been answered in any available online documentation, there’s now a new resource for you. Joe Block has created a Google Group for The Luggage. Feel free to join and post comments, submit feature requests, or ask any questions that might have had trouble conveying in 140 character tweets. I’ll be trolling the list as well, so I’m sure you’ll see me chime in every so often.

http://groups.google.com/group/the-luggage/

Passing MCollective Muster: Using the Registration Agent with Nagios

Once you’ve started to deploy MCollective down to the Desktop level (or even to multiple servers), you suddenly find yourself with the need to inventory MCollective-enabled nodes. There are many ways to do this, but today I’m outlining the Registration Agent plugin that’s available within PuppetLabs' MCollective-Plugins collection.

How it works:

The MCollective Registration agent (as of today’s date and including documentation) is a 51 line wonder that will simply create a file in the /var/tmp/mcollective directory for every MCollective node that checks-in. Nodes will check-in every 300 seconds (by default), which means that you can easily check the last time your nodes have checked-in by doing a stat of the mtime of their node file. The file that the registration agent creates contains a YAML dump of all currently installed MCollective agents, but this can be customized if you feel like modifying the registration agent. The registration agent also need only be installed on ONE machine (the machine saving the node check-in files) – all other nodes will simply add two lines to their /etc/mcollective/server.cfg file. If you’re managing MCollective’s config files with Puppet (which you should be doing), then the modification is a simple commit that can be pushed out to all of your nodes.

Installation

1. First, you’ll need to grab a copy of the MCollective plugins project from the PuppetLabs github repo. You want this specific file to setup registration on a target machine (which we’ll refer to as “the server”).
2. Save the file mentioned in step 1 into the MCollective library’s “agent” folder (which, by default, should be /usr/libexec/mcollective/mcollective/agent) on the server machine. If you did it correctly, your server machine should have a file named /usr/libexec/mcollective/mcollective/agent/registration.rb
3. Restart MCollective on the server so it recognizes the registration agent.

4. On a test node, open the /etc/mcollective/server.cfg file and add the following two lines:

   registerinterval = 300
   registration = Agentlist

5. Restart MCollective on your node machine.

6. On the server machine, check out the /var/tmp/mcollective folder and see if there’s a file created named after the “identity” configuration setting of your test node. If everything went correctly, this file should be created and should contain a YAML dump of that node’s currently enabled MCollective Agents.

The registration interval can be changed to whatever you like, and you can even change the default directory to which the registration plugin will store its node files. For more information, check out the Puppetlabs Documentation.

Lets Go Meta!

If you want even MORE information in every one of your /var/tmp/mcollective node files, check out the Registration Metadata Plugin for use on your nodes. This plugin will send all fact, class, and plugin information back to the server for storage in your node files. There are two changes:

1. You WILL need to install the meta.rb plugin file in the MCollective Library’s agent directory (/usr/libexec/mcollective/mcollective/agent) on EVERY node that reports back to your server.

2. Your two lines in server.cfg must read as follows:

   registerinterval = 300
   registration = Meta

Once it’s implemented and MCollective is restarted on your node, you should see all your fact, class, and plugin information in the node file on the server!

Nagios Integration

The real power with using the registration agent comes in monitoring when your nodes stop responding to MCollective. A script called check_mcollective.rb is included with the registration plugin that can be used by Nagios to report the status of your nodes. This script will report CRITICAL: if any nodes do not respond for 5 minutes (of course, this too is customizable using the —interval argument.). If you’re monitoring a couple of servers, then I would enable alerting according to this script. You’re monitoring hundreds of desktops (and several laptops), then you’re not going to be overly concerned if a couple aren’t checking in. This script, tied to Nagios, is more importantly a metric for how many nodes are currently online and reporting.

Even if you decide NOT to tie this script into Nagios, you can use the /var/tmp/mcollective directory as a snapshot of which machines have enabled MCollective. Simple “ls -lt” commands will order things according to mtime and you can “grep lab01” or “grep lab01 | wc -l” to search for individual machines or the total number of machines in a particular lab.

Conclusion

The most important aspect of the registration agent is that it give you another metric to monitor your nodes. Its power is in the utter simplicity of creating a file for every node (if you need more control than that, feel free to modify the script to your heart’s content). I look forward to people implementing and improving on the concept. As always, send me an email or a tweet if you do so I can check it out!

A Ruby Script to Check Apple Warranty Information

First, a big “Hi!” and “Thanks!” to everyone at Oslo who’s giving my blog a workout this week! Next, let’s look at a cool little function that I’ve adapted into a Ruby script and a Facter Fact with the help of a couple of people on Twitter (@rockpapergoat and @chilcote). Apple provides a Self Servicing site for you to check the Warranty Information of your machines, but it also provides a web conduit for scripts to utilize the same information. Recently, Apple changed the formatting of this script – so we had to go back to the shed and retool a couple of variables. Here’s the script in its entirety (which can also be accessed here.)

#!/usr/bin/env ruby
#
# File:  Warranty.rb
#
# Decription: Contact's Apple's selfserve servers to capture warranty information
#              about your product. Accepts arguments of machine serial numbers.

require 'open-uri'
require 'openssl'

# This is a complete hack to disregard SSL Cert validity for the Apple
#  Selfserve site.  We had SSL errors as we're behind a proxy.  I'm
#  open suggestions for doing it 'Less-Hacky.' You can delete this 
#  code if your network does not have SSL issues with open-uri.
module OpenSSL
  module SSL
    remove_const:VERIFY_PEER
  end
end
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

def get_warranty(serial)
  hash = {}
  open('https://selfsolve.apple.com/warrantyChecker.do?sn=' + serial.upcase + '&country=USA') {|item|
    item.each_line {|item|}
    warranty_array = item.strip.split('"')
    warranty_array.each {|array_item|
      hash[array_item] = warranty_array[warranty_array.index(array_item) + 2] if array_item =~ /[A-Z][A-Z\d]+/
    }

    puts "\nSerial Number:\t\t#{hash['SERIAL_ID']}\n"
    puts "Product Decription:\t#{hash['PROD_DESCR']}\n"
    puts "Purchase date:\t\t#{hash['PURCHASE_DATE'].gsub("-",".")}"
    puts (!hash['COV_END_DATE'].empty?) ? "Coverage end:\t\t#{hash['COV_END_DATE'].gsub("-",".")}\n" : "Coverage end:\t\tEXPIRED\n"
  }
end

if ARGV.size > 0 then
  serial = ARGV.each do |serial|
    get_warranty(serial.upcase)
  end
else
  puts "Without your input, we'll use this machine's serial number."
  serial = %x(system_profiler SPHardwareDataType | awk '/Serial/ {print $4}').upcase.chomp
  get_warranty(serial)
end

This script will accept an argument of one or more serial numbers (i.e. warranty.rb qp12345678912 w8456154642) and look up the respective warranty information for each serial number. The script can also be run without an argument (at which time it pulls the serial number from the current machine programmatically). It’s pretty simple code to follow – all I’m doing is pulling the important bits that Apple returns into a hash of data and accessing that information where necessary (I’m also substituting periods for dashes as it’s nicer on Facter/MCollective that way. Feel free to remove the gsub() calls if you prefer the dashes).

Playing with Facter and Puppet

From here you can turn this data into individual Facter Facts pretty quickly. Here’s the custom fact I created:

#
# File:       warranty.rb
# Type:       Facter fact
# Decription: Returns the dates of purchase and warranty expiration,
#              plus determines if the warranty is active or inactive.
#              We're using open-uri which caused SSL errors behind 
#              proxies (hence the hack in lines 16-20).

require 'facter'
require 'open-uri'
require 'openssl'

# This is a complete hack to disregard SSL Cert validity for the Apple
#  Selfserve site.  We had SSL errors as we're behind a proxy.  I'm
#  open to suggestions for doing it 'Less-Hacky.'
module OpenSSL
  module SSL
    remove_const:VERIFY_PEER
  end
end

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE


# Constraining to the Darwin Kernel - it's only useful for Macs
if Facter.value('kernel') == 'Darwin'
  warranty_array = []
  hash = {}
  sn  = %x{system_profiler SPHardwareDataType | awk -F": " '/Serial/{print $2}'}
  open('https://selfsolve.apple.com/warrantyChecker.do?sn=' + sn.upcase + '&country=USA') {|item|
         item.each_line {|item|}
         warranty_array = item.strip.split('"')
         warranty_array.each {|array_item|
           hash[array_item] = warranty_array[warranty_array.index(array_item) + 2] if array_item =~ /[A-Z][A-Z\d]+/
         }
  }
end

Facter.add("purchase_date") do
  confine :kernel => "Darwin"
  setcode do
    hash['PURCHASE_DATE'].gsub("-",".")
  end 
end

Facter.add("warranty_out") do
  confine :kernel => "Darwin"
  setcode do
    (hash['HW_COVERAGE_DESC'] == 'Out of Warranty') ? "Yes" : "No"
  end 
end

Facter.add("warranty_end") do
  confine :kernel => "Darwin"
  setcode do
    (!hash['COV_END_DATE'].empty?) ? hash['COV_END_DATE'].gsub("-",".") : "Expired"
  end 
end


Facter.add("product_description") do
  confine :kernel => "Darwin"
  setcode do
    hash['PROD_DESCR'].chomp
  end 
end

If you don’t feel like copying/pasting this for yourself, you can click the link above that will take you directly to my Github account where you can download the raw file. This fact will create four custom facts: purchase_date, warranty_out, warranty_end, and product_description (all are self-explanatory).

Have fun and happy inventorying!

Managing Puppet SSL Certificates

A simultaneous bane and boon of using Puppet is the fact that it uses ssl certificates for added security. This works well if you’re managing 1 – 10 servers, but when your infrastructure starts scaling upward (or you start managing desktops) it’s time for a new approach. Mine may not be THE BEST approach, but it certainly works well for me. I manage everything with the combination of a Puppet wrapper script, certificate signing and cleaning cgi scripts, and autosigning on our master.

The Puppet Wrapper Script

I started this script by stealing the one that Allan Marcus uses/used for Los Alamos National Labs and then ported it over to Ruby in the end of 2009. From there I’ve bolted on various checks that work for our infrastructure. It works like this:

1. Determine which puppet master server we need to contact by checking our IP Address. You can use any method that works best for you; since we’re relatively small, I can use our subnets to determine the correct puppet mast server.
2. Pull the certname variable out of nvram I store my node’s certname in nvram using the mac_uid variable. The certname is based on the machine’s serial number (since it doesn’t change) which keeps the node free to change its machine name. I’m going to move away from this in my next revision and store the certname in puppet.conf, but for now this is how we do it. If the mac_uid variable doesn’t exist in nvram, we pull the serial number and then set the mac_uid variable.
3. Next, we run puppet in —onetime and —no-daemonize mode and look for two strings in the puppet output that signal a certificate error I don’t daemonize puppet but instead call it via a launchd plist file that calls this wrapper script. This allows me to perform various checks before puppet is called. Technically you COULD do this with a puppet prerun_command in puppet.conf, but I prefer my wrapper script.
4. If a certificate error exists, clean the certs and re-run puppet. This is done via our clean_certs method which will delete the local $vardir and $ssldir as well as calling our certificate cleaning CGI script on the puppet master server (which I’ll detail in a later paragraph). Once the certs are cleaned locally and on the server, we will wait 5 seconds and run puppet again (so it can re-generate certs). Since I’m using autosigning on the server, the puppet master will automatically sign the certificate and we’ll be good to go.

The entire script can be found on my Git repo, so feel free to check it out in its entirety.

An aside on certnames

When a puppet node (client) contacts the puppet master server, it uses the hostname of the machine (by default) as the certificate name (or certname variable). This is bad because any time a client changes its Bonjour name (or renames their machine) Puppet is going to have certificate errors (as the certificate name will no longer match the node). To remedy this, as I mentioned before, we use the node’s serial number as the certname. The pro is that this will NEVER change (unless the motherboard is replaced) and you won’t need to worry about certname mismatches. The con is that you will need to know your node’s serial number if you’re running something like Puppet Dashboard and want to look up statistics. There ARE ways around this, though (and they’re beyond the scope of this article).

Certificate Cleaning and Signing CGI Scripts

To be fair, I ALSO stole these from Los Alamos' Puppet setup. Since I’ve enabled autosigning on the server, I don’t need to worry about the signing CGI anymore (it’s still functional, though, and may serve your purposes). Here’s what the cert cleaning CGI script looks like:

#!/usr/bin/ruby
# pclean.rb
# cgi to clean a puppet ssl cert
class Puppetca
    # removes old certificate if it exists
    # The certname parameter is the node's certname
    # Make sure to add the _www user to your sudoers file
    # added using visudo:
    # _www    ALL = NOPASSWD: /usr/bin/puppet, !/usr/bin/puppet cert --clean --all
    def self.clean certname, addr
        command = "/usr/bin/sudo /usr/bin/puppet cert --clean #{certname}"
        # for some reason the "system" command causes Mac apache to crash
        # when used here
        %x{#{command}}
        %x{"logger #{addr} cleaned #{certname}"}
        return true
    end
end
=begin
CGI starts here
=end
# get the value of the passed param in the URL Query_string
require 'cgi'
cgi=CGI.new
certname = cgi["certname"]
# define the characters that are allow to avoid an injection attack
# 0-9, a-z, period, dash, and colon are allowed. All else is not
pattern = /[^a-z0-9.\-:]/
# determine if any other characters are in the certname
reject = (certname =~ pattern) ? 1 : 0
if ((reject == 0) && Puppetca.clean(certname, ENV['REMOTE_ADDR']))
    cgi.out("status" => "OK", "connection" => "close") {"OK #{certname} cleaned from testing.huronhs.com\n"}
else
    cgi.out("status" => "BAD_REQUEST", "connection" => "close") {"Not Processed: #{certname}\n"} 
end

Replace “testing.huronhs.com” with your server’s hostname. In order for this script to run, we will need to edit our sudoers file and allow the _www user to use sudo when calling the puppet cert —clean command. There are a couple of lines that needed to be added to our sudoers file (which can be edited with visudo):

_www    ALL= NOPASSWD: /usr/bin/puppet, !/usr/bin/puppet cert --clean --all

This line allows our web user (via the CGI Script) to use sudo only with the /usr/bin/puppet command. Depending on what OS you are using, you may have to add a line that will allow sudo to be run by a user that isn’t currently logged in (but with OS X this is not necessary). Since I use OS X Servers, this is all that is needed in my sudoers file.

Script Installation

If you’re using the default /Library/WebServer path for your webroot, then the pclean.rb script should be installed in /Library/WebServer/CGI-Executables/ folder and you’ll need to enable CGI-Execution in Server Admin. Also, make sure it’s executable with the following command:

sudo chmod 755 /Library/WebServer/CGI-Executables/pclean.rb

Finally, you can call your script with the following web address:

http://your.server.ip/cgi-bin/pclean.rb?certname=machinecertname

Replace your server’s hostname and “machinecertname” with the certname of your puppet node to clean the SSL cert of your node. You can test and make sure everything worked by monitoring the server’s system log.

A note on Autosigning

Autosigning all ssl certs is a security risk and can totally defeat the purpose of USING SSL in the first place. You CAN exclude machines by setting up an autosign.conf file that details exactly WHICH nodes should be autosigned. The way we do this is with a single line in our autosign.conf file:

*.huronhs.com

With this, puppet will ONLY autosign nodes with a certname of |something|.huronhs.com. Our Puppet wrapper script tacks on “.huronhs.com” to the end of our node’s serial number so it conforms to our autosign.conf file. Granted this is very easy to compromise, but because I’m in a k-12 environment it’s enough security for the time being.