Staffing in I.T. Departments

A lot can be said about the number of people in your I.T. department. There are too many, too few, and of course, "We don't need more because Other School XYZ and/or Competing Business XYZ has fewer."

In my experience, the reality is that the staffing is more about perception and budgets than anything else. When management sees I.T. as a cost center, they give as few resources as possible to it. When management sees I.T. as a way to make staff more efficient or make instruction more effective, then more resources are allocated -- including staff. New hardware is "sexy" and everyone likes being "Santa" by providing new gadgets. The effects of proper staffing, on the other hand, are usually overlooked.

Consider this: How often does your computer break down, confuse you, or require an update? Include any time that you're working for the computer, instead of the other way around. Let's say you lose a full 8 hours to "overhead" out of every 2000 hours. That sounds pretty generous, right? You probably have computer-headaches much more often than that, right? Well, that is roughly one full working day per 1 working year. If you assume 52 weeks per year minus some time for vacations, holidays, and sick days, you have 200 - 240 working days per work-year. (Less if you're in education.) From that, we get 200 to 240 days times 8 hours per day, or 1600 to 2048 hours per year. So at least one full day out of every 200-240 full days is lost to computer overhead like fixing things, figuring out how to do something, installing new software, replacing dead parts, installing upgrades, etc.

This isn't terribly scientific, but its a start. You should have one full time I.T. worker (1.0 FTE) per 200-240 computers. That is just in regards to the PCs, laptops, and tablets. You still need staff for your networking, servers, website, phones, printers and photocopiers, projectors and TVs, document cameras (a.k.a. "Elmos"), streaming video (a.k.a. "Apple TV"), and so on. After all, why have computers if they can't print, share files, access your databases, use the web, or send email to customers?

Yes, that is a lot of staff. Yes, that is a lot of money.

Let's look at a parallel idea for a moment. The cost of a bigger office building is more than just additional rent. Its electricity for the lighting, janitors, and cleaning products. If a manufacturer short staffs the assembly line, then fewer products can be built each week and fewer sales are made. Everything has upkeep costs. Computers are the same. If your staff is really good, you could get away with short-staffing for a while. Of course, good staff will move to jobs that give them less stress. At that point, since there weren't enough I.T. workers, the institution will really feel the pain of the loss. In addition to things taking even longer to get done while searching for a replacement worker, the institution will suffer a brain-drain. The new worker will take months to figure out how things are implemented. A sudden loss of a worker that doesn't have a fall-back co-worker with the same knowledge and skills can cripple a company. Also, what do you do when one of your All Star team of three gets into a car accident or catches bronchitis and needs several weeks to recover?

Its a touch choice. You need staff to keep things running, but staff costs money that you want to use elsewhere.

If you're reading this and you're in management, my suggestion is that you start a dialog with your I.T. department. I think you'll be surprised at how many things they maintain and how deeply dependent on them your institution is. More importantly, ask them what they wish they could do for the institution. Chances are good that they can improve the company or school in ways that you haven't even considered yet. The I.T. department isn't just a cost center. They can be -- and want to be -- a profit center. Most I.T. workers became excited about the field because they thought they could help people and be heroes. Let them. They'll make you look good if you give them a chance.

If you're reading this and you're in I.T., my suggestion is to start a list of things you and your team do. Don't be shy; be realistic. If something goes wrong with the copiers, phones, or TVs, are you going to be asked to help? If so, put it on your list. Then break up your list into visible (user-facing) and invisible (back-end) systems. For the invisible ones, list out the impact of it going down. If the boot disk in your Active Directory server dies, what does that mean to your computers, phones, copiers, wireless networking, BYOD, instructional environment, etc? You're not trying to make it scary, but be honest. The day will come when this list will be helpful. You want to have it ready, because if you have to think of things on the spot it won't be accurate or complete.

Other lists to consider include projects that you think would improve any part of your workplace for your coworkers (new services, consolidate systems, ways to make work easier), ways to make your team or anyone else more efficient, ways to decrease downtime (including scheduled maintenance and redundancies like UPSs, RAID, and VM), and ways to decrease costs. If you know of any legal regulations, keep a list of those, too. For example, K-12 public schools should know about FERPA, COPPA, CIPA, eRate, software licensing regulation, etc.

Whatever you do, you should start with the question of what your institution wants to have running and at what level of reliability. You can work backwards from that and find the right number of staff. Chances are good its not the same number that you have. If you keep an open mind, you might even realize why it takes so long to get a reply from your I.T. department.

Book Recommendation: Time Management for System Administrators

Just a quick tip for today. If you work as a network administrator, system administrator, director of I.T., or almost anything else in the Information Technology field, you need to buy Time Management for System Administrators.

Seriously. You need this book. I bought it years ago and still consider it one of the three best I.T. books I've ever bought.

Its an easy read. Most ideas are broken up into one chapter. Something called The Cycle System takes up three chapters, but its still presented in an easily digested fashion. My advise would be to read each chapter by itself and ponder it for a day or more. Its really good stuff.

The suggestions in it aren't dependent on any vendor or even any era. It works just as well for Windows-centric shops as for Linux or Mac shops, web services, businesses or schools. It was just as useful in 2004 as it is in 2014.

Disable "Add Account" Prompt on Mac

When you login to Gmail (or other popular web services) from Safari, you may be prompted to add the account to Mail, Calendar, Contacts, and other programs that Apple bundles with Macs. While very helpful for home users, this can be a problem for schools. Students and faculty could potentially have many gigabytes of data in Gmail, making the login process take longer and using up limited server space.

With more and more schools moving to Google Apps For Education, this is becoming a more common issue. Depending on your system, it could also mean longer waits during login and logout. If you store user data on a server, its also a lot of unnecessary disk space used (and used again in your backup system.)

To prevent this, I wrote the following script and included it in my Mac setup process inside Deploy Studio. If you use it in your environment, replace "YOUR-DOMAIN-HERE" with your Google Apps domain or remove it. If you want to disable the "Add Account" prompt for a service not listed in this script, just add it after "yahoo.com" in quotation marks.

#!/bin/sh

defaults write /Library/Preferences/com.apple.Safari DomainsToNeverSetUp -array "YOUR-DOMAIN-HERE" "google.com" "me.com" "aol.com" "facebook.com" "flickr.com" "twitter.com" "vimeo.com" "yahoo.com"

exit 0

If your Macs are already in service and you don't want to image them again, this script will still work. Just push it to them and run it as the user "root". You can do this in Apple Remote Desktop, Casper, FileWave, Munki (as a postflight script in a PKG installer), or other tools.

Change Default Dock Items

When users login to a new version of MacOS for the first time, it will fill their Dock with new icons that Apple has chosen. This might not always meet the needs of your business, school, or organization. For example, if you have your users' home directories on a file server, you might not want to encourage your users to use iMovie, GarageBand, iTunes, and iPhoto due to their tendency to use large amounts of disk space. If you use Google Apps for Work or Google Apps for Education, you might not want to present Mail, Maps, or Calendar. Each institution must make their own decision about what is best for them.

At my job, we use home directories on a file server with anywhere from 35 to 100 users on it at the same time. As you can imagine, this means a classroom of 25 students using iMovie could cause the entire school to experience serious slow-downs. We also use Google Apps for Education, so I don't want to encourage users to utilize Apple's Mail program. That would mean several GB more disk space used per user over hundreds of users. In other words, my servers would unnecessarily have about an extra 1TB (or more) disk space used, my backup system would get filled up, and everyone's Macs would be much, much slower.

To solve this, I give my end users an empty Dock. They're shown how to use Google Apps for Education and how to add items to the Dock. Then when they think "I want to see my email," they go to Gmail instead of the Mail program.

To do this, I wrote the following script and put it in our Deploy Studio server. It runs just after the Mac is imaged with its OS.

(Note: This is for MacOS 10.9. For MacOS 10.7, remove the # from the beginning of the appropriate line and add it to the beginning of the line for 10.9.)

#!/bin/bash
#
# This script will attempt to avoid adding all of Apple's default items from the
# Dock for new users.  This default behavior of MacOS is very useful for home
# users but terrible for institutional deployments, where the needs of Apple
# and the needs of the institution which purchased Macs might not be aligned.
#
# 2014-04-03-1615, Jaime Kikpole

DEFAULTS="/System/Library/CoreServices/Dock.app/Contents/Resources/en.lproj/default" # For 10.9
# DEFAULTS="/System/Library/CoreServices/Dock.app/Contents/Resources/English.lproj/default" # For 10.7
DEFAULTSFILE="${DEFAULTS}.plist"

FIXUP="/Library/Preferences/com.apple.dockfixup" # For 10.7
FIXUPFILE="${FIXUP}.plist"

echo Changing default settings for the Dock for new users...

if [ -w ${DEFAULTSFILE} ]; then
 echo - Changing settings in "${DEFAULTS}"...
 defaults write ${DEFAULTS} persistent-apps ""
 defaults write ${DEFAULTS} persistent-others ""
 chmod +r ${DEFAULTS}
 echo ...done.
else
 echo "${DEFAULTSFILE}" not found or writable.
fi

if [ -w ${FIXUPFILE} ]; then
 echo - Changing settings in "${FIXUP}"...
 defaults write /Library/Preferences/com.apple.dockfixup add-app ""
 chmod +r ${FIXUP}
 echo ...done.
else
 echo "${FIXUPFILE}" not found or writable.
fi

echo End of script.

Backup Configuration Files

Anyone who manages a Unix-like system has edited a configuration file from time to time.  At an absolute minimum, you should copy these files before editing them.  Otherwise, you could get through a few lines of editing, break something (or just want to go back to your old setup), and not know how to go back.

Here is a quick trick I came up with for handling this.  There are better ways, but this can be dropped right into any Unix-like system with very little effort. That easily includes MacOS, Linux, and FreeBSD.

First, I started with this trick from http://osxdaily.com/2012/06/11/make-a-quick-backup-of-a-file-from-the-command-line/.

cp file.txt{,.backup}

This will copy file.txt to file.txt.backup .

Unfortunately, the next time its done, file.txt.backup will be overwritten and the history of our changes over time will be lost.

So I decided to end the backup files with the date. Done correctly, this will alphabetize them into chronological order and allow some rudimentary change-tracking in the future using the diff command. So let's look at this:

date "+%Y%m%d%H%M%S"

It outputs the current date and time in YYYYMMDDHHMMSS format. In other words, it prints out the year, then month, then day of month, then the time. So 9:59am on July 11, 2014 will come out as 201407110959.

Note that this is done with "padding zeros" to make sure that it alphabetizes into chronological order. You can see it in the "07" for July, which will now come before December. This is because 07 is alphabetically before 12, even though July and Jul are alphabetically after December and Dec.

So using that and the cp command from earlier, we now have this.

cp file.txt{,.`date "+%Y%m%d%H%M%S"`}

This copies file.txt to file.txt.<timestamp>. Using the example date above (9:59am on July 11, 2014), that would be file.txt.201407110959. This is the effect we want.

In a bash shell script, "$1" references the first string after the command, as parsed by whitespace delimiting.

So using that, we get this shell script:

#!/bin/bash
cp $1{,.`date "+%Y%m%d%H%M%S"`}

Let's put that in a file called script.sh and then make sure its executable with chmod 755 script.sh. This allows "./script.sh file.txt" to copy file.txt to file.txt.<timestamp> without having to manually calculate the timestamp.

Name this "backup" and copy it to your ~/bin, /usr/local/custom, or some other location in your $PATH. Then set it to chmod 0755, if you haven't done so yet.
The next time you have to edit a configuration file, just type "backup config-file.txt" before you edit it.

So, to re-cap:

1. Make a file called "backup" and put the above two line script in it.
2. Put "backup" someplace where it will be in your $PATH, such as ~/bin or /usr/local/bin or /usr/local/custom.
3. Make sure that location is actually in your $PATH.
4. Use chmod 755 /path/to/backup to make sure its executable.
5. From now on, before editing configuration files you should type "backup <config file>" to make a quick copy that you can revert to.

Note: There are better ways to track your server's configuration changes, but they're not always an option. Sometimes policy gets in the way. Sometimes convincing the other system administrators to use the tools is hard. This solution should work on any system and provides a predictable and reliable tool with a minimum of training and without installing any additional software. I recommend starting with this and, where available, eventually moving to tools like RCS or git.