FileWave and Let's Encrypt

Let's Encrypt offers free SSL certificates. These are usually used for websites, but they can be used for other things. Here I demonstrate how I made them function with FileWave. This removes the need to (a) manually install new certificates every year and (b) pay for those certificates.

For the unfamiliar, FileWave is a tool for managing your endpoint computers. It can send files, run scripts, install programs, update the OS, and other "overhead" tasks for Windows and MacOS. It can also act as an MDM for any of Apple's platforms (e.g. MacOS, iOS, iPadOS, etc.) as well as Android. In order to function properly, it needs to have secure connections between the endpoint devices and the server which coordinates these actions. Usually you would buy a certificate to achieve this and have to replace it every year.

However, Let's Encrypt intentionally designed their system so you could automate the renewals and they don't charge for their certificates. This makes it the perfect tool for eliminating this manual work and reduce your upkeep costs. I run FileWave on CentOS and I use Certbot to automate renewals with Let's Encrypt, so I'll show how I used those tools. If you're running a FileWave server on a Mac, these general ideas should be easily adaptable. The Certbot website gives directions on how to install it on Macs using Homebrew.

First: Install Certbot

Go to the command line on your FileWave server and install certbot. You can find directions on Certbot's website. Specifically, I followed the directions for CentOS 7 and "other" applications.

Make sure that any firewall or packet filtering settings on your server are going to allow Certbot to work. For CentOS 7, I used these commands:

sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --reload

Second: Get A Certificate

At this point, you should be able to get a certificate for the server. Remember that it must have a public IP and a publicly resolvable hostname. Otherwise, Let's Encrypt can't issue it a certificate. To get the certificate, run this command and answer the questions.

sudo certbot certonly --standalone

Assuming your hostname is filewave.example.com, then you'll have certificates in /etc/letsencrypt/live/filewave.example.com. This is fine for some programs, but the FileWave server needs to be "tricked" into using it. That takes a few steps. First, move the original self-signed certificate out of the way. Second, replace it with the certificate that Let's Encrypt signed for you. You can do that with these commands:

sudo -s
cd /usr/local/filewave/certs
mv server.key server.key_bak
mv server.crt server.crt_bak
cp /etc/letsencrypt/live/fielwave.example.com/fullchain.pem server.crt
cp /etc/letsencrypt/live/filewave.example.com/privkey.pem server.key
/usr/local/bin/fwcontrol server restart
exit

At this point, you might be asking why I didn't just use symbolic links. I tried that first, but the dashboard in FileWave Admin claimed that an SSL certificate wasn't installed.

Third: Automate Certificate Renewals

Lastly, to make sure the certificates renew themselves a few weeks before they expire, you'll need to make a script to renew the certificates and move them into place periodically. You could run this every day or every week, as you prefer. You'll need to adjust the script's FQDN variable to be the fully-qualified domain name of your server, but it otherwise looks like this:

#!/bin/bash
FQDN="filewave.example.com"
/bin/certbot renew
cp -uf /etc/letsencrypt/live/${FQDN}/fullchain.pem /usr/local/filewave/certs/server.crt
cp -uf /etc/letsencrypt/live/${FQDN}/privkey.pem /usr/local/filewave/certs/server.key
yes | /usr/local/filewave/python/bin/python /usr/local/filewave/django/manage.pyc update_dep_profile_certs
/usr/local/bin/fwcontrol server restart
exit 0

Save the script at /usr/local/bin/certbot-renew.sh. Also, run "sudo chmod +x /usr/local/bin/certbot-renew.sh" to make sure it is executable. Then make it run every morning by adding this line to the bottom of /etc/crontab:

0 5 * * 6 root /usr/local/bin/certbot-renew.sh

References

Some of the above was put together thanks to things I read from the following sources.

1. https://community.letsencrypt.org/t/script-that-has-been-working-for-years-stopped- working-after-feb/122142
2. https://github.com/nycon/filewave-installer/blob/main/filewaveAIO.sh

SSHGuard

If you run any Unix-like systems, you probably know that the bad guys are trying to break into your server over SSH. You can use fail2ban or sshguard to greatly reduce the chances of a break-in. Below, I show how I setup sshguard on my FreeBSD servers. Its quick and relatively easy, so I consider it a requirement for all of my FreeBSD servers.

First, install sshguard from the ports collection:

cd /usr/ports/security/sshguard
make install

Next, tell FreeBSD to allow it to start by adding this line to /etc/rc.conf:

sshguard_enable="YES"

Then start it and stop it. This will create a configuration file that we'll use.

service sshguard start
service sshguard stop

Now the file /usr/local/etc/sshguard.whitelist should exist. Edit that and add any IP addresses that you might need to whitelist. For example, I whitelisted my network monitor. My reasoning was that it would test if the SSH service was still running every 60 seconds. So this would look like a break-in attempt and the monitor would be blocked after a few minutes.

You might also wish to whitelist certain other hosts, but I don't recommend whitelisting everything in your internal network. If someone did manage to break into your web server or some other system, they could then use "island hopping" to get to this host and break into it, too. So sshguard makes that harder on the attacker.

To add items to the whitelist, just add one IP address or FQDN per line. If you want to insert a comment, you can start the line with a "#" and then write your comment. I highly recommend putting a comment just above every entry. A few years from now, after an IP address is assigned to a different server, you might not remember why it is in your whitelist. Comments can help your future self save time.

Now just start the service back up with this:

service sshguard start

To confirm that it's running, try this:

service sshguard status

That is all it takes. If sshguard sees a suspicious attempt to login, it will add the IP address to the top of /etc/hosts.allow as a "deny" rule. It will take care of things all by itself from now on.

If you find that it blocked an IP by mistake, you can remove the block by just removing its IP from the hosts.allow file. Just be sure that you can really trust that IP. Maybe someone put a rootkit or bot on that host and sshguard is doing exactly what it should be doing. So be confident that its an error before removing the IP.

A few last notes: First, sshguard will log some data in /var/db/sshguard/blacklist.db. As far as I can tell, this is more or less just a log. I think sshguard uses it at startup time, but I'm not sure. If you need to remove the blacklisting from an IP, edit /etc/hosts.allow instead.

Second, there are a few different ways to setup sshguard. One of them involves piping data from syslog into sshguard. Others involve using PF or IPFW instead of hosts.allow. I haven't used those option and don't know the relative advantages and disadvantages of each method. What I've presented here is what works for me. Please feel free to research the options further and do what works for you. If you know why I should consider another method, please leave a comment on this article. I would truly appreciate the advice.

Lastly, don't forget that this is looking for persistent failed logins from a single IP. Advanced Persistent Threats can use botnets to try out one or two passwords from an IP and then one or two from another IP and so on. Patient attackers might also try one or two passwords every few hours until they guess right. People who know you or looked at the password list you keep under your keyboard are much less likely to be stopped by sshguard. The bottom line is simple: sshguard helps reduce risk but security is a mindset and not something any single product can give you. Be smart and be safe out there.

SFTP Connections from PowerSchool

At my job, we store student data in a program called PowerSchool. One of PowerSchool's features is AutoSend. AutoSend can make a text file full of data and send it to another computer over SFTP. This is very useful, as it allows student data to be entered once (in PowerSchool) but appear in many systems.

Recently, I replaced and updated the FreeBSD system that runs our SFTP server. After the upgrade, PowerSchool couldn't send data to the FreeBSD SFTP server. Other SFTP programs, such as FileZilla, were able. This issue only seemed to affect PowerSchool's AutoSend. I couldn't figure it out at first. The FreeBSD community couldn't. Our tech support couldn't. They escalated the issue over and over until it reached "engineering" and I never heard back from engineering.

After working on this on-and-off through the summer and eventually found a way to make it work. Since it took me so long and confused so many other people, I wanted to put this out there for others to benefit.

The short version is this: I had to change the default settings of the SSH server.

This may sound strange to some, but SFTP on many Unix systems -- such as FreeBSD and Linux -- is based on OpenSSH. OpenSSH is a system that is all about making encrypted connections and preventing the bad guys from seeing what you're transmitting. However, its default settings moved to a more strict standard at some point and this was why I saw a difference between the old FreeBSD server and the new one.

The setting in question deals with how OpenSSH handles authentication. In plainer language, it's all about the login process. It seems that "keyboard-interactive" is the mode used by programs that interact with humans, such as FileZilla. However, "password authentication" is the kind used by automated systems, such as PowerSchool. Personally, I found the name "password authentication" to be confusing for a while, but that is what it is called.

So I edited /etc/ssh/sshd_config to allow the password authentication system. However, the OpenSSH developers disabled it for a reason. They know more about computer security than I do, so I struck a compromise. I changed the settings so that password authentication was valid if and only if the connection came from my PowerSchool server.

If you need to do this, just find your system's copy of sshd_config and add the following lines to the bottom of the file. If you have any lines beginning with "Match", this should go immediately above those.

# Turn off PasswordAuthentication in general, i.e. without a Match statement to change it.
PasswordAuthentication no
# Allow only the IP address of the PowerSchool server to use the
# PasswordAuthentication directive. Note the "PasswordAuthentication no" 
# above all Match statements.  That is part of this configuration, but 
# must be before any Match blocks.
Match Address 999.999.999.999/32
        PasswordAuthentication yes

You should change "999.999.999.999" to the IP address of your PowerSchool server. Everything else should be fine exactly as I wrote it above.

Backup Configuration Files

Anyone who manages a Unix-like system has edited a configuration file from time to time.  At an absolute minimum, you should copy these files before editing them.  Otherwise, you could get through a few lines of editing, break something (or just want to go back to your old setup), and not know how to go back.

Here is a quick trick I came up with for handling this.  There are better ways, but this can be dropped right into any Unix-like system with very little effort. That easily includes MacOS, Linux, and FreeBSD.

First, I started with this trick from http://osxdaily.com/2012/06/11/make-a-quick-backup-of-a-file-from-the-command-line/.

cp file.txt{,.backup}

This will copy file.txt to file.txt.backup .

Unfortunately, the next time its done, file.txt.backup will be overwritten and the history of our changes over time will be lost.

So I decided to end the backup files with the date. Done correctly, this will alphabetize them into chronological order and allow some rudimentary change-tracking in the future using the diff command. So let's look at this:

date "+%Y%m%d%H%M%S"

It outputs the current date and time in YYYYMMDDHHMMSS format. In other words, it prints out the year, then month, then day of month, then the time. So 9:59am on July 11, 2014 will come out as 201407110959.

Note that this is done with "padding zeros" to make sure that it alphabetizes into chronological order. You can see it in the "07" for July, which will now come before December. This is because 07 is alphabetically before 12, even though July and Jul are alphabetically after December and Dec.

So using that and the cp command from earlier, we now have this.

cp file.txt{,.`date "+%Y%m%d%H%M%S"`}

This copies file.txt to file.txt.<timestamp>. Using the example date above (9:59am on July 11, 2014), that would be file.txt.201407110959. This is the effect we want.

In a bash shell script, "$1" references the first string after the command, as parsed by whitespace delimiting.

So using that, we get this shell script:

#!/bin/bash
cp $1{,.`date "+%Y%m%d%H%M%S"`}

Let's put that in a file called script.sh and then make sure its executable with chmod 755 script.sh. This allows "./script.sh file.txt" to copy file.txt to file.txt.<timestamp> without having to manually calculate the timestamp.

Name this "backup" and copy it to your ~/bin, /usr/local/custom, or some other location in your $PATH. Then set it to chmod 0755, if you haven't done so yet.
The next time you have to edit a configuration file, just type "backup config-file.txt" before you edit it.

So, to re-cap:

1. Make a file called "backup" and put the above two line script in it.
2. Put "backup" someplace where it will be in your $PATH, such as ~/bin or /usr/local/bin or /usr/local/custom.
3. Make sure that location is actually in your $PATH.
4. Use chmod 755 /path/to/backup to make sure its executable.
5. From now on, before editing configuration files you should type "backup <config file>" to make a quick copy that you can revert to.

Note: There are better ways to track your server's configuration changes, but they're not always an option. Sometimes policy gets in the way. Sometimes convincing the other system administrators to use the tools is hard. This solution should work on any system and provides a predictable and reliable tool with a minimum of training and without installing any additional software. I recommend starting with this and, where available, eventually moving to tools like RCS or git.