Building and Restoring a MIAB (Mail in a Box) Server

Last Updated: 31-December-2021 (editorial, added Resources)

Why write this up?

Surely this is documented elsewhere. Sure, it is, but as I worked through this on my own, I discovered that the official MIAB guide is missing a key point on achieving a successful restore/move of an MIAB instance.

MIAB's "Moving to a New Box/Testing Backups" is where you start, so that's where I started this journey. For completeness, I'll repeat the steps given in the MIAB maintenance guide, but make some notes along the way.

Why I'm doing this

I've been running "my own" mail server on a Digital Ocean VPS for a few years now, and I have no complaints with DO at all. However, you never know if your backup is any good if you don't test it, and there's not much more than we rely on these days than a reliable email service.

Since I'm now the weakest link in my email services, it's way past time that I kick my own tires. To add a little more spice to the test, I decided to set up the new location for the mail server on a Linode VPS, rather than on DO.

Here's what we'll do

The order of the day is as follows: * Secure the original server (check for any system or MIAB updates, close the front door to any inbound traffic, and make one last backup) * Spin up the Ubuntu new server * Do the necessary DNS things on the Domain Registrar to no longer point to the DO server but now to the new Linode server * Install MIAB on the new server * Restore all of the MIAB data to the new server (and there is a key GOTCHA in this step, which inspired me to write this post), and wait on DNS propagation. * Test a few things, and wait...

OK, let's lock down the old box

I keep a pretty clean server, but you never know when an update will come through and the new server will be all updated - so let's check anyway.

Update the old system

suser@box:~$ sudo apt-get update && sudo apt-get dist-upgrade

Nothing today, onward with closing the firewall to all inbound traffic except ssh 

sudo ufw reset
sudo ufw allow 22
sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
suser@box:~$

I edied out some of the feedback, for brevity, but that last bit was true. This killed my ssh session, so I log back in... no problem.

Backing up your old MIAB's data

MIAB offers many backup options; local to the machine (and then you need to figure out how you want to copy the backup data files to somewhere else for safe keeping), or remote to some other server. Note: MIAB added a BackBlaze B2 option a few releases ago, but it simply does not work. Someone did some digging, and it turns out to be a problem in the MIAB implementation of the protocol. I use rsync to another server. 

suser@box:~$ cd mailinabox
suser@box:~/mailinabox$ sudo management/backup.py
[sudo] password for suser: 
suser@box:~/mailinabox$

There was ZERO feedback during that command. You'll think that nothing is happening or you typed in your password incorrectly. It's probably working. BTW, it simply executes the current "Backup" settings on your MIAB. In my case, I was logged in to my Backup destination server and observed that additional backup files were created/transferred, so I'm good...

Of course, make sure that you have your backup secret key available. It is located at: 

/home/user-data/backup/secret_key.txt

on your server, and you should already have a copy of its contents somewhere safe. You will need it to restore the backed up data on your new server.

At this point, our old MIAB server is still running but not accepting any connections, we have made a final backup, and we're ready to build our new server.

Building the new Server

The MIAB "The Machine" Installation Guide is our reference, and there's really nothing to add to that guide, but I'll make some notes here along the way on some specific choices I made.

Throughout this guide, I'll use "box.mydomain.com" as my MIAB box name.

I'll be using Linode as my hosting provider in this example.

From the Create Linode Screen, I choose the following:

  • Ubuntu 18.04 LTS (MIAB v55 is not supported on LTS 20.04)
  • Region - same region as I use for my other Linodes.
  • Plan - Shared CPU, Nanode 1Gb RAM, 25Gb storage(currently the smallest plan) at $5/month
  • Label - setting it to "box.mydomain.com"
  • Root Password - Required by Linode. I will disable it later.
  • SSH Keys - selected my default set of admin SSH keys already established with my Linode account.
  • Attach a VLAN - not necessary, but if you want to do so, and you want to host in the USA, you have one choice for your Region(see above); Atlanta.
  • Add-ons; Backups, of course. Private IP - for use with or without your VLAN if you want.

Hit the "Create Linode" button and you're off to provisioning...

Getting the Server ready for MIAB

Kill that root password and name your server

I already have established SSH keys, so we'll use those to scoot on in to the shiny new server and disable the Root password that we were required to enter above. 

ssh root@170.187.158.22
passwd -l root
adduser suser
usermod -aG sudo suser
apt-get update && apt-get dist-upgrade

These commands log me in as root to the new server using ssh, disable/epire the password on user root, add a new user to the system that I will use as an 'sudo' user, and then run the system updates - there will be many updates.

While the updates are running in one shell window, I open another terminal on my workstation to push my 'suser's ssh key up to the corresponding account on the new server.

ssh-copy-id -i ~/.ssh/id_rsa.pub suser@74.207.227.185

Once the updates are complete, I log in using my 'suser' ssh keys to the server:

ssh suser@74.207.227.185

The Message of the Day will tell us that the system needs to be rebooted (because of all of those updates), but let's do a few necessary things while we're here. 

sudo hostnamectl set-hostname box
sudo hostnamectl set-hostname "box" --pretty
sudo shutdown -r now

This is the best way that I know of to set the hostname on a new server, followed by a reboot.

(optional) Tighten up SSHD a little bit

We already disabled the password on the 'root' account, so now let's do a few more things to secure ssh a little more on the server, including disallowing 'root' from logging in via ssh.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo vi /etc/ssh/sshd_config

I then changed the following settings in sshd_config with the values below: PasswordAuthentication no PermitEmptyPasswords no PermitRootLogin no MaxAuthTries 3 MaxSession 5

Saved and exited vi and restarted the ssh service. 

sudo /etc/init.d/ssh restart

MIAB takes care of the firewall, but for now, we need to make those DNS changes on the Domain Register so that hopefully by the time we have MIAB installed and our data restored, the Glue Records and server A record will be pointing to our new IP addres that will serve box.mydomain.com.

Glue Records and rDNS

I go to GoDaddy and change my three name records for the 'box' server to the new box's IP address, as well as the two nameservers to that same new email address. This part is probably more interesting visually because of the non-obvious way that GoDaddy hides the place you need to go down under some ellipses.

Fundamentally, you need to update three A records; box, ns1.box, and nx2.box for your mydomain.com domain at the registrar. Set them all to point to the IP address of your new IP address; in my case, 74.207.227.185.

( insert pictures here and some description of/links about Glue Records ) This will take some time to propagate.

Reverse DNS (rDNS) settings.

It is important to set your Reverse DNS in the Linode control panel, however, the Linode control panel will not allow you to set it to box.mydomain.com until Linode can resolve that name. This can only happen once the new DNS records have propagated a bit more.

Install MIAB on the new server

This part is really easy. It's why MIAB exists the way that it does. Just log in to to the new server and run the following:

curl -s https://mailinabox.email/setup.sh | sudo -E bash

I rebooted here - just because. At this point, however, I'm still waiting on the DNS changes to propegate and the Reverse DNS instructions don't seem to be working, probably because the name servers aren't updated yet...

The server is now installed and we are finally ready to restore the data from the old box.

Restoring the backup data to the new box

I'm sure there are better ways to do this, but my backup files size isn't crazy, and rather than add any new credentials/permissions, etc. to my world, I just copied the MIAB backup data to an intermediate location (my workstation) from the server where I had been making the automated backups. So, from my workstation, I ran the following command to copy the backup data down to my workstation: 

rsync -avhe ssh --dry-run suser@some.otherserver.com:/home/boxbackup/boxbackups .

BTW, I love to use the --dry-run option to make sure got the comannd right before dumping a bunch of files somehwere I don't want them to go. Of course, remove that bit before actually running the command. :)

and then pushed it up to the new MIAB server to my 'suser's home directory for now:

rsync -avhe ssh --dry-run boxbackups suser@74.207.227.185:/home/suser

Pay attention here, kids

Before you run the command to restore the MIAB backup data to the new server, you'll need to do two things: * update /home/user-data/backup/secret_key.txt with the contents of that file from the original server or you won't be able to decrypt the backup data. * DELETE the ssl data generated by the new MIAB installation because the restore process will fail if anything is in that ssl directory.

There WILL be data in that directory as it was created by the MIAB install process. This folder MUST be empty before you restore from the backup, or you'll have to install the new box again; from scratch.

To clear the ssl files, do this:

sudo rm -rf /home/user-data/ssl/*

and then you can successfuly restore the backup data using these commands:

export PASSPHRASE=$(sudo cat /home/user-data/backup/secret_key.txt)
sudo -E duplicity restore --force file:/home/suser/boxbackups /home/user-data

That should all success without error. If so, re-run the MIAB installer to stitch the system back together.

sudo mailinabox

After a while, and the usual dialogs (nothing to change in there), you should see the following familiar text...

-----------------------------------------------

Your Mail-in-a-Box is running.

Please log in to the control panel for further instructions at:

https://74.207.227.185/admin

woohoo!

Testing a few things

At this point, I was still having some difficulty with the Reverse DNS entries, but that was the only issue. After 15-20 more minutes, probably about an hour after I first changed the "Glue" records on the domain registrar, Linode did not complain when I set the Reverse DNS records there for both the IPv4 and IPv6 addresses. On the box, every now and then the Status page would complain that the IPv4 rDNS value was still the old 'slug' provided by Linode, but eventually things settled down.

I sent myself some emails, and sent some outgoing emails as well. Linode struggles with spammers just like all VPS service providers do, but as of this writing, Linode blocks/allows SMTP traffic to Linodes on a per-account basis; not per-server. Fortunately, I had already gone through the approval process on a separate MIAB server with Linode, so there was nothing to do from my side.

I also used a few tools to check that my new server was running acceptably. All seemed good!

Resources

  • MXToolbox - Simple check using "https://mxtoolbox.com/emailhealth/mydomain.com/". That site has a lot of very specific additional tests and monitors that you can explore.
  • Mail Tester - Send an email from your server to a randomly-generated mail-tester.com address and see how 'spammy' your server and mail content looks to them.

Until next time...

social