Thursday, November 3, 2016

Rear- Linux Disaster Recovery Solution

Rear” (Relax-and-Recover) fits perfect in implementing a bare metal disaster recovery solution & image migration as well.  Rear is the leading Open Source disaster recovery solution. It is a modular framework with many ready-to-go workflows for common situations.

Most backup software solutions are very good at restoring data but do not support recreating the system layout. It is very good at recreating the system layout but works best when used together with supported backup software.

In this combination Relax-and-Recover recreates the system layout and calls the backup software to restore the actual data. Thus there is no unnecessary duplicate data storage and the Relax-and-Recover rescue media can be very small. There are a lot of articles or documents available for "Rear", hence, this documented I created is for my reference, but available publicly, so if anyone finds it helpful then I'd say "I'm glad".

As per “Red Hat” it is officially supported from RHEL7.2 on-wards, however, relax-and-recover web site got rear packages for RHEL5/6 as well. I was able to trace “rear” package in the ISO image of RHEL 6.8 version.
With this solution "sys admins" should be able to recover a failed system with minimal downtime. Saying so, “Rear” is not a backup solution rather it is recovery solution. Hence, external precautions and procedure should be followed up to take data backup.

There are two main steps involved in this process:-     

  • Backup system
  • Recover system

Let’s see how we could ‘backup’ and ‘recover’ system using rear now.  

Scope: This document is mainly for Linux systems and intention is recovering a failed system, again, not a backup solution.

Testing Environment: This solution would be tested on RHEL 6.7 virtual system running in VMware Workstation 11. Backup of the system image would be stored on RHEL 6.8 virtual system in same network.

Step 1: Backup System

In this step, I’m going to take system backup using “rear mkbackup” command. This is a test VM (virtual machine) which I’d destroy later and then try to recover using “rear” solution which would be explained in step 2.

- A snap of this system:

- Install “rear” package.

This is not available in the ISO image, hence, I had to download it from the site and also available in EPEL, and other resources.

- Once installed, need to edit /etc/rear/local.conf file and make necessary changes to tell the system about backup method being used, backup server etc. So, I’ve made the changes to this file as shown below:-

Here, the backup would create a bootable ISO image file in the format "rear-$(hostname).iso", and it is using default “NETFS” backup method (internal archive method being used by rear), and finally the backup files including the ISO files are stored on the remote system as mentioned in the parameter “BACKUP_URL” via NFS protocol.  This option is mandatory when backup method is specified as “NETFS”.

Refer this link for more details on the other options:-

- Let’s start making backup of the system now. At this stage, we are ready to run the command “rear -v mkbackup” to start making backup of our system as shown below:-

Depending on the amount of data to be taken as backup and network speed, it would take some time to complete. When checked on the server where this backup is stored, found that the total size of backup was 1.1 GB as shown:-

- Now, I'd delete the boot files from /boot partition of this system "test" to make it non-bootable.

- After the reboot, the system dropped into grub shell and unable to boot up. Since I had removed all the boot files it is difficult to get those files backup, hence, I’d need to restore using “rear” now.

Step 2: Recover Failed System

Now, at this stage, I’d need to boot the failed system using the bootable ISO file that was created in the earlier stage. So, let’s mount this ISO image and boot the system up.  When checked on my system this file was size of 73 MB as shown here:-

Boot the system with this ISO file. Once systems comes up, screen should look similar to this one:

- So, go ahead and select “Recover” and hit enter to start the recovery process. This would check and build the necessary layout for the system and drops into rescue shell.

At the shell, need to run the command “rear recover”. After a few minutes, systems failed to recover and shown the below error message:-

When checked and found that it was because dynamic IP assigned to this system running in rescue, hence, I changed it to static IP after which I was able to recover successfully.

Starting recovery…….

End of recovery…..

At the prompt type “reboot” and system should be able to reboot without any errors. File system re-labeling would kick off since SELinux is in enforcing mode.

Once system is up, those files that I had stored under /data was there as before:-

That's all :)


Allington Howsham said...

Exactly.. disaster recovery amazon
is highly available and scalable internet storage which is designed to concurrent loss of data in following facilities including buckets and has web service API’s to allow for S3 interactions.

Anonymous said...

can you please provide steps for mounting the iso file and booting the system with the iso file? having a really hard time with this.

Thanks in advance

SimplyLinuxFAQ said...

You can't boot from an ISO image mounted on running system, if you really wish to use the same image to boot then move the image out of the system and then boot from it. Please explain in more detail what you wish to accomplish here. Send your query with details to this email address :

Koti Deva said...

nice information,

Thanks and best regards: Software development company

Matthew Hayden said...

its really good source of information as after reading this post i come to know about this story.....Managed IT Services