I have a standalone linux IDS system that I have been putting together. It runs MySQL locally, as well ntop, nagios, base, snort, apache etc. I want to be able to make a backup of the system that has everything from the old system so that I can do a quick dd from a static image of the system and then restore it to the last state.
To my knowledge I cannot DD a live system, which leaves me at a loss as to how to get everything from the server backed up.
I have never used rsync in the past, and I think it may be the solution, but I am not certain that I can use that on a live mysql DB.
I am pretty sure this is going to need a layered approach, but any input would be helpful.
edit:
The systems being monitored are low traffic most of the time, and apache is only for the web UIs for the applications, so its not as overloaded as it may seem
You know, I'm a REALLY big fan of shutting OFF the server and using something like Clonezilla or Norton Ghost to image that sucker.
Even if its 11pm on a Friday night, one a month... you get the idea.
A couple hours of "scheduled downtime" is worth HOURS and HOURS of crash recovery.
Like I tell my assistant, "I'm not saying this because I'm smart. I'm saying this because I WAS stupid at one point and learned my lesson the HARD WAY!"
Thank you for your update. Okay, so no downtime for this server.
Is it possible to build a "warm standby" you can use for the 2-3 hours it will take to clone the drive?
Is taking the network down at 3am on a Sunday morning an option?
Surely there has to be SOME sort of way to schedule some downtime?
I'm just trying to give you a "simple" solution here. :-)
For mysql you can use mysqldump to dump the database contents to a file. Other than that, you usually only need to back up the config files (
/etcand any other directories you modified), any other data directories, log files, and the list of installed packages. You should be able to restore the system by doing a fresh install of the OS, installing the packages from the list, and replacing the data and configuration files.I see two options in your situation:
You can use rsync on a live DB but you could get data corruption. Stop the db, run the rsync, and start it again. You can also dump the DB and rsync that file.
I am really a big fan of using dirvish for backups. It is basically just a perl script that makes taking backups with rsync easy.
For the mysql database you really do need to use the mysqldump utility or something specifically designed for that purpose. Copying database files with rsync is not a good idea.
Usually one can just rsync the file system contents to another location and that's that. The incremental backup should be pretty fast (a few minutes? depending on the bandwidth available to rsync and how often you backup) so you can even take problematic services down for the duration with almost no effect (I did note the "no down time" requirement).
The usual offender for this practice is the MySQL database that doesn't handle snapshotting well. You can tackle this problem by either using Kamil's suggestion of dumping to SQL file and backing up that, or doing some sort of "hot backup". The advantage with the latter approach is that rsync likes it much better - creating a new SQL dump for each backup tends to cause rsync to copy the entire database everytime which is a lengthy process for large databases. using "hot backup", you can take advantage of rsync's ability to copy only changes to the data. Innobase (creators of InnoDB) sell a commercial product to do hot backup to the entire database. I found that as long as you do not have a lot of DB writes, you can get away with
LOCK TABLES, rsync the binary files, andUNLOCK TABLES. when you restore MySQL thinks you had a server crash and recovers nicely (you do want to runmysqlcheckafter recovery).Another method for backing up your server, slightly like Dave's first suggestion, is to backup a standby raid device. This is a very different approach then my first answer so I'm putting it in another answer.
What you can do is to setup your server using software raid (mdraid) with mirroring. Then when you want to take a snapshot, disconnect one raid device (
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1),ddthe contents of the disconnected device of the server, and reattach it to the raid (mdadm --re-add /dev/sdb1). The raid sync will copy all the data that was changed while the device was off.When you restore you can
ddthe same data to both raid devices (its a mirror anyway). The restored image would look, at worst, as if the server lost power during a transaction so you should use a journaling file system and transactional databases anyway.You'd probably want to use a 3 drive system in such a case so your system can handle a disaster during backup, in which case you'd either want to have a 3-way mirror and disconnect the 3rd device for backup, or even have it mounted on a hot-switch bay and just take it away physically as described in this thread (please read the thread through if you want to do it this way - the original poster has some issues with the specific commands and their order)
i'm using
rsync --delete --times --recursive --perms --owner --group --verbose --progress --stats -e ssh [email protected]:/backup1/ /backup1/
on 2nd server on that server i have lvm config so there is very easy to run backup just taking a snapshot you can follow link
http://www.howtoforge.com/linux_lvm_snapshots
Take a look at backupEdge at www.microlite.com
(Too bad they don't have a windows version)
Cloning isn't a viable backup scheme for a production system.
Look for the book "Unix Backup and Recovery" by Curtis Preston. Backing up a live system is possible with lots of commercial applications, as well as open-source applications like Amanda. There's going to be a learning curve, though.
Typically the most challenging aspect of backing something up is recovery. Start planning for recovery first. If you cannot afford to take the system down for backup, you can't afford to restore it eihter.