Damon's questions

I am looking to begin a tape backup regimen and am looking to keep data flowing to the tape drive in a sufficient manner (120+MBs target sustained) but cannot figure out how to do so without a dedicated source drive/array that idles when not writing tapes. The documentation for our specific drive mentions no minimum throughput required.

Enviroment

• Linux Debian writing to tape using mt & tar backing up RAR archives with recovery record, each ~1GB-300GB in size
• LTO-4 Tapes on Quantum TC-42BN tape drive via SAS over external SFF Cable
• Server is used for file backups only, no network services or fileserving.
• MD RAID arrays with data intermittently being read/written in spurts throughout the day/night.

If the source array has significant reads/writes (from scheduled backups) during a tape write, throughput to the tape would drop dramatically even if temporarily. So some questions centered around source array/tape write throughput:

1. I am assuming a sustained drop in throughput to below 10-20MB/s (or less) on the source during a tape write would be a problem?
2. Do I need to have a source guaranteed to have no backups scheduled to it? Essentially 2 arrays minimum; one for backups and one for archives and tape writing?
3. Is there a QOS for drives/arrays that could prioritize the tape writing over all else?
4. LTO-4 tape drives throttle, so is there a common lower throughput limit to maintain for LTO-4 or does it vary widely per drive? Again, documentation mentions max designed speed and "variable speed transfers", but no mention of how variable.
5. Am I missing something in this source-throughput equation, or have unfounded worries?

Update:

I decided to tax things minimally with a single I/O stream via a 600GB archive job reading from the array at about 30MB/s sustained while a tar was being written to the tape from a 4 drive RAID 6 with consumer SATA. The tape definitely slowed to a crawl via listening to the drive but did NOT seem to run out of data or shoe shine. This tells me to NOT expect things to keep up during a full scheduled backup for our hardware configuration but it can handle a less taxing I/O job wile writing to tape.

As of note, the LOT4 tapes must do 56 end-to-end passes so effectively it writes in ~14GB chunks before it stops for some seconds to slow down and then "go" the other direction. I think this helped keep the drive "fed" with data under lower throughput as I have read ahead and async writes set in the stinit.def.

Another note is a read of "dd if=/dev/st0 of=/dev/null" only produced a result of 107MB/s. This, I would assume, is the real-world max effective throughput of this the drive and NOT 120 MB/s. The drive is currently on a dedicated SAS PCIe HBA with no other PCIe cards installed

In the meantime, I setup a 1TB RAID0 as a Disk2Tape buffer and had to add another disk to server to make this feasible.

I would still love to find away to do some sort of QOS for the tape drive and set writing to tape top priority so we can simplify our arrays and reduce parasitic hardaware costs, but in the mean time, I'm not seeing a way to NOT get around having a dedicated disk2tape buffer if I want to ensure continuous writes no matter what scheduled jobs hit the array.

I just recently setup a 3 drive 4TB MDRAID 5 array for mirroring and an online backup of our server.

I am preparing for a future hardware (drive) failure and wanted to mitigate a recovery failure from a URE.

Typically I think of the process for rebuilding an array to be:

1. Remove and replace failed drive.
2. Rebuild array

From my understanding, in a degraded RAID 5 array you can still access data; but when the failed drive has been replaced and a the array is rebuilding, if a URE is detected, the recovery will fail and the data on the array will immediately be rendered unreadable and unrecoverable.

If my understanding is correct then it does not seem prudent to recover the array until all the (readable) data has been duplicated.

This leaves me with a process of:

1. Duplicate data from array.
2. Remove and replace failed drive.
3. Rebuild array

Is there another process that would mitigate rebuild failures (aside from a second drive failure during rebuild)? Is it safe to rebuild array without duplicating the data first? Are my assumptions wrong such as the re build fails on URE but data is still available in degraded state?

Is an effectively nested Xen Server storage repository a problem for stability?

Consider the graphic for the data path of the web server to a physical drive: (Starting on the right)

In the graphic Xen Server and the VM DRBD/NFS server is hosted on an internal USB flash drive. Hard drives are a local MDRAID10 array. The other VMs would be stored on a storage repo via NFS on a DRBD via a VDI attached as a block device on the MDRAID array. VMS are typically Debian. We DO NOT have a dedicated shared storage device (NAS, SAN, etc.).

If performance is not a concern (5-10 low use users for sporadic web access to a company site from the field), will there be stability concerns based on the data path being a storage repository served by another storage repository via NFS on DRBD.

TL;DR

In lieu of dual SANs with supporting networking, we are trying to use DRBD on two servers with local storage to allow for easy manual failover during an outage on the primary server. During an outage the secondary server would become the primary and the VMs could (theoretically) be instantly be fired up with very little configuration. The servers are same RAM and CPU generations even.

Xen seems to have its quirks and I forsee Xen Server having a problem that wipes out the entire server until it gets "fixed" given Xen is running EVERYTHING. I doubt we would have permanent data loss, but storage repositories disappearing happens more often than I would think based on the reading; and unless all your disaster recovery ducks are in a perfect little row, it could take a bit of time to bring things back up correctly if we had to reinstall Xen from scratch while carefully filling in the holes we were missing in our documentation as we went.

With DRBD thought, in the event of a problem we could be running on our backup server very quickly with active file and VM mirrors. Then worst case we could easily just start from scratch on the primary server if need be and not have to worry about "fixing" anything.

What is not shown are a couple more VMS to serve files, but the data would be housed on another SR, so only the VM data itself which is rather static would be served by the same path depicted in the graphic.

Given XenServer (7 currently) is based on CentOS, does that mean it works just like CentOS in terms of updating, CLI, administration (non-Xen specific like mdadm and boot loaders) etc?

Basically, if I want to use XenServer, then am I committing to using, learning, and working in the CentOS "way"?

We have a new (to us) server on the way and now is the time to switch hypervisors and we are set on using Xen.

Our current setup that I am familiar with and can administer efficiently is a Debian host with a couple VMs using Virtual Box which is less than ideal to say the least. Due to this, I am familiar to working in Debian and have made a conscious choice to use Debian for our servers. I administer only our servers for our small business so I do not have the diversity of other setups and other distributions to work from.

From my understanding, the way Redhat does things is a bit different from Debian based distributions and would require a learning curve of unknown amount; but a learning curve for sure.

So if I use XenServer, am I also committing to the Redhat learning curve?

I am aware that I can install Xen with a Debian based Dom0 but the consensus I have read seems to say XenServer works the best overall. However there will be a bit of configuration I will need to do such as getting our local RAID arrays up and running for the Dom0, Xen, and Network Shares, along with getting boot loaders and grub in order. I can do this configuration rather easily in Debian so I am trying to weigh the cost in time of trying to do the same configuration the CentOS way which I am afraid will add a considerable amount of time to get the new server in to Production given IT by myself for our company slowly happens in the afterhours of business; hence the question.

We are trying to eliminate a single point of failure for our files without a SAN. We currently have a single server for file serving (locally), web server, and database (in VM). Looking at using DRBD with GFS2 to create a cluster file system and then run as much as possible from that file system, including the Host OS, VM's, and shared files.

We would like to have two servers setup in a master/slave setup running from a shared source; but the servers would have slightly different hardware configurations (drive port assignments, actual brands and models of main board, NICs, etc) so the hardware configuration files would be kept local (fstab and mdadm for example).

Ideally, one server would act as primary. Package management would happen by the primary server only on the clustered file system because the slave server should not need to if it is using the same data (data from same block device).

The master server would also be the single point of administration for both servers simply because they share the data and configuration files, and the second server simply sits as a redundant set of hardaware to run the software that is configured on server 1 and does so via STONITH, Hearbeat, Pacemaker, etc.

But as I read, it seems as though most people administer the servers independently of each other, and try to automate using cron jobs and rsync.

It also seems that most of the files are fairly static and user modify; that is the number of files/folders that are actively modified by the OS (logs, swapfile, etc) independent of user input and/or dependent on hardware are very small.

We would like to keep the files served by the native OS, (not in a VM).

So now the questions.

1. Can the the software administration be separated from the hardware administration? It seems the only thing that needs administered independently is the hardware on each machine; the software should be managed at once via the clustered file system. (Could be wrong, hence the question)

2. Can the Host OS be ran from a shared set of files by either of the two machines, one at a time? That is, can both machines be ready to administer the Host OS, but only the master does until failover at which point the slave machine picks up the administration work? Or is this assumption inaccurate?

3. Can we set a machine into a slave/idle state until the master goes down? (Suspend cron jobs, suspend services, etc) I am assuming some of this will be a matter of of the fencing software?

(This is not a question of how to run both machines from the same set of data at the same time; they would be effectrively running from the same set of data at different times. The files the servers each need to constantly modify would be on the local file system)

We simply want to setup a redundant server and minimize the redundant administration.

Looking at using DRBD or a clustered files system to help with up-time when downtime strikes in a small business environment.

We currently use a server box for a file server using Linux and samba, then running the web server and Database in a VM. Was looking at adding a second server and putting the files and the VM onto the distributed file system. The base OS is more static and easily can be managed more manually (copy config files at time of change, copy base OS if needed from full backups, etc)

Question is about the fail over scenario if manually done. If server 1 goes down and fail over is manually done, is fail over completed by simply setting the static IP of server 2 to server 1 (again server 1 is down and would be in a state of needing repair), starting Samba, and starting the VM which would have the same static IP's as they had when running on server 1, and starting the backup services?

This sounds like a quick and simple process, almost too simple. Am I missing something? This could easily be automated as well through a script or something that someone with little proficiency could be directed to run in the event of a failure.

Down time if we have a hardware failure could easily be days without the support of on call IT support and the parts needed without a second server, but with the the second server, down time would be at the maximum a matter of hours (if no one is the office proficient enough to perform such operations, minutes if someone was)

Backround: We are in need of a HA server in a small office environment and are looking at DRBD to provide it. We only have about 100GB that needs to be on the HA server and server load will be extremely low. The data will probably increase about 10%-25% per year if we archive older office data, and 50%-75% each year if we don't.

Point is we use a mix of consumer grade and used enterprise grade hardware which WILL be a problem if we don't preemptively plan for it; and pre-built quality servers DO fail, so redundant servers seems like the way to go.

The Plan: We are thinking it would be good to find (2) of the best bang-for-our-buck used servers and synchronize them. We simply need SATA/SAS capable servers and space for as many drives as can be had for the price. These servers seem like they can be had for $100-$200 (+some parts and additional drives) if you catch a deal.

This would theoretically mean a server could fail and if we took days to get to it, as long as we didn't have another coincidental failure, things would still hum along until our IT department (me) could get to it. We would use Debian as an OS.

Some Questions

1. (A) How does DRBD handle drive or controller failure? That is This shows DRBD before the storage driver, so what happens when the controller fails and writes dirty data or the drive fails but doesn't crash immediately? Is the data mirrored to the other server or not and is there risk of data corruption across servers in cases like these?

2. (B) What are the fail points for DRBD; that is theoretically as long as one server is up and running there are no issues EVER. But we know that there are issues so what are the fail modes using DRBD since most of them should theoretically be software?

3. If we are going to have two servers for this, would it be reasonable to run VM's on each with MYSQL and Apache for database and web server replication? (I am assuming so)

4. Is DRBD reliable enough? If not, is the unreliability isolated to certain tasks, or is it more random. Searching turned up people with various issue but this IS the internet with seemingly more bad info than good.

5. If data is being synchronized over LAN, does DRBD use double the bandwidth? That is, should we double up on NICS and do some link aggregation and trunking? Then maybe put them on separate routers on separate circuits and UPS's in separate rooms and now you really have some redundancy!

6. Is this too crazy for an office in terms of server management? Is there a simpler REALTIME alternative (granted DRBD seems simple in theory).

We already have a server. So it seems to me a second USED server with a dedicated drive for DRBD could easily be had for around $150-$250 with some smart shopping. Add a second router, more drives, more NIC's (Used), and (2) UPS's and were talking $1,000 +/-. That is relatively cheap! And I am hoping this would mainly buy us time during a server fault. Drive failures seem like the easier thing to handle with RAID these days. It's other hardware failures like controllers, memory, or power supplies that might require downtime to diagnose and fix that are the concern.

Redundant servers for us means used hardware becomes more viable with more up time and more flexibility for me to fix things when my schedule allows vs having to stop everything to repair the server.

Hopefully I didn't miss that these questions have easy searchable answers. I did a quick search and didn't find what I was looking for.