Questions[backupexec](server)

What is an acceptable level of hard write errors on tape? Specifically, what is acceptable on HP LTO-2 media? Is it a hard number of errors, a ratio of hours in use to errors, or something else entirely?

Further background

We are using a MSL6000 library with one LTO-2 drive using Backup Exec 11d (for now). Backup Exec always shows some soft errors for most of the drives, but some are starting to show hard errors. Backups are done with immediate verification, and the verify has yet to fail, so I don't have reason to be alarmed right now.

While I can find the duty cycle for the drive (250,000 hrs), I can't seem to find any hard numbers as to when a particular tape should be should just be retired.

If there's a best practice for rotating media out, I'd love to hear that, too. We're also soon migrating to LTO-4 media, so thoughts on errors there would also be helpful.

Edited to add:

I don't have hard error on every tape. To give an idea of what I'm looking at:

Tape    Hours in Use    Hard Errors
A       142             11
B       255             0
C       159             2

The vast majority of my tapes are like B and C. A is the outlier.

I'm looking for some sort of best practice here. The tapes are verifying OK. I don't want to have a tape fail just when I want to restore, but I also don't want to toss a tape with a handful of errors if it's unnecessary.

A bit of a mashup of systems here, so bear with me. Essentially, I'm having some trouble using the Backup Exec agent for Oracle, while trying to backup a remote Linux server. The BE agent appears to use RMAN to backup the databases

The backup server is on one VLAN and the target server on another, with a Cisco ASA firewall providing the only link between them. This is by design, as the backup server is to support numerous clients and each client must be on its own VLAN to prevent them from accessing each other. I have added the recommended ports to the firewall to at least allow the agent to talk to the media server.

The backup starts well enough (indeed a smaller Oracle database on the same server completes without issue) but a 200GB database, which would clearly take a few hours to complete, is not able to complete.

I believe the problem to be related to http://www.symantec.com/business/support/index?page=content&id=TECH59632, which says that a CORBA session is established on port 5633 at the start of the backup and used before each RMAN operation but, while data is being transferred, the CORBA session's socket receives no packets. Since the connection timeout on the firewall is 60 mins, the CORBA session is dropped and, when the RMAN agent tries to perform its next action, the whole process bombs. Symantec say this problem was fixed in an earlier version of Backup Exec, but do not detail any additional settings to enforce it.

Setting the connection timeout on the firewall to something high-enough to cover the backup window (e.g. 12 hours) seems like the wrong thing to do, as it is an estate-wide change, which would also affect the connection lifetime of (for example) web requests to another client's web server.

Moving the Linux server into the same LAN as the backup server is out of the question.

I'm not a Linux guru, but I roughly know my way around. So far, I have tried starting using libkeepalive (http://libkeepalive.sourceforge.net/) to force the beremote process' socket creation to be made with a KEEPALIVE TCP flag, but a quick netstat -top indicates that it is not taking. Either I'm using libkeepalive incorrectly, or it doesn't work for the beremote binary

I guess I am looking for an option that fits with the environment I am in. I figure I'm looking for one or more of the following:

• a way to configure the BE agent to keep the connection alive?
• a way to inject the keepalive flag to the existing TCP connection (e.g. via a cronjob)?
• a way to tell the Cisco device to increase the connection timeout for a specific source/target (maybe a policy-map)?

Any/all (other) ideas welcome...

J.

RE: Comment by @Weaver

As requested, class-map, policy-map and service-map entries...

class-map CLS_INSPECTION_TRAFFIC
 match default-inspection-traffic
class-map CLS_ALL_TRAFFIC
 match any
class-map CLS_BACKUPEXEC_CORBA
 description Oracle/DB2 CORBA port for BackupExec traffic
 match port tcp eq 5633
!
!
policy-map type inspect dns PMAP_DNS_INSPECT_SETTINGS
 parameters
  message-length maximum client auto
  message-length maximum 1280
policy-map PMAP_GLOBAL_SERVICE
 class CLS_INSPECTION_TRAFFIC
  inspect dns PMAP_DNS_INSPECT_SETTINGS 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ipsec-pass-thru 
  inspect icmp 
  inspect snmp 
 class CLS_BACKUPEXEC_CORBA
  set connection timeout idle 1:00:00 dcd 
 class CLS_ALL_TRAFFIC
  set connection decrement-ttl
!

I currently have ~12TB of data for a full disk to tape (LTO3) backup. Needless to say, it's now requiring over 16 tapes so I'm looking at other solutions. Here is what I've come up with. I'd like to hear the community's thoughts.

• Server for Disk-to-Disk
• BackupExec 2010 Using De-duplication Technology
• 20+TB worth of SATA drives
• LTO5 robotic library connected via SAS
• 1Gbps NIC connected to network

What I envision is doing a full backup of my entire network which will initially take a long time over the 1Gbps NIC but once the de-duplication kicks in backups should be quick. I will then use the LTO5 to make disk to tape backups and archive those accordingly.

What does everyone think? Any faster way of doing the initial full backup over the 1Gbps NIC? What will be my pain points? Is there a better way of doing what I'm trying to achieve?

OK. some givens to factor in

I use Backup Exec 12.x/13.x, Have server 2003/2008 environment including Exchange.

I have Backup to Disk (Full/Diff) happening that is independent of the backup to LTO (Full/Diff). For more than one reason I'd rather not just go to backing up from disk to tape I'd like to keep the backup direct to LTO happening.

I currently have a single LTO-3 drive with no sort of loader/robot/library. The box serving the LTO drive has an Adaptec 39160 Ultra160 SCSI card in it. I currently use one tape for Full (one per week) and one tape for Diff (four days a week before the tape is taken out). The Full backup is bumping up against the 372.5GB barrier and when it does the backup doesn't finish Saturday it's still waiting for a tape on Monday morning.

Ward mentioned putting the second LTO3 full tape in on Monday afternoon/evening after normal business hours. The problem with this is compared below:

Normal flow

• Friday insert LTO3 tape 1 for Full backup for week 1
• Monday insert LTO3 tape for differential
• Tuesday, Wednesday, Thursday differentials use tape that was inserted Monday
• repeat for week 2

2 LTO3 tapes for Full backup flow

• Friday insert LTO3 tape 1 for Full backup for week 1
• Monday insert LTO3 tape 2 for Full backup for week 1
• Monday insert LTO3 tape 1 for Full backup for week 1 (for verify process)
• Monday insert LTO3 tape 2 for Full backup for week 1 (for verify process)
• Tuesday insert LTO 3 tape for Differential
• Wednesday, Thursday differentials use tape that was inserted Tuesday
• repeat for week 2

The extra tape swaps eat 6+ hours into Monday (starting from the time I put in the second tape). If I did that at 5PM I'd be here until almost midnight swapping tapes. That's not counting the idle time on Sat/Sun/Mon waiting for a tape.

Now I could turn off the verify process and save two tape swaps and shorten the "backup" process by several hours but I can't just put tape 2 in and walk away at the end of the day if I don't turn verification off. Having the backup spill over onto a 2nd tape lengthens the backup process but it also

• Increases the number of tapes in the rotation (cost)
• Increases the number of tapes in transport (size/weight of briefcase going to off-site storage)
• Increases the complexity of the backup process by making me stay on site after hours for verification process
• Increases the complexity of managing backup/restores from my office which is not right next to the server room. This goes quadruple for dealing with such issues from home.

And yes I'm not going in on Saturday to sit there for 6+ hours and babysit the tape drive. I'd like to have a life outside of work. 12 hour days M-F are bad enough when they happen. I'm not going to permanently tie myself to a 6 day workweek.

The tape drive is a Dell PowerVault 110T LTO3. The backup server is on Gigabit Ethernet using only a single NIC and can fill a full tape in about 12 hours.

I can change the backup process to separate one of the more intensive servers to a full backup on its own LTO to temporarily hold this decision off but soon I think I'll need to choose one of these options:

1. Buy a LTO-3 drive and take advantage of just having a second physical tape available. This is a less desirable option and only makes sense if LTO-3 drives are considerably cheaper than LTO-4 drives which is not the case.

2. Buy a LTO-4 drive and use the LTO-4 tapes for full backups and use LTO-3 tapes for differentials until the LTO-3 tapes get rotated out and new LTO4 tapes match the price of LTO3 tapes. This will probably get me through the weekend backup for years to come without having to swap tapes. This also partially addresses shoeshining since LTO4 has lower minimum speed than LTO3.

3. Buy something that can feed tapes in automatically. I'm assuming there isn't something I can add to the PowerVault 110T and this would mean a purchase of a new device that has the tape and loader in a single unit. This is probably not cost effective vs just getting a drive and manually loading tapes but going autoloading LTO4 would be the ultimate in convenience. I'll let the boss above me decide between single tape drive and autoloading drive.

Evan Anderson mentioned in another solution that you could buy drives around this price range

 LTO-4 (internal drive, 1 tape / day) - $2,766.00  
 LTO-4 (autoloader, 1 tape / day) - $4,566.00

but I don't know specifics on what he or you would recommend for the actual drive and if necessary controller. Show me a newegg URL (or Dell, or HP, or whatever your favorite vendor would be) for your solution if you don't mind looking it up or just give me a brand and a model number and I'll be glad to do the leg work myself.

I'm looking to make a needed purchase some time down the road before this backup rotation gets to be too cumbersome. I probably have a few months.

Xenny mentions the age of the servers and speed of backup. The Exchange server is 6 years old (though the hard drives are much newer). There are a couple of 4 year old servers in the mix with consumer grade sata drives (WD6400AAKS). Servers I consider "new" are 2 years old at this point.

Backup to disk from the old exchange server has been as fast as 2184 MB/min but in general backup to disk is just as slow as backup to tape in this setup. In fact backup to disk is sometimes slower than backup to the LTO-3 tape drive. I've also had issues with drives failing and lack of bays to add more drives. In general backup to disk is even more of an issue than the LTO3/4 transition but that belongs on a different question on serverfault if I wanted input on that subject.

I'll just pick some numbers from a recent backup to give you an idea on speeds. This is not a complete list but gives you an idea on the variety of speeds involved. I plan to update this soon in the format of oldspeed MB/min newspeed MB/min where oldspeed is the old SCSI 320 LTO3 and newspeed is the SAS LTO4.

DC C: ~850 MB/min
DC system state ~700 MB/min
Exchange Server C: and system state ~500 MB/min ~600 MB/min
Exchange Server D: ~1400 MB/min ~1200 MB/min
Exchange Server First Storage Group ~1100 MB/min ~700MB/min
Webserver C: ~600 MB/min ~950 MB/min
Webserver E: ~1700 MB/min ~1950 MB/min
Fileserver C: ~500 MB/min
Fileserver E: ~1500 MB/min ~2200 MB/min
Fileserver G: ~1800 MB/min ~2400 MB/min
Fileserver system state ~650 MB/min
faxserver C: ~400 MB/min ~550 MB/min
Accounting server C: ~1300 MB/min ~1775 MB/min
Accounting server D: ~1500 MB/min ~2250 MB/min
Accounting SQL instance ~1600 MB/min
application server C: and system state ~700 MB/min ~900 MB/min
backup server C: 700 MB/min ~1800 MB/Min
backup server E: 1350 MB/min ~2900 MB/min

Monitoring the Fileserver I saw numbers that make me think the raid controller is holding back the transfer rates. The controller is SATA 1.5 but the drives are 3.0 capable. I noticed after changing volumes from RAID 1 to RAID 10 and getting no increase in speed for the backups. Unfortunately doubling the sustained read speed had no affect on backup to the LTO3 tape drive.

In general backup straight to LTO gives me a decent benchmark of where my servers are I/O limited. The servers that are backing up below 1500 MB/min are generally slow disk wise and the ones between there and 2400 MB/min are still low hanging fruit. For example the Exchange 2003 server is getting low on disk space and continues to expand the database for the First Storage Group out to slower portions of the disks. This server will be replaced with a Exchange 2010 server with faster processors and more disks. The other servers will get disk upgrades and/or SSDs added.

http://en.wikipedia.org/wiki/Tape_drive mentions "When shoe-shining occurs, it significantly affects the attainable data rate, as well drive and tape life." but it doesn't mention shoe-shining reducing effective capacity of a tape. After looking at archival tapes from the bank I can confirm about 2% to 15% space wasted on the LTO3 tapes. Nowhere near enough to keep me from moving to LTO4 or an autoloader but it could be significant. For those of you with Backup Exec you can calculate your shoeshining waste by:

• Making a backup job that will backup around 100% of the tapes native capacity without compression. Disable compression on the drive and software when running the test.
• look in the media tab of backup exec and compare the "used capacity" column to the "Data" column. If compression is off and the numbers match you aren't shoeshining at all.

In my case I had a archival LTO3 tape with 272.4 GB "used" but only 233.67 GB "data" and another with 400.6 GB versus 395.19 GB. I also tried a backup to LTO4 without compression and got 833 GB "used" with only 786.77 GB "data". Obviously the shoeshining will vary from my environment to yours but before this I didn't think to test it. Hopefully this will make it clear to you how to figure out how much wasted tape you have in your backup environment.

edit: new info at http://www.fujifilmusa.com/shared/bin/LTO_Overview.pdf showing minimum tape speeds for LTO3 and LTO4. It looks like the IBM LTO4 actually has a lower minimum speed than the IBM LTO3. Either way my average server is too slow to feed LTO3/4 without shoeshining. I'm concerned even my backup to disk local volumes will be too slow to feed the drive quickly but I'll have to test that.

Pulling IBM full height drive info from PDF above I get

LTO4 : 30-120MB/s 800GB native (45-240MB/s compressed)
LTO3 : 40- 80MB/s 400GB native (60-160MB/s compressed)
LTO2 : 18- 35MB/s 200GB native (27- 70MB/s compressed)
LTO1 : 15- 15MB/s 100GB native (30- 30MB/s compressed)  

Update: The server I was using for backup started giving me stop errors so I moved the tape drive to another server. The old SCSI controller was an Adaptec 160 the "new" controller is a LSI based 320 (at least I assume the external connector is a 320 as the 4 hard drives inside the server mention 320 SCSI in the server management).

The new server situation leaves me without backup to disk temporarily until I get an external enclosure for direct attached storage. In general this LTO discussion has pointed me towards buying more hard drives for my servers. I will have work to do reconfiguring RAID arrays to increase the speed of the backup and hopefully increase the reliability of the overall setup.

Update 2:Comparison below uses a old fileserver whose raid controller bottlenecks all transfers at ~40MB/s so ideal would be about 2400MB/min. This is about the speed needed to test the edge of shoe shining. Presumably the data flow will not be perfectly regular and will force speed matching almost all the way through the test.

I no longer know the buffer size and buffer count I used on the speed test of the old LTO3 drive but it doesn't change it much at all I got maybe 100MB/min gain by tuning buffers. The test data is about 20GB of scanned tifs and jpgs. I did these tests on a Friday afternoon and I didn't repeat the tests enough times to average the data or otherwise weed out invalid data. Testing after hours, choosing different data, and other variables could noticeably affect these tests.

The same servers are used in all tests. The old drive is on a 320 SCSI LVD controller that is PCIx. The new drive is on a PCIe LSI 3801E SAS controller. It is possible that the drive controller and/or the LTO3 tape drive are bottlenecks. I won't be testing the individual components, only the old pairing vs the new pairing. The server running Backup Exec has 4GB ram, 32bit Server 2008 standard, Pentium D 3.2GHz dual core CPU.

Network connectivity is by way of a 1Gb switch both servers are on the same switch. I have a Remote Desktop Connection open but with the backup going + that connection the Gb connection is less than 50% utilized at worst and averages more like 25% usage.

So as rough as the test methods are I feel reasonably confident that the bottlenecks aren't in a variable that I'm ignoring.

Short Test Results:
~1500 MB/min using Dell LTO3 drive and LTO3 tape compression ON, 64KB block size (many buffer counts tested, best result listed here)

~1800 MB/min using Quantum Superloader3 LTO 4 drive with a LTO3 tape (same tape as above) compression ON, 64KB block size, 64KB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON

~2150 MB/min using Quantum Superloader3 LTO 4 drive with a LTO3 tape (same tape as above) compression ON, 256KB block size, 256KB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON
~2200 MB/min using Quantum Superloader3 LTO 4 drive with a LTO3 tape (same tape as above) compression OFF, 256KB block size, 256KB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON

~2050 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression ON, 256KB block size, 256KB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON
~2250 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression OFF, 256KB block size, 256KB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON

~2050 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression ON, 256KB block size, 1MB buffer size, buffer count 10,highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON
~2300 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression OFF, 256KB block size, 1MB buffer size, buffer count 10, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON

~2200 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression ON, 256KB block size, 1MB buffer size, buffer count 20, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON
~2300 MB/min using Quantum Superloader3 LTO 4 drive with a LTO4 tape compression OFF, 256KB block size, 1MB buffer size, buffer count 20, highwater count 0, Write Single block mode ON, Write SCSI pass-through mode ON

It's clear that tuning block size is more important than buffer size. No matter the block or buffer size you use, you will get better performance turning off compression if your source data can't keep up with the tape drives minimum data matching rate. Unfortunately that is a per drive setting not a per job or per tape format setting so you can't just restrict compression to full backups or to LTO3 only. You will also have to test how much of an issue it is with your combination of hardware/software. Of course that hit in performance is minor and the more important tests will be to optimize the Full backup of 600GB to 800GB instead of 20GB. I'll try to update again once I have a few weeks or months of backups done.

I'm in the process of designing my first ever backup scheme. I'm completely new to managing data backup, and there are some concepts that I don't totally understand. Here's what I've got so far, and what equipment I'll be using.

There are only three servers that I will be backing up, total data is approx 200Gb. I will be doing weekly full backups on Saturdays, followed by differential backups Mon thru Fri nights. There will also be an end of month full backup that will be stored off site for DR purposes.

Equipment being used: -8 slot tape backup drive -LTO2 tapes -Backup Exec 12.5 with Exchange and SQL Agents

I will be using two sets of tapes, the first for Week 1, and then another for Week 2, which will be alternated back and forth every other week.

So my question is this, how many tapes should I be using in each set? Do I have to use eight since the backup drive accepts up to eight tapes? Will it be thrown off if I put less in?

Secondly, since the diff backups each weeknight will probably only be about 5Gb or so at most, do I need to put in five LTO2 tapes (which hold up to 400Gb) into the media pool, one tape for each night? Or is one sufficient, since it could theoretically hold many weeks worth of diffs?

What I don't understand is if BE selects a new tape for each day, or if it will just continue to append to the same tape until it's full and then roll over to the next one.

Perhaps the easier question to ask, is, if you had the backup equipment and servers to backup listed above, what would your backup design be?

Many thanks....