jemmille's questions

I have a server that holds ZFS snapshots that I export via NFS to the servers they are backing up so you can restore via a custom application written in-house. The issue is as follows:

NOTE: I am not using ZFS built-in NFS for a reason, so please don't tell me to use that!

This is all NFS v4
The host is running CentOS 6.2 The client is running CentOS 5.7

I have 8 nfs servers started by default on the host.

On the backup server that holds the NFS shares, I can traverse the directory structure as deep as needed and see all expected files.

On the client, I can traverse the filesystem, but sometimes, and it really seems random, when I go 2 or more directories deep, I end up seeing the files from another server.

Here is an example:

[NFSSERVER /nfs/share]#  ls -l
total 60
drwx--x--x 30 root root 4096 Feb 25 00:15 20120225
drwx--x--x 30 root root 4096 Feb 26 00:05 20120226
drwx--x--x 30 root root 4096 Feb 27 00:06 20120227
.....

so on

[NFSCLIENT /app/backups]# ls -l
total 60
drwx--x--x 30 nobody nobody 4096 Mar  2 00:25 20120225/
drwx--x--x 30 nobody nobody 4096 Mar  2 00:25 20120226/
drwx--x--x 30 nobody nobody 4096 Mar  2 00:25 20120227/
......

so on

You can see those are identical, as they should be.

This is where the problem starts. If i go into:

[NFSCLIENT /app/backups/20120225/home] #  ls -l

When I run this ls -l on the client sometimes I see the proper files, sometimes I see the home dir of another server.

If I got to [NFSSERVER /nfs/share/20120225/home]# ls -l

When I run this ls -l I see the proper files. If I delete a folder in /nfs/share/ I can see the result on the client immediately. It is only when i go deeper that I see these "cross-mounted" filesystems.

Here is a portion of my /etc/exports (hostnames changed)

/nfs *.domain.com(fsid=0,ro,nohide,no_root_squash)
/nfs/server1/20120308 *.domain.com(ro,nohide,no_root_squash)
/nfs/server1/20120309 *.domain.com(ro,nohide,no_root_squash)
/nfs/server1/20120310 *.domain.com(ro,nohide,no_root_squash)
/nfs/server1/20120311 *.domain.com(ro,nohide,no_root_squash)
/nfs/server2/20120308 *.domain.com(ro,nohide,no_root_squash)
/nfs/server2/20120309 *.domain.com(ro,nohide,no_root_squash)
/nfs/server2/20120310 *.domain.com(ro,nohide,no_root_squash)
/nfs/server2/20120311 *.domain.com(ro,nohide,no_root_squash)
/nfs/server3/20120204 *.domain.com(ro,nohide,no_root_squash)
/nfs/server3/20120205 *.domain.com(ro,nohide,no_root_squash)
/nfs/server3/20120206 *.domain.com(ro,nohide,no_root_squash)
/nfs/server3/20120207 *.domaincom(ro,nohide,no_root_squash)

IF I remove all lines from etc exports EXCEPT the one that is cross-mounting, then reload the exports file (ie, only leaving one entry in /etc/exports), it shows all of the proper directories on the client machine.

So, stale NFS handles? More NFS servers running by default? Something else? Any ideas? I've been banging my head for a couple weeks now on this one.

UPDATE

This is the line of code my script runs that is setting up the directories that are being exported:

mount -t ext4 -o noload,ro /dev/zvol/backups/$HOST@$DATE"-00" /nfs/$HOST/$DATE

The /nfs/$HOST/$DATE folders are the ones being exported (as you can see in the exports file above)

I have a storage unit with 2 backplanes. One backplane holds 24 disks, one backplane holds 12 disks. Each backplane is independently connected to a SFF-8087 port (4 channel/12Gbit) to the raid card.

Here is where my question really comes in. Can or how easily can a backplane be overloaded? All the disks in the machine are WD RE4 WD1003FBYX (black) drives that have average writes at 115MB/sec and average read of 125 MB/sec

I know things would vary based on the raid or filesystem we put on top of that but it seems to be that a 24 disk backplane with only one SFF-8087 connector should be able to overload the bus to a point that might actually slow it down?

Based on my math, if I had a RAID0 across all 24 disks and asked for a large file, I should, in theory should get 24*115 MB/sec which translates to 22.08 GBit/sec of total throughput.

Either I'm confused or this backplane is horribly designed -- at least for a performance-based environment.

I'm looking at switching to a model where each drive has it's own channel from the backplane (and new HBA's or raid card).

EDIT: more details

We have used both pure linux (centos), open solaris, software raid, hardware raid, EXT3/4, ZFS.

Here are some examples using bonnie++

4 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
194MB/s   19%    92MB/s    11%    200MB/s   8%     310/sec  
194MB/s   19%    93MB/s    11%    201MB/s   8%     312/sec  
--------- ----   --------- ----   --------- ----   ---------
389MB/s   19%    186MB/s   11%    402MB/s   8%     311/sec

8 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
324MB/s   32%    164MB/s   19%    346MB/s   13%    466/sec  
324MB/s   32%    164MB/s   19%    348MB/s   14%    465/sec  
--------- ----   --------- ----   --------- ----   ---------
648MB/s   32%    328MB/s   19%    694MB/s   13%    465/sec

12 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
377MB/s   38%    191MB/s   22%    429MB/s   17%    537/sec  
376MB/s   38%    191MB/s   22%    427MB/s   17%    546/sec  
--------- ----   --------- ----   --------- ----   ---------
753MB/s   38%    382MB/s   22%    857MB/s   17%    541/sec

Now 16 Disk RAID-0, it's gets interesting

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
359MB/s   34%    186MB/s   22%    407MB/s   18%    1397/sec 
358MB/s   33%    186MB/s   22%    407MB/s   18%    1340/sec 
--------- ----   --------- ----   --------- ----   ---------
717MB/s   33%    373MB/s   22%    814MB/s   18%    1368/sec

20 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
371MB/s   37%    188MB/s   22%    450MB/s   19%    775/sec  
370MB/s   37%    188MB/s   22%    447MB/s   19%    797/sec  
--------- ----   --------- ----   --------- ----   ---------
741MB/s   37%    376MB/s   22%    898MB/s   19%    786/sec

24 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
347MB/s   34%    193MB/s   22%    447MB/s   19%    907/sec  
347MB/s   34%    192MB/s   23%    446MB/s   19%    933/sec  
--------- ----   --------- ----   --------- ----   ---------
694MB/s   34%    386MB/s   22%    894MB/s   19%    920/sec

(anyone starting to see the pattern here?) :-)

28 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
358MB/s   35%    179MB/s   22%    417MB/s   18%    1105/sec 
358MB/s   36%    179MB/s   22%    414MB/s   18%    1147/sec 
--------- ----   --------- ----   --------- ----   ---------
717MB/s   35%    359MB/s   22%    832MB/s   18%    1126/sec 

32 Disk RAID-0, ZFS

WRITE     CPU    RE-WRITE  CPU    READ      CPU    RND-SEEKS
354MB/s   35%    190MB/s   22%    420MB/s   18%    1519/sec 
354MB/s   35%    190MB/s   22%    418MB/s   18%    1572/sec 
--------- ----   --------- ----   --------- ----   ---------
708MB/s   35%    380MB/s   22%    838MB/s   18%    1545/sec 

More details:

Here is the exact unit:

http://www.supermicro.com/products/chassis/4U/847/SC847E16-R1400U.cfm

I have a fleet of VPS servers, each have two /27's running to it. On only one of the machines, a set of VPS's on a particular subnet drops connection for 5-10 minutes at a time to the internet. The VPS themselves do not crash, just networking drops but returns after some time.

These are in bridged mode so no tricky routing going on. I have had our datacenter check and double check routes and they claim there is no issue (so please don't tell me to have the DC check at this time) I literally have 20 other nodes in this exact same config without issue.

The other /27 routed to this server has had zero issues and as mentioned and it's only 4-5 VM's on the other subnet that have the issue. I'm investigating this issue more as I await any reply on here to help.

Let me know if any particular log files or similar will help anyone.

As for the datacenter, this was their reply:

The X.X.X.X/27 IP block is routed as a secondary block to the Vlan as it should be. The gateway for the X.X.X.X/27 IP block is X.X.X.X (it was right). Also, I confirmed that there are no overlapping routes for this block that might have been given to another customer. From what I can, the router is configured properly for this subnet. Typically, when single IPs are having issues like this, it is usually caused by some sort of server side configuration. Unfortunately, I am not too familiar with Xen to know what might be causing the issue. However, seeing as this is intermittent, I would definitely suspect a server side configuration issue over a network configuration issue.

Thanks!

I have an iSCSI fail over implementation setup so if one of my storage units fails the other takes over immediately (it also runs the NFS shares). When fail over occurs, volumes are exported, the IP is switched to the other machine and the targets are reconfigured. The fail over of the storage system itself works just fine. I use NexentaStor for my filer.

When I do a test (manual) fail over of my storage the following occurs:

Note: I run the admin VM's on NFS and customer based VM's on iSCSI

1. All NFS based VM's remain up and working perfectly through the failover and after
2. All VM 's running on iSCSI eventually report the following:
   • An error about not being able to write to a particular block
   • An error about journaling not working
   • Then the file system goes RO

To get the VM's working again I have to do the following:

1. Force shutdown of the "broken" VM's.
2. Detach the iSCSI SR
3. Re-attach the iSCSI SR
4. Boot the VM on a different server (5 in my pool) If I don't boot on a different server I get this error "Internal error: Failure("The VDI <uuid&gt; is already attached in RW mode; it can't be attached in RO mode!")" The only way I have found to fix that error is to reboot the entire server it was running on previously which is obviously a huge pain.

Currently multipathing is NOT enabled (but can be and the same thing still occurs). I have edited much of the /etc/iscsid.conf file to work with the timeout settings but to no avail.

In short, my storage fails over properly but XenServer does not keep the connection alive. As a thought, the error that shows up in #4 above might be the ultimate cause and fixing that would fix everything?

Any help would be appreciated more than you know.

I may be confusing myself here so I'm asking for some clarification.

I have a large XenServer install that is currently using lvmoiscsi. I am looking for ways to reduce actual disk space usage. I am running ONLY linux VM's in this environment.

The sparse provisioning and chaining that the VHD format supports seems like the ticket but it seems to only support Windows bases VM's using NFS SR's or the the supported StorageLink SR's.

So, can I have linux VM's using sparse provisioning and chaining? Everything I keep reading and what I understand says no. I'm willing to re-work my storage setup if necessary. I currently am using Nexenta filer to serve up my iSCSI LUN's but it can serve most anything out there.

Someone please help me clear this up!!!

I have been doing lots of I/O testing on a ZFS system I will eventually use to serve virtual machines. I thought I would try adding SSD's for use as cache to see how much faster I can get the read speed. I also have 24GB of RAM in the machine that acts as ARC. vol0 is 6.4TB and the cache disks are 60GB SSD's. The zvol is as follows:

pool: vol0
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        vol0                     ONLINE       0     0     0
          c1t8d0                 ONLINE       0     0     0
        cache
          c3t5001517958D80533d0  ONLINE       0     0     0
          c3t5001517959092566d0  ONLINE       0     0     0

The issue is I'm not seeing any difference with the SSD's installed. I've tried bonnie++ benchmarks and some simple dd commands to write a file then read the file. I have run benchmarks before and after adding the SSD's.

I've ensured the file sizes are at least double my RAM so there is no way it can all get cached locally.

Am I missing something here? When am I going to see benefits of having all that cache? Am I simply not under these circumstances? Are the benchmark programs not good for testing the effect of cache because of the the way (and what) it writes and reads?

If you have two network storage boxes (iSCSI specifically) can you livemigrate (either with Xen or XenServer) across the storage units or does it just have to be done when the VM's are paused/shutdown due to the storage being on a different physical machine?

I was wondering if anyone has ever tried to setup HA-like functionality without paying for the Essentials Package using XenServer. I love XenServer but would rather not fork over $2.5k per server for the Essentials Package.

Also, is the HA that comes with the Essentials Package worth it when compared to the time it would take me to script a custom HA package?

Any thoughts would be greatly appreciated.

Thanks.

P.S. I know you can set this up using Xen and Heartbeat but that's not my question :-)

I'm looking for a way to change a partition type on a linux system without starting fdisk. (In this particular case from "linux" to "softwareRAID")

The ultimate goal here is a script that is going to take /dev/sda and automatically create a software RAID1 with /dev/sdb

I'm not looking for anyone here to write me a script, just trying to provide some info changing the partition type in a script. I can take care of the rest.

Thanks in advance.