Questions[gfs2](server)

I am working in an environment with an embedded High-Availability NVMeoF cluster, and need this cluster to be available within minutes of a cold power-on of all nodes, and trying to set up a RAID-10 on this cluster.

The legacy infrastructure that I am working with is based on GFS2 and LVM2. Unfortunately, the raid-10 option for LVM appears to only allow one journal for GFS2, when I need more. I started working down the path of manually setting up a series of RAID-1 arrays with DRBD over which I could set up a RAID-0 with LVM2. I have been able to set this up without too much trouble. However, at this point, I run into a snag: How do I cleanly shut down and start back up the nodes for seamless data transition?

My initial, basic attempts have resulted in each board booting in a state with synchronization at 0%, and it takes hours for them to resync. I used the following commands on each of four nodes to attempt to shut down cleanly:

vgchange -a n g1 #g1 is the logical volume laid overtop the physical volumes of r0 and r1
vgchange --lockstop

drbdadm down r0 #drbd resource configured as a physical volume
drbdadm down r1 #drbd resource configured as a physical volume

Then each board is power cycled, and I attempt to start back up with the following commands:

drbdadm up r0
drbdadm up r1
if [ `hostname` = "appropriate-host" ]; then drbdadm primary --force r0; fi
if [ `hostname` = "appropriate-host2" ]; then drbdadm primary --force r1; fi
vgchange --lockstart
vgchange -asy g1

I have noticed that sometimes, this just works. Other times, I am told that my metadata is invalid and I will need to recreate it. After running drbdadm create-md r0 or r1, synchronization via drbd occurs from a 0% starting point, which takes hours, which my effort can't sustain. I am unsure if a specific start-up/shut-down sequence might allow me to reliably avoid synchronization concerns; if there is a way to force DRBD to speed up re-synchronization efforts; if swapping to a RAID-01 configuration where DRBD is overlaid on top of two logical volumes might ensure a more reliable starting configuration that can skip initial synchronization; or if I'm using entirely the wrong tools for the job.

Does ServerOverflow have any insight to help me tool up my configuration for reliable and clean start-ups and shut-downs? Any help would be greatly appreciated!

I'm configuring a two node A/A cluster with a common storage attached via iSCSI, which uses GFS2 on top of clustered LVM. So far I have prepared a simple configuration, but am not sure which is the right way to configure gfs resource.

Here is the rm section of /etc/cluster/cluster.conf:

<rm>
    <failoverdomains>
        <failoverdomain name="node1" nofailback="0" ordered="0" restricted="1">
            <failoverdomainnode name="rhc-n1"/>
        </failoverdomain>
        <failoverdomain name="node2" nofailback="0" ordered="0" restricted="1">
            <failoverdomainnode name="rhc-n2"/>
        </failoverdomain>
    </failoverdomains>
    <resources>
        <script file="/etc/init.d/clvm" name="clvmd"/>
        <clusterfs name="gfs" fstype="gfs2" mountpoint="/mnt/gfs"  device="/dev/vg-cs/lv-gfs"/>
    </resources>
    <service name="shared-storage-inst1" autostart="0" domain="node1" exclusive="0" recovery="restart">
        <script ref="clvmd">
            <clusterfs ref="gfs"/>
        </script>
    </service>
    <service name="shared-storage-inst2" autostart="0" domain="node2" exclusive="0" recovery="restart">
        <script ref="clvmd">
            <clusterfs ref="gfs"/>
        </script>
    </service>
</rm>

This is what I mean: when using clusterfs resource agent to handle GFS partition, it is not unmounted by default (unless force_unmount option is given). This way when I issue

clusvcadm -s shared-storage-inst1

clvm is stopped, but GFS is not unmounted, so a node cannot alter LVM structure on shared storage anymore, but can still access data. And even though a node can do it quite safely (dlm is still running), this seems to be rather inappropriate to me, since clustat reports that the service on a particular node is stopped. Moreover if I later try to stop cman on that node, it will find a dlm locking, produced by GFS, and fail to stop.

I could have simply added force_unmount="1", but I would like to know what is the reason behind the default behavior. Why is it not unmounted? Most of the examples out there silently use force_unmount="0", some don't, but none of them give any clue on how the decision was made.

Apart from that I have found sample configurations, where people manage GFS partitions with gfs2 init script - https://alteeve.ca/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Defining_The_Resources

or even as simply as just enabling services such as clvm and gfs2 to start automatically at boot (http://pbraun.nethence.com/doc/filesystems/gfs2.html), like:

chkconfig gfs2 on

If I understand the latest approach correctly, such cluster only controls whether nodes are still alive and can fence errant ones, but such cluster has no control over the status of its resources.

I have some experience with Pacemaker and I'm used to that all resources are controlled by a cluster and an action can be taken when not only there are connectivity issues, but any of the resources misbehave.

So, which is the right way for me to go:

1. leave GFS partition mounted (any reasons to do so?)
2. set force_unmount="1". Won't this break anything? Why this is not the default?
3. use script resource <script file="/etc/init.d/gfs2" name="gfs"/> to manage GFS partition.
4. start it at boot and don't include in cluster.conf (any reasons to do so?)

This may be a sort of question that cannot be answered unambiguously, so it would be also of much value for me if you shared your experience or expressed your thoughts on the issue. How does for example /etc/cluster/cluster.conf look like when configuring gfs with Conga or ccs (they are not available to me since for now I have to use Ubuntu for the cluster)?

Thanks you very much!

I've got a system that is doing a continuous steady stream of writes to GFS2 filesystems, and what happens is that the cache builds up to vm.dirty_ratio and then load skyrockets during a synchronous flush

I've already set vm.dirty_background_ratio=1 and vm.dirty_ratio=10 and that definitely helped, but there's still a cycle of cache building up and a synchronous flush, albeit a smaller one.

What I really want is to make the background flush work harder. I've got CPU to spare, so I'd rather spread the flushing work out as evenly as possible and never have it be synchronous. Any advice on the best way to do that? The next thing I'll try is lowering vm.dirty_writeback_centisecs, but I've seen warnings about that. http://www.westnet.com/~gsmith/content/linux-pdflush.htm

This is 2.6.32 kernel.

I'm planning to share a GFS2 filesystem via iSCSI to my web servers.

Are the files read by my web servers cached locally using the linux disk cache stored in memory or are they only cached on the GFS2 cluster?

I need a cluster filesystem for an iSCSI device. The problem is that the servers to which it is connected generate datafiles which must be read by every other servers. Except for the writing and deleting of such files, I do not need a full locking scheme like in OCFS2 or GFS2.

So, can I use a distributed lock manager (DLM) on top of an ext3 filesystem or must I use only specialized filesystem ?