Niko S P's questions

What would be the preferred way to update the configuration of one node when provisioning new nodes?

E.g. let's say there is a cluster-like environment

i am provisioning nodes, let's say these nodes need to access a memcached or MySQL Server on some other node (also managed by puppet)

now when adding a new node, how do i go about updating iptables configuration or any other initial configuration step that is needed on another node in order to enable the new node to start its job?

i am using a homegrown "provisioning" system for these tasks currently, is there a convenient way to use puppet for something like this?

I am running several fileservers using this controller, filesystem and disk setup.

All of them are suffering from poor write performance, once the 256MB BBU Write Cache is full i get really high iowait (>40) and the write speed goes down to a few MB/s
It get's even worse if the servers are encountering medium to heavy reads while writing.

I am looking for suggestions on how to tweak the controller or the filesystem to improve write performance.

Some data about the Raid Array and Controller:

RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
Size:5.456 TB
State: Optimal
Stripe Size: 64 KB
Number Of Drives:4
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Enabled
Encryption Type: None

Product Name    : PERC 6/i Integrated
FW Version         : 1.22.12-0952
BIOS Version       : 2.04.00

Data about the filesystem:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Default mount options are used and the filesystem was created using the default options of the mkfs.ext4 command.

edit:
Just to illustrate my use-case i will describe what these servers are doing
They are serving files via lighttpd at 40-80 MB/s, new files are periodically being downloaded to the servers via ftp.
The files are between 800MB and 6GB.
Serving the files works great, without any noticable IOWait, but everytime the ftp transfers kick in to get new files you can see it really struggeling.

as requested, here is the bonnie++ output:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
XXXXXXX          8G   580  99 94284  14 61903   9  2853  83 189033  11 420.5   8
Latency             14004us     825ms    1548ms     105ms     202ms   98036us
Version  1.96       ------Sequential Create------ --------Random Create--------
XXXXXXX             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  5 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               406us     535us     598us     374us      21us      60us

The disks in use are D-WMAY03176700WDC WD2002FAEX-007BA0 on all servers

I set up a new server last week, did some testing with different mkfs.xfs options on mdadm raid5, i finally found some statisfying mkfs options.
Now i am about to set up a new server having exactly the same disk setup
Unfortunately i did not write down the mkfs.xfs options i was using the last time, so here is my question:

Is there a way to get the mkfs.xfs options used from a running (and currently mounted) xfs filesystem?

I am currently using Master-Slave Replication, separating reads from writes at the application level. The only kind of fail-over currently implemented is application level detection of master failure, making the slave the new master so the application can run uninterrupted. As one can guess, it is a lot of work to switch back, get in sync, tidy up and so on.

I have searched a lot, and read dozens of articles, but I did not come across a solution for HA MySQL if you have mostly MyISAM Tables, Mysql Cluster, Heartbeat/DRBD, Schooner, these are all fine for InnoDB-only setups, but not suitable for MyISAM.

I would appreciate some suggestions or real-world experience with HA Setups. It does not need to be open-source (read free), it just needs to work.

I am struggling with tc prio qdisc for a few hours now. I have read the lartc Documentation, Examples and HowTos, but this whole thing is kinda new for me and somewhat confusing :)

So this is my scenario: A couple of fileservers serving a high-volume of http and ftp traffic. I need to prioritize mysql traffic, because often when the links are full the sql traffic becomes slow and/or garbled, leading to connection errors, timeouts and so on.

This is what i have so far:

# tc qdisc add dev eth0 root handle 1: prio
# tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dport 22 0xffff flowid 1:1
# tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dport 3306 0xffff flowid 1:1
# tc filter add dev eth0 parent 1: prio 3 protocol all u32 match u32 0 0 flowid 1:3
# tc -s qdisc ls dev eth0
qdisc prio 1: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 125836067 bytes 87549 pkt (dropped 0, overlimits 0 requeues 347) 
backlog 0b 0p requeues 347 

If i am not mistaken by the lartc docs this should put ssh and mysql traffic into prio band 1 and everything else into prio band 3, according to the docs, prio qdisc has 3 bands by default, lower bands should have higher priority

Can anyone confirm or deny this or do you have other thoughts? I do not want to test this on the production systems before i can be absolutely sure it will work. I am put off by the statistics, since they do not show a clear seperation of traffic

edit: i just did some more testing with this configuration, doing a ping on the server, loading up the links, ping goes from 40ms to 170ms. Doing this:

# tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip protocol 1 0xff flowid 1:1

Ping goes down to 40ms, so this might actually be working already :)

edit2: after some more testing i came to up with the following:

tc qdisc add dev eth0 root handle 1: prio
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dport 22 0xffff flowid 1:1
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dport 3306 0xffff flowid 1:1
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip protocol 1 0xff flowid 1:1

for matching "any other traffic" one might use:

tc filter add dev eth0 protocol ip parent 1: prio 2 u32 match ip src 0/0 flowid 1:2

or

tc filter add dev eth0 parent 1: prio 2 protocol all u32 match u32 0 0 flowid 1:2

but i found not specifying a "catch all" filter did work as well, seems like the default prio band is already low.