bart613's questions

I recently acquired Juniper SRX 210BE; until now I've been using Cisco ASA 5505 but I really want to switch over to Juniper world (big FreeBSD fan and all that).

I've got FTTC (fiber to the cabnet) internet feed that goes into a modem that changes the VDSL signal into ethernet and then from modem I connect Juniper with Ethernet cable and perform PPPoE auth on the Juniper.

Everything is working fine, but... I'm getting a major packet-loss according to both ping and traceroute. The really weird thing is that it seems like this PL is only on ICMP / traceroute messages and it's not permanent but rather intermittent. Eg. I'm going to get 10 packets lost then it's going to be fine for ~ 40 seconds and then another burst of PL.

The PL includes the first hop which is the Juniper itself! Please see below.

Host                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.17.17.62                                                29.3%   454    1.3   1.5   0.9  21.6   1.7
 2. 109.224.191.50                                             13.2%   453   14.8  17.3  14.5 111.8   9.0
 3. 149.6.9.170                                                12.8%   453   15.3  18.5  14.5 149.1  13.2
 4. gi8-10.mag01.lhr01.atlas.cogentco.com                      13.7%   453   15.4  25.8  14.6 218.9  33.1
 5. te0-5-0-5.rcr21.lhr01.atlas.cogentco.com                   12.6%   453   16.3  17.5  15.0  69.7   6.5
 6. be2040.ccr41.lon13.atlas.cogentco.com                      13.5%   453   16.7  17.9  16.0  72.4   4.9
 7. be2194.ccr41.ams03.atlas.cogentco.com                      12.6%   453   23.9  25.8  23.4  74.6   6.1
 8. be2186.ccr41.ham01.atlas.cogentco.com                      12.8%   453   33.7  35.6  32.5  94.6   8.3
 9. be2147.rcr21.ber01.atlas.cogentco.com                      13.7%   453   36.9  39.8  36.2 104.3   9.1
10. te3-1.ccr01.poz01.atlas.cogentco.com                       13.2%   453   41.1  58.7  40.4 251.0  41.1
11. 149.6.28.22                                                13.2%   453   48.8  52.0  46.8 126.8   9.9
12. 81.210.127.195                                             13.5%   453   55.7  57.6  53.0 110.2   8.4
13. 83.238.250.131                                             38.9%   453   52.6  54.4  51.7  98.0   5.8
14. GdynR005RT01-RedaR001RT01.inetia.pl                        17.9%   453   53.5  54.8  52.4  93.4   5.9
15. ???
16. rtr4.rtr-int-1.adm.wp-sa.pl                                13.7%   453   55.5  57.3  54.7 120.9   6.8
17. www.wp.pl                                                  13.5%   453   55.3  56.8  54.5 111.9   6.3

10.17.17.62 is Juniper's internal IP -- how weird, isn't it? The config is pretty much stock one with PPPoE auth configured as well as NAT. See below:

## Last commit: 2014-10-17 23:19:14 BST by root
version 12.1X44-D40.2;
system {
    time-zone Europe/London;
    root-authentication {
        encrypted-password "xxx"; ## SECRET-DATA
    }
    name-server {
        208.67.222.222;
        208.67.220.220;
    }
    services {
        ssh;
        telnet;
        xnm-clear-text;
        web-management {
            http {
                interface vlan.16;
            }
            https {
                system-generated-certificate;
                interface vlan.16;
            }
        }
        dhcp {
            domain-name lon.domain;
            pool 10.17.17.0/26 {
                address-range low 10.17.17.32 high 10.17.17.55;
                default-lease-time 7200;
                name-server {
                    8.8.8.8;
                    8.8.4.4;
                }
                router {
                    10.17.17.62;
                }
            }
        }
    }
    syslog {
        archive size 100k files 3;
        user * {
            any emergency;
        }
        file messages {
            any critical;
            authorization info;
        }
        file interactive-commands {
            interactive-commands error;
        }
    }
    max-configurations-on-flash 5;
    max-configuration-rollbacks 5;
    license {
        autoupdate {
            url https://ae1.juniper.net/junos/key_retrieval;
        }
    }
}
interfaces {
    interface-range interfaces-trust {
        member ge-0/0/1;
        member fe-0/0/3;
        member fe-0/0/4;
        member fe-0/0/5;
        member fe-0/0/6;
        member ge-0/0/0;
        member fe-0/0/2;
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/0 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    ge-0/0/1 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/2 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/3 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/4 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/5 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/6 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-trust;
                }
            }
        }
    }
    fe-0/0/7 {
        unit 0 {
            encapsulation ppp-over-ether;
        }
    }
    pp0 {
        traceoptions {
            flag all;
        }
        unit 0 {
            point-to-point;
            ppp-options {
                chap {
                    default-chap-secret "xxx"; ## SECRET-DATA
                    local-name "xxx";
                    passive;
                }
            }
            pppoe-options {
                underlying-interface fe-0/0/7.0;
                idle-timeout 0;
                auto-reconnect 30;
                client;
            }
            no-keepalives;
            family inet {
                negotiate-address;
            }
        }
    }
    vlan {
        unit 16 {
            family inet {
                address 10.17.17.62/26;
            }
        }
    }
}
routing-options {
    static {
        route 0.0.0.0/0 {
            next-hop pp0.0;
            metric 0;
        }
    }
}
protocols {
    stp {
        disable;
        interface ge-0/0/0.0 {
            disable;
        }
        interface ge-0/0/1.0 {
            disable;
        }
        interface fe-0/0/2.0 {
            disable;
        }
        interface fe-0/0/3.0 {
            disable;
        }
        interface fe-0/0/4.0 {
            disable;
        }
        interface fe-0/0/5.0 {
            disable;
        }
        interface fe-0/0/6.0 {
            disable;
        }
    }
}
security {
    flow {
        tcp-mss {
            all-tcp {
                mss 1380;
            }
        }
    }
    screen {
        ids-option untrust-screen {
            ip {
                source-route-option;
                tear-drop;
            }
            tcp {
                syn-flood {
                    alarm-threshold 1024;
                    attack-threshold 200;
                    source-threshold 1024;
                    destination-threshold 2048;
                    timeout 20;
                }
                land;
            }
        }
    }
    nat {
        source {
            rule-set trust-to-untrust {
                from zone trust;
                to zone untrust;
                rule source-nat-rule {
                    match {
                        source-address 10.17.17.0/26;
                    }
                    then {
                        source-nat {
                            interface;
                        }
                    }
                }
            }
        }
    }
    policies {
        from-zone trust to-zone untrust {
            policy trust-to-untrust {
                match {
                    source-address any;
                    destination-address any;
                    application any;
                }
                then {
                    permit;
                }
            }
        }
    }
    zones {
        security-zone trust {
            host-inbound-traffic {
                system-services {
                    all;
                }
                protocols {
                    all;
                }
            }
            interfaces {
                vlan.16 {
                    host-inbound-traffic {
                        system-services {
                            dhcp;
                            all;
                        }
                    }
                }
                fe-0/0/5.0;
            }
        }
        security-zone untrust {
            interfaces {
                pp0.0 {
                    host-inbound-traffic {
                        system-services {
                            ssh;
                            ping;
                            traceroute;
                        }
                    }
                }
            }
        }
    }
}
vlans {
    vlan-trust {
        vlan-id 16;
        l3-interface vlan.16;
    }
}

When I run ping to anything from the Juniper itself, sometimes the ping is fine but at some points I'm getting PL as well. Even to my GW. If I switch back to Cisco ASA 5505 -- I'm getting literally no PL.

Does anyone have any suggestions what to look at? I tried different ports, wipe out config and try again etc. nothing helps. Given the other protocols work fine during the PL on ICMP / traceroute I suspect some kind of limiting kicking in, but how to find which one it is??? I'm pretty sure I disabled all screens etc...

Any advice, much appreciated.

EDIT: I'm running latest recommended software version -- 12.1X44-D40.2

root> show version
Model: srx210be
JUNOS Software Release [12.1X44-D40.2]

EDIT 2: I just made a test; I connected a Raspberry PI directly to the interface and started MTRing that. Below is the result:

                                            Packets               Pings
 Host                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.17.17.62                           47.3%   112    1.2   1.2   1.0   3.3   0.3
 2. 10.50.50.2                             0.0%   111    1.7   1.9   1.3  14.4   1.7

With 10.17.17.62 being the GW again.

After further tests, it seems like ICMP is not actually affected. It's just MTR which is using UDP for traceroute. However, I can't really track down what the hell is limiting / dropping that.

I was wondering whether anyone could advice on best practices as to the best way to redirect multiple ports for single host in the internal network.

As I understand this you can only nat a single port per object. This caused a bit of a litter around in a config as I have to specify individual objects for the same host (IP) to redirect individual ports.

It looks as follows:

object network ratatouille-4569
 nat (inside,outside) static interface service udp 4569 4569 
object network ratatouille-ssh
 nat (inside,outside) static interface service tcp ssh ssh 

Now this approach have few obvious problems (especially if you've more complex/more rules) eg. if you need to change that internal host's IP address, you've to do this for each object individually.

In this specific case, I have 7 ports to redirect to this specific host.

I'd appreciate any advice about best-practice as how to do this.

I currently manage 6 Cisco ASA devices (2 pairs of 5510s and 1 pair of 5550s). They all work quite nicely and are stable so this is more of a best-practice advice question rather then "OMG it's broken help me fix it".

My network is split into multiple VLANs. Pretty much each service role has its own VLAN so DB servers would have their own VLAN, APP servers, Cassandra nodes.

Traffic is being managed on an allow only specific, deny rest basics (so default policy is to drop all traffic). I do this by creating two ACLs per network interface, eg.:

• access-list dc2-850-db-in ACL that's being applied to dc2-850-db interface in "in" direction
• access-list dc2-850-db-out ACL that's being applied to dc2-850-db interface in "out" direction

It's all pretty tight and works as expected, however I was wondering whether this is the best possible way to go? At the moment I got to a point where I have over 30 VLANs and I must say it's becoming slightly confusing at some points to manage those.

Probably something like common/shared ACLs would help here which I could inherit from other ACLs but AFAIK there's no such thing...

Any advice much appreciated.

I seemed to have hit quite interesting problem recently while investigating my previous problem -- One ColumnFamily places data on only 3 out of 4 nodes

We had a very long row with over 800.000 columns in it. It stored user details; one column per one user's details. Not getting in reasons behind this kind of designs, according to documentation this should be fine, however we were having massive performance problems.

It seemed as the whole row was cache by the operating system cache and Cassandra -- because it was fairly often used row -- was spending most of the CPU time on serialising the row in order to get data out of it (even though the applications queries were about specific column within the row).

First thing I did was enabling row-cache on the ColumnFamily which stored this row. This seemed to have instantly solved the high CPU usage problem and solved performance problems... Until first scheduled manual repair.

At some point during manual repair (replicable; it was happening every single time I tried running manual repair), node was loosing contact with other nodes (and vice-versa!) and all affected nodes (usually two out of four) were reporting that heap is almost full (Heap is 0.9977127109734825 full...). Affected nodes were simply dying (as in, Cassandra, not the host itself).

We have solved the root problem -- changed the way we store data within the problematic ColumnFamily, however I still don't understand why I was getting into those problems and would appreciate if someone could shed some light as to potential source of the problem?

Was there anything really bad about what we did?

We are using 0.8.7, over 4 nodes cluster with RF=3. Heap is set to 16GB and those are 48GB boxes.

I have posted this on cassandra-user mailing list however didn't get any responses just yet and I was wondering whether someone here on serverfault.com will have any ideas.

I seem to have came across rather weird (at least for me!) problem / behaviour with Cassandra.

I am running a 4-nodes cluster on Cassandra 0.8.7. For the keyspace in question, I have RF=3, SimpleStrategy with multiple ColumnFamilies inside the KeySpace. On of the ColumnFamilies however seems to have data distributed across only 3 out of 4 nodes.

The data on the cluster beside the problematic ColumnFamily seems to be more or less equal and even.

# nodetool -h localhost ring
Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               127605887595351923798765477786913079296     
192.168.81.2    datacenter1 rack1       Up     Normal  7.27 GB         25.00%  0                                           
192.168.81.3    datacenter1 rack1       Up     Normal  7.74 GB         25.00%  42535295865117307932921825928971026432      
192.168.81.4    datacenter1 rack1       Up     Normal  7.38 GB         25.00%  85070591730234615865843651857942052864      
192.168.81.5    datacenter1 rack1       Up     Normal  7.32 GB         25.00%  127605887595351923798765477786913079296     

Schema for the relevant bits of the keyspace is as follows:

[default@A] show schema;
create keyspace A
  with placement_strategy = 'SimpleStrategy'
  and strategy_options = [{replication_factor : 3}];
[...]
create column family UserDetails
  with column_type = 'Standard'
  and comparator = 'IntegerType'
  and default_validation_class = 'BytesType'
  and key_validation_class = 'BytesType'
  and memtable_operations = 0.571875
  and memtable_throughput = 122
  and memtable_flush_after = 1440
  and rows_cached = 0.0
  and row_cache_save_period = 0
  and keys_cached = 200000.0
  and key_cache_save_period = 14400
  and read_repair_chance = 1.0
  and gc_grace = 864000
  and min_compaction_threshold = 4
  and max_compaction_threshold = 32
  and replicate_on_write = true
  and row_cache_provider = 'ConcurrentLinkedHashCacheProvider';

And now the symptoms - output of 'nodetool -h localhost cfstats' on each node. Please note the figures on node1.

node1

Column Family: UserDetails
SSTable count: 0
Space used (live): 0
Space used (total): 0
Number of Keys (estimate): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0

node2

Column Family: UserDetails
SSTable count: 3
Space used (live): 112952788
Space used (total): 164953743
Number of Keys (estimate): 384
Memtable Columns Count: 159419
Memtable Data Size: 74910890
Memtable Switch Count: 59
Read Count: 135307426
Read Latency: 25.900 ms.
Write Count: 3474673
Write Latency: 0.040 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 120
Key cache hit rate: 0.999971684189041
Row cache: disabled
Compacted row minimum size: 42511
Compacted row maximum size: 74975550
Compacted row mean size: 42364305

node3

Column Family: UserDetails
SSTable count: 3
Space used (live): 112953137
Space used (total): 112953137
Number of Keys (estimate): 384
Memtable Columns Count: 159421
Memtable Data Size: 74693445
Memtable Switch Count: 56
Read Count: 135304486
Read Latency: 25.552 ms.
Write Count: 3474616
Write Latency: 0.036 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 109
Key cache hit rate: 0.9999716840888175
Row cache: disabled
Compacted row minimum size: 42511
Compacted row maximum size: 74975550
Compacted row mean size: 42364305

node4

Column Family: UserDetails
SSTable count: 3
Space used (live): 117070926
Space used (total): 119479484
Number of Keys (estimate): 384
Memtable Columns Count: 159979
Memtable Data Size: 75029672
Memtable Switch Count: 60
Read Count: 135294878
Read Latency: 19.455 ms.
Write Count: 3474982
Write Latency: 0.028 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 119
Key cache hit rate: 0.9999752235777154
Row cache: disabled
Compacted row minimum size: 2346800
Compacted row maximum size: 62479625
Compacted row mean size: 42591803

When I go to 'data' directory on node1 there is no files regarding the UserDetails ColumnFamily.

I tried performing manual repair in hope it will heal the situation, however without any luck.

# nodetool -h localhost repair A UserDetails
 INFO 15:19:54,611 Starting repair command #8, repairing 3 ranges.
 INFO 15:19:54,647 Sending AEService tree for #<TreeRequest manual-repair-89c1acb0-184c-438f-bab8-7ceed27980ec, /192.168.81.2, (A,UserDetails), (85070591730234615865843651857942052864,127605887595351923798765477786913079296]>
 INFO 15:19:54,742 Endpoints /192.168.81.2 and /192.168.81.3 are consistent for UserDetails on (85070591730234615865843651857942052864,127605887595351923798765477786913079296]
 INFO 15:19:54,750 Endpoints /192.168.81.2 and /192.168.81.5 are consistent for UserDetails on (85070591730234615865843651857942052864,127605887595351923798765477786913079296]
 INFO 15:19:54,751 Repair session manual-repair-89c1acb0-184c-438f-bab8-7ceed27980ec (on cfs [Ljava.lang.String;@3491507b, range (85070591730234615865843651857942052864,127605887595351923798765477786913079296]) completed successfully
 INFO 15:19:54,816 Sending AEService tree for #<TreeRequest manual-repair-6d2438ca-a05c-4217-92c7-c2ad563a92dd, /192.168.81.2, (A,UserDetails), (42535295865117307932921825928971026432,85070591730234615865843651857942052864]>
 INFO 15:19:54,865 Endpoints /192.168.81.2 and /192.168.81.4 are consistent for UserDetails on (42535295865117307932921825928971026432,85070591730234615865843651857942052864]
 INFO 15:19:54,874 Endpoints /192.168.81.2 and /192.168.81.5 are consistent for UserDetails on (42535295865117307932921825928971026432,85070591730234615865843651857942052864]
 INFO 15:19:54,874 Repair session manual-repair-6d2438ca-a05c-4217-92c7-c2ad563a92dd (on cfs [Ljava.lang.String;@7e541d08, range (42535295865117307932921825928971026432,85070591730234615865843651857942052864]) completed successfully
 INFO 15:19:54,909 Sending AEService tree for #<TreeRequest manual-repair-98d1a21c-9d6e-41c8-8917-aea70f716243, /192.168.81.2, (A,UserDetails), (127605887595351923798765477786913079296,0]>
 INFO 15:19:54,967 Endpoints /192.168.81.2 and /192.168.81.3 are consistent for UserDetails on (127605887595351923798765477786913079296,0]
 INFO 15:19:54,974 Endpoints /192.168.81.2 and /192.168.81.4 are consistent for UserDetails on (127605887595351923798765477786913079296,0]
 INFO 15:19:54,975 Repair session manual-repair-98d1a21c-9d6e-41c8-8917-aea70f716243 (on cfs [Ljava.lang.String;@48c651f2, range (127605887595351923798765477786913079296,0]) completed successfully
 INFO 15:19:54,975 Repair command #8 completed successfully

As I am using SimpleStrategy I would expect the keys to be split, more or less, equally across the nodes, however this don't seem to be the case.

Has anyone came across similar behaviour before? Has anyone have any suggestions what I could do to bring some data into node1? Obviously, this kind of data split means node2, node3 and node4 need to do all the read work which is not ideal.

Any suggestions much appreciated.

Kind regards, Bart

Until now I have been involved in using Cisco hardware in professional environment but on a small level. Recently, requirement has came up for our new facility to be built with PCI compliance in mind and since we have quite a lot of traffic passing between the servers we have decided to invest some money into high-end Cisco hardware (at least high-end with what I have been using so far - Cisco 5505 / Cisco 5510s).

We got:

• 2* Cisco ASA 5550
• 4* Cisco Catalyst 2960G-48TC

There will be two racks, in each 1 ASA and 2 switches. I have been thinking for a while on the best way to cross connect everything and came out with following schema:

• so 2 ASA working in HA mode, Active/Standby (we can't use Active/Active as we have a lot of VPN sessions)
• ASAs are connected with each other using two links 0/3 - 0/3 as FOLINK and 1/3 - 1/3 as STATELINK
• 0/0s in ASAs are feeds from our ISP
• ASAs are connected with each switch in local rack using LACP bundles (two cables each)
• switches are also connected with each other using LACP bundles

Now a request for advice here - is there anything wrong in my thinking as for this design? Could anything be improved / changed to make it better?

With network cables it is easy. I use velcro-ties, coloured cables, 0U vertical cable organizers from APC on side of the rack -- easy and neat.

However with power cables it was always giving me headaches.

With some of the racks I am using Vertical PDUs from APC (0U) so it's kind of neat as am getting 0.3m/0.5m cables and connecting them to servers which are close to the PDU sockets. It's kind of neat, but it could be better.

With horizontal PDUs it is really bad -- no matter how hard I try there is always few cables that are too long and are hanging loose below the APC.

Eg. http://www.flickr.com/photos/92011471@N00/25695150/ -- this will cause the cables on the bottom to be different length and its going to get quite messy...

What are best practices for power cabling? How do you guys do this so your cables once put in once are neat and you don't have to worry about them for years? Do you also cut them to match right length? Do you use some special organizers? Or maybe you know some tricks like hiding them between the side of the server and rack?

I went through many many posts on SF however most of the are oriented around network cabling and very few are touching slightly on the power problem.

Of course I am talking about having network cables on one side and power cables on the other side.

Any suggestions, recommendations, suggestions much appreciated.

I've got a server from hetzner.de (EQ4) with 2* SAMSUNG HD753LJ drives (750G 32MB cache).

OS is CentOS 5 (x86_64). Drives are combined together into two RAID1 partitions:

1. /dev/md0 which is 512MB big and has only /boot partitions
2. /dev/md1 which is over 700GB big and is one big LVM which hosts other partitions

Now, I've been running some benchmarks and it seems like even though exactly the same drives, speed differs a bit on each of them.

# hdparm -tT /dev/sda

/dev/sda:  Timing cached reads:   25612 MB in  1.99 seconds = 12860.70 MB/sec  Timing buffered disk reads:  352 MB in  3.01 seconds = 116.80 MB/sec
# hdparm -tT /dev/sdb

/dev/sdb:  Timing cached reads:   25524 MB in  1.99 seconds = 12815.99 MB/sec  Timing buffered disk reads:  342 MB in  3.01 seconds = 113.64 MB/sec

Also, when I run eg. pgbench which is stressing IO quite heavily, I can see following from iostat output:

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   231.40  0.00 298.00     0.00  9683.20    32.49     0.17    0.58   0.34  10.24
sda1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00   231.40  0.00 298.00     0.00  9683.20    32.49     0.17    0.58   0.34  10.24
sdb               0.00   231.40  0.00 301.80     0.00  9740.80    32.28    14.19   51.17   3.10  93.68
sdb1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00   231.40  0.00 301.80     0.00  9740.80    32.28    14.19   51.17   3.10  93.68
md1               0.00     0.00  0.00 529.60     0.00  9692.80    18.30     0.00    0.00   0.00   0.00
md0               0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00  0.00   0.60     0.00     4.80     8.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00 529.00     0.00  9688.00    18.31    24.51   49.91   1.81  95.92

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   152.40  0.00 330.60     0.00  5176.00    15.66     0.19    0.57   0.19   6.24
sda1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00   152.40  0.00 330.60     0.00  5176.00    15.66     0.19    0.57   0.19   6.24
sdb               0.00   152.40  0.00 326.20     0.00  5118.40    15.69    19.96   55.36   3.01  98.16
sdb1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00   152.40  0.00 326.20     0.00  5118.40    15.69    19.96   55.36   3.01  98.16
md1               0.00     0.00  0.00 482.80     0.00  5166.40    10.70     0.00    0.00   0.00   0.00
md0               0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00 482.80     0.00  5166.40    10.70    30.19   56.92   2.05  99.04

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   181.64  0.00 324.55     0.00  5445.11    16.78     0.15    0.45   0.21   6.87
sda1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00   181.64  0.00 324.55     0.00  5445.11    16.78     0.15    0.45   0.21   6.87
sdb               0.00   181.84  0.00 328.54     0.00  5493.01    16.72    18.34   61.57   3.01  99.00
sdb1              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00   181.84  0.00 328.54     0.00  5493.01    16.72    18.34   61.57   3.01  99.00
md1               0.00     0.00  0.00 506.39     0.00  5477.05    10.82     0.00    0.00   0.00   0.00
md0               0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00  0.00   0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00 506.39     0.00  5477.05    10.82    28.77   62.15   1.96  99.00

And this is completely getting me confused. How come two exactly the same specced drives have such a difference in write speed (see util%)? I haven't really paid attention to those speeds before, so perhaps that something normal -- if someone could confirm I would be really grateful.

Otherwise, if someone have seen such behavior again or knows what is causing such behavior I would really appreciate answer.

I'll also add that both "smartctl -a" and "hdparm -I" output are exactly the same and are not indicating any hardware problems. The slower drive was changed already two times (to new ones). Also I asked to change the drives with places, and then sda were slower and sdb quicker (so the slow one was the same drive). SATA cables were changed two times already.