Gediz GÜRSU's questions

I have 8x18TB Raid10 mdadm volume on Ubuntu server. When I connect from Windows iSCSI speeds 220MB/sec read 375MB/sec write. I guess it is a iSCSI protocol bottleneck could not improve it. It leaves me multipath.

I have 2 nics on both server and client. I tried many things but could not figure out how to use Multipath or multiple sessions (in windows interface). Tried to connect sessions as

192.168.122.100 --> 192.168.122.110
192.168.123.100 --> 192.168.123.110

I see both portals in portals and create favorite as MPIO but when I use MCS it says it is already connected. Also tried to

192.168.122.100 --> 192.168.122.110
               |--> 192.168.122.111

Could not get it work. What adjustment should I do server side. I am clueless at the moment. By the way windows is windows10 PRO 20H2 so no restrictions on it I guess.

note: We try to make it work fluently with avid media composer. Since it asks for disks. If I cannot get the performance I need from ISCSI, I plan to switch to nbd ? Any good alternatives to iSCSI or other suggestions which works over 10Gbe ethernets and switch ?

Edit: I stumbled upon CNA cards RDMA ISER solution gonna try it ... Suggestions are still welcome.

I have checked the logs. System has been working for a month then suddenly they could not see the drive on network. It is a linux file server with a virtual windows on top of it ... Virtual windows has the virtio drive of 12*10 TB Disks shared over network...

The error on windows is : you should format this disk. The error on linux is : mount syscall cannot be ... ,structure needs cleaning.

When I peek to drive via gparted I only see an ext partition cannot see ntfs partition in md0 it should contain an ntfs partition createn in virtual windows.

I was trying to simulate a scenario with mdadm and bcache. I created 4 raw img files using qemu-img. I have created raid 10 using mdadm and created tmpfs on /mnt/ramdisk and created a raw ramdisk.img file on ramdisk. Tried to connect ramdisk.img and other 4 mdadm raid10 img files using bcache...

It looks like this.

loop0           7:0    0   512M  0 loop   
└─md0           9:0    0  1020M  0 raid10 
  └─md0p1     259:0    0  1018M  0 md     
    └─bcache0 252:0    0  1018M  0 disk   /media/gediz/vmbcache
loop1           7:1    0   512M  0 loop   
└─md0           9:0    0  1020M  0 raid10 
  └─md0p1     259:0    0  1018M  0 md     
    └─bcache0 252:0    0  1018M  0 disk   /media/gediz/vmbcache
loop2           7:2    0   512M  0 loop   
└─md0           9:0    0  1020M  0 raid10 
  └─md0p1     259:0    0  1018M  0 md     
    └─bcache0 252:0    0  1018M  0 disk   /media/gediz/vmbcache
loop3           7:3    0   512M  0 loop   
└─md0           9:0    0  1020M  0 raid10 
  └─md0p1     259:0    0  1018M  0 md     
    └─bcache0 252:0    0  1018M  0 disk   /media/gediz/vmbcache
loop101         7:101  0   255M  0 loop   
└─loop101p1   259:1    0   228M  0 loop   

mdadm stat:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid10 loop3[3] loop2[2] loop1[1] loop0[0]
      1044480 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

Backing superblock:

bcache-super-show /dev/md0p1 
sb.magic        ok
sb.first_sector     8 [match]
sb.csum         51E3C78C7BED1DBF [match]
sb.version      1 [backing device]

dev.label       (empty)
dev.uuid        bb01d667-c8bf-442e-994b-94c108864cf5
dev.sectors_per_block   1
dev.sectors_per_bucket  1024
dev.data.first_sector   16
dev.data.cache_mode 0 [writethrough]
dev.data.cache_state    0 [detached]

cset.uuid       acd29060-d604-4441-961a-413ce85cc604

Cache superblock :

sb.magic        ok
sb.first_sector     8 [match]
sb.csum         D66FD05126FF5088 [match]
sb.version      0 [cache device]

dev.label       (empty)
dev.uuid        774d0fab-82ab-4825-ba19-43a83e4b4f53
dev.sectors_per_block   1
dev.sectors_per_bucket  1024
dev.cache.first_sector  1024
dev.cache.cache_sectors 465920
dev.cache.total_sectors 466944
dev.cache.ordered   no
dev.cache.discard   no
dev.cache.pos       0
dev.cache.replacement   0 [lru]

cset.uuid       f3c837cb-a743-4615-9a7f-4f647d603eaf

I have first tried with make-bcache -B /dev/md0 -C /dev/loop101 feeling sure that it will work without p1 partitions and it did not work with raw devices. Then I have created gpt partitions and formatted backing disk to ext4 left cache disk w/o formatting (as a guide states) and tried to attached didn't worked either ...

# echo f3c837cb-a743-4615-9a7f-4f647d603eaf > /sys/block/bcache0/bcache/attach
-bash: echo: write error: No such file or directory

And the state is:

# cat /sys/block/bcache0/bcache/state
no cache

It is deatached and cannot be attached ...

What did I do wrong ?

My aim is to simulate raid 10 array formed from virtual disks with ramdisk cache using mdadm and bcache.

Thank You.

Is it possible to connect all 24 disks to one raid card ? How do I connect these physically ? Do I need an expander ? Or is it-mode flushing sufficient ? I read it-mode flushing permits up to 1024 drives and software raid. (mdadm bcache megacli etc usage is not a problem ?

The Hardware I got :

• OS : Linux
• Backplane : BPN-SAS3-846EL-1 https://www.supermicro.com/manuals/other/BPN-SAS3-846EL.pdf
• Raid Cards : LSI 9260-8i , 2 x LSI 2108 pci-e x8
• Cable: SFF-8643 to SFF-8087

One of raid cards were giving multibit ecc ram module failure , thats why I am moving 60TB of data from one server to another ...

Getting 60-100 Mb/s read speeds 3 Megaraid sas raid1 s 8+8+6 x 10TB Disks and spanning them with raid 0 using mdadm when compresisng a folder with tar pigz to another network raid5 folder or same server. I rarely beat 100mb/s for a couple of seconds in an hour (up to 250mb/s) ...

However When I copy a huge file 8 GB it copies with 10G Ethernet 987mb/s. Also hdparm -tT /dev/md0p1 gives 1.5 GB/s.

Cluster size of disks and raid array all 4096 k %75 of files are >30mb there GB files too How can I tar.gz reliably and as fast as possible ... What is wrong with this machine

One of raid cards were giving multibit ecc ram module failure so can it be the culprit behind slow speeds and or is it because disks are mechanical however keep in mind file distribution is like this :

Bytes           Number of Files:        
            0  14
            16  24
            32  21
            64 603
           128 207
           256 1677
           512 2361
          1024  45
          2048  90
          4096 112
          8192 358
         16384 315
         32768 235
         65536 309
        131072 296
        262144 2275
        524288 1148
       1048576 2187
       2097152 3204
       4194304 2708
       8388608 2148
      16777216 703
      33554432 1585
      67108864 906
     134217728 259
     268435456  71
     536870912  42
    1073741824  33
    2147483648  38
    4294967296  16

Here is a little bit human readable version : https://docs.google.com/spreadsheets/d/15J3LsU5G_km70mW0yE6ehiK4oHZmOLuUxRpBCM25-r0/edit?usp=sharing

Any practical workaround to copy all data before I change faulty raid card ?