Questions[open-iscsi](server)

I'm using Proxmox VE platform, version 6 or 7, and I'm currently documenting the initial setup with the idea of automating it later. So I'm looking for all command line commands to do all that I must do on the web UI.

My problem is in the storage configuration step: I try to configure an LVM volume on top of an iSCSI LUN.

If I do this setup with the web UI, the /etc/pve/storage.cfg looks like

iscsi: <ID1>
    portal <portal>
    target <target>
    content none

lvm: <ID2>
    vgname <vgname>
    base <ID SCSI>
    content rootdir,images
    shared 1

I can use the pvesm utility to do it with the following commands

pvesm add iscsi <ID> --portal <SERVER> --target <TARGET> --content none

pvesm add lvm <ID> --vgname <vgname> --base <ID SCSI>

The problem is I don't know how to find this ID SCSI, neither using any PVE or open-iscsi tool.

ps: the "ID SCSI" looks like <ID1>:0.0.0.scsi-<1234000afbc....>

I have two oldish "NAS" devices, specifically Buffalo TeraStation Rackmount iSCSI ("TS-RIXL/R5"). Those are basically embedded computers running Linux with four SATA slots and two Ethernet ports, with a firmware that shows up as an iSCSI target.

With both devices, i did the following:

I put in four new 16TB Seagate Exos X16 drives, used Buffalo's Windows "Firmware update" tool to install the latest (1.66) firmware on them, and created a RAID5 array.

Then, on an amd64 machine running Debian Buster with standard kernel and tools, I tried to setup an encrypted filesystem:

# find the device
iscsiadm -m discovery -t st -p <ip_address>:3260
# connect it to the initiator's SCSI subsystem
iscsiadm  -m node  --targetname "iqn.2004-08.jp.buffalo:<device_id>:vol1" --portal <ip_address>:3260,1" --login
# this made /dev/sda appear

# create an encrypted device
cryptsetup luksFormat /dev/sda
cryptsetup open /dev/sda nas

# create a filesystem and mount it
mke2fs -t ext4 /dev/mapper/nas
mount /dev/mapper/nas /mnt/nas/

# make a directory
mkdir /mnt/nas/store

This appeared to work fine, but immediately made the follwoing error appear in dmesg and syslog:

EXT4-fs error (device dm-0): ext4_validate_block_bitmap:384: comm mkdir: bg 18736: bad block bitmap checksum

Copying some gigabytes of data to the device then also made errors appear such as

EXT4-fs (dm-0): Delayed block allocation failed for inode 479102523 at logical offset 8708096 with max blocks 128 with error 74
EXT4-fs (dm-0): This should not happen!! Data will be lost

This is reproducible with both NAS devices, ruling out some hardware fault.

The devices have been working flawlessly for years (with iSCSI, without encryption).

If I do the same thing now without the "cryptsetup" part, make the filesystem on /dev/sda directly and mount it, I can work with multiple gigabytes of data without errors appearing in the syslog, and without having any data corruption.

On the other hand, using the same procedure on the Debian machine, setting up an encrypted filesystem on a hard disk connected via USB (instead of iSCSI), works fine as well.

So:

ext4 => dm-crypt => iSCSI => Buffalo NAS: "This should not happen!! Data will be lost"

ext4 => iSCSI => Buffalo NAS: works

ext4 => dm-crypt => external USB disk: works

(where "works" means: i copied around 10'000 files with a size of about 48 GB, and verified using md5sum that all files were there and were not corrupted)

I then repeated the tests on a machine running Ubuntu hirsute (21.04), and got the same results.

Is there some known problem running dm-crypt on top of iSCSI? Do I maybe need to have a special configuration for the iSCSI part, for example some configuration for the caching or block size?

I am currently setting up a SAN for diskless boot. My backend consists of ZFS-Vol shared via iSCSI. So far everything is working just fine except for TRIM/UNMAP. For test puposes I setup two VMs running Ubuntu20.04 in VirtualBox networked together via an internal network with static IPv4 addresses. On the target (tgt) got a second virtual drive formatted with ZFS. On this zpool I created a zVol and formatted it with GPT and ext4.

/etc/tgt/conf.d/iscsi.conf
<target example.com:lun1>
    <backing-store /dev/zvol/tank/iscsi_share>
        params thin_provisioning=1
    </backing-store>
    initiator-address 192.168.0.2
</target>

On the initiator (open-iscsi) I use this command to provoke a TRIM operation:

sudo mount /dev/sdb1 /iscsi-share
sudo dd if=/dev/zero of=/iscsi-share/zero bs=1M count=512
sudo rm /iscsi-share/zero
sudo fstrim /iscsi-share

but the shell responds with "fstrim: /iscsi-share: the discard option is not supported". If I issue those commands on the target machine the "REFER" property of the zVol decreases as expected.

As I found nothing while searching the web I found no hint as to why this is not working or if this is even possible at all.

Edit: As I got the advice to use the option thin_provisioning.

After I repartitioned the drive and mounted it on the initiator I got error message blk_update_request: critical target error, dev sdb, sector 23784 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0 for several sectors and after creating and deleting my testfile, fstrim send the message

blk_update_request: I/O error, dev sdb, sector 68968 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0
fstrim: iscsi-share: FITRIM ioctl failed: Input/output error

Edit: As there were Answers refering to LIO I now also tried targetcli. There I setup a target with my zVol under /backstores/block/iscsi and set attribute emultate_tpu=1. After importing this into my initiator I repartitioned, formatted and mounted it on the initiator. Then I created my test file, deletetd it and issued the fstrim command and it worked. Thanks for the help.

I have two iSCSI target servers. The iSCSI Initiator is an Ubuntu host (Ubuntu 14.04). I am using Ubuntu's open-iscsi to mount the iSCSI targets from both iSCSI targets servers to my Initiator.

The problem is that the symlinks in /dev/disk/by-id are "overwritten".

When I do the iscsiadm discovery and the iscsiadm -m node -l command to the first iSCSI target server, the targets are mounted and there are symlinks under /dev/disk/by-id.

For example, doing an ls -l /dev/disk/by-id after logging in to the first iSCSI target server might show the following:

scsi-3600000e00010001 -> ../../sdc
scsi-3600000e00010002 -> ../../sdd
scsi-3600000e00010003 -> ../../sde
wwn-0x60000e00010001 -> ../../sdc
wwn-0x60000e00010002 -> ../../sdd
wwn-0x60000e00010003 -> ../../sde

After doing a discovery and a login to the 2nd iSCSI target server (without logging out of the iSCSI target server), doing an ls -l /dev/disk/by-id shows something like this:

scsi-3600000e00010001 -> ../../sdaa
scsi-3600000e00010002 -> ../../sdab
scsi-3600000e00010003 -> ../../sdac
wwn-0x60000e00010001 -> ../../sdaa
wwn-0x60000e00010002 -> ../../sdab
wwn-0x60000e00010003 -> ../../sdac

1) I checked that all the iSCSI targets are mounted on the Ubuntu initiator by issuing an fdisk -l command. The iSCSI targets are all mounted.

2) But the symlinks point to targets from the 2nd server now. It looks like the symlinks to the targets from the 1st server were overwritten by symlinks to the 2nd server.

Is it possible to have all the iSCSI targets, from both iSCSI target servers, have separate symlinks and be listed under /dev/disk/by-id?

I am not an iSCSI expert so I thought I'd ask. I Googled this problem and searched for relevant tags here but I didn't see any answers. Maybe I am not configuring something correctly?

A workaround would be for me to write a script that ccreates individual, unique symlinks for all the iSCSI targets but I'd first like to see if I am misconfiguring something.

So after upgrading to CentOS 6.2, I am seemingly no longer able to login into my iSCSI targets. I have multiple interfaces on different subnets on the system, and I first thought that it had to do with the fact that I may not be binding correct interfaces, which seems to be the case when looking at netstat, as this is clearly wrong:

[root]⌘ netstat -na|grep .90
tcp        0      1 10.10.100.60:42354          10.10.8.90:3260             SYN_SENT    
tcp        0      1 10.10.100.60:40777          10.10.9.90:3260             SYN_SENT 

I then went ahead and disabled all but one interface, and so as a result netstat appears to be correct, but the issue with login remains. I am positive that the target never sees a packet, because I see nothing by SYN_SENT. I know the problem is on my client, because the target is servicing multiple systems, none of which are CentOS 6.2. At this point I am pretty confident that some things changed between CentOS 6.0/6.1 and 6.2. So, if anyone have any thoughts, or ran into this, I would very much like to hear your thoughts.

[root]⌘ iscsiadm --mode node --targetname iqn.2011-12.dom.homer:01:lab-centos-servers-00001 --portal 10.10.8.90:3260,2 --interface=sw-iscsi-0 --login
Logging in to [iface: sw-iscsi-0, target: iqn.2011-12.dom.homer:01:lab-centos-servers-00001, portal: 10.10.8.90,3260] (multiple)
iscsiadm: Could not login to [iface: sw-iscsi-0, target: iqn.2011-12.dom.homer:01:lab-centos-servers-00001, portal: 10.10.8.90,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals


[root]⌘ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.8.0       0.0.0.0         255.255.255.0   U         0 0          0 eth2.7
10.10.9.0       0.0.0.0         255.255.255.0   U         0 0          0 eth3.7
10.10.100.0     0.0.0.0         255.255.252.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth3
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth2.7
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth3.7
0.0.0.0         10.10.100.1     0.0.0.0         UG        0 0          0 eth0

Output of ip addr show for the two interfaces involved:

[root]⌘ for i in 2.7 3.7; do ip addr show eth$i; done
6: eth2.7@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:0c:29:94:5b:8d brd ff:ff:ff:ff:ff:ff
    inet 10.10.8.60/24 brd 10.10.8.255 scope global eth2.7
    inet6 fe80::20c:29ff:fe94:5b8d/64 scope link 
       valid_lft forever preferred_lft forever
7: eth3.7@eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:0c:29:94:5b:97 brd ff:ff:ff:ff:ff:ff
    inet 10.10.9.60/24 brd 10.10.9.255 scope global eth3.7
    inet6 fe80::20c:29ff:fe94:5b97/64 scope link 
       valid_lft forever preferred_lft forever

Update 01/06/2012:

This issue is getting even more interesting by the day it seems. I went a few weeks back and grabbed a snapshot of this system from before upgrading to 6.2. I spun up a new system from the snapshot, and reconfigured interface info and host keys, as well as iSCSI initiator and iscsi interface info to match new MACs. Changed nothing else.

Then, I attempted to connect to my targets, and no issues at all. I cannot say that this was unexpected. I then went ahead and compared sysctl settings from both systems and there were differences after the upgrade, but nothing seemingly relevant to iSCSI or IP that could contribute to this. I also noticed that by default now two sessions per connection were enabled after the upgrade, but I changed it back to 1 session in /etc/iscsi/iscsid.conf.

On the problematic system we can see that source interface is seemingly wrong, but even when I disable the 10.10.100 interface, problems persist. So, while this may be relevant, I could not validate it for certain. Needless to say, further research is necessary. Something is clearly different between releases. Working system is on 6.1, and non-working is 6.2.

::Working System::
tcp        0      0 10.10.8.210:39566           10.10.8.90:3260             ESTABLISHED 
tcp        0      0 10.10.9.210:46518           10.10.9.90:3260             ESTABLISHED 

[root]⌘ ip route show
10.10.8.0/24 dev eth2.6  proto kernel  scope link  src 10.10.8.210 
10.10.9.0/24 dev eth3.7  proto kernel  scope link  src 10.10.9.210 
10.10.100.0/22 dev eth0  proto kernel  scope link  src 10.10.100.210 
169.254.0.0/16 dev eth0  scope link  metric 1002 
169.254.0.0/16 dev eth2.6  scope link  metric 1006 
169.254.0.0/16 dev eth3.7  scope link  metric 1007 
default via 10.10.100.1 dev eth0

::Non-working System::
tcp        0      1 10.10.100.60:44737          10.10.9.90:3260             SYN_SENT    
tcp        0      1 10.10.100.60:55479          10.10.8.90:3260             SYN_SENT

[root]⌘ ip route show
10.10.8.0/24 dev eth2.6  proto kernel  scope link  src 10.10.8.60 
10.10.9.0/24 dev eth3.7  proto kernel  scope link  src 10.10.9.60 
10.10.100.0/22 dev eth0  proto kernel  scope link  src 10.10.100.60 
169.254.0.0/16 dev eth0  scope link  metric 1002 
169.254.0.0/16 dev eth2.6  scope link  metric 1006 
169.254.0.0/16 dev eth3.7  scope link  metric 1007 
default via 10.10.100.1 dev eth0 

And the result is still same:

[root]⌘ iscsiadm: Could not login to [iface: sw-iscsi-0, target: iqn.2011-12.dom.homer:01:lab-centos-servers-00001, portal: 10.10.8.90,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: sw-iscsi-1, target: iqn.2011-12.dom.homer:02:lab-centos-servers-00001, portal: 10.10.9.90,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals

Update 01/08/2012:

I believe I have been able to figure out the answer to my issue. It is quite obscure and I doubt this will happen to anyone else any time soon. It turns out that setting iface.iscsi_ifacename and iface.hwaddress in the interfaces configuration file is not legal. When one manually adds an iscsi target, such as below, all settings from the interface config file are copied into the node config file, that gets created by the below command. Result is parameters iface.iscsi_ifacename and iface.hwaddress together in the same config file. These parameters are seemingly mutually exclusive, which does not exactly make sense, or there is perhaps an oversight in the codepath. Perhaps I will investigate further.

# iscsiadm -m node --op new -T iqn.2011-12.dom.homer:01:lab-centos-servers-00001 -p 10.10.8.90,3260,2 -I sw-iscsi-0
# iscsiadm -m node --op new -T iqn.2011-12.dom.homer:02:lab-centos-servers-00001 -p 10.10.9.90,3260,2 -I sw-iscsi-1

Notice, below I commented out iface.hwaddress and iface.ipaddress, after which I re-added targets, with same command as above. All works just fine.

[root]⌘ cat *
# BEGIN RECORD 2.0-872.33.el6
iface.iscsi_ifacename = sw-iscsi-0
iface.net_ifacename = eth2.6
#iface.hwaddress = XX:XX:XX:XX:XX:XX 
#iface.ipaddress = 10.10.8.60
iface.transport_name = tcp
iface.vlan_id = 6
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD
# BEGIN RECORD 2.0-872.33.el6
iface.iscsi_ifacename = sw-iscsi-1
iface.net_ifacename = eth3.7
#iface.hwaddress = XX:XX:XX:XX:XX:XX
#iface.ipaddress = 10.10.9.60
iface.transport_name = tcp
iface.vlan_id = 7
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
# END RECORD

Again, chances of this happening to someone else are slim to none, so likely waste of time typing this up. But, if someone does encounter this issue, I hope this post will help.