iscsi initiator failed authorization

[ourhelpdesk]

I'm trying to add a new iSCSI SR on a Synology box, to Xen Hypervisor 7.6 (Build date 2018-08-24)

 

All is well through XenCenter > Storage > New SR > iSCSI > Full Provisioning >  Scan Target Host (without CHAP auth) > Target IQN/LUN (with CHAP auth) -- so far all is good. Some trickery has to be done running the scan target host without chap but acquiring the Target with chap, but that seems to be a known quirk with XenCenter.   At  this time all is fine, look:

 

After clicking finish, it scans for a moment and then fails:

 

I feel I've tried all sorts of combinations using or not using CHAP, but there are no less than 4 different places to add username/passwords in /etc/iscsi/iscsid.conf, but I get furthest with having them all commented out.

 

#node.session.auth.authmethod = CHAP

 

#node.session.auth.username = rcd
#node.session.auth.password = thepassword

 

#node.session.auth.username_in = rcd
#node.session.auth.password_in = thepassword

#discovery.sendtargets.auth.authmethod = CHAP

#discovery.sendtargets.auth.username = rcd
#discovery.sendtargets.auth.password = thepassword

#discovery.sendtargets.auth.username_in = rcd
#discovery.sendtargets.auth.password_in = thepassword

 

In Synology I have tried similarly, with various combinations, but find I get furthest with enabling CHAP, but not Mutual CHAP.

 

 

Help, anyone?

[Mark Syms]

XenServer expect both discovery and login to be using CHAP (or not), there is no support for a mix.

[ourhelpdesk]

Like i wrote, I think I have tried all possible combinations, and ended up with the above getting me the furthest.

 

Could you be a bit more specific please?  Should I uncomment any of the above entries, or is it a matter of entering the login details in the gui? 

[Mark Syms]

You shouldn't change any of the installed files on the XenServer. With a very exceptions changing the files will put you into an unsupported state. The iSCSI storage must accept/expect CHAP authentication on both discovery and session login (or no CHAP on either) for XenServer to be able to connect to it. You cannot for instance have unauthenticated discovery and authenticated session login.

[ourhelpdesk]

Well it didn't work so I had to try something.  I had a look here and came across this thread https://discussions.citrix.com/topic/334529-xenserver-62-synology-dsm-42-iscsi-chap/ which tried different things, including making changes to iscsi.conf.

 

If rather than tell me what I shouldn't do, you could give me hints to how I could get this fixed it would be most appreciated, thanks.    The Synology iscsi setup page does not differentiate between discovery and session login, I would assume the chap applies to both.  As I already explained, I tried both with and without chap, but couldn't get it to work either way.

 

 

[ourhelpdesk]

 

One thing I just noticed though, but I don't know if it's important - while it is scanning it says it's scanning for LVM volumes:

 

 

But when it fails it says scanning for GFS2 SR's (thin provisioning)

 

 

Since I don't have a license (or need) for thin provisioning that will obviously fail.

 

 

But perhaps that's just a bug?

 

 

 

[ourhelpdesk]

Anyone?

 

Btw, I already rolled /etc/iscsi/iscsid.conf back - everything commented out again, as per standard.

 

[Tobias Kreidl]

GFS2 support with thin provisioning requires an Enterprise license. Not sure why you see that error if you're scanning for a standard LVM configuration.

I'd probably do an iscaiadm discovery process at this point on each host and make sure also the iSCSI initiator settings (IQN) are all recognized by your storage device.

 

-=Tobias

[ourhelpdesk]

 

$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.122
192.168.1.122:3260,1 iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644
[fe80::211:32ff:fe78:a48f]:3260,1 iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644

 

It seems the same as on the first picture I included.  Does this help?

[Tobias Kreidl]

That's the first step, to be followed by scanning for nodes. I'd suggest following this very good guide:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/online_storage_reconfiguration_guide/scanningnewdevs-iscsi

At least you can see the storage device, so that's a good start.

-=Tobias

[ourhelpdesk]

$ sudo service iscsi status
- open-iscsi.service - Login to default iSCSI targets
   Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2019-02-27 22:23:46 CET; 8h ago
     Docs: man:iscsiadm(8)
           man:iscsid(8)
 Main PID: 114152 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 19660)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/open-iscsi.service

Feb 27 22:23:46 Debian-Prod systemd[1]: Starting Login to default iSCSI targets...
Feb 27 22:23:46 Debian-Prod iscsiadm[114150]: iscsiadm: No records found
Feb 27 22:23:46 Debian-Prod systemd[1]: Started Login to default iSCSI targets.

$ sudo iscsiadm --mode node --targetname iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644 --portal 192.168.1.122:3260,1
# BEGIN RECORD 2.0-874
node.name = iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644
node.tpgt = 1
node.startup = manual
node.leading_login = No
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.gateway = <empty>
iface.subnet_mask = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.bootproto = <empty>
iface.dhcp_alt_client_id_state = <empty>
iface.dhcp_alt_client_id = <empty>
iface.dhcp_dns = <empty>
iface.dhcp_learn_iqn = <empty>
iface.dhcp_req_vendor_id_state = <empty>
iface.dhcp_vendor_id_state = <empty>
iface.dhcp_vendor_id = <empty>
iface.dhcp_slp_da = <empty>
iface.fragmentation = <empty>
iface.gratuitous_arp = <empty>
iface.incoming_forwarding = <empty>
iface.tos_state = <empty>
iface.tos = 0
iface.ttl = 0
iface.delayed_ack = <empty>
iface.tcp_nagle = <empty>
iface.tcp_wsf_state = <empty>
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.tcp_timestamp = <empty>
iface.redirect = <empty>
iface.def_task_mgmt_timeout = 0
iface.header_digest = <empty>
iface.data_digest = <empty>
iface.immediate_data = <empty>
iface.initial_r2t = <empty>
iface.data_seq_inorder = <empty>
iface.data_pdu_inorder = <empty>
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
iface.chap_auth = <empty>
iface.bidi_chap = <empty>
iface.strict_login_compliance = <empty>
iface.discovery_auth = <empty>
iface.discovery_logout = <empty>
node.discovery_address = 192.168.1.122
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 192.168.1.122
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD

 

Ok, that produced a lot of output - and a lot of <empty>

 

So the next step, according to Redhat, is to "mount the iSCSI device sdb".  That's where the chain falls off again - where does it get sdb from?  The only device name I see is Target-1, is that what they are looking for?

 

 

 

[Tobias Kreidl]

You may now be able to see an sdb device; run "fdisk -l" and see if you do. If so, you may also still have to partition and format the disk (also possible with fdisk).

Here's a good article about this, specific even to Synology storage: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Virtualization/How_to_set_up_and_use_iSCSI_target_on_Linux

 

-=Tobias

[ourhelpdesk]

Still no device, but after having checked the synology page it seems the previous "iscsiadm --mode node" command was missing --login:

root@Debian-Prod:~# iscsiadm -m discovery -t st -p 192.168.1.122
192.168.1.122:3260,1 iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644
[fe80::211:32ff:fe78:a48f]:3260,1 iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644
root@Debian-Prod:~# iscsiadm --mode node --targetname iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644 --portal 192.168.1.122:3260,1 --login
Logging in to [iface: default, target: iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644, portal: 192.168.1.122,3260] (multiple)
Login to [iface: default, target: iqn.2000-01.com.synology:DS216j.Target-1.c4487e1644, portal: 192.168.1.122,3260] successful.
root@Debian-Prod:~# fdisk -l /dev/sda
Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

So, in other words, I can connect to the datastore now. 

So far without authentication. There seem some way to use CHAP authentication but it is not really clearly explained, but this is on my private network so not a huge deal. Would be nice to have though.

So, the good news is I am now able to connects to an iscsi device on synology, and a quick tests shows it is very fast, about as fast as the internal disks in the server, and about 3 times faster than accessing the disks through USB3:

# /tmp = internal FS
root@Debian-Prod:~# dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 4.80215 s, 83.8 MB/s

# /mnt = iscsi device
root@Debian-Prod:~# dd if=/dev/zero of=/mnt/output conv=fdatasync bs=384k count=1k; rm -f /mnt/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 4.32812 s, 93.0 MB/s

# /media/usb = same device attached through USB
root@Debian-Prod:~# dd if=/dev/zero of=/media/usb/share/output conv=fdatasync bs=384k count=1k; rm -f /media/usb/share/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 11.0772 s, 36.3 MB/s

 

The bad news is, the xencenter interface still can't connect.   I can mount a LUN for data within the VM but in order to use a LUN as system drive it would have to be configured in xencenter or the xen control node.   

I made some changes to /etc/iscsi/iscsid.conf at an earlier time, maybe that is the problem. Anyone have an example of a correct /etc/iscsi/iscsid.conf ?

[Rafael Pereyra]

Hello,

 

I just went through the same problem with a Synology device.

 

I was able to add the Storage from CLI following the instructions here https://www.tecmint.com/xenserver-create-and-add-storage-repository/

 

This points to a bug in the SR Wizard logic (for example using the CLI it never asked to format the device as a LVM, my guess is the Wizard expects the block device to be empty and Synology do format the LUN when it creates it). This only happens with Synology device by the way, on the same rack I have a FreeNAS server and it hosts the main SR without any complains from XenCenter, very straight forward.

 

I haven't fully tested it yet as the machine was shipped to one of our branches today, however for testing purposes I created a template from a Snapshot and then move the VDI to the new SR without any issues (in my case this will be the repository of old snapshots for DSR purposes).

 

Good luck, hopefully I'm not too late with a possible solution!

[jamie bennion1709153033]

Things like this are why i am leaving XenServer / Citrix Hypervisor.

 

I have the same problem with a QNAP that I've even previously connected to just fine. Right next to another QNAP that IS connected to just fine.

 

I don't know if anyone from Citrix reads these sections, but the oddball problems that pile up and never get addressed are just too much. If you are running XS in production you can't exactly justify tinkering around under the hood trying to find workaround #46 to yet another problem.

Here I've even got CHAP turned off. I've tried every combination of settings.

 

If you want to do this via CLI :

 

get a list of IQNs :
 

xe sr-probe type=lvmoiscsi \
device-config:target=[ip address] device-config:port=3260 \
device-config:chapuser="chapdude" device-config:chappassword="chapdudepw"

output like :
.. blah...
iqn.2004-04.com.qnap:ts-ec1279u-rp:iscsi.targetname.d18579
.. blah ..
..
.

get SCSIid for an IQN :
 

xe sr-probe type=lvmoiscsi device-config:target=[ip address] \
device-config:port=3260 \
device-config:chapuser="chapdude" device-config:chappassword="chapdudepw" \
device-config:targetIQN=iqn.2004-04.com.qnap:ts-ec1279u-rp:iscsi.targetname.d18579

output like :
36e843b6b622f450d8d8fd4a56d8a77dc

create the SR :
 

xe sr-create name-label="my bitchin iscsi sr lol" type=lvmoiscsi \
content-type=user shared=true device-config:target=[ip address] \
device-config:port=3260 \
device-config:targetIQN=iqn.2004-04.com.qnap:ts-ec1279u-rp:iscsi.targetname.d18579 \
device-config:chapuser="chapdude" device-config:chappassword="chapdudepw" \
device-config:SCSIid=36e843b6b622f450d8d8fd4a56d8a77dc

Also it seems to not like exclamation points in the CHAP username or password. Probably other BASH reserved symbols I bet.

 

If you are attaching to a POOL, that shared=true is important. If you aren't running a pool and you don't shared=true then you have to also supply the host-uuid you want to add the SR to.