XenServer + Dell Compellent +Multipathing

ABICO Licensing · September 6, 2019

Hi all -

I have a problem similar to the one discussed here:

https://discussions.citrix.com/topic/397933-xenserver-add-session-path/

but it's not quite the same.

I have eight XenServers in a pool, connected to a pair of Dell Compellents… each Compellent has four iSCSI interfaces, they are configured in replication so effectively present as unit. Each Compellent serves six SRs. The pool is multipathed to the Compellent. Five of the SRs are old, one is new., The "problem" is with the new one.

I am able to scan/attach absolutely fine, however for the five old SRs I see:

4 of 4 paths active (4 iSCSI sessions)

whereas on the new SR I see:

4 of 4 paths active (2 iSCSI sessions)

mpathutil status confirms four active sessions:

An example old SR:

36000d31000c8ac000000000000000003 dm-6 COMPELNT,Compellent Vol

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=enabled
|- 8:0:0:1 sdh 8:112 active ready running
|- 9:0:0:1 sdk 8:160 active ready running
|- 10:0:0:1 sdp 8:240 active ready running
`- 21:0:0:1 sdv 65:80 active ready running

The new SR:
36000d31000e17e00000000000000004f dm-15 COMPELNT,Compellent Vol
size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:7 sdab 65:176 active ready running
|- 7:0:0:7 sdag 66:0 active ready running
|- 6:0:0:7 sdaf 65:240 active ready running
`- 10:0:0:7 sdae 65:224 active ready running

The only odd thing I've found (other than 4 vs 2) is:

[root@xen01 iscsi]# iscsiadm -m node -p 10.0.15.45 --rescan
Rescanning session [sid: 15, target: iqn.2002-03.com.compellent:5000d31000e17e22, portal: 10.0.15.45,3260]
Rescanning session [sid: 16, target: iqn.2002-03.com.compellent:5000d31000e17e25, portal: 10.0.15.45,3260]
iscsiadm: invalid error code 65280
iscsiadm: Could not execute operation on all sessions: (null)

I get the same result of any of the Compellent's IPs.

Related, I have tried attaching to both the "primary" and "secondary" Compellent with the same result.

Finally, I will mention that this install predates me, and I notice NO custom config in the multipath.conf as I'd expect for the Compellent. However, seeing as six of seven SRs have been working for literally years, whatever cause for concern here must be minor.

I cannot understand what causes the new SR to have fewer sessions to the *exact same* iSCSI target. TBH, I'm not sure it matters! I just would like to understand the discrepancy more than anything.

Hopefully someone can help... as you might imagine I'm out of ideas. :(

Tobias Kreidl · September 6, 2019

Did you do an iscaiadm discovery process and then from XenCenter, use the wildcard interface to connect the new server to the storage interface?

-=Tobias

Alan Lantz · September 6, 2019

I would blame storage as the SR is seeing all of the paths, but its not able to log into all of the sessions.

Something has to explain the difference.

--Alan--

ABICO Licensing · September 6, 2019

I am inclined to blame the storage as well, but I'm not exactly sure how. :)

I did do discovery - eg:

iscsiadm -m discovery -t sendtargets -p 10.0.15.40

(there are multiple interfaces, obviously)

I initially tried logging in automatically:

iscsiadm -m node -L all

but that didn't work, so I then detach/forgot the SR and tried logging in individually, eg:

iscsiadm -m node iqn.2002-03.com.compellent:5000d31000c8ac20 --login

but netted the same result.

The connection in XenCenter was made using the wildcard, which you can effectively see in the autogenerated description:

iSCSI SR [10.0.15.40 (*; LUN 7: 0000e17e-0000004f: 1024 GB (COMPELNT))]

What I would expect to see is *also* 10.1.15.40 * but I don't.

The thing is, this storage is presented by the exact same device/controller/IP/IQN/everything as the existing SRs. If it's using 4 sessions there, wouldn't the iscsi db direct a new LUN on existing logons appropriately?

Am I correct that when creating a new iSCSI SR from XenCenter, the actual processing is happening from the pool master?

Tobias Kreidl · September 6, 2019

Odd, as the iscsiadm discovery should reveal all paths. Any issue with the IP assignments, possibly multiple IQNs with the same name, or something like that, or firewall restrictions? That you can see part of the paths is odd. Does the multipath.conf file reflect the same configuration as the other hosts?

And, yes, most actually commands run via the pool master as it has to update the database to track changes in the configuration.

-=Tobias

Alan Lantz · September 6, 2019

I don't know Compellant, so I can't be much help. But there could be backend permissions, LUN access, different target assignment

or something else causing this. I wonder if you detached from an old LUN (if you can), if a reattachment logs in as two or four

sessions. This sort of thing is very hard to sort out. Also, I would search/ask for a multipath.conf that matches XenServer for your

hardware. Just because it work doesn't necessarily mean its working right. Incorrect settings in multipath.conf could be introducing

this weirdness.

--Alan--

ABICO Licensing · September 9, 2019

Tobias: Should be no issue with IP/IQN, and I'd guess that the fact 5 of 6 SRs mount properly on all eight hosts. The only thing that separates this sixth SR is that it's a different LUN. Same target! There is no "special" multipath configuration in multipath.conf which I *know* is wrong... but unfortunately I cannot afford any downtime in which to fix it. The folks that originally configured this didn't do very much by the book, which always leads to problems when troubleshooting issues like this. The only thing I really have to work from is "it worked before, for three years." :)

Alan: I am SERIOUSLY considering trying to detach/reattach an existing LUN, but that's very problematic as it means shutting down a slew of VMs as they have nowhere to live during the experiment. I don't have that many that are non-essential. I do agree about multipath.conf, and even though "mostly working right" could give the appearance of "all working right," I certainly don't like *knowing* someone missed a configuration point.

I am going to try and promote another host to poolmaster and see if I get a different result.

Is there any way (or any harm from) purging the session db? Will that disrupt existing sessions?

I really appreciate your feedback!

Alan Lantz · September 9, 2019

Yuck, I hate doing anything like purging sessions on a production system. I wouldn't do that myself.

--Alan--

Tobias Kreidl · September 9, 2019

I would hope a full detaching would not be necessary; you've got a connection that needs be be "woken up".

See if maybe this article helps, despite its age: https://docplayer.net/10933147-Using-dell-equallogic-and-multipath-i-o-with-citrix-xenserver-6-2.html

-=Tobias

Tobias Kreidl · September 9, 2019

And perhaps try this multipath.conf entry:

device {
		vendor "COMPELNT"
		product "Compellent Vol"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
}

Alan Lantz · September 9, 2019

Since Multipath is per server you can try those settings Tobias give you on just one for testing.

--Alan--

ABICO Licensing · September 9, 2019

Ah, I did not realize multipath.conf was per server ... I have two totally idle hosts I can try that on first. Thank you!

As of now, I have moved the pool master (mostly for grins) and completely killed this "rogue" SR - purged it from Xen AND Compellent. I'm going to let it stew overnight. Tomorrow I will insert that multipath.conf (I think an earlier search may have lead me to your prior response with that!), and recreate the LUN and related SR after "rediscovering" the four iSCSI targets & logging in. Maybe that will get me where I need to be.

Edit: Is there any preference on restarting iscsid vs rebooting the entire host after changing multipath.conf?

I will be sure to follow up.

Tobias Kreidl · September 9, 2019

I suggested earlier to check to make sure the multipath.conf file was correct for all servers.

I think restarting the multipath daemon should be all that is needed:

% service multipathd restart

or

% systemctl restart multipathd

You may be able to just run:

% systemctl reload multipathd

-=Tobias

Alan Lantz · September 10, 2019

I would probably put the server in maintenance mode, restart multipathd, and take the server out of maintenance mode.

But thats just me being overly cautious on how XenServer interacts with that service.

--Alan--

Tobias Kreidl · September 10, 2019

No guts, no glory!

Alan Lantz · September 10, 2019

I'm getting to where I don't even want to install hotfixes.

Tobias Kreidl · September 10, 2019

I hear you, man. The biggest issues I have had are the changes that sometimes take place in the CPU masks, making XenMotion impossible via XenCenter since now various nodes of the pool are not compatible with each other. Why XenCenter doesn't have an override option is beyond me.

ABICO Licensing · September 10, 2019

I bailed out of XenServer around 2015 and went full VMware due to these types of issues, but I've changed jobs and found myself back in the thick of it! :) There are some things I missed about Xen, but some things I did not.... like never knowing specifically what action is going to break what thing. :)

I'm about ready to get going on this... will report back.

Tobias Kreidl · September 10, 2019

Good luck -- remember, it's not a job, it's an adventure!

Alan Lantz · September 10, 2019

Had XenServer gained marketshare and been competitive it could have been an awesome platform.

At this stage of development against VMWare and Hyper-V its losing appeal. There are some niche

areas like customers with heavy Citrix investment or GPU scenarios, but core virtualization functionality

there is just too many options and too much competition.

--Alan--

ABICO Licensing · September 10, 2019

Update -

I tried the multipath config above, and that netted a bunch of multipath errors - specifically a reduction in active paths:

2 of 4 paths active (4 iSCSI sessions)

I went ahead and redid discovery & login, and now everything is back to normal on that server for all "old" SRs:

4 of 4 paths active (4 iSCSI sessions)

It *seems* things are improved, as a single login command logged into *all* possible paths as it should:

[root@xen07]# iscsiadm -m node iqn.2002-03.com.compellent:5000d31000c8ac1f --login
Logging in to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e24, portal: 10.1.15.45,3260] (multiple)
Logging in to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e22, portal: 10.0.15.45,3260] (multiple)
Logging in to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e21, portal: 10.1.15.45,3260] (multiple)
Logging in to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e25, portal: 10.0.15.45,3260] (multiple)
Login to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e24, portal: 10.1.15.45,3260] successful.
Login to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e22, portal: 10.0.15.45,3260] successful.
Login to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e21, portal: 10.1.15.45,3260] successful.
Login to [iface: default, target: iqn.2002-03.com.compellent:5000d31000e17e25, portal: 10.0.15.45,3260] successful.

I want a good way to test this config to be sure there aren't issues, so I will work on that...

I'm not sure whether it's prudent to either:

1. Move the VMs from the pool master to this machine before recreating & reattaching the new LUN or

2. Promote this host to pool master before reattaching & recreating the new LUN or

3. Do nothing and just see what happens. :)

#1 is the difficult one, so, you know, I'd rather not. :)

ABICO Licensing · September 10, 2019

1 hour ago, Alan Lantz said:

Had XenServer gained marketshare and been competitive it could have been an awesome platform.

At this stage of development against VMWare and Hyper-V its losing appeal. There are some niche

areas like customers with heavy Citrix investment or GPU scenarios, but core virtualization functionality

there is just too many options and too much competition.

--Alan--

Yes, for sure. In the 2000s I bet on Xen and it turned out I'd made the wrong call. :)

I have several XCP-ng hosts running now and have been having good luck with it and XOA. I may make that move for these machines someday. But, honestly, aside from some quirks Xen 6.x and 7.x are pretty stable and run pretty well. It's hard to be genuinely mad at the platform. My issue is that *good* backup solutions (like Veeam, Unitrends, etc.) are simply non-existent for Xen, and XCP does solve that issue for me.

Tobias Kreidl · September 10, 2019

Not the snazziest thing in the world as far as backup/restore utilities for XenServer / Citrix Hypervisor are concerned, but totally free: https://github.com/NAUbackup/VmBackup

Alan Lantz · September 10, 2019

And not to knock NAUBackup, but UniTrends does do backup of XenServer.

--Alan--

ABICO Licensing · September 10, 2019

14 minutes ago, Alan Lantz said:

And not to knock NAUBackup, but UniTrends does do backup of XenServer.

--Alan--

It does? Since when? This is very exciting!

Edit: Wait, I seem to remember it always did... but there is some limitation... like it can't do differentials (incrementals, depending on when you were born :) or can't test them or something... Am I misremembering?

Apparently I am... man, you just changed my week!

Alan Lantz · September 10, 2019

It can do incrementals/differentials, and there of course are many ways to go about it.

But yes, one limit is it does snapshots and merges that into its database, so you do

need a lot of extra space. Full backups is initially done then you can do forever incrementals

that merge into this base image.

What oddity I ran into is with a physical appliance you can't merge into that XenServer pool

as cleanly anymore, so what I did is I install their client on all of my VM's and do a image

backup from within. The big advantage is it shortened my backup time tremendously.

The downside recovery is more difficult, but I rarely do restores.

--Alan

XenServer + Dell Compellent +Multipathing

Question

Link to comment

46 answers to this question

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Create an account or sign in to comment

Create an account

Sign in