Jump to content


Photo

Upgrade Xenserver 5.5 to 5.6 - performance issues

Started by Wesley Petterson , 01 June 2010 - 02:53 PM
68 replies to this topic

Wesley Petterson Members

Wesley Petterson
  • 4 posts

Posted 01 June 2010 - 02:53 PM

Hello,

We have upgraded our farm Hosts and VM's to Xenserver 5.6. Now we are experiencing Performance issues on all of our hosts and VM's.

(Almost all VM's are updated, some not. Those who are not updated to 5.6 are running fine. The host of these VM's are updated to 5.6)

We notice that everything is very slow and sometimes the VM's hang for some time.

Is there maybe a bug in this update or did we forgot something.

Hope some one can help.

+We think it has something to do with the upgrade of the Xen Tools.+



Dan LeBaron Members

Dan LeBaron
  • 273 posts

Posted 01 June 2010 - 08:26 PM

Have you tried to uninstall the XenTools completly, reboot the install XenTools 5.6 ? Also, make sure to only install from the XenCenter console and not through an rdp session mounting the drive.



Wesley Petterson Members

Wesley Petterson
  • 4 posts

Posted 01 June 2010 - 09:38 PM

I was affrayed that this answer would come up. It is almost impossible to reboot the servers (VM's). I'm affrayed that we are forced to work at night to realize this. I will try it tomorrow night with one of our VM's. (I'm talking about 50 VM servers and 8 hosts)

Is there no one else that has experienced this issue? I'm hoping that you answer is the solution.

If some one else has another idea I'm open for it.



James Warburton Members

James Warburton
  • 6 posts

Posted 02 June 2010 - 09:30 PM

I too am having extreme issues with Windows XP Pro VMs on XenServer 5.6 - but again, only when I have installed the 5.6 Xen Tools.

If I log on into safe mode and remove the Xen Tools from the add/remove programs control panel app, the performance returns to acceptable levels (but not as good as with the Xen Tools 5.5 loaded obviously).

All my Windows 2003/R2/2008 servers have worked perfectly. My VMs have all had the Xen Tools loaded from XenCenter using the 'default desktop' so no RDP sessions.



James Warburton Members

James Warburton
  • 6 posts

Posted 02 June 2010 - 10:56 PM

I've now started having similar issues with some Windows 2003 servers (R2 and 2008/R2 don't appear to be affected). It is definitely driver related as Safe Mode works at full speed.

I hope someone can shed some light on this soon as I have 5 VMs that I can no longer use XenMotion on (as they don't have the Xen Tools installed).



Oved Lourie Citrix Employees

Oved Lourie
  • 7 posts

Posted 03 June 2010 - 12:55 AM

Can you try creating a new vm and attaching the storage from one of the vm's that you are having an issue with to see if the problem persists? This way you will be using a new template for the os from 5.6. This would be my first recommendation. Is the performance related to network or just general sluggishness within the vm?
If you leave the XenServer tools installed and boot the machine up with no nic, does the machine act normal?

Edited by: Oved Lourie on Jun 2, 2010 8:56 PM



James Warburton Members

James Warburton
  • 6 posts

Posted 03 June 2010 - 09:03 AM

I have to admit, I am really angry with Citrix - this upgrade has caused major issues for me and clearly for some others. The Xen Tools 5.6 will not work correctly on any of my Windows 2003 Server or Windows XP Pro guests. It makes them completely unusable - network ping times in excess of 1000ms with regular drops, terrible performance on the console access or via RDP.

I've had to reinstate a 5.5 server just to resolve this issue - the added pain is that the 5.6 Xen Tools change the network configuration of some of the machines when it is removed meaning you've got to reset all the IP addresses once you've gone through the uninstall process. Without the Tools installed you can't move the VMs between hosts and I can't find a way to get the 5.5 version which worked perfectly, back onto these machines.

What kind of testing was done on this latest patch? - you force the userbase to install the upgrade on the master server first which is just downright dangerous. Did no one think to try out the latest Tools on 2003 and XP guests? Did nobody mention this in the Beta program?

My mistake was taking advice from an email flyer telling me to upgrade to 5.6. At least I knew not to do this on the production environment. It still doesn't help recoup the time though.



Oved Lourie Citrix Employees

Oved Lourie
  • 7 posts

Posted 03 June 2010 - 01:32 PM

James,
Please contact me to address this issue. I have sent all of my contact info over in a separate reply.
Regards,
Oved



Wesley Petterson Members

Wesley Petterson
  • 4 posts

Posted 03 June 2010 - 03:22 PM

I totally agree with James, We have done the upgrade because it was recommended by Citrix. Also the new features are great and important to our environment. We have 8 xenserver hosts (on HP DL380 G5) with 50 vm's which include 30 XenApp (Terminal) servers (mainly Windows 2008 server). All the VM's are running on a in combination with a NetApp FAS.

We are running XenServer 5.6 on our production environment. (So you understand that it has a great impact on our environment, and that’s why we are not happy..) We have installed XenServer tools 5.6 on some servers not all. On these servers some applications perform (random) slow. On some servers we have still running the XenServer tools 5.5. It seems that these VM's perform normally.

Now we have changed the default settings (actually disabled) on the XenApp servers with XenServer tools 5.6 some settings on the properties off the Citrix PV Ethernet, we have tried the following settings:

Allow blank checksums : No
Offload IP Checksum : Disabled
Offload TCP Checksum : Disabled
Offload UDP Checksum : Disabled

Now it seems that these XenApp servers with the applications act normal and also the performance is back.
First we did also disable the settings on the domain controllers, but this wasn't a very good idea. We just get then more problems with the DHCP-server and DNS-server running on these DC's. So we decided to implement these settings only on the XenApp servers.

We have read some issues with DisableTaskOffload register settings http://support.microsoft.com/kb/904946 . I think this has something to do with each other. We have chosen to disable it on the NIC properties. For now this seems to work. But these settings are not default.



Dan LeBaron Members
  • #10

Dan LeBaron
  • 273 posts

Posted 03 June 2010 - 03:51 PM

We have had to use the DisableTaskOffload register settings since XenApp 5.0 on any Windows 2003 server due to performance issues, though this hasnt been needed on Windows 2008 or 2008 R2.

I have our Test Xen 5.6 server up and running with a few guests running Windows 2008 R2 server, Windows 2003 and Windows 2008. It seems in our testing that Windows 2008 R2 hasnt had any issues, bu tthe other flavors of the OS are experiencing performance issues....we beleive them to be the network stack. CTX any thoughts on these network stack issues with the Xentoools and is there anything the forum members can provide to help you find a long term fix for these ongoing issues?



Dan LeBaron Members
  • #11

Dan LeBaron
  • 273 posts

Posted 03 June 2010 - 06:59 PM

By the way the other item we notice with the XenTools stack is on Windows Clusters. The network cards have been dropping randomly, thus you cannot bring the cluster back online without deleting the ip from the cluster and local adapter and readding the ip and adapter back to the cluster.



Gerasim Shaverdoff Members
  • #12

Gerasim Shaverdoff
  • 47 posts

Posted 04 June 2010 - 07:45 AM

did you export you pool host-data and vm before upgrade to 5.6?

i think you must deinstall xentools 5.5 on whole of you VM then

Following examples will be helpful for backup & restore.
to export

#xe pool-dump-database file-name=”xen1.pool”
#xe host-backup file-name=”xen1.xbk”
#xe vm-export filename=VM.xva vm=”VM”

delete VM on you xenserver
upgrade it the make import

*to import*
#xe pool-restore-database file-name=”xen1.pool”
#xe host-restore file-name=”xen1.xbk”
#xe vm-import filename=VM.xva

and then install new 5.6 xen-tools



IVAYLO KRUMOV Members
  • #13

IVAYLO KRUMOV
  • 73 posts

Posted 05 June 2010 - 09:10 AM

We have performance issues as well. Our test environment is running XenServer on 2 HP ProLiant DL 380 G5 servers. We have Windows 2003 Server and Windows 2008 Server guests. I am not on fire about the Windows Machines, since they are running on the test servers, but they are experiencing performance and stability issues after the upgrade. What I do worry about are the 4 NetScaler VPX guests. They were totaly crashed. The Netscalers are organized as two HA pairs forming in GSLB infrastructure. Before the upgrade HA pair nodes were working fine, after the upgrade both HA pairs are experiencing performance issues. Also they are loosing the synchronization and once in awhile they loose connection between the nodes participating in the HA configuration.
On Behalf of all the IT people i would like to send this serious request

CITRIX, VM Ware, Microsoft, and the rest of you, PLEASE DO NOT RUSH PRODUCTS ON THE MARKET WITHOUT THOROUGHLY TESTING THE PRODUCT!!!



LEVON BRAGG Members
  • #14

LEVON BRAGG
  • 4 posts

Posted 05 June 2010 - 01:58 PM

Just my 2 cents.

I have quite a few Xen servers out there in production, but I have yet to upgrade to 5.6.
I have not upgraded yet because I have not setup a test environment and tested migration with my machines specific to my installations.

This is one of the best ways to discover problems BEFORE upgrading your production system.

I agree that Citrix needs to test upgrades before releasing, but it is OUR responsibility to THROUGHLY test in OUR environment before WE deploy these patches to the production environment....



Larry Heier Members
  • #15

Larry Heier
  • 153 posts

Posted 05 June 2010 - 02:34 PM

Hi there,

I have used DisableTCPOffload with certain NIC's to resolve network performance issues between VM's.

Unless you have a new install, you should definitely test any hypervisor upgrade in a lab first before rolling into production!

-Larry

Edited by: larry heier on Jun 5, 2010 11:48 AM



Mathias DELLACHERIE Members
  • #16

Mathias DELLACHERIE
  • 6 posts

Posted 30 June 2010 - 02:39 PM

Hi !

I made my way to 5.6 from 5.0 because i was told i was " late ". Did from 5.0 to 5.5, then to 5.6.

I had the same others do, network connections falling, and of course, all the users coming in my office, thing that most of you all know i guess.

I used the " DisableTCPOffload " thing, it worked yesterday. Today, 18 hours later, one of both xenservers had the problem. I migrated all " VERY slowly " to the " working " Xenserver, and everything came back to a normal state.

I don't know what to do, am lost.

I found out a script here, but i don't really understand all the commands, so i'll keep from using it :
http://www.xenappblog.com/2010/citrix-xenserver-slow-network-performance/comment-page-1/#comment-1291

Can someone explain what else it does compared to the " DisableTaskOffload " thing ?

So, here are my questions :

- Is it worth setting up all those :
-> The DisableTaskOffload registry thing on the vms
-> The shell script on the Xen servers themselvers
-> Changing parameters on the virtual interfaces of the vms (Allow blank checksums : No, Offload IP Checksum : Disabled, Offload TCP Checksum : Disabled, Offload UDP Checksum : Disabled)
-> removing Xentools (loosing mobility, but at least employees will be able to work without bad surprise, the time Citrix finds a solution) or using previous 5.6 Xentools on the VMs.

Finally, as i didn't run Xenserver 5.5, is it possible to go there from 5.6 ? Did some here have the same problem using 5.5, or was in fine under 5.5, and went wrong under 5.6 ?

I'm the only IT man here, and " theorically " going on holidays on friday with my family. I'm quite angry now.



Dan LeBaron Members
  • #17

Dan LeBaron
  • 273 posts

Posted 30 June 2010 - 03:07 PM

I am surprised this is still an issue for Citrix....does anyone have a locked in answer and resolution as to what is causing this? Citrix any thoughts on your side? I have 200 VM's I am patiently waiting to update until this has been resolved....i guess maybe i need to see how Microsofts VM solution performs and look to move to that....



James White Members
  • #18

James White
  • 1,039 posts

Posted 30 June 2010 - 04:24 PM

After 5.5 and the Windows 2003 issues, I'm waiting until at least update 1 to even put this in our dev/test environments. Honestly, deploying immature software into production is always a big risk.

And they STILL haven't fixed the TaskOffload bug.



Dan LeBaron Members
  • #19

Dan LeBaron
  • 273 posts

Posted 30 June 2010 - 04:53 PM

Has anyone had these issues on microsofts vm platform? I'm beginning to think Citrix isn't taking the virtulization seriously if they let crap like this linger



Mathias DELLACHERIE Members
  • #20

Mathias DELLACHERIE
  • 6 posts

Posted 30 June 2010 - 04:54 PM

Dan, James, do you confirm that under 5.5 these problems don't exist ?
If so, i'll rollback to 5.5 and wait until 5.6 is fixed, but i thought these problems occured under 5.5 too ?

Another thing, is rolling back to 5.5 as simple as putting the cd in the server, booting and choosing the 5.5 state menu to get back to there ?

Darn, i'm always checking, even from home, hour after hour, if everything's alright at the company :-s