XenCenter Console unavaible

[Daniel Bokori]

HI, 

 

we have a XenServer 6.5 SP1 and the console window is blank in the XenCenter 6.5.2 or 7.0.1 applications when we would like to access the VMs.

 

The XC apps are running on W10 Pro x64 pc-s, i had removed the %AppData%\\Citrix\XenCenterMain.exe_Url_e3jqcmyhhuvw2c1t3wydiid0qz3e5jr3 folder than i reconfigured the server access, but it didn't solved the problem. I removed the app completely, then reinstalled but no luck, same issue on other pc-s too. 

 

This problem is affects all of our VMs (W10 Pro x64, Debian 7-8) which are hosted on this server. Xen Guest tools were updated on the vm-s.

I would like to install a new vm, but i can't because this problem crops up at the new VMs too. 

 

installed fixes: 

 

XS65E001

XS65E002

XS65E003

XS65E005

XS65E006

XS65E007

XS65E008

XS65E009

XS65E010

XS65E013

XS65E014

XS65E015

XS65E017

XS65E018

XS65ESP1

XS65ESP1002

XS65ESP1003

XS65ESP1004

XS65ESP1005

XS65ESP1008

XS65ESP1009

XS65ESP1010

XS65ESP1011

XS65ESP1012

XS65ESP1013

XS65ESP1014

XS65ESP1016

XS65ESP1018

XS65ESP1019

XS65ESP1020

XS65ESP1021

XS65ESP1022

XS65ESP1023

XS65ESP1024

XS65ESP1025

XS65ESP1026

XS65ESP1027

XS65ESP1028

XS65ESP1029

XS65ESP1032

XS65ESP1033

 

[Alan Lantz]

I know for mine under tools/options/console I unchecked automatically switch to the remote desktop console when it becomes available since that was annoying me. I have everything else checked. Nothing comes to mind other that time sync issues. Seems like there were issues like this in previous older versions, but haven't seen any odd console issues recently.

 

--Alan--

[Daniel Bokori]

Hi Alan, 

 

thanks for your reply. I'v tried with both options, but it hasn't solved the problem. :( I tested on 6.52 and 7.0.1 too.

[Daniel Bokori]

I know for mine under tools/options/console I unchecked automatically switch to the remote desktop console when it becomes available since that was annoying me. I have everything else checked. Nothing comes to mind other that time sync issues. Seems like there were issues like this in previous older versions, but haven't seen any odd console issues recently.

 

--Alan--

Anything else what i can try to solve the issue? 

[Boby John1709155536]

Are you using a Windows 10 VM to access the XenCenter ?

What happens if you try from a VM inside the same Xenserver via RDP?

( I am assuming that RDP works fine )

[Daniel Bokori]

Are you using a Windows 10 VM to access the XenCenter ?

What happens if you try from a VM inside the same Xenserver via RDP?

( I am assuming that RDP works fine )

 

Yes. 

I can access the vm-s via RDP, SSH.

[Daniel Bokori]

Are you using a Windows 10 VM to access the XenCenter ?

What happens if you try from a VM inside the same Xenserver via RDP?

( I am assuming that RDP works fine )

 

Yes. 

I can access the vm-s via RDP, SSH.

[Mike Maddison]

Hello All.

I seem to have one Xenserver running 6.5 build 90233c that is having this problem and many more. I have 6 CentOS VMs running, all with xenserver tools installed.  If I physically reboot the host machine I regain access to the consoles through the XenCenter program.  If I come back a day or two later xen host will no longer allow any consoles on the VMs through XenCenter.   I can't even use XenCenter to stop or restart the VMs.  I tried log in to one of the VMs directly through SSH to do a reboot and the VM seems to have gone down but never boots back up.  XenCenter still shows the VM as up and running.  Performance graphs are empty.
 

Jul  7 10:42:08 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:455e7007f47f|xapi] Session.destroy trackid=703ab52d1d5666f41cf6fe0c703fa7a3

Jul  7 10:50:08 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:b8ca42e105e0|xapi] Session.destroy trackid=4da0c60d600f219f7eab757fea8d7e79

Jul  7 10:50:38 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:4917f0a68556|xapi] Session.destroy trackid=0c67f1ef1fe53b93e9f79ff8e814a8bf

Jul  7 11:20:39 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:a3f1bc7f4690|xapi] Session.destroy trackid=e63e92869a6ee542b796a517b0c97c24

Jul  7 11:20:39 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:a3f1bc7f4690|xapi] Session.destroy trackid=1c87bb5b8640582ab2e8454363f70229

Jul  7 11:20:39 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:a3f1bc7f4690|xapi] Session.destroy trackid=8ba9f86cb4116d6e3ba3023efdcee630

Jul  7 11:41:52 rotary xapi: [ info|rotary.mynetwork.com|26420 UNIX /var/xapi/xapi|session.slave_login D:0be7276cfc33|xapi] Session.create trackid=24ef6993b6b2045c36009d2f150245ac pool=true uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49

Jul  7 11:41:52 rotary xapi: [ info|rotary.mynetwork.com|26423 UNIX /var/xapi/xapi|session.logout D:e9565d0bc16a|xapi] Session.destroy trackid=24ef6993b6b2045c36009d2f150245ac

Jul  7 15:12:20 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:d5d8b52fe42a|xapi] Session.destroy trackid=a4585d741a4c7be231bc5c07e601bef2

Jul  7 15:12:20 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:d5d8b52fe42a|xapi] Session.destroy trackid=2c92505fc33e4ee937f7bcb5358d3dde

Jul  7 15:14:20 rotary xapi: [ info|rotary.mynetwork.com|27060 UNIX /var/xapi/xapi|session.login_with_password D:2b1a7efc1f00|xapi] Session.create trackid=26155253866fa4b18078e9a8fd7bc359 pool=false uname= originator=mpathalert is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49

Jul  7 15:19:20 rotary xapi: [ warn|rotary.mynetwork.com|25 db_gc|DB GC D:60c4916893c8|db_gc] GCed old task that was still in pending state: 29e2dc34-e9f7-6e0b-162b-aeff5c8aadc8

Jul  7 16:40:24 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:0637eb0eb54a|xapi] Session.destroy trackid=0a58419a74cc722aa0cf6c3e55bf8ca6

Jul  7 21:27:07 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:a2b7c4023da8|xapi] Session.destroy trackid=807014cba00f30ed1883d4b70d9e4711

Jul  7 21:27:37 rotary xapi: [ warn|rotary.mynetwork.com|25 db_gc|DB GC D:b5a8799d2259|db_gc] GCed old task that was still in pending state: 4f0c3856-00a5-0fa4-bc30-665a6b578a99

Jul  7 21:27:37 rotary xapi: [ info|rotary.mynetwork.com|25 db_gc|DB GC D:b5a8799d2259|xapi] Session.destroy trackid=caea2930aabbbcb07ec365d74dd86b1d

Jul  8 06:54:01 rotary logrotate: 36MB >= 36MB so running logrotate with size 4247k (/var/log/audit.log)

Jul 08 06:54:01 rotary syslogd 1.4.1: restart.

Jul  8 09:43:29 rotary kernel: [169392.901983] vif vif-7-0 vif7.0: Guest Rx stalled

Jul  8 09:43:32 rotary kernel: [169395.756090] vif vif-7-1 vif7.1: Guest Rx stalled

Jul  8 08:14:04 rotary RINGWATCH-ALERT: RingWatch(vbd3-2-768/io_ring)[sTCK]: RingState(size=32, Req(prod=23943, cons=23943, event=23943), Rsp(prod=23943, pvt=2

3943, event=23943)): io: complete, req: complete, rsp: pending

Jul  8 08:18:04 rotary RINGWATCH-ALERT: RingWatch(vbd3-1-51728/io_ring)[sTCK]: RingState(size=32, Req(prod=45266271, cons=45266270, event=45266271), Rsp(prod=4

5266245, pvt=45266245, event=45266246)): io: pending, req: pending, rsp: complete

Jul  8 09:22:04 rotary RINGWATCH-ALERT: RingWatch(vbd3-1-51712/io_ring)[sTCK]: RingState(size=32, Req(prod=1586997, cons=1586984, event=1586985), Rsp(prod=1586976, pvt=1586976, event=1586977)): io: pending, req: pending, rsp: complete

Jul  8 09:32:42 rotary xapi: [error|rotary.mynetwork.com|38811 INET 0.0.0.0:80|Get RRD updates. D:3f4e9a26fc50|xapi] Failed to transfer fd to /var/xapi/xcp-rrdd.forwarded: Unix.Unix_error(63, "connect", "")

 

I'm not sure if these errors are any help. I have 2 other machines with similar specs that are running 6.5 and do not seem to have this problem.  They also do not seem to have these errors in the logs.
 

xe patch-list | grep name-label

              name-label ( RO): XS65ESP1016

              name-label ( RO): XS65ESP1022

              name-label ( RO): XS65ESP1018

              name-label ( RO): XS65ESP1010

              name-label ( RO): XS65ESP1028

              name-label ( RO): XS65E001

              name-label ( RO): XS65E018

              name-label ( RO): XS65E015

              name-label ( RO): XS65E016

              name-label ( RO): XS65E013

              name-label ( RO): XS65ESP1011

              name-label ( RO): XS65ESP1005

              name-label ( RO): XS65E007

              name-label ( RO): XS65E006

              name-label ( RO): XS65E008

              name-label ( RO): XS65ESP1002

              name-label ( RO): XS65E005

              name-label ( RO): XS65ESP1027

              name-label ( RO): XS65ESP1004

              name-label ( RO): XS65ESP1026

              name-label ( RO): XS65ESP1025

              name-label ( RO): XS65ESP1012

              name-label ( RO): XS65ESP1023

              name-label ( RO): XS65ESP1021

              name-label ( RO): XS65E017

              name-label ( RO): XS65ESP1019

              name-label ( RO): XS65E002

              name-label ( RO): XS65ESP1024

              name-label ( RO): XS65ESP1013

              name-label ( RO): XS65ESP1009

              name-label ( RO): XS65ESP1003

              name-label ( RO): XS65E014

              name-label ( RO): XS65ESP1020

              name-label ( RO): XS65ESP1008

              name-label ( RO): XS65E009

              name-label ( RO): XS65E003

              name-label ( RO): XS65E010

              name-label ( RO): XS65ESP1

              name-label ( RO): XS65ESP1014

 

 

[Mike Maddison]

Isn't anyone else experiencing this?

[Russ Purinton1709154694]

Im not sure if this is related...

 

http://support.citrix.com/article/CTX215284

 

Symptoms or Error

Console tab in XenCenter is blank and cannot show the VM screens; White screen only, sometimes not even the empty console box shows.

Solution

Clear XenCenter cache.
Settings are kept in %AppData%\Citrix\XenCenterMain.exe_Url_xxxxxxxxxxxxxxx.
Rename the cache folder to .old

[Mike Maddison]

I tried clearing the cache but unfortunately that had no effect.

[Daniel Bokori]

Cache cleaning hasn't solved my problem neither. 

[Mike Maddison]

New information.  On this new machine I have a number of VMs running CentOS/Debian.  I noticed that problems seem to start whenever I have CentOS7 VMs activated.  To test my idea I had to do a physical reboot of the host machine and then proceeded to start up come CentOS 5 VMs but did NOT start up any CentOS 7 VMs.  The machine remained stable for days and the consoles were fine.  Today I fired up one CentOS 7 VM and within 20-30m I lost console, performance data, etc. to all the VMs and can no longer manage them.

 

That sounds to me like a problem with XenServer and CentOS 7.  All VMs have the xstools installed.
 

[Russ Purinton1709154694]

Its possible.  I don't run any CentOS 7 VMs and I've never experienced this issue.

[Mike Maddison]

Just to keep people informed, this is still a problem.  I'm currently researching moving away to RedHat / KVM as I'm finding no way to stabilize this machine with Xenserver.

[Tobias Kreidl]

We run CentOS 7 as well as RHEL 7 and do not experience this issue. Access to the console via XenCenter should also have nothing to do with whether or not XenTools are installed.

[Mike Maddison]

It doesn't seem to matter if XenTools are installed or not (but they are).  CentOS 5 and 6 VMs do not cause the problem.  Start up a CentOS 7 vm and within a few hours to a few days or so all consoles on the machine (including the host itself) become unusable from XenCenter.  I've installed all they XenServer 6.5 updates to date with no effect.

I also tried installing XenServer 7 but unfortunately there are no drivers for almost all the RAID cards we use so I can't test if that fixes the issue. 

[Mike Maddison]

I'm not sure if this is affecting anyone else but now I have 2 standalone Xenserver machines in my network that randomly stop handling consoles.  To make matters worse whenever this happens ALL VMs on the machine are locked in an ON state and cannot be shut down gracefully.  The only solution I have been able to find to get the machine and VMs back to normal operation is to run:

 

echo 1 > /proc/sys/kernel/sysrq 

echo b > /proc/sysrq-trigger

 

 

on the XenServer Host.  This is a terrible solution when all you want is to reboot one VM.  One machine is XenServer 6.5 and the other is 6.1
 

[Howard Turner]

Has anyone found a solution to this?

 

Having this issue as well with ONLY the pool master in a small XenServer 7.1 pool. When the master first boots up, it will display consoles and then after a while the consoles are no longer accessible. The only VM's running in this pool are Debian 7

 

I've tried the clear cache option with no success. The only thing that fixes it so far has been to reboot the host.

[Mike Maddison]

I'm not exactly sure if the problem is solved for us as it always seemed rather random when it would occur, but I haven't had the issue for at least month or so now.   A few things have changed in our set up and I don't know if any one or if all of these things contributed to the stability but here's my list:
1.  Removed all older CentOS 7 VMs and now we're only using the latest 7.1 updates.
2.  Reduced ram to 48GB on 64GB servers (problem only happened on hosts with 64GB RAM).
3.  Installed latest XS patches.
4.  Set up iptables on XS hosts to allow only traffic from the private network.
5.  Switched one machine from local RAID5 to local RAID10 storage (performance on R5 was terrible).

Maybe it's just a fluke that it hasn't happened lately.

[Howard Turner]

I'm not exactly sure if the problem is solved for us as it always seemed rather random when it would occur, but I haven't had the issue for at least month or so now.   A few things have changed in our set up and I don't know if any one or if all of these things contributed to the stability but here's my list:

1.  Removed all older CentOS 7 VMs and now we're only using the latest 7.1 updates.

2.  Reduced ram to 48GB on 64GB servers (problem only happened on hosts with 64GB RAM).

3.  Installed latest XS patches.

4.  Set up iptables on XS hosts to allow only traffic from the private network.

5.  Switched one machine from local RAID5 to local RAID10 storage (performance on R5 was terrible).

 

Maybe it's just a fluke that it hasn't happened lately.

 

 

Hmm...

 

You may be onto something with the 64GB thing... The host in question is the only 64GB host in the pool (the others are only 32GB).  Seems odd that that would be the issue but that's an easy test. This unit is already using RAID10 and updated with the latest patches.

 

I'll pull some chips from that host hopefully later today and see if the issue surfaces again.

[Mike Maddison]

Interesting development to support the 64GB theory.  I just had our one remaining 64GB machine that I "thought" was not affected by this issue flake out and the consoles all went away.

[Boby John1709155536]

Has anyone tried restarting the xenconsoled service ?

This can also happen if the slave is out of time sync with master. 

[Howard Turner]

Has anyone tried restarting the xenconsoled service ?

This can also happen if the slave is out of time sync with master. 

 

Tried: systemctl restart xenconsoled.service -> Still no consoles

 

Looked into the time in the pool, the master had proper UTC time with the pool members but it was set to an invalid timezone. The master had the time zone in quotes "America/Time/Zone" where the others had the time zone in no quote America/time/zone.

 

Used: timedatectl set-timezone "America/time/zone" (with the proper timezone obviously)

 

Then rebooted the server, which obviously fixed the consoles for now. Will run some more VM's and check tomorrow to see if the consoles are still accessible.

[Howard Turner]

Update: So far the unit with 64GB of RAM is still displaying consoles. It's been about 18 hours since the timezone was fixed on the server as well. Normally the console has disappeared within an hour or two.

 

Mike, can you check the timezone on your units that are having issues?

[Mike Maddison]

I went ahead and updated the timezone setting yesterday so now we wait.  I've had this machine work for weeks without locking consoles.  I've also had it happen more than once in the same day.