Jump to content


Photo

Xen API cannot start after updates on 6.5

Started by George Christodoulou , 10 October 2017 - 08:55 AM
14 replies to this topic

George Christodoulou Members

George Christodoulou
  • 24 posts

Posted 10 October 2017 - 08:55 AM

Yesterday I made some updates on my 6.5 pool.

I followed the link bellow 

https://support.citrix.com/article/CTX138115#XenServer_6.5

 

And installed SP1027, SP1045, SP1048, SP1055. SP1060 and SP1061

A whole lot of them..

I haven't been restarting after each hotfix. I decided to do a restart after applying the last hotfix.

I first restarted the slave and not the master, but when the slave started I got the following message and since then it cannot be added to the pool nor I can ping it:

 

"The underlying Xen API xapi is not running. This console will have reduced functionality. Would you like to attempt to restart the xapi?".

 

Which I am doing but with no luck.

 

 



Alan Lantz Members

Alan Lantz
  • 7,399 posts

Posted 10 October 2017 - 01:38 PM

With XenServer 6.5 make sure you didn't run out of space on your partition. You can remote /var/patch (just not the stuff in /var/patch/applied). See if xe-toolstack-restart will then work. 

 

--Alan--



George Christodoulou Members

George Christodoulou
  • 24 posts

Posted 10 October 2017 - 02:02 PM

A df command on sda1 gives me 74% usage. 1027876 blocks free. It's aroung 1GB. Isn't it?



George Christodoulou Members

George Christodoulou
  • 24 posts

Posted 10 October 2017 - 02:12 PM

Alan, when I did xe-toolstack-restart it returns OK in every step and now I can ping and ssh to it.

Still in maintenance mode though. Shall I try to exit maintenance mode from Xen Center?



Alan Lantz Members

Alan Lantz
  • 7,399 posts

Posted 10 October 2017 - 04:04 PM

Yes, that should be fine to exit maintenance mode. Hopefully it will exit properly and return you to a round green icon. 

74% usage was plenty of free space, I always kept mine below 80% usage.

 

--Alan--



Alan Lantz Members

Alan Lantz
  • 7,399 posts

Posted 10 October 2017 - 04:06 PM

just verify after all of the hot fixes are installed that no other pending reboots are needed then go ahead and migrate the VM's and upgrade the master.

 

--Alan--



George Christodoulou Members

George Christodoulou
  • 24 posts

Posted 11 October 2017 - 08:03 AM

Well, I tried putting the server out of maintenance mode but it disappeared from the pool again. Although I am able to SSH on it.

I think the problem is that I should have restarted the master first after the hotfixes update. Correct?
The problem is that I have to switch all VMs off and restart the Master and the I hope after restarting the slave it will auto-join the pool again.

What scares me is the possibility to loose both server and have all my VMs down! 



Alan Lantz Members

Alan Lantz
  • 7,399 posts

Posted 11 October 2017 - 01:37 PM

Yes, on 6.5 you will likely need to restart your host for pending updates. When you highlight each server in XenCenter and under the general tab look at updates you should see a notice if a pending reboot is required to activate the hotfix.

 

Rebooting the master shouldn't cause the master VM's to be down. The slave on the other hand could be. If the slave won't see the pool correctly and pool recovery attempts fail you could force it to become its own master so you could then start those VM's and work on getting the pool squared away after VM's are all running.

 

Yes, very scary. This is why I like pools of 3 servers or more and remote storage. I realize that is not a possibility for everyone, but it makes the host not as critical a component as it is when you have local storage.

 

--Alan--



George Christodoulou Members

George Christodoulou
  • 24 posts

Posted 11 October 2017 - 02:00 PM

Alan, first of all all my running VMs are now migrated to the master. All my VMs in general are on a remote storage. Nothing locally.

 

On the "updates" under the "General" tab the "faulty" slave has only applied updates listed. Which I think is what we want.

The master has several that show that it has to be rebooted in order for the update to take effect.

 

If I restart the master without shuting down the VMs wouldn't this be a problem? In which server will they be running???

Don't I have to shut down all of them? 



Alan Lantz Members
  • #10

Alan Lantz
  • 7,399 posts

Posted 12 October 2017 - 05:58 PM

If the VM's are running on the master and you restart it all of the VM's will either be suspended or shut down. I think what I would do is make sure time/time sync is good on both servers and do a xe-toolstack-restart to see if slave comes around. If not I would kick it out of the pool since you don't have anything on local storage, reset network with XSConsole back to a single eth0 management interface, and rejoin it to the pool. Once its a part of the pool and all hotfixes are in sync then migrate the VM's from the master to the slave so you can restart the master to apply pending updates.

 

--Alan--



George Christodoulou Members
  • #11

George Christodoulou
  • 24 posts

Posted 13 October 2017 - 08:18 AM

Hi Alan,

to remove it from the pool should I just select "Destroy" from XenCenter? The server is shown as red (not available) in XenCenter so I think I have to run something locally on the server too. Don't I?

 

Edit: If I restart xapi service I can see the server for a few minutes in the XenCenter console. Should I select from there "Remove Server from the Pool".

 

 

Also I didn't understand what to do with the management interface... Sorry for that. Too many questions, I know.


Edited by George Christodoulou, 13 October 2017 - 10:45 AM.


Alan Lantz Members
  • #12

Alan Lantz
  • 7,399 posts

Posted 13 October 2017 - 01:58 PM

If you can see it for a few minutes it sounds like your servers aren't in sync with the same time source. Maybe starting over isn't the solution, you need to look at your time first.

 

But yes, remove from pool if you go that route and from a command line type in xsconsole. Under network and management interface there is an emergency network reset. This will set the management interface back to eth0 so you can basically start over with your XenServer networking then when you joint that host to the pool it will re-learn all of its networks.

 

--Alan--



George Christodoulou Members
  • #13

George Christodoulou
  • 24 posts

Posted 13 October 2017 - 02:13 PM

The time is synced all right. I can see it for a few minutes but when I lose it I check the xapi status and is dead again. I tried to remove it from the XenCenter console but I think it lost connection and couldn't complete the task.
 

So is there a way to remove it from command line?



Alan Lantz Members
  • #14

Alan Lantz
  • 7,399 posts

Posted 13 October 2017 - 03:27 PM

Always.

 

xe host-list and grab the host uuid you are wanting to remove.

xe host-forget uuid=<uuid of server> should kick it out of the pool.

 

--Alan--



George Christodoulou Members
  • #15

George Christodoulou
  • 24 posts

Posted Yesterday, 09:34 AM

Great, 

 

I will get back later this week because I will be out of office for a couple of days and I don't want to ruin anything before I leave...