Jump to content


Photo

Machines don't always start after scheduled Reboot

Started by Tarik Chougui , 20 April 2015 - 12:41 PM
32 replies to this topic

Tarik Chougui Members

Tarik Chougui
  • 20 posts

Posted 20 April 2015 - 12:41 PM

Hello everyone,

 

I'm having an issue on my XenDesktop 7.6 (server OS  2008 R2) machines.

I scheduled a reboot directly trhough studio at 3 am for all my delivery groups.

 

The problem is only half of my machines startup on the morning. The Vsphere console tells me that the machine shutdown at 3 am then bootup then shutdown again 20 minutes later ( i've added a screenshot of the vsphere events)

 

I turn  AutomaticPowerOnForAssigned and AutomaticPowerOnForAssignedDuringPeak to $false for all my delivery group.

 

But the issue still occurs.

I'm gonna try to bypass it and use a vmware restart task and see how it goes.

 

Any idea why its not working through Studio ?

 

Thanks for the help !

 



William Charnell Citrix Employees

William Charnell
  • 336 posts

Posted 20 April 2015 - 04:49 PM

The shutdown after 20 minutes sounds like the machine never managed to register after being started during that 20 minutes. The broker is designed to perform this shutdown after a timeout waiting for a registration after a VM start, and this timeout defaults to 20 minutes but can be configured via a registry entry.

 

For multi-session server OS machines, these are never permanently assigned, so the AutomaticPowerOnForAssigned and AutomaticPowerOnForAssignedDuringPeak properties don't apply.

 

Do the event logs on the failed machines show anything about a failure to register?



Tarik Chougui Members

Tarik Chougui
  • 20 posts

Posted 20 April 2015 - 05:23 PM

Since my machines are "non persistant" I can't see VDA registration on them.

 

But my DDC tells me :

 

3am it launch : The Citrix Broker Service successfully performed power action 'Shutdown' (origin: Schedule) on virtual machine BLA BLA

and at 3am01 : The Citrix Broker Service successfully performed power action 'TurnOn' (origin: Schedule) on virtual machine BLA BLA

 

 

The Citrix Broker Service has completed the processing of the 'GroupReboot' for 'DG_BLABLA'. The reboot cycle (UID = '296') successfully completed. Summary: '1' machines successfully rebooted, '0' machines failed to reboot, '2' machines were not processed.

 

Also : The Citrix Broker Service marked a power action as failed because virtual machine BLABLA failed to register with the site after being started, restarted, or resumed.

 

I switched to vmware tasks in the meantime. I'm gonna check that registry entry and see if I can make it to 1hour (But it should not take that much time to register in my opinion)



Campbell Kay Members

Campbell Kay
  • 60 posts

Posted 25 June 2015 - 08:22 PM

Hi Guys,

 

DId you manage to figure out what causes this.

 

we had about half of our Windows 2012 R2 machines not turn on after the schedule reboot. we are running XenApp 7.6

 

i get some errors like this

 

The Citrix Broker Service successfully communicated with the Citrix Machine Creation Service, but the Citrix Machine Creation Service failed to reset the image for virtual machine '4229c782-a2f9-0fcc-4893-6b37b10b8993' on host connection 'PSN vCenter' to its default state. This problem could be due to a timing issue or an internal error. 
 
Error details: 
Exception 'Failure in ResetVM, GeneralFailure' of type 'Citrix.Cds.HostingManagement.McsResetVMFailedException'.

 

 

as if its lots connection to my vcenter server, also in the vcenter logs you can see the startup command never got issues.

 

THanks,



Joe Pawlicki Members

Joe Pawlicki
  • 79 posts

Posted 26 June 2015 - 06:14 PM

.



Sam Rayes Members

Sam Rayes
  • 74 posts

Posted 27 June 2015 - 09:17 PM

we use an mdt deployment server and need to stop the wds service ( with script .. net stop xxx ) when citrix servers are goin up or we got the same result .. maybe that is what you need to do also ..



Prakash V Members

Prakash V
  • 7 posts

Posted 04 March 2016 - 03:56 PM

we are seeing similar behavior with our Xenapp 7.6. The VDA servers boot through PVS 7.6 and after every weekly scheduled reboot half of the servers get shutdown automatically.

 

Running the "Get-BrokerhostingPoweraction" command on the delivery controller clearly show the shutdown action has been triggered by the broker services.

 

We have also tried  increasing the VDA registering from default 20 mins to 40 mins but no good signs. Any fix or suggestion..



Daniel Lucas Members

Daniel Lucas
  • 17 posts

Posted 12 March 2016 - 08:42 PM

Hello guys,

 

Im having the same problem.

 

I´m adding another event that I get:

 

"The Citrix Broker Service has terminated the processing of the 'GroupReboot' for 'XXXX'. The reboot cycle (UID = '10564') ended with the final state 'Abandoned'. Summary: '2' machines successfully rebooted, '0' machines failed to reboot, '1' machines were not processed.
 
Broker Service 3105"
 
It´s weird, because after this I got the machine stuck in power off.
And I don´t know how to handle this.
 
What you think about to change the reboot schedule for a script?
Is it better? Or something that I can do?
 
And another problem is that my broker service is assign users to machines that the state is "Unregistered" and I always need to force the disconnect and sometime force a reboot.
 
Regards,
 
Daniel.

 



Catherine Neily Members

Catherine Neily
  • 19 posts

Posted 06 April 2016 - 07:01 PM

Daniel - did you find a resolution to this?  I am seeing the same thing, but I think the main reason is that only about half of the XenApp 7.6.300 VDA's register with the Controllers after a reboot.  

 

My post is here: http://discussions.citrix.com/topic/376910-intermittent-vda-registration-763/



Morten Brager Members
  • #10

Morten Brager
  • 2 posts

Posted 07 April 2016 - 06:38 AM

This looks like the issue I'm having. The users takes so long time logging off the servers, that the server is still in a shutdown state when the Power On signal is sent. The problem is cookies on the users profiles which is mirrored and copied between the servers on logon and logoffs. I'm using the recommended "cookie policy", but still I have to manually delete cookies on userprofiles 2-3 times each year. Then after the cookie cleanup, the servers power on after daily reboot more often.

 

But I need a more permanent solution to this. 



jfarrell360 Members
  • #11

jason farrell
  • 17 posts

Posted 05 January 2017 - 08:57 AM

Having a similar issue in XD 7.6 where server OS  2008 R2 machines are becoming unregistered randomly throughout the day and after a scheduled reboot.

 

VDA version is 7.6.300

Hyper v 2012r2 standard

Performance counters are working

DNS & service principal names checks out ok 

There are multiple mini WAN ports

VDA health care assistant checks out with no faults  

 

event logs show desktop service lost contact wih DDC

event id 1014

event id 1002 

 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          05/01/2017 01:04:50
Event ID:      1014
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      hostname.local
Description:
The Citrix Desktop Service lost contact with the Citrix Desktop Delivery Controller Service on server 'hostname.local'.
The service will now attempt to register again.
 
Error details:
Exception 'There was no endpoint listening at http://ipaddress/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.' of type 'System.ServiceModel.EndpointNotFoundException'
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Citrix Desktop Service" />
    <EventID Qualifiers="32768">1014</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-01-05T01:04:50.000000000Z" />
    <EventRecordID>9163</EventRecordID>
    <Channel>Application</Channel>
    <Computer>hostname.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>hostname.local</Data>
    <Data>There was no endpoint listening at http://ipaddress/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.</Data>
    <Data>System.ServiceModel.EndpointNotFoundException</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          05/01/2017 01:04:55
Event ID:      1002

Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      hostname.local
Description:
The Citrix Desktop Service cannot connect to the delivery controller 'http://hostname.local:80/Citrix/CdsController/IRegistrar' (IP Address '192.168.15.43')
 
Check that the system clock is in sync between this machine and the delivery controller. If this does not resolve the problem, please refer to Citrix Knowledge Base article CTX117248 for further information.
 
Error Details:
Exception 'Error occurred when attempting to connect to endpoint at address http://hostname.local:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.Security.SecurityNegotiationException: SOAP security negotiation with 'http://192.168.15.43/Citrix/CdsController/IRegistrar' for target 'http://192.168.15.43/Citrix/CdsController/IRegistrar' failed. See inner exception for more details. ---> System.ComponentModel.Win32Exception: The Security Support Provider Interface (SSPI) negotiation failed.
   at System.ServiceModel.Security.WindowsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetOutgoingBlobProxy.GetOutgoingBlob(ChannelBinding channelBinding)
   at System.ServiceModel.Security.RequestSecurityToken.GetBinaryNegotiation()
   at System.ServiceModel.Security.WSTrust.Driver.WriteRequestSecurityToken(RequestSecurityToken rst, XmlWriter xmlWriter)
   at System.ServiceModel.Security.RequestSecurityToken.WriteTo(XmlWriter writer)
   at System.ServiceModel.Channels.Message.OnWriteMessage(XmlDictionaryWriter writer)
   at System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota)
   at System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset)
   at System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message message, Boolean shouldRecycleBuffer)
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Citrix.Cds.BrokerAgent.ControllerConnectionFactory.AttemptConnection[T](EndpointReference endpoint, Boolean throwOnError, Boolean allowNtlmAuthentication, String connectUsingIpThisIpAddress, Boolean cacheFactory)' of type 'Citrix.Cds.BrokerAgent.ConnectionFailedException'..
Event Xml:
<Event xmlns="">
  <System>
    <Provider Name="Citrix Desktop Service" />
    <EventID Qualifiers="32768">1002</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-01-05T01:04:55.000000000Z" />
    <EventRecordID>9164</EventRecordID>
    <Channel>Application</Channel>
    <Computer>hostnamel</Computer>
    <Security />
  </System>
  <EventData>
    <Data>http://hostname.local:80/Citrix/CdsController/IRegistrar</Data>
    <Data>192.168.15.43</Data>
    <Data>Error occurred when attempting to connect to endpoint at address http://hostname.local:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.Security.SecurityNegotiationException: SOAP security negotiation with 'http://192.168.15.43/Citrix/CdsController/IRegistrar' for target 'http://192.168.15.43/Citrix/CdsController/IRegistrar' failed. See inner exception for more details. ---&gt; System.ComponentModel.Win32Exception: The Security Support Provider Interface (SSPI) negotiation failed.
   at System.ServiceModel.Security.WindowsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetOutgoingBlobProxy.GetOutgoingBlob(ChannelBinding channelBinding)
   at System.ServiceModel.Security.RequestSecurityToken.GetBinaryNegotiation()
   at System.ServiceModel.Security.WSTrust.Driver.WriteRequestSecurityToken(RequestSecurityToken rst, XmlWriter xmlWriter)
   at System.ServiceModel.Security.RequestSecurityToken.WriteTo(XmlWriter writer)
   at System.ServiceModel.Channels.Message.OnWriteMessage(XmlDictionaryWriter writer)
   at System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota)
   at System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset)
   at System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message message, Boolean shouldRecycleBuffer)
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Citrix.Cds.BrokerAgent.ControllerConnectionFactory.AttemptConnection[T](EndpointReference endpoint, Boolean throwOnError, Boolean allowNtlmAuthentication, String connectUsingIpThisIpAddress, Boolean cacheFactory)</Data>
    <Data>Citrix.Cds.BrokerAgent.ConnectionFailedException</Data>
  </EventData>
</Event>



jhystad940 Members
  • #12

Jan Edvard Hystad
  • 3 posts

Posted 10 January 2017 - 07:21 AM

Hey, i have the same problem aswell. I have servers that should by booted by MCS. They do the reboot but fails to register and shutsdown after 20minutes. Havent got a hold of the eventlog on the servers yet. if you have had any progress investigating the problem please reply.



Tim Mallon Members
  • #13

Tim Mallon
  • 15 posts

Posted 11 January 2017 - 03:23 AM

jhystand940,  what have you found, I have on Machine Catalog, several Delivery groups that won't reboot, started after 7.12 install. another group works great



jhystad940 Members
  • #14

Jan Edvard Hystad
  • 3 posts

Posted 11 January 2017 - 08:44 AM

Im just about to move the eventlog to the cachedisk so i can check it after next reboot schedule.



Christian Hock Members
  • #15

Christian Hock
  • 96 posts

Posted 13 January 2017 - 06:28 AM

Hello,

 

we have the same problem after the upgrade to the VDA 7.12. The scheduled reboot created in studio or with powershell do not work.

Create multiple VDA restart in 7.12:

http://www.jgspiers.com/create-multiple-restart-schedules-xenapp-xendesktop-7-12/

 

In the eventlog on the controller we have the warning 3105 Citrix broker Service.

 

"The Citrix Broker Service has terminated the processing of the 'GroupReboot' for 'XXXX'. The reboot cycle (UID = '106') ended with the final state 'Abandoned'. Summary: '0' machines successfully rebooted, '0' machines failed to reboot, '1' machines were not processed.

 

When we created a new Delivery Group, we have the same error.

 

Any Ideas?

 

Regards Chris



L. T. Zwanenburg Members
  • #16

L. T. Zwanenburg
  • 6 posts

Posted 16 January 2017 - 10:09 AM

We found that one in another post. We have the same problem.

it is an 7.12 update issue.

 

run in powershell

remove-brokerrebootschedule -desktopgroupname "xxxxx"

 

then recreate the schedule.



Christian Hock Members
  • #17

Christian Hock
  • 96 posts

Posted 16 January 2017 - 01:00 PM

Hello,

 

we remove the schedule with the powershell command an recreate it in the Studio.

But error is the same.



pjoppen371 Members
  • #18

Patrick Joppen
  • 4 posts

Posted 17 January 2017 - 07:01 AM

We have the same problem.

We think it's an issue of 7.12

 

The Powershell Command dosen't work for us.

 

Problem is still the same.. no reboot schedule is working.



Christian Hock Members
  • #19

Christian Hock
  • 96 posts

Posted 17 January 2017 - 07:22 AM

Yes, I think it too,

I have create a new OS with VDA 7.11 and the reboot works. After the upgrade to version 7.12 the do not reboot.



Mirco Geibel Members
  • #20

Mirco Geibel
  • 2 posts

Posted 17 January 2017 - 09:52 AM

Hello everybody..

the scheduled restarts worked fine with VDA 7.9... but since we had some Problems with the logoff in VDA 7.9 Server 2012 R2 (sessions stucked at logoff Screen) we upgraded to VDA 7.12. Now the logoffs work fine without Problems..

What I can say about the scheduled restarts in VDA 7.12:

Yesterday I was wondering why only four of our six XenApps could handle the scheduled restarts.. So I checked the Citrix Studio policy and everything was fine.. but then I saw that there were some cut sessions on the Servers which did not restart. After I logged off those users in Citrix Studio the two Servers began to restart. So for me it seems that active or cut sessions got not logged off automatically by the restart policy. The user is receiving the message from that policy but he has to manually logoff. Otherwise the Server is not rebooting. It Looks like a bug in 7.12 since  it worked like a charm with 7.9. So maybe check the Citrix Studio for active or cut sessions and logoff those users and try the scheduled reboot again.