Jump to content
Welcome to our new Citrix community!
  • 1

Machines don't always start after scheduled Reboot


Tarik Chougui

Question

Hello everyone,

 

I'm having an issue on my XenDesktop 7.6 (server OS  2008 R2) machines.

I scheduled a reboot directly trhough studio at 3 am for all my delivery groups.

 

The problem is only half of my machines startup on the morning. The Vsphere console tells me that the machine shutdown at 3 am then bootup then shutdown again 20 minutes later ( i've added a screenshot of the vsphere events)

 

I turn  AutomaticPowerOnForAssigned and AutomaticPowerOnForAssignedDuringPeak to $false for all my delivery group.

 

But the issue still occurs.

I'm gonna try to bypass it and use a vmware restart task and see how it goes.

 

Any idea why its not working through Studio ?

 

Thanks for the help !

 

Link to comment

Recommended Posts

  • 2

Before attempting to add the Registry keys as mentioned in the CTX KBs 239537 & 218117, i have made one more change before that to drastically reduce the Failure rate. 

 

I have changed the Restart Duration from 30 mins to 3 hrs in the Delivery group, post that the failure rate (Powered OFF VDAs) dropped from 50% to 5%. Though there are still 5% VDAs that are failing to power on, need to apply the registry and completely arrest the issue.

  • Like 2
Link to comment
  • 1

Anyone tried the fix in this Citrix KB and found it helpful ?

 

https://support.citrix.com/article/CTX239537

 

https://support.citrix.com/article/CTX218117

 

These are the events and my observations so far...

 

When everything works fine, we only get Event ID 3013 for PowerAction: Turn OFF/ Turn ON (Origin: Schedule)
When the Machines are not powered On Back, then  I see 2 Event IDs 3013, 3012
Event ID 3012 - The Citrix Broker Service detected that power action 'Shutdown' (origin: Schedule) on virtual machine 'Test\Test01' failed. Exception 'PowerStateWait Operation failed' of type  'PluginUtilities.Exceptions.ManagedMachineGeneralException'.
Event ID 3013-  Power Action :Turn OFF (Origin Schedule) The Citrix Broker Service successfully performed power action 'Shutdown' (origin: Schedule) on virtual machine 'Test\Test01'.

For these 2 events (Successful reboot and Failed reboot) if I check the Last Registration Failure Event, they are marked as The Desktop Powered Down and Agent Shutdown. In which Agent Shutdown is a graceful shutdown and being handled by controller. The Desktop Powered Down event is the only one event which is causing the issue.

 

This is were Server OS VDAs did not shutdown gracefully.
 

  • Like 1
Link to comment
  • 0

The shutdown after 20 minutes sounds like the machine never managed to register after being started during that 20 minutes. The broker is designed to perform this shutdown after a timeout waiting for a registration after a VM start, and this timeout defaults to 20 minutes but can be configured via a registry entry.

 

For multi-session server OS machines, these are never permanently assigned, so the AutomaticPowerOnForAssigned and AutomaticPowerOnForAssignedDuringPeak properties don't apply.

 

Do the event logs on the failed machines show anything about a failure to register?

Link to comment
  • 0

Since my machines are "non persistant" I can't see VDA registration on them.

 

But my DDC tells me :

 

3am it launch : The Citrix Broker Service successfully performed power action 'Shutdown' (origin: Schedule) on virtual machine BLA BLA

and at 3am01 : The Citrix Broker Service successfully performed power action 'TurnOn' (origin: Schedule) on virtual machine BLA BLA

 

 

The Citrix Broker Service has completed the processing of the 'GroupReboot' for 'DG_BLABLA'. The reboot cycle (UID = '296') successfully completed. Summary: '1' machines successfully rebooted, '0' machines failed to reboot, '2' machines were not processed.

 

Also : The Citrix Broker Service marked a power action as failed because virtual machine BLABLA failed to register with the site after being started, restarted, or resumed.

 

I switched to vmware tasks in the meantime. I'm gonna check that registry entry and see if I can make it to 1hour (But it should not take that much time to register in my opinion)

Link to comment
  • 0

Hi Guys,

 

DId you manage to figure out what causes this.

 

we had about half of our Windows 2012 R2 machines not turn on after the schedule reboot. we are running XenApp 7.6

 

i get some errors like this

 

The Citrix Broker Service successfully communicated with the Citrix Machine Creation Service, but the Citrix Machine Creation Service failed to reset the image for virtual machine '4229c782-a2f9-0fcc-4893-6b37b10b8993' on host connection 'PSN vCenter' to its default state. This problem could be due to a timing issue or an internal error. 
 
Error details: 
Exception 'Failure in ResetVM, GeneralFailure' of type 'Citrix.Cds.HostingManagement.McsResetVMFailedException'.

 

 

as if its lots connection to my vcenter server, also in the vcenter logs you can see the startup command never got issues.

 

THanks,

Link to comment
  • 0

we are seeing similar behavior with our Xenapp 7.6. The VDA servers boot through PVS 7.6 and after every weekly scheduled reboot half of the servers get shutdown automatically.

 

Running the "Get-BrokerhostingPoweraction" command on the delivery controller clearly show the shutdown action has been triggered by the broker services.

 

We have also tried  increasing the VDA registering from default 20 mins to 40 mins but no good signs. Any fix or suggestion..

Link to comment
  • 0

Hello guys,

 

Im having the same problem.

 

I´m adding another event that I get:

 

"The Citrix Broker Service has terminated the processing of the 'GroupReboot' for 'XXXX'. The reboot cycle (UID = '10564') ended with the final state 'Abandoned'. Summary: '2' machines successfully rebooted, '0' machines failed to reboot, '1' machines were not processed.
 
Broker Service 3105"
 
It´s weird, because after this I got the machine stuck in power off.
And I don´t know how to handle this.
 
What you think about to change the reboot schedule for a script?
Is it better? Or something that I can do?
 
And another problem is that my broker service is assign users to machines that the state is "Unregistered" and I always need to force the disconnect and sometime force a reboot.
 
Regards,
 
Daniel.

 

Link to comment
  • 0

This looks like the issue I'm having. The users takes so long time logging off the servers, that the server is still in a shutdown state when the Power On signal is sent. The problem is cookies on the users profiles which is mirrored and copied between the servers on logon and logoffs. I'm using the recommended "cookie policy", but still I have to manually delete cookies on userprofiles 2-3 times each year. Then after the cookie cleanup, the servers power on after daily reboot more often.

 

But I need a more permanent solution to this. 

Link to comment
  • 0

Having a similar issue in XD 7.6 where server OS  2008 R2 machines are becoming unregistered randomly throughout the day and after a scheduled reboot.

 

VDA version is 7.6.300

Hyper v 2012r2 standard

Performance counters are working

DNS & service principal names checks out ok 

There are multiple mini WAN ports

VDA health care assistant checks out with no faults  

 

event logs show desktop service lost contact wih DDC

event id 1014

event id 1002 

 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          05/01/2017 01:04:50
Event ID:      1014
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      hostname.local
Description:
The Citrix Desktop Service lost contact with the Citrix Desktop Delivery Controller Service on server 'hostname.local'.
The service will now attempt to register again.
 
Error details:
Exception 'There was no endpoint listening at http://ipaddress/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.' of type 'System.ServiceModel.EndpointNotFoundException'
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Citrix Desktop Service" />
    <EventID Qualifiers="32768">1014</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-01-05T01:04:50.000000000Z" />
    <EventRecordID>9163</EventRecordID>
    <Channel>Application</Channel>
    <Computer>hostname.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>hostname.local</Data>
    <Data>There was no endpoint listening at http://ipaddress/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.</Data>
    <Data>System.ServiceModel.EndpointNotFoundException</Data>
  </EventData>
</Event>

 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          05/01/2017 01:04:55
Event ID:      1002

Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      hostname.local
Description:
The Citrix Desktop Service cannot connect to the delivery controller 'http://hostname.local:80/Citrix/CdsController/IRegistrar' (IP Address '192.168.15.43')
 
Check that the system clock is in sync between this machine and the delivery controller. If this does not resolve the problem, please refer to Citrix Knowledge Base article CTX117248 for further information.
 
Error Details:
Exception 'Error occurred when attempting to connect to endpoint at address http://hostname.local:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.Security.SecurityNegotiationException: SOAP security negotiation with 'http://192.168.15.43/Citrix/CdsController/IRegistrar' for target 'http://192.168.15.43/Citrix/CdsController/IRegistrar' failed. See inner exception for more details. ---> System.ComponentModel.Win32Exception: The Security Support Provider Interface (SSPI) negotiation failed.
   at System.ServiceModel.Security.WindowsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetOutgoingBlobProxy.GetOutgoingBlob(ChannelBinding channelBinding)
   at System.ServiceModel.Security.RequestSecurityToken.GetBinaryNegotiation()
   at System.ServiceModel.Security.WSTrust.Driver.WriteRequestSecurityToken(RequestSecurityToken rst, XmlWriter xmlWriter)
   at System.ServiceModel.Security.RequestSecurityToken.WriteTo(XmlWriter writer)
   at System.ServiceModel.Channels.Message.OnWriteMessage(XmlDictionaryWriter writer)
   at System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota)
   at System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset)
   at System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message message, Boolean shouldRecycleBuffer)
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Citrix.Cds.BrokerAgent.ControllerConnectionFactory.AttemptConnection[T](EndpointReference endpoint, Boolean throwOnError, Boolean allowNtlmAuthentication, String connectUsingIpThisIpAddress, Boolean cacheFactory)' of type 'Citrix.Cds.BrokerAgent.ConnectionFailedException'..
Event Xml:
<Event xmlns="">
  <System>
    <Provider Name="Citrix Desktop Service" />
    <EventID Qualifiers="32768">1002</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-01-05T01:04:55.000000000Z" />
    <EventRecordID>9164</EventRecordID>
    <Channel>Application</Channel>
    <Computer>hostnamel</Computer>
    <Security />
  </System>
  <EventData>
    <Data>http://hostname.local:80/Citrix/CdsController/IRegistrar</Data>
    <Data>192.168.15.43</Data>
    <Data>Error occurred when attempting to connect to endpoint at address http://hostname.local:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.Security.SecurityNegotiationException: SOAP security negotiation with 'http://192.168.15.43/Citrix/CdsController/IRegistrar' for target 'http://192.168.15.43/Citrix/CdsController/IRegistrar' failed. See inner exception for more details. ---> System.ComponentModel.Win32Exception: The Security Support Provider Interface (SSPI) negotiation failed.
   at System.ServiceModel.Security.WindowsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetOutgoingBlobProxy.GetOutgoingBlob(ChannelBinding channelBinding)
   at System.ServiceModel.Security.RequestSecurityToken.GetBinaryNegotiation()
   at System.ServiceModel.Security.WSTrust.Driver.WriteRequestSecurityToken(RequestSecurityToken rst, XmlWriter xmlWriter)
   at System.ServiceModel.Security.RequestSecurityToken.WriteTo(XmlWriter writer)
   at System.ServiceModel.Channels.Message.OnWriteMessage(XmlDictionaryWriter writer)
   at System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota)
   at System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset)
   at System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message message, Boolean shouldRecycleBuffer)
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Open()
   at Citrix.Cds.BrokerAgent.ControllerConnectionFactory.AttemptConnection[T](EndpointReference endpoint, Boolean throwOnError, Boolean allowNtlmAuthentication, String connectUsingIpThisIpAddress, Boolean cacheFactory)</Data>
    <Data>Citrix.Cds.BrokerAgent.ConnectionFailedException</Data>
  </EventData>
</Event>

Link to comment
  • 0

Hello,

 

we have the same problem after the upgrade to the VDA 7.12. The scheduled reboot created in studio or with powershell do not work.

Create multiple VDA restart in 7.12:

http://www.jgspiers.com/create-multiple-restart-schedules-xenapp-xendesktop-7-12/

 

In the eventlog on the controller we have the warning 3105 Citrix broker Service.

 

"The Citrix Broker Service has terminated the processing of the 'GroupReboot' for 'XXXX'. The reboot cycle (UID = '106') ended with the final state 'Abandoned'. Summary: '0' machines successfully rebooted, '0' machines failed to reboot, '1' machines were not processed.

 

When we created a new Delivery Group, we have the same error.

 

Any Ideas?

 

Regards Chris

Link to comment
  • 0

Hello everybody..

the scheduled restarts worked fine with VDA 7.9... but since we had some Problems with the logoff in VDA 7.9 Server 2012 R2 (sessions stucked at logoff Screen) we upgraded to VDA 7.12. Now the logoffs work fine without Problems..

What I can say about the scheduled restarts in VDA 7.12:

Yesterday I was wondering why only four of our six XenApps could handle the scheduled restarts.. So I checked the Citrix Studio policy and everything was fine.. but then I saw that there were some cut sessions on the Servers which did not restart. After I logged off those users in Citrix Studio the two Servers began to restart. So for me it seems that active or cut sessions got not logged off automatically by the restart policy. The user is receiving the message from that policy but he has to manually logoff. Otherwise the Server is not rebooting. It Looks like a bug in 7.12 since  it worked like a charm with 7.9. So maybe check the Citrix Studio for active or cut sessions and logoff those users and try the scheduled reboot again.

Link to comment
  • 0

I think there is a differens in the update order.

We Always first update The Delivery Controllers\Director and then the VDA's.

I think the official way is first VDA's and then the Controllers.

 

Remove the reboot schedule and then recreate it is only working when the controllers are 7.12

(Controllers take the action to reboot an server)

 

After updated the controllers the reboot schedules where not working. (all versions VDA's)

ater recreating them it works fine.

 

So i do not know what happens when you first update the VDA's to 7.12 and the controler is still an older version.

I Think recreating the scheduler is not working in that case because the controller take the action to reboot.

Link to comment
  • 0

We have the same problem since we´ve updated from 7.6 FP3 to 7.12. Schedule reboots don´t work no more for each existing desktop group. I´ve created a new one with 7.9 or new features, added a server to this group.and set up a schedule reboot... That works!

 

We updated controller first, then the VDA´s. I´m going to remove the schedule reboot, recreate it and let you know tomorrow if that was the solution-

 

BR

Martin

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...