Citrix Xenapp 7.6 FRP3 - Servers randomly crashing

[Martin Godfrey]

We have an issue with Citrix servers randomly rebooting themselves.

 

Since applying kb4012216 Windows Updates onto Citrix Xenapp servers (VDA 7.6.0, Windows Server 2012 R2), we have noticed that from our 20 Citrix servers, at least once a day a random server at a random time would reboot itself.  We use vSphere 6 to manage the ESXi hosts and VMs, this simply reports that the Virtual Machine's guest operating system has crashed.  Looking at the Event Viewer logs, around the time of the crashes, this is the only event log that appears to be relevant:

Application popup: BrokerAgent.exe - Application Error : The exception unknown software exception (0xe0434352) occurred in the application at location 0x2d7995fc.

 

The users do not receive a message on their screen and their lose their connection.

 

Since this started to happen we have updated the VDA on all server to version 7.6.300 and the operating system has had all updates applied.  Despite this, the servers still randomly crash and reboot themselves.

 

Any suggestions would be most welcome.

 

Regards

Martin

[Chris Schrameyer]

So it only started happening after the kb4012216 update?

[Martin Godfrey]

Yes Chris,  it wasn't immediate though.  Our servers reboot every night and a fresh version of the image is loaded.  After a couple of weeks, only 1-3 servers were randomly rebooting daily, but now its between 5-10 each day.  It's very strange.

[Deepanshu Soni]

Hello Martin,

 

Is the event id 26?

Do you see kernel log after getting BrokerAgent.exe - Application Error ? saying :

 

The kernel power manager has initiated a shutdown transition.
Shutdown reason: Kernel API
Event ID: 109        

 

 

Do you have any antivirus running?

I have seen Symnetic endpoint protection causing such issues.

 

 

-Deepanshu

[Martin Godfrey]

Good evening,

 

Yes it is Event ID 26 and then 3 seconds later, see event id 109 kernel power manager initiated a shutdown transition.  We are running Sophos AV.

 

Regards Martin

[Deepanshu Soni]

Martin,

 

Try to disable Sophos AV and test. let me know if it helps.

 

-Deepanshu

[Martin Godfrey]

OK, we've re-examined the Windows updates that have been installed, particularly around kb4012216.  So we've uninstalled the Windows update from May and kb4012216 and started again by installing the update from December 2016.  We are running these updates from WSUS whereas kb4012216 we installed from the Microsoft catalog.  So lets see how we get on.

[Martin Godfrey]

For the community - Uninstall/reinstalling updates made NO difference.  We are still experiencing random crashes.

[Martin Godfrey]

For the community - We have applied the VDA hotfix from ctx20311.  This made no difference.  Last night - 20/06/17 - we uninstall the VDA on our servers and reinstalled 7.6.300 + hotfix.  So again we are monitoring. At the moment we have resisted disabling Sophos on our Citrix servers.  Out of interest the reboots only occur during core hours 9am GMT to 5pm GMT, no reboots happen outside of this time.

[Adam Wilkinson1709155951]

For the community - We have applied the VDA hotfix from ctx20311.  This made no difference.  Last night - 20/06/17 - we uninstall the VDA on our servers and reinstalled 7.6.300 + hotfix.  So again we are monitoring. At the moment we have resisted disabling Sophos on our Citrix servers.  Out of interest the reboots only occur during core hours 9am GMT to 5pm GMT, no reboots happen outside of this time.

 

Hi Martin,

 

How did you resolve this is in the end please?

 

We have tried a new catalog with brand new machines and get the same issue. We have also rolled back to a snapshot on the master/golden image which was working a month ago but produces the same error message.

 

Out of the catalog we have a different VDA rebooting each day at random times which has the error message event ID 41 kernel power failure.

 

If you could provide any help at all we would appreciate it :)

 

Thanks

[Michael DiFiglia1709155180]

Anything new on this topic? I have Citrix 7.6 Application servers shutting down randomly. This started on 10/10 on two servers and has happened on 10/11 on three servers. There has been no warning at all. I did receive Errors 36888 Schannel: the following fatal alert was generated 10. The internal error state is 10.

 

 I also have a ton of Schannel info alerts 36880 in my System Event logs that fill it up on a daily basis making it harder to troubleshoot. I did see I can turn those off by editing the Registry, but I'm hesitant on doing that.  

[Ryan Jewell1709155084]

Hi, interestingly we are receiving very similar issues - can you let us know if you disabled Sophos AV\modified your settings to resolve this?

[Michael DiFiglia1709155180]

This happened again on our last Citrix server which hadn't shutdown yet. So now all of our Citrix servers have shutdown in the past month. Also we had this happen to another server which is used as a Terminal server for an application that isn't in Citrix.

 

I haven't disabled Sophos on these servers yet. I'm also hesitant in doing this.

[Darren Etheridge]

Anyone have an update on this? We're experiencing the EXACT same issues. Random sh hosts. Exact same event id's this is on 7.15 LTSR

[MICHAEL HADDOCK]

I had this same frustrating issue with random reboots.  The last time I upgraded VMtools on the vDisk, I changed SVGA Driver from Entire Feature Will Be Unavailable to Will Be Installed.  Rebooted, then reinstalled VMtools, this time returning the SVGA Driver to Unavailable.  Our random reboots have stopped completely.  

[Sharath Attur Prakash]

On 6/26/2018 at 8:20 PM, MICHAEL HADDOCK said:

I had this same frustrating issue with random reboots.  The last time I upgraded VMtools on the vDisk, I changed SVGA Driver from Entire Feature Will Be Unavailable to Will Be Installed.  Rebooted, then reinstalled VMtools, this time returning the SVGA Driver to Unavailable.  Our random reboots have stopped completely.  

Hi , 

 

Can you please share further details on what was done. We have similar issue with our infra but we never upgraded any VM tools. We always see this events on event viewer but the issue with random restarts has been observed since couple of weeks.