Absolutely weird time sync issue causing VDA disconnects. - Citrix VDA 7.15

[Carlo Piva1709156772]

Hopefully someone can lead me in the right direction.

About 3 or 4 weeks ago we started getting weird disconnects on the VDA's.

At first we thought it was an issue with a switch, but it looked okay. We even changed cabling. 

The symptoms were random, and at first they would only occur once per every other day, but now it is occuring every few minutes (to a few hours between incidents) , but eventually the disconnection does resolve itself. 

The other day, I was able to dig further into the problem and discovered that when the incidents occur that the time on the VDA changes by about -4.5 hours (as in, it changes to 4.50 hours into the past).

Knowing now that it is probably a time sync issue, i investigated further.

I have been able to confirm that the time on the XenServer hosts has not changed during this incidents

I have been able to confirm that the time on the Domain Controllers has not changed when this occurs.

I have also been able to confirm that the time does not change on the Citrix Delivery Controllers, Netscalers, or Storefront servers when this occurs

The only servers affected are the Citrix VDA’s. This is what made me initially think it could be group policy related (Netscaler doesn’t get GPO’s, The Storefronts and Delivery Controllers are in different groups and are not influenced by GPO’s that are associated to the VDA's. However checking the output of gpresult came up empty.

 

I have verified that the time zone is configured correctly on all affected serves - again - this just started occurring out of the blue a few weeks ago and these servers have been live for about 3 years.

I noticed a pattern whereby 3 servers appear to be more affected than the others.  2 being Server 2016 servers, and the other a Server 2012R2 server.

Three other servers are also affected but to a much lesser extent, so not sure why these servers specifically are affected by the time changes more than the others.

There is a total of 9 VDA servers on this domain, 6 of them are affected in one way or another. 

 

I attempted to go through the log files on all affected servers, the delivery controller and the domain controllers and have not been able to find anything that correlates the the sudden time changes or it's equally sudden resolution.

 

I had thought it could be the BIOS on the 2 XenHosts, but quickly ruled that out because there are multiple servers on them, including the Delivery Controllers and Storefronts. If it was the bios, those servers would have also seen a time change.

 

The time between the changes are random. Can happen at any time, and periods in between changes are completely random as well. 

 

Trying to eliminate the domain controllers as a possible cause (as they act as our internal time servers) , I manually changed the NTP server on two of the affected servers using the following commands: 

net stop w32time

w32tm /config /syncfromflags:manual /manualpeerlist:"ca.pool.ntp.org"

net start w32time

w32tm /config /update

w32tm /resync /rediscover

This immediately works in the sense that the clock re-adjusts itself to the current time, however again, it will occur again, and the time will read -4.5 hours from the actual current time.

 

I am at a loss to troubleshoot this further.

Everytime I call Citrix for support, but the time I get an agent, the issue has resolved itself and there is nothing they can do. They do gather logs, and do a variety of checks, but even they admit that its hard to troubleshoot something that appears to be working. 

 

I'm just hoping that someone can point me in the right direction, give me a few next steps to look into.

 

 

[Kasper Johansen1709159522]

Hi,

 

It sounds like something I have seen a 2 different customers. 1 of them found the solution in this article helpful:
https://discussions.citrix.com/topic/399684-xenserver-tools-timesync-disable/

[Carlo Piva1709156772]

I checked out that discussion and applied the reg fix this morning

Not sure why I haven't been receiving email notifications saying that there was a response to my query, I had assumed it went unanswered until last night when I decided to check (I blame our spam firewall).

 

Anyhow I'll see if the the reg fix works. We should know shortly. I applied it to three servers that were affected most.

My manager is also having me talk to Citrix to see if there is private fix available.

 

Thanks again and I'll keep you posted!

[Carlo Piva1709156772]

Update: I applied the regkey changes as per the article Kasper had mentioned above, and those servers have not lost time sync once since doing this.

My discussion with Citrix about the issue has lead me to believe that they are aware of an issue affecting some (not all) users with the latest XenServer Management Tools. We get ours updated through Windows updates, and the issues started to occur around the same time that the servers updated their XenServer Management Tools - around July 11, 2019). On one server that was affected, I had placed into its own segregated Delivery Group for testing purposes, I reverted the registry key changes and the time sync problem returned. Citrix Support had me uninstall the XenServer Management Tools from the server and install a previous version to see if the issue persists. I am keeping an eye on it an will report findings to them (and here).