XenServer tools timesync disable

[Patrik Nyberg]

I recently encountered a very annoying problem on my Windows 10/Server 2016 VMs running on XenServer 7.1 CU1.

On startup of the VMs the time was advanced 54 minutes when XenInterface/XenAgent started, when the w32time service started on the VM the time was then synchronized to the DCs of the domain and was ok again.

 

My issues I got was that some of my VMs lost connection to the domain - because the time was off with more than 5 minutes, and this was happening randomly - depending on when the Xen Agents was starting. The impact for me was that the startup scripts configured in the domain was not running on the VMs...

 

My troubleshooting was to check NTP settings on the hosts, on the DCs and the HW clock on the Hosts - all to no avail - meaning no clock issues on any of them. Also the timeoffset on the VM metadata I tried to set differently - but also did not change the behaviour.

 

So in the end, my only workaround left was to try to disable the timesync between the VMs XenAgent (XenInterface) and the host. But no official documentation stated that this was possible..

However, after a bit of digging I found in the Xen source code for the XenAgent - it seemed there was a registry setting inside the VM that was checked for a specific value - if that was found it would synchronize the VMs' time with the host. So in order to disable that you can just set tha t value to anything else - or remove the value all together would also do the trick I believe.

 

The registry key/value to disable time sync with host (works on Windows 10 VMs (tested on 1709),  Server 2016, XenServer tools 7.1.1305, XenServer 7.1 CU1):

 

HKLM\Software\Citrix\XenTools\HostTime="Local"

HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime="Local"

 

Default value is HostTime="UTC" - so set anything else than that - should disable sync with the host.

 

For me the problem went away completely and I have implemented this registry setting on all my VMs now.

 

 

[Michael Martis]

I also experienced the time sync. issue after the management agent was updated from 7.1.1327 to 7.1.1396
Both keys below had value of UTC
HKLM\SOFTWARE\Citrix\XenTools\HostTime

HKLM\SOFTWARE\Wow6432Node\Citrix\XenTools\HostTime 

 

Just for peace of mind, cleared both values , and it fixed the issue. 

 

I found it odd, for Wow6432Node key, HostTime was the only value. No other values or sub keys.  ~shrug~

[Mike Berry]

Quote

I also had this issue and now running the most current XenTools. 7.2.1555.  I have some 2012R2 Servers (Not PVS'd, just standard vm)  that we removed the UTC from those reg keys and are spot on with time.  PVS'd 2012R2 Servers, even though have same fix and driver I am finding have time drift of anywhere from 2 minutes before "correct time" to two minutes after.  Below is just a sampling...  webxxx you can see are in sync as opposed to the hsxxx pvs'd servers.  Our XenServers are LSTR and current with all patches, VM's have up today tools (7.2.1555) and XenApp CU5, PVS is 7.15 LTSR with Current CU5 as well.  No matter what I try, w32tm /resync, etc.....  they are never in "sync" like other Servers in our environment.  

 

web015  04/05/2020 00:04:26
web016  04/05/2020 00:04:26
web017  04/05/2020 00:04:26
web018  04/05/2020 00:04:26
web019  04/05/2020 00:04:26
web020  04/05/2020 00:04:26
web021  04/05/2020 00:04:26
web022  04/05/2020 00:04:26
web023  04/05/2020 00:04:26
web024  04/05/2020 00:04:26
web025  04/05/2020 00:04:26
web026  04/05/2020 00:04:26
web027  04/05/2020 00:04:26
web028  04/05/2020 00:04:26
web029  04/05/2020 00:04:26
web030  04/05/2020 00:04:26

 

HS001  04/05/2020 00:04:20
HS003  04/05/2020 00:04:24
HS004  04/05/2020 00:04:21
HS005  04/05/2020 00:04:16
HS006  04/05/2020 00:04:17
HS007  04/05/2020 00:04:27
HS008  04/05/2020 00:04:18
HS009  04/05/2020 00:04:27
HS010  04/05/2020 00:04:29
HS011  04/05/2020 00:04:26
HS012  04/05/2020 00:03:57
HS013  04/05/2020 00:03:58

 

[Ralf Grashoff]

I´ve found one task the agent will do(?): the server reboot configured in Citrix Director... but not every server did it the last 3 days.

 

Best regards,

Ralf

[IT Carbogen Amcis]

On 4/24/2020 at 5:28 PM, Mike Berry said:

Update for time sync issue on Xen Server 8.1.x for escalation

 

As you may already know, we opened this other case to track your time sync issue in your Hypervisor 8.1.

Just as a quick overview, there is a known issue on Citrix Hypervisor 8.1, on Windows VMs with latest XenTools, V9 drivers, where there is a time sync drift every 30 minutes or so.

Basically, there is a race between XenAgent time sync and AD time Sync.

There are some workarounds for this that we know:

1. Disable AD time sync and only allow XenAgent time sync.

2. Disable XenAgent and leave AD time sync, but by doing this you will loose graceful shutdown, reboot and some other features of XenAgent on VMs.

3. The last workaround is to install a third party time sync tool. We know of NetTime v3.14 available for free from www.timesynctool.com. This utility can be set to sync against domain controllers internally and resync after a period of time. We currently have it set to sync every 15 minutes and adjust time if the skew is more than 1 second then adjust the time....This has so far been working great on their test environment.

Just as a side note, we haven't completely tested these workarounds, some of them were tested by other users and we don't know if there are other implications so far, so I just provided them just in case you are in any urgency and would like to test.

On the other hand, we have an internal case with our developers, they are working on a possible fix that basically will allow you to disable XenAgent time sync only without disabling other features of XenAgent, and leaving the users to be able to use AD time sync only for the VMs.

For this, we don't have an ETA yet but I will provide details as soon as we get them.

 

 

Hi 

 

Any updates here? We seem to have the same with XenTools V9 on Win2012 and 2016 Server. Stop the Xen Tools service worked but can't be the solution.

 

Thanks

Patrick

[Mike Berry]

Citrix has given me a private fix to test.  Will be moving it into one pool tonight and will provide feedback in the am.

[Ard-Jan Verhage]

10 minutes ago, Mike Berry said:

Citrix has given me a private fix to test.  Will be moving it into one pool tonight and will provide feedback in the am.

Good news.

 

I'm now running my PVS vDisks with an older version of XenTools to avoid the issue.

[Kyle Peterson]

Turns out TimeSyncMode has not been working in versions 9.2.2 and Im going to guess 9.2.1 as well. Updated to 9.2.3 and now the key works as it should.

 

See https://support.citrix.com/article/CTX235403

[Alan Lantz]

I've not noticed anything like that. Is the time/time zone on the XenServer host proper ?

I guess you have something that works, but again seems strange that no one else has

had to do this.

 

--Alan--

 

[Patrik Nyberg]

Yepp, the time/time zone is correct on the host - I agree very strange - especially since the time difference/change is advanced about 54 minutes - suggesting it is not just time zone issue... Started the VM on all my hosts - just to rule out if the HW clock is bad on one of the hosts... but the problem persisted...

I thought I have tested everything "obvious" - also uninstalled latest Windows 10 CU - but the problem persisted, also reverted tools to the previous version, but no change..

 

So when I found this workaround - I was happy - because the problem went away right away.

 

Patrik

[Tobias Kreidl]

I assume the time on the VM is not being synchronized independently, as otherwise it'll just take on the time of the host in which it's running. The other option is to of course set it up to independently sync to preferably the same NTP source as the XenServer host. In any case, that behavior you experienced seems odd. I've not seen this with any Windows or Linux VMs.

 

-=Tobias

[Nick Panaccio]

I know I’m digging up an old thread, but did you ever find another workaround for your issue? We have been fighting a time sync problem that exists solely on our Server 2016 VMs (XA 7.15 LTSR CU running on XS 7.1 CU1). The XenServers are all configured for the same NTP servers, the time is correct on all of them, yet only the 2016 guests boot to wildly off times (more than 2 hours in some instances). And as such, all group policy processing fails and the box is virtually useless.

 

We’ve follower every Citrix guide on this, including a more recent one telling you to run w32tm /resync /nowait with the vDisk in Private mode to no avail. Your HostTime registry key is the only thing that seems to work in my limited testing. That I can’t find a single other reference to this is bothering me.

[Alan Lantz]

I vaguely remember some hardware that has issues, but a 2016 server on the domain should follow the 

domain controller time.  The NTP servers for the XenServers are all external to the XenServer infrastructure,

correct? Running NTP within XenServer would really mess things up.  A local time source or syncing out

to the Internet is required.

 

--Alan--

 

[Patrik Nyberg]

Nick,

 

No unfortunately not, I haven't found the root cause for this as of now. Very strange indeed. In the mean time I have deployed the registry "fix" on all my VMs - and I haven't had a single problem with time sync since then...

Usually I also want to find the root cause - but there are limits on how much time you put on an issue - especially if you have a pretty simple workaround...

 

But I will update this post if I ever find the root cause.

Thanks for sharing your experiences Nick, I thought I was the only one...

 

[Nick Panaccio]

14 hours ago, Alan Lantz said:

I vaguely remember some hardware that has issues, but a 2016 server on the domain should follow the 

domain controller time.  The NTP servers for the XenServers are all external to the XenServer infrastructure,

correct? Running NTP within XenServer would really mess things up.  A local time source or syncing out

to the Internet is required.

 

--Alan--

 

Yes, the NTP servers are all external to XS, nothing running on the XS itself.

 

5 hours ago, Patrik Nyberg said:

Nick,

 

No unfortunately not, I haven't found the root cause for this as of now. Very strange indeed. In the mean time I have deployed the registry "fix" on all my VMs - and I haven't had a single problem with time sync since then...

Usually I also want to find the root cause - but there are limits on how much time you put on an issue - especially if you have a pretty simple workaround...

 

But I will update this post if I ever find the root cause.

Thanks for sharing your experiences Nick, I thought I was the only one...

 

Our XA admin was ready to bang her head through a wall over this issue, because it's been killing us. Before I stumbled across your post, she had written a script to force a lot of stuff at startup just to get through testing, but since making this change everything has been good. I'm going to just roll with it, as well, but I do believe that she submitted a ticket with Citrix last week. If that goes anywhere, I'll absolutely post the findings here.

[Danil Pichugin]

Same problem. I discovered it after updating XenTools to 7.1.1327 (from 7.1.1120). The time when the XenServer Agent service restarts (C: \ Windows \ system32 \ xenagent_8_2_1_111.exe) offset +4:30.

 

XenServer 7.1 CU2, HPE Server Gen8, Gen9. Time and Time Zone is correct:
# date
Mon Jun 17 16:10:55 +05 2019
] # hwclock - show
Mon 17 Jun 2019 04:11:01 PM +05 -0.560539 seconds
# cat / etc / adjtime
0.0 0 0.0
0

Guest OS - Windows server 2012 R2.

Removing the parameter value of the HostTime in the registry or set to "Local", or disable service solved it problem. 

[Ralf Grashoff]

Hi,

 

we´ve got the same issue with a new windows server 2016 vm running on a 7.1 CU2 host (hp dl360g9). With eyery reboot the time of the vm added 2h48m - so there is no way for authentication with AD and time sync.

Removing the parameter fixed the problem.

 

Update: with installation of Citrix VDA 1906 the Regkey will be set back to UTC!

 

Best regards,

Ralf

[MICHAEL BOROSAK]

On 11/16/2018 at 6:56 AM, Patrik Nyberg said:

I recently encountered a very annoying problem on my Windows 10/Server 2016 VMs running on XenServer 7.1 CU1.

On startup of the VMs the time was advanced 54 minutes when XenInterface/XenAgent started, when the w32time service started on the VM the time was then synchronized to the DCs of the domain and was ok again.

 

My issues I got was that some of my VMs lost connection to the domain - because the time was off with more than 5 minutes, and this was happening randomly - depending on when the Xen Agents was starting. The impact for me was that the startup scripts configured in the domain was not running on the VMs...

 

My troubleshooting was to check NTP settings on the hosts, on the DCs and the HW clock on the Hosts - all to no avail - meaning no clock issues on any of them. Also the timeoffset on the VM metadata I tried to set differently - but also did not change the behaviour.

 

So in the end, my only workaround left was to try to disable the timesync between the VMs XenAgent (XenInterface) and the host. But no official documentation stated that this was possible..

However, after a bit of digging I found in the Xen source code for the XenAgent - it seemed there was a registry setting inside the VM that was checked for a specific value - if that was found it would synchronize the VMs' time with the host. So in order to disable that you can just set tha t value to anything else - or remove the value all together would also do the trick I believe.

 

The registry key/value to disable time sync with host (works on Windows 10 VMs (tested on 1709),  Server 2016, XenServer tools 7.1.1305, XenServer 7.1 CU1):

 

HKLM\Software\Citrix\XenTools\HostTime="Local"

HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime="Local"

 

Default value is HostTime="UTC" - so set anything else than that - should disable sync with the host.

 

For me the problem went away completely and I have implemented this registry setting on all my VMs now.

 

 

I had a similar issue but different situation.

 

Running hybrid cloud solution, Netscaler in the cloud with the Xenservers on prem. Using MCS to create 25 Windows 10 VMs. The VMs would all lose time by hours and minutes. Usually 4 hours and 30 minutes either behind or future. When this time change would happen the VMs would should up as Lost Connection in cloud portal and could not be reached by those requesting a desktop through the storefront. I would manually sync the windows clocks and the time would correct but usually within 20 minutes the time would change again to some strange time. I had followed many other Windows clock troubleshooting steps and suggestions but the time would never stay in sync. I have probably worked on this issue for over 40 hours without resolving, including working with Citrix support, until I found this forum post.

 

I was literally at wits end over this issue. Can't tell you how glad I am to have this resolved. Hopefully Citrix can come to some solution.

[Michael Warrillow1709160721]

We also have this issue. We're running XenServer 7.6 hotfix update 2.  

Time on hosts and NTP all working correctly. 

Our guests are a mixture of OS's 2016 and 2012 . 

Those guests with the xenServer agent installed have the issue. 

This is C:\windows\system32\xenagent_8_2_1_110.exe

 

Running "C:\Program Files\Citrix\XenTools\XenGuestAgent.exe"  version 7.0.1.212

 

We found this as GPO fails to apply for a while until NTP corrects the time flip of about 3 to 4.5 hours. 

 

Going to install the latest XenServer hotfix and see if this helps.

The registry hack does help but I'd like Citrix to resolve this . 

We'll likely open a ticket if the hotfix doesnt help. 

 

Anyone running XenServer 8 and have this issue ?

 

This seems 100 % like a citrix bug.

 

[Michael Warrillow1709160721]

Installed a new version of the Citrix management agent... 
https://support.citrix.com/article/CTX235403    7.1.0.1396

 

Control panel programs etc .  had been showing version 7.1.1327 

 

Had been using 

 

this changed the value of HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime  to be empty/ blank :  ie there is no data for the value but it is present. 

 

This also seems to resolve the error. 

[Martijn Kools]

On 09/08/2019 at 10:48 PM, Michael Warrillow1709160721 said:

Installed a new version of the Citrix management agent... 
https://support.citrix.com/article/CTX235403    7.1.0.1396

 

Control panel programs etc .  had been showing version 7.1.1327 

 

Had been using 

 

this changed the value of HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime  to be empty/ blank :  ie there is no data for the value but it is present. 

 

This also seems to resolve the error. 

 

I run the same management agent version which is what started the issue for us.

 

That key is empty indeed but HKLM\Software\Citrix\XenTools\HostTime still has the value UTC, I emptied that as well which solved the issue (I think).

 

So with version 7.1.1396 of the management agent I have the issue.

With version 7.1.1327 no issues at all even though the key values are the same for both versions (empty in wow6432node, UTC in normal node).

[Julian Jakob]

On 21.8.2019 at 2:21 PM, Martijn Kools said:

 

I run the same management agent version which is what started the issue for us.

 

That key is empty indeed but HKLM\Software\Citrix\XenTools\HostTime still has the value UTC, I emptied that as well which solved the issue (I think).

 

So with version 7.1.1396 of the management agent I have the issue.

With version 7.1.1327 no issues at all even though the key values are the same for both versions (empty in wow6432node, UTC in normal node).

 

Can confirm this, also using XS 7.1 LTSR CU2 with Agent Version 7.1.1396 at Windows Server 2012 R2. Only registry change fixxed the time sync.

[Torsten Thomsen]

Hi

 

My issue with time sync started after I updated my 2016 VDA´s with Windows patches (KB3172985) and XenServers was patched with XS76E004 and XS76E005 . I am running 7.1.1270 and have done that for a long time.

A VDA running a PVS image from before Windows patches did not have the time issue.

A VDA running on same XenServer after Windows patches had the issue.

 

We really saw issues caused by this. Outlook popping up with meeting notifications two hours before start, Navision client lost connection to Nav server - this was caused by this time sync.

I implemented the reg settings, and everything is stable again.

Thanks for the hint

 

/ Torsten

 

 

 

[Konrad Ruess]

Hey P,

Wow, thank you so much for sharing your problems and that solution with the HostTime registry value.

We were suffering from the very same bug on 7.1.1396 XenTools (uups, sorry: Mgmt Agent) - the time on our servers always jumped 6480s to the future and this repeated every +/- 32 minutes.

Older Mgmt Agents come up with a HostTime value of <empty> but .1396 now shows "UTC".

I'd really like to know what Citrix was trying to get implemented with that change. But unfortunately, release documentation is rare, those days.

So, let's enjoy stable time sync and wait for the next quirk... :-/

BfN, Konrad

[Leonardo Prates]

Thanks for sharing your problem and your solution. I started my trajectory in the citrix this September and came across this problem that seemed to me astonishing. I contacted support several times within a 2 week period and just reading this post found PEACE !! Thank you community !!!

[Jeff Bambas]

Thank you for posting!  I could not figure out why my new VM kept going back 6 hours and 18 minutes.  It only started after I installed XenApp 7.15 CU3 Virtual Delivery Agent on the VM.  Before that install, it kept its time. 

This happened on a different VM previously and I restored it from a snapshot not knowing what the cause was.  That VM created 9/17/19, same XenServer pool. 

Today, testing upgrading the VM from XenApp 7.15 CU3 to CU4, and the time went bad again.  6:18 hours behind.  The "HostTime" registry setting had UTC listed again.  I cleared it and the time is normal again.  WOW6432Node setting is still blank.    

System Info:

XenServer 7.1 CU2 Updated to XS71ECU2016

VM: Microsoft Server 2016 created 10/10/19, domain joined, fully updated

Happened after installing XenApp 7.15 CU3 Virtual Delivery Agent

XenServers get their time from 3 different internal NTP servers.  Time, date, time zone are all correct on the XenServers.

The older VMs do not have this problem.  This seems to affect newly created VMs within the past month.  I hope Citrix can track down this odd bug.