XenApp 7.16 / Windows 2016 random disconnects

[Martijn Kools]

So updated to XenApp 7.16 today, got some random published desktop disconnects myself and some other users reported similar behavior. After this happens you can just reconnect to your session.

 

Any idea what might cause this? There's no event log messages when it happens. No error on the client either.

 

 

 

[Grant Brunton]

We have recently upgraded to 7.16 to fix another fault and the result of that upgrade was a series of disconnects for users on different servers at random points in time.

We have identified the cause for those events is when the Citrix Group Policy Engine performs an RSOP scan for computer policies.

We have disabled the CitrixCSEEngine service for now and Citrix are investigating the fault to find a bug fix.

 

If you filter the application event log on the servers for event IDs 11 & 1049 you should be able to see evidence of this quite clearly if you have the same issue.

Here is an example of what we were seeing:

applog.csv 1040  Information,7/12/2017 4:12:02 p.m.,Citrix Desktop Service,1049,None,The session '69f8fd5c-5ff2-49b1-b47e-47c776c0d32c' was disconnected.
applog.csv  1041  Information,7/12/2017 4:12:01 p.m.,Citrix Desktop Service,1049,None,The session '5ddf3d1e-bdec-4d11-bbb8-e6b1d2b51543' was disconnected.
applog.csv  1054  Information,7/12/2017 4:11:58 p.m.,Citrix Desktop Service,1049,None,The session '36d9ed0c-5fba-4ddb-af51-d03ca04ef033' was disconnected.
applog.csv  1055  Information,7/12/2017 4:11:57 p.m.,Citrix Desktop Service,1049,None,The session 'ca28a1c3-0347-4235-88b7-dde1b6a735e7' was disconnected.
applog.csv  1060  Information,7/12/2017 4:11:56 p.m.,Citrix Desktop Service,1049,None,The session '27de3ab0-7854-4b62-8e9b-a442fbfc7753' was disconnected.
applog.csv  1066  Information,7/12/2017 4:11:54 p.m.,Citrix Desktop Service,1049,None,The session '0e11b36f-3dbd-4add-acfc-89589b1ac79f' was disconnected.
applog.csv  1067  Information,7/12/2017 4:11:54 p.m.,CitrixCseEngine,11,None,The Citrix Group Policy service completed RSOP calculation for computer.
applog.csv  1068  Information,7/12/2017 4:11:54 p.m.,CitrixCseEngine,10,None,The Citrix Group Policy service started RSOP calculation for computer.


applog.csv 3042  Information,7/12/2017 2:32:53 p.m.,Citrix Desktop Service,1049,None,The session '4e355447-5db8-4bd8-8394-af2dc8bf9677' was disconnected.
applog.csv  3049  Information,7/12/2017 2:32:51 p.m.,Citrix Desktop Service,1049,None,The session '6cfc9af3-8c88-4694-8756-5152c4814c4d' was disconnected.
applog.csv  3114  Information,7/12/2017 2:32:35 p.m.,Citrix Desktop Service,1049,None,The session '69f8fd5c-5ff2-49b1-b47e-47c776c0d32c' was disconnected.
applog.csv  3119  Information,7/12/2017 2:32:33 p.m.,Citrix Desktop Service,1049,None,The session '5ddf3d1e-bdec-4d11-bbb8-e6b1d2b51543' was disconnected.
applog.csv  3124  Information,7/12/2017 2:32:32 p.m.,Citrix Desktop Service,1049,None,The session '6cfc9af3-8c88-4694-8756-5152c4814c4d' was disconnected.
applog.csv  3133  Information,7/12/2017 2:32:29 p.m.,Citrix Desktop Service,1049,None,The session 'ca28a1c3-0347-4235-88b7-dde1b6a735e7' was disconnected.
applog.csv  3138  Information,7/12/2017 2:32:27 p.m.,Citrix Desktop Service,1049,None,The session '4e355447-5db8-4bd8-8394-af2dc8bf9677' was disconnected.
applog.csv  3147  Information,7/12/2017 2:32:26 p.m.,Citrix Desktop Service,1049,None,The session '27de3ab0-7854-4b62-8e9b-a442fbfc7753' was disconnected.
applog.csv  3148  Information,7/12/2017 2:32:25 p.m.,Citrix Desktop Service,1049,None,The session '69004f0f-1997-4f4b-9bc1-55f6c42264ce' was disconnected.
applog.csv  3153  Information,7/12/2017 2:32:23 p.m.,Citrix Desktop Service,1049,None,The session '0e11b36f-3dbd-4add-acfc-89589b1ac79f' was disconnected.
applog.csv  3159  Information,7/12/2017 2:32:22 p.m.,Citrix Desktop Service,1049,None,The session 'ceefbc20-7912-4794-b787-e5ab6e0bb0ac' was disconnected.
applog.csv  3160  Information,7/12/2017 2:32:22 p.m.,CitrixCseEngine,11,None,The Citrix Group Policy service completed RSOP calculation for computer.
applog.csv  3161  Information,7/12/2017 2:32:22 p.m.,CitrixCseEngine,10,None,The Citrix Group Policy service started RSOP calculation for computer.

 

[Derek Thorslund]

There have been numerous reports of random ICA session disconnects after a policy refresh, and the private fix to LC8937 appears to have solved the issue. This issue happens when customers configure machine policies that affect ICA listeners in combination with PVS/MCS. Our Engineering team are taking the fix through additional internal testing. As soon as we have a fully tested solution, we plan to post a re-spin of the 7.16 VDA to the Downloads site, plus a public patch for customers who have already installed the current 7.16 VDA. I've asked for a warning message to be put on the Downloads page for the current 7.16 VDA; some customers may still want to download it to evaluate new features but I advise holding off on putting the 7.16 VDA into production until you have obtained the fix for LC8937.

[Martijn Kools]

23 minutes ago, Troels Dietz said:

https://support.citrix.com/article/CTX231127 - looks like the "real" fix for the issue.

 

 

 

Limited release, like only a few people have this issue ;) 

Thanks for the link!

[Juan Rivera]

Are you going through NS or directly connecting to the VDA? I assume all default settings, right? Including EDT enabled and using latest Receiver?

[Martijn Kools]

1 minute ago, Juan Rivera said:

Are you going through NS or directly connecting to the VDA? I assume all default settings, right? Including EDT enabled and using latest Receiver?

 

Directly to the VDA. all default settings (no special policies configured) and latest receiver yes.

 

However, We've also added a GPU, Nivida M6 and enabled hardware acceleration so it might be related to that I'm not sure yet, I will first roll-back to 7.15 to see if it makes any difference. We get kicked out of our sessions several times a day so it wouldn't be hard to reproduce. Our production farm still runs 6.5 and no issues there so it's def. not network related.

[Eduard Hübner]

I´ve seen this issue also a few times were all connected users to a VDA will get disconnected.

 

Wondering about the fact, you are also using NVIDIA M6 and EDT turned on.

We had also the M6 for Eval period.

The eval period expired just a few days ago.

 

I will have a look, if the issue still persists for me...

 

To be precise...

We use XA7.16, Receiver 4.8 on the Clients.

[Martijn Kools]

24 minutes ago, Eduard Hübner said:

I´ve seen this issue also a few times were all connected users to a VDA will get disconnected.

 

Wondering about the fact, you are also using NVIDIA M6 and EDT turned on.

We had also the M6 for Eval period.

The eval period expired just a few days ago.

 

I will have a look, if the issue still persists for me...

 

To be precise...

We use XA7.16, Receiver 4.8 on the Clients.

 

I'm back on 7.15 now with the private fix to fix the black windows and so far so good. Still too early to tell though.

 

Might be a different issue though because not all users on the VDA are disconnecting, only a couple but always at the same time.

 

 

[Josh Farlow1709157142]

Was your site upgraded to 7.16 as well or was it an older version? We are having some random disconnects as well with 7.16 and Server 2016. No GPUs here though and the site is still on 7.13. Working on upgrading receivers now to 4.10. 

[Martijn Kools]

2 minutes ago, Josh Farlow1709157142 said:

Was your site upgraded to 7.16 as well or was it an older version? We are having some random disconnects as well with 7.16 and Server 2016. No GPUs here though and the site is still on 7.13. Working on upgrading receivers now to 4.10. 

 

I haven't tested 7.16 without GPU, only reason I went to 7.16 was the black screen fix. Now back on 7.15 and no random disconnects anymore, at least today. It might be too early to tell but it did happen every day, sometimes one user 4 - 5 times.

 

Yes site was updated to 7.16 as well. So now we have a 7.16 site with 7.15 VDA. I already tested with Receiver 4.10, makes no difference, same random disconnects.

 

[Josh Farlow1709157142]

2 minutes ago, Martijn Kools said:

 

I haven't tested 7.16 without GPU, only reason I went to 7.16 was the black screen fix. Now back on 7.15 and no random disconnects anymore, at least today. It might be too early to tell but it did happen every day, sometimes one user 4 - 5 times.

 

Yes site was updated to 7.16 as well. So now we have a 7.16 site with 7.15 VDA. I already tested with Receiver 4.10, makes no difference, same random disconnects.

 

Ok, good to know. I guess I will be downgrading as well. Thanks for the update. 

[Sebastiano Ingallo1709155720]

We got the same issue, upgraded from 7.15 CU1 to 7.16 for the keyboard issue on iPads, but now we have this bug. Cannot go back to 7.15 since our users love the fact that when they type ‘z’ on ipad an actual ‘z’ is displayed in their session!! 

 

The amount of bugs and issues in their products is becoming ridiculous, my customers are losing trust in CITRIX and so I am.

 

please let us know if you finally get a fix for 7.16

[Martijn Kools]

9 hours ago, Sebastiano Ingallo1709155720 said:

We got the same issue, upgraded from 7.15 CU1 to 7.16 for the keyboard issue on iPads, but now we have this bug. Cannot go back to 7.15 since our users love the fact that when they type ‘z’ on ipad an actual ‘z’ is displayed in their session!! 

 

The amount of bugs and issues in their products is becoming ridiculous, my customers are losing trust in CITRIX and so I am.

 

please let us know if you finally get a fix for 7.16

 

It's not like there are more bugs now then back in the Metaframe days, I think there are actually less. Only back then we've had public hotfixes so if one person had a bug and it got fixed it got fixed for everybody. Now we all need to call Citrix individually because they want to sell maintenance contracts, or you have to wait 3 months for the next release which contains new bugs and you're back at square one. It's a vicious circle.

 

You can choose: run 7.xx with bug A or upgrade to 7.xx and have bug B.. Choose your bug. I haven't seen a single release since 7.9 that didn't contain a bug that made the entire release unusable for production use.

 

Either way I think the patching method right now just sucks, I think it would be a lot better to have public hotfixes again.

[Eduard Hübner]

Just now did an gpupdate /force /target:computer.

This caused a couple of users, NOT all, to disconnect from the VDA.

 

As already described these events can be found on the Application Eventlog:

Log Name:      Application
Source:        CitrixCseEngine
Date:          12/12/2017 3:08:28 PM
Event ID:      10
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      xxx
Description:
The Citrix Group Policy service started RSOP calculation for computer.
 

Log Name:      Application
Source:        CitrixCseEngine
Date:          12/12/2017 3:08:29 PM
Event ID:      11
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     xxx
Description:
The Citrix Group Policy service completed RSOP calculation for computer.
 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          12/12/2017 3:08:31 PM
Event ID:      1049
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      xxx
Description:
The session 'e2c6a1b2-618b-442f-92a4-7ba700325a34' was disconnected.
 

Log Name:      Application
Source:        Citrix Desktop Service
Date:          12/12/2017 3:08:31 PM
Event ID:      1049
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      xxx
Description:
The session '08842d4c-5244-4348-a805-d5a26897c5eb' was disconnected.
 

 

 

[Jeffrey Holsapple]

Did you happen to change any policies in Citrix Studio before the disconnects occurred?

[Eduard Hübner]

10 minutes ago, Jeffrey Holsapple said:

Did you happen to change any policies in Citrix Studio before the disconnects occurred?

Yes.

I changed something concerning Printer driver mapping.

But this should be only a User setting.

 

Or does the /force parameter proceed the Citrix Computer policies to reapply?

[Jeffrey Holsapple]

2 hours ago, Eduard Hübner said:

Yes.

I changed something concerning Printer driver mapping.

But this should be only a User setting.

 

Or does the /force parameter proceed the Citrix Computer policies to reapply?

 

A printer policy shouldn't affect this.

 

The VDA should be monitoring for machine policy changes, i.e. to the HDX Adaptive Transport, Session Reliability Enabled, ICA Port Number, Session Reliability Port Number, and Multi-Port policies, and if any of these policies changed, it will restart the listeners (and for EDT connections, this will cause a disconnect and require a reconnect). Also, when TLS/DTLS is configured on a VDA, the listeners are restarted.

 

These machine policies very rarely change, so you shouldn't be seeing disconnects. But, there may be a bug in this code. If you can repro this and get CDF traces for CitrixServicesManager_Service and MF_Dll_wsxica then I can investigate further.

 

In the next release, machine policy changes will require a reboot of the VDA to take affect (so as not to disconnect active sessions).

[Martijn Kools]

4 minutes ago, Jeffrey Holsapple said:

 

A printer policy shouldn't affect this.

 

The VDA should be monitoring for machine policy changes, i.e. to the HDX Adaptive Transport, Session Reliability Enabled, ICA Port Number, Session Reliability Port Number, and Multi-Port policies, and if any of these policies changed, it will restart the listeners (and for EDT connections, this will cause a disconnect and require a reconnect). Also, when TLS/DTLS is configured on a VDA, the listeners are restarted.

 

These machine policies very rarely change, so you shouldn't be seeing disconnects. But, there may be a bug in this code. If you can repro this and get CDF traces for CitrixServicesManager_Service and MF_Dll_wsxica then I can investigate further.

 

In the next release, machine policy changes will require a reboot of the VDA to take affect (so as not to disconnect active sessions).

 

I hardly touch the machine policies, I don't think we even have any we use all default values. It's not hard to repro just run a 7.16 vda put 15 users on it (idle is enough) wait 4-8 hours and watch half of them drop from the server.

 

Ironic btw how the next release prevents disconnecting sessions while this release drops them because of a bug. Seems like new features are being introduced faster then bugs can be fixed. I'd rather have it the other way around.

 

Anyway if somebody wants to test this and run a CDF trace go ahead. I'll stick to 7.15 I'm done doing Citrix's home work, it's not like I'm getting a discount or get paid to do it. I'll check back on the next LTSR.

[Grant Brunton]

What we discovered was that we could trigger the Citrix Policy Engine to perform a RSOP scan (triggering event 11 and disconnecting users) by either doing a gpupdate /force or by updating policy settings in Citrix Studio so definitely looks like the same issue happening here.

 

We have a top analyst working with us on the issue and he has identified the problem occuring in the CDF trace logs we have sent them so are working on the cause of the bug.

 

After disabling the CSE Engine service though we are still getting disconnects and session freezes happening to our users but a lot less than before so there are still more issues to identify somewhere.

 

[Sebastiano Ingallo1709155720]

On 12/11/2017 at 8:54 PM, Grant Brunton said:

We have recently upgraded to 7.16 to fix another fault and the result of that upgrade was a series of disconnects for users on different servers at random points in time.

We have identified the cause for those events is when the Citrix Group Policy Engine performs an RSOP scan for computer policies.

We have disabled the CitrixCSEEngine service for now and Citrix are investigating the fault to find a bug fix.

 

If you filter the application event log on the servers for event IDs 11 & 1049 you should be able to see evidence of this quite clearly if you have the same issue.

Here is an example of what we were seeing:

applog.csv 1040  Information,7/12/2017 4:12:02 p.m.,Citrix Desktop Service,1049,None,The session '69f8fd5c-5ff2-49b1-b47e-47c776c0d32c' was disconnected.
applog.csv  1041  Information,7/12/2017 4:12:01 p.m.,Citrix Desktop Service,1049,None,The session '5ddf3d1e-bdec-4d11-bbb8-e6b1d2b51543' was disconnected.
applog.csv  1054  Information,7/12/2017 4:11:58 p.m.,Citrix Desktop Service,1049,None,The session '36d9ed0c-5fba-4ddb-af51-d03ca04ef033' was disconnected.
applog.csv  1055  Information,7/12/2017 4:11:57 p.m.,Citrix Desktop Service,1049,None,The session 'ca28a1c3-0347-4235-88b7-dde1b6a735e7' was disconnected.
applog.csv  1060  Information,7/12/2017 4:11:56 p.m.,Citrix Desktop Service,1049,None,The session '27de3ab0-7854-4b62-8e9b-a442fbfc7753' was disconnected.
applog.csv  1066  Information,7/12/2017 4:11:54 p.m.,Citrix Desktop Service,1049,None,The session '0e11b36f-3dbd-4add-acfc-89589b1ac79f' was disconnected.
applog.csv  1067  Information,7/12/2017 4:11:54 p.m.,CitrixCseEngine,11,None,The Citrix Group Policy service completed RSOP calculation for computer.
applog.csv  1068  Information,7/12/2017 4:11:54 p.m.,CitrixCseEngine,10,None,The Citrix Group Policy service started RSOP calculation for computer.


applog.csv 3042  Information,7/12/2017 2:32:53 p.m.,Citrix Desktop Service,1049,None,The session '4e355447-5db8-4bd8-8394-af2dc8bf9677' was disconnected.
applog.csv  3049  Information,7/12/2017 2:32:51 p.m.,Citrix Desktop Service,1049,None,The session '6cfc9af3-8c88-4694-8756-5152c4814c4d' was disconnected.
applog.csv  3114  Information,7/12/2017 2:32:35 p.m.,Citrix Desktop Service,1049,None,The session '69f8fd5c-5ff2-49b1-b47e-47c776c0d32c' was disconnected.
applog.csv  3119  Information,7/12/2017 2:32:33 p.m.,Citrix Desktop Service,1049,None,The session '5ddf3d1e-bdec-4d11-bbb8-e6b1d2b51543' was disconnected.
applog.csv  3124  Information,7/12/2017 2:32:32 p.m.,Citrix Desktop Service,1049,None,The session '6cfc9af3-8c88-4694-8756-5152c4814c4d' was disconnected.
applog.csv  3133  Information,7/12/2017 2:32:29 p.m.,Citrix Desktop Service,1049,None,The session 'ca28a1c3-0347-4235-88b7-dde1b6a735e7' was disconnected.
applog.csv  3138  Information,7/12/2017 2:32:27 p.m.,Citrix Desktop Service,1049,None,The session '4e355447-5db8-4bd8-8394-af2dc8bf9677' was disconnected.
applog.csv  3147  Information,7/12/2017 2:32:26 p.m.,Citrix Desktop Service,1049,None,The session '27de3ab0-7854-4b62-8e9b-a442fbfc7753' was disconnected.
applog.csv  3148  Information,7/12/2017 2:32:25 p.m.,Citrix Desktop Service,1049,None,The session '69004f0f-1997-4f4b-9bc1-55f6c42264ce' was disconnected.
applog.csv  3153  Information,7/12/2017 2:32:23 p.m.,Citrix Desktop Service,1049,None,The session '0e11b36f-3dbd-4add-acfc-89589b1ac79f' was disconnected.
applog.csv  3159  Information,7/12/2017 2:32:22 p.m.,Citrix Desktop Service,1049,None,The session 'ceefbc20-7912-4794-b787-e5ab6e0bb0ac' was disconnected.
applog.csv  3160  Information,7/12/2017 2:32:22 p.m.,CitrixCseEngine,11,None,The Citrix Group Policy service completed RSOP calculation for computer.
applog.csv  3161  Information,7/12/2017 2:32:22 p.m.,CitrixCseEngine,10,None,The Citrix Group Policy service started RSOP calculation for computer.

 

 

Hello gbrunto,

do you have the number of your ticket case at Citrix? I ask it because they say that they don't have other tickets related to this issue.

My ticket number is 75210007.

[CitrixSuisse]

can you guys have a look at Microsoft-Windows-TerminalServices-LocalSessionManager/Operational

just wondering if that's an RDS issue and not a citrix issue

[Sebastiano Ingallo1709155720]

 

52 minutes ago, CitrixSuisse said:

can you guys have a look at Microsoft-Windows-TerminalServices-LocalSessionManager/Operational

just wondering if that's an RDS issue and not a citrix issue

 

Thanks for your reply CitrixSuisse, however, the issue started as from the upgrade of the VDA to version 7.16. With VDA 7.15 and 7.15 CU1 the issue does not occur.

On the EventViewer, in the location you indicated, I see messages like this: Session 6 has been disconnected, reason code 12, I guess it's normal

[CitrixSuisse]

Having the same issue but also with VDA 7.15 on server 2016.

It's not GPO refresh and the interval between random disconnection varies.

 

Using receiver 4.7

[CitrixSuisse]

Just opened a support case.

[Sebastiano Ingallo1709155720]

3 minutes ago, CitrixSuisse said:

Just opened a support case.

ok, if you wish you can give them my ticket number so they can link the two of them

[Samuel Jacob]

I am the Engineer  working the issue with the Disconnect on 7.16. It is primarily occurring due to listener refreshes because of  machine policy update. If you have 7.16 ServerOS VDA, and are seeing the disconnects please fire up a support case and let me know here. 

 

I will supply you a PrivateFix once  engineering develops a build for it.