2203 LTSR issues

[Franc van de Westelaken]

After upgrading from 2106 to 2203 LTSR we have two major issues:

 

- disconnected session sometimes aren't being reconnected when a user logs on again. Instead a new session is created. But, since we are using FSLogix, this seconds session fails due to the fact that the VHDX containers are still being used in the other disconnected session. This issue is not related to the issue below, since the disconnected that don't reconnect are shown in Director.

- at random, some sessions, especially disconnected sessions, don't show up in director. They are only visible when logging on to the VDA directly and opening taskmanager -> users. Clients are Windows 10 clients using Citrix Workspace app 2112 or newer.

 

Anyone else experiencing this also?

[Björn Schläfli]

just saw that Citrix released Virtual Apps & Desktops 2203 CU1 and found these fixed issues in their list for vda 2203 cu1:

https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/2203-ltsr/whats-new/cumulative-update-1/fixed-issues.html

 

"The wfshell.exe process might exit unexpectedly, causing new application launches from published applications to fail. [CVADHELP-17310]"

"The wfshell.exe process might exit unexpectedly when the virtual channel disconnects. [CVADHELP-17982]"

 

I will now install and test the vda 2203 cu1 and report back.

[Eric Rottier1709163337]

Citrix support gave me a workaround that should alleviate the issue. It's looking good for me after 2 days in both 2203 LTSR CU1 and 2209 versions.

 

To repeat my casenr.: SR#81394369

I did do the regkey "EnforceUserPolicyEvaluationSuccess" on its own and that did NOT work. See below for what DID work for me (for the past 2 days at least).

 

Enter these three registry keys on each affected VDA:

Do note that you may have to reboot the VDA.

 

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect

Name: FastReconnect

Type: REG_DWORD

Value: 0

 

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect

Name: DisableGPCalculation

Type: REG_DWORD

Value: 1

 

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Ica\GroupPolicy

Name: EnforceUserPolicyEvaluationSuccess

Type: REG_DWORD

Value: 0

 

I do not know (yet) if the cause is known and if\or\when and in which version it will be fixed.

 

Hope it helps!

Edited October 20, 2022 by Eric Rottier
Bit clearer and added case nr again

[Andy Vanderbeken]

 

Update: After testing and troubleshooting alot more yesterday and today I noticed this solution is not working after all. I noticed this after I implemented it on all VDA servers yesterday, waited for the nightly reboot to kick in with those reg keys present and then today upon my great surprise I could not reconnect to any new sessions on any of those VDA servers. Next I started wondering what was different yesterday when I tested it and found it working: The restarting of the "Citrix Desktop Service" service after setting the reg keys in order to make them go into effect. And YES, as soon as I restarted that service again today to verify this all existing MAC sessions on that VDA server could suddenly be reconnected to properly. 

 

What's more, I quickly discovered that when that service restarts it re-registers all existing sessions on the VDA server with the DDC and moreover correctly this time ! That means that you see the unnamed MAC user sessions in Studio suddenly become named just as the windows user sessions !  And from that moment they can be reconnected to perfect as well.

 

So in conclusion I deduce that whatever the VDA agent does during logon time in combination with the broker in order to pass on the username, brokering username, etc… seems to be the source of the problem causing the MAC sessions never to be registered correctly with the DDC and therefore not properly exist and not be found by the broker during next reconnection attempt.  Sessions from Windows clients on the other are registered properly by the VDA agent during logon and therefore don't have this problem. That my working theory atm.

 

As a temporary workaround I'm now trying out a script that restarts this service every 15 min to make sure that all existing sessions on that VDA server become properly registered with the DDC and available for proper reconnection.

[Björn Schläfli]

I have the same issue as Bruynzeel IT Support (NO session is in Director or studio (But on the VDA itself in windows you see a disconnected session with open applications)) with VDA 2203 and VDA 2103 before. Log off session in Director or Studio is the only way to get user working again as I know so far. VDA is operating on W2k16, so no FastReconnect feature on it. 

[Andy Vanderbeken]

BREAKING NEWS everyone on the specific issue of MAC (and possible other non-windows) CWA clients getting randomly disconnected unexpectedly with the "connection interrupted" pop-up message: After months of troubleshooting with both the Netscaler engineering team and VDA engineering team I just finished testing a new special build with a private fix in it that seems to solve the problem completely. After all this troubleshooting I have have identified that:

 

-The problem only happens when the connection is happening through a Netscaler Gateway. So you won't see the problem when connected through a third party VPN or internal network.

-The CWA client must be a MAC (or potentially other non-windows). The issue will not happen with a Windows CWA

-This particular problem was introduced as early as of VDA agent 2112 and is present in all later editions.

-The problem only happens when inside the Citrix session from the MAC a client-redirected device or drive is being accessed or used in any way. The easiest way to force and bring out the problem is therefore to start a continuous copy of a very large file inside the Citrix session on the from the MAC client machine towards the VDA or remote network inside the VDA. Then start clicking with your mouse on the empty desktop until the problem happens. Using this way I can consistently bring out the problem within 5-30 seconds.

 

Suspected Root cause: It's still under investigation but if this is correct it seems to be a problem introduced by a change in the winstation driver wdica.sys

Solution: obtain a new/old version of the driver through a private fix (you can refer to my Citrix CASE 81283528) and replace the current faulty version with it. Alternatively wait for a future public fix version to appear.

 

I hope this makes your day or at least helps you ! Feel free to use the upvote button

[npatel287]

Hello,

 

Just wanted to report we're seeing the same issue with a fresh new Server 2019 environment and LTSR 2203 CU1. Users trying to reconnect to disconnected sessions are randomly starting new sessions which fail due the our FSLogix VHDX still tied to the existing session. The only fix is to log them off.

 

We're using the Citrix Cloud gateway services and management. Are there any particular logs we should be looking at?

[Gian Amstutz]

Hello everyone

We are still facing the same problems with Citrix DaaS VDA 2203 CU1 LTSR with FsLogix. User have random disconnects and are not able to reconnect to their session. We always have to log them off from the worker and disconnect manually the fslogix VHDX. What is Citrix planning to do with this issue? At least a workarround and a rollback to a Version where this issues dont appear.

[npatel287]

19 hours ago, Gian Amstutz said:

I updated all our VDA's to 2206 and everything is now working fine. According to Citrix Support, the issues are all resolved in Version 2206.

Citrix support got back to me this morning and said the issue it's related to a FSLogix issue they've been tracking. They suggested we turn off FSLogix, which we obviously cannot do.

Based on what I'm seeing in the logs, I firmly believe it's a session reliability issue. I actually had the issue happened to myself a few days ago:

 

1) I came back to my computer and saw the connection interrupted (Session Reliability)

2) A second new Citrix Viewer session automatically opened up

3) I was presented with the Windows login page for my session (not normal)

4) When entering in my credentials, the screen become stuck and did not proceed any further

5) I disconnected from that session and tried to launch the Citrix viewer again from the Cloud Storefront page

6) I was connected to a brand new session and in Director my existing session disappeared

7) My old session was still on the server even though Director showed no existing session

? Had to log off that session in order to reconnect back successfully

 

EDIT: Citrix support stated they have customers on VDA 2206 with the same issue and are escalating it further, there's no ETA for a fix. I'm not confident they are associating this issue correctly.

 

@Gian Amstutz Is this the behavior you were seeing?

Edited September 3, 2022 by npatel287
Updated notes

[Eric Rottier1709163337]

On 9/2/2022 at 7:48 PM, npatel287 said:

 

 

6) I was connected to a brand new session and in Director my existing session disappeared

7) My old session was still on the server even though Director showed no existing session

8 ) Had to log off that session in order to reconnect back successfully

 

 

@Gian Amstutz Is this the behavior you were seeing?

 

If I may chime in? (doing it anyway)

 

In my 2203 LTSR CU1 environment I never see why something is gone wrong, users always come back from a bit of AFK, like lunch. Then your points 6 - 8 are seen.

We do have FSLogix, but only for O365 Outlook caching.

Users can start two sessions at a time, but that may be a FSLogix bug(?). They do not have multi-sessions FSLogix access, so no Outlook caching.

I do have the feeling it's less then without CU1 somehow.

 

Always the same story. People come back from AFK, they cannot launch Edge (we have a redirected folder, so only one sessions at a time can start Edge) and I'm called.

I'll go to the fileserver to check which two servers the users is connected to. Then I'll check the Director.

The server the user is not connected to, I'll log the users off from.

[npatel287]

1 hour ago, Eric Rottier1709163337 said:

 

If I may chime in? (doing it anyway)

 

In my 2203 LTSR CU1 environment I never see why something is gone wrong, users always come back from a bit of AFK, like lunch. Then your points 6 - 8 are seen.

We do have FSLogix, but only for O365 Outlook caching.

Users can start two sessions at a time, but that may be a FSLogix bug(?). They do not have multi-sessions FSLogix access, so no Outlook caching.

I do have the feeling it's less then without CU1 somehow.

 

Always the same story. People come back from AFK, they cannot launch Edge (we have a redirected folder, so only one sessions at a time can start Edge) and I'm called.

I'll go to the fileserver to check which two servers the users is connected to. Then I'll check the Director.

The server the user is not connected to, I'll log the users off from.

 

It seems like you have to be disconnected for more than 20 minutes for the occur more often. Here's another thread: CVAD 2203 LTSR lost user session - XenDesktop 7.x - Discussions (citrix.com)

 

Have you considered downgrading to 1912 CU5? I think we may have no choice.

[Andy Vanderbeken]

1 minute ago, Eric Rottier1709163337 said:

Already posted there, I can here due to your post ?

 

Yes and no. I won't consider it because I want to stay on LTSR.

 

I have to things to figure out and try. First I'm going to update FSLogix to the latest versions. (I'm 1 version+hotfix behind) And I noticed on a disconnected user, after logging that person off, the AppData\local\Microsoft\Office\16.0\Licensing folder is left behind. I cannot delete it or takeown. This is what I'm working on atm.

Just for the record I'm still on that old FSlogix version as well and still am even after successfully solving all the 2203 issues (reconnect problem, interrupted problem, ...) so I don't think FSlogix is the problem in any of these cases.

[Andy Vanderbeken]

On 9/2/2022 at 7:48 PM, npatel287 said:

Citrix support got back to me this morning and said the issue it's related to a FSLogix issue they've been tracking. They suggested we turn off FSLogix, which we obviously cannot do.

Based on what I'm seeing in the logs, I firmly believe it's a session reliability issue. I actually had the issue happened to myself a few days ago:

 

1) I came back to my computer and saw the connection interrupted (Session Reliability)

2) A second new Citrix Viewer session automatically opened up

3) I was presented with the Windows login page for my session (not normal)

4) When entering in my credentials, the screen become stuck and did not proceed any further

5) I disconnected from that session and tried to launch the Citrix viewer again from the Cloud Storefront page

6) I was connected to a brand new session and in Director my existing session disappeared

7) My old session was still on the server even though Director showed no existing session

? Had to log off that session in order to reconnect back successfully

 

EDIT: Citrix support stated they have customers on VDA 2206 with the same issue and are escalating it further, there's no ETA for a fix. I'm not confident they are associating this issue correctly.

 

@Gian Amstutz Is this the behavior you were seeing?

@npatel287

 

This sounds completely like the "reconnect" issue. 2 questions:

 

1. What are the client OS's of the affected machines ?

2. The next time it happens to a person, before entirely logging off that session can you please try the following workaround: Look which VDA server their disconnected (non-reconnectable) session is located on and restart the "Citrix Desktop Service" service on that particular VDA server. If all goes well 15sec after restarting that service the existing session should be reconnectable once more as restarting this service will re-register the existing session in the Citrix Studio database.

 

If this works it is 100% the reconnect issue and if that's the case I can tell you it is NOT an FSlogix issue and it's for sure NOT solved in 2206 because I tested all that myself.

[Andy Vanderbeken]

On 9/27/2022 at 4:34 AM, npatel287 said:

@Andy Vanderbeken What registry fixes are you using?

 

FastReconnect

DisableGPCalculation

EnforceUserPolicyEvaluationSuccess

 

Support wants us to test FastReconnect alone without the other two, but we did see EnforceUserPolicyEvaluationSuccess work as my noted in my previous post.

 

Oh totally missed this question somehow.

 

EnforceUserPolicyEvaluationSuccess I haven't used up till now but I'm going to start trying it out this very week to see if it does anything additional to my environment

DisableGPCalculation I don't use at all apparently in my build

I have been using Fastreconnect since March 2020 because it was a known fix for occasional reconnection issues because of a new windows 2019 mechanic. I immediately opened a case even before bringing this new build productional which eventually lead me to this particular reg key fix. I just dove into my old personal build notes from back then and there I find the following relevant part:

 

-Added the following Registry key:
 [HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect]
"FastReconnect"=dword:00000000

Note: this solves an issue where people sometimes cannot reconnect anymore to a (disconnected) session and thus lose all their open applications and unsaved data. 
(https://support.citrix.com/article/CTX256900)
 
 

*this build will go live on all relevant servers at 21-03-2020
 

 

 

[Eric Rottier1709163337]

Oh btw, I just heard that it should've been fixed in the next CU's!

So that is some great news to look forward to, right?

[Andy Vanderbeken]

Hi Franc,

 

yes. the MAC users are having this constantly and the reason -after endless troubleshooting and digging-  seems to be in the session reliability mechanism. Try disabling it through a Citrix policy, then gpupdate on the VDA server to make it go into effect and then test again or wait for the next day to see if things change.

 

I have support tickets open with Citrix to hunt down the problem, trace logs gathered and sent to the Citrix engineering team for analysis. I'm convinced they broke something in VDA 2112 as well as 2203 LTSR compared to for instance the golden standard 1912 LTSR CU3 which runs flawless.

[Franc van de Westelaken]

Hi,

 

we have the issue with Windows 10 clients. For mac users the issue seems to be resolved with the latest Workspace app for MAC.

 

Regards,

Franc.

[Dawid Bbn]

Hi all,

What's the latest on this issue?

We have the same problem with 2112 & 2203 VDAs and CWA for Linux.

Andy, did Citrix go back to you?

Thanks
 

[Andy Vanderbeken]

Hi all,

 

 

the latest status on my side is that I'm still waiting on feedback from their backend engineering team. We have been able to reproduce the issues in a trace and log collect. They have seen the problems and escalated all the way up to the software devs who are now trying to come up with a solution.

[Hardeep Sidhu1709163344]

14 hours ago, Andy Vanderbeken said:

Hi all,

 

 

the latest status on my side is that I'm still waiting on feedback from their backend engineering team. We have been able to reproduce the issues in a trace and log collect. They have seen the problems and escalated all the way up to the software devs who are now trying to come up with a solution.

Hi There, I wouldn't hold my breath on this one. It is a known issue with no fix. We decided to downgrade our clients from 2112 and 2109 to 2106 and have not had the issue again. 2106 is supported until the end of this year so there is time for Citrix to finally fix this bug. I did consider going to 2203 however it appears this issue has continued to through to that version as well. Another option would be 1912 CU3 which came out around the same time as 2106 and seems to be stable.

[Dawid Bbn]

Thanks for update Andy,

I suppose Citrix is aware that these bug stops many customers from uprading to 2112/2203 LTSR in order to take advantage of recent HDX & Adaptive Audio improvements..



 

[Joe Robinson]

On 5/20/2022 at 6:28 PM, Hardeep Sidhu1709163344 said:

Hi There, I wouldn't hold my breath on this one. It is a known issue with no fix. We decided to downgrade our clients from 2112 and 2109 to 2106 and have not had the issue again. 2106 is supported until the end of this year so there is time for Citrix to finally fix this bug. I did consider going to 2203 however it appears this issue has continued to through to that version as well. Another option would be 1912 CU3 which came out around the same time as 2106 and seems to be stable.

 

Just wanted to point out, 2106 is vulnerable to an escalation of privledges attack.  You can read more about it here.

https://support.citrix.com/article/CTX319750

 

As far as I can tell ther eis no "good" version; you're either vulnerable, or stuff doesn't work.

[Andreas Roeslmaier]

Just want to let you know, we had the same problems after updating our published desktops to VDA 2203 (no reconnect to existing session, and new session did not show up in director).

After rollback the VDAs to 2106 the problems were gone.

[Martin Garber]

Hi, we have the same problem with reconnection since VDA upgrade from 1912 CU5 to 2203.

Is there a solution from Citrix in the meantime?

[Andy Vanderbeken]

5 minutes ago, Martin Garber said:

Hi, we have the same problem with reconnection since VDA upgrade from 1912 CU5 to 2203.

Is there a solution from Citrix in the meantime?

Hi Martin,

 

no, my open Citrix case has been having the following status for the last month now:

 

Case Status:
==================================
Multiple Engineering Teams engaged

Internal Investigation ongoing

 

 

furthermore I got confirmation that the Citrix VDA team is crawling through the source code trying to find the root cause. So it could be soon. Or not... ?

[Martin Garber]

Hi,

 

try to set this reg-keys

 

Local machine - software - Citrix created a new folder called "reconnect"

added dword key "DisableGPCalculation" with 1

added dword key "FastReconnect" with a 0

 

This fixed our problem