CVAD: ConnectionFailure after install LTSR 2203 CU2 on Server 2022

[Patrik Grill]

Hi!

After updating a VDA from Citrix 2203 LTSR to CU2, connections can no longer be established. After an app is started, the connection is immediately terminated. No error message appears for the end user either.  There is nothing interesting in the Windows event log except an information from the Citrix Desktop Service:

ID 1030
The Citrix Desktop Service has detected that a user session has been terminated. Session 'a37c579b-6cd5-4277-ad1e-4c4c6f9cb263' for user 'test' was terminated; cause: ConnectionFailure.

This issue only affects VDAs running Windows Server 2022. The VDAs with 2019 do not affect this problem.

Any ideas?

[JAKOB JENSEN1709163176]

17 hours ago, Jan Klimek1709163519 said:

 

Hi Jakob,

could you tell us which Powershell script from Citrix you are refering to? 

 

Hi Jan, here you have the cleanup script provided by Citrix:  

 

--------------------------------------------------

$iddHwId = "VEN_5853&DEV_1003"
$pnpDispDevPath = "HKLM:\SYSTEM\CurrentControlSet\Enum\SWD\RemoteDisplayEnum"

try
{
    $iddDevEntries = Get-ChildItem -Path $pnpDispDevPath -ErrorAction Stop| Where-Object Name -Match $iddHwId
}
catch
{
    Write-Output "Failed to query remote ID device entries"
    Write-Output $_
    return
}

if ($iddDevEntries.Count -eq 0)
{
     Write-Output "No remote ID device entries found."
     return
}

$iddDevEntries | ForEach-Object {
    
    $devName = ($_.Name -Split "\\")[-1]
    
    try
    {
        Remove-ItemProperty -Path $_.PsPath * -ErrorAction Stop
        Write-Output "Cleared values for $devName"
    }
    catch
    {
        Write-Output "Failed to clear values for $devName"
        Write-Output $_
    }
}

----------------------------------------------

[JAKOB JENSEN1709163176]

We had same issue : 

 

After updating a VDA from Citrix 2203 LTSR to CU2, connections can no longer be established. After an app is started, the connection is immediately terminated. No error message appears for the end user either.  There is nothing interesting in the Windows event log except an information from the Citrix Desktop Service:

ID 1030
The Citrix Desktop Service has detected that a user session has been terminated. Session 'a37c579b-6cd5-4277-ad1e-4c4c6f9cb263' for user 'test' was terminated; cause: ConnectionFailure.

 

Opened case with Citrix and following resolved our issue.

1.  Disabled Antivirus on VDA

2.  Uninstalled VDA

3. Ran cleanup VDA cleanup tool

4. Rebooted VDA

5. Ran Microsoft Powershell script provided by Citrix

6. Rebooted VDA

7. Ran VDA 2203 LTSR CU2 installation again.

 

Problem resolved 

  

[Alan Hodson]

Is the licensing up to date for the environment? What was the previous VDA version on the 2022 machine(s)?

[David Bernal1709161555]

Same Issue here.

Windows Server 2022 with CVAD 2203 LTSR CU1 updated to CU2. I can't connect to this VDA since the update. 

Regards.

[Buro Happold IT]

We had/have the same issue ever since 2112.  Any VDA upgrades on Server 2022 break it - permanently.  Server 2019 is not affected.

 

We tried various fixes, such as uninstall, reinstall, different GPOs, Citrix policies, even going back to the previous VDA version (which was working) didn't fix it, so the OS is effectively written off once its in that state.

 

We had a ticket raised with Citrix and MS, both blaming each other so in the end we cut our losses and rebuilt any Server 2022 servers and a clean VDA, same GPOs and Citrix policies and all good

 

Would like to know the root cause as clearly its not affecting everyone and rebuilding the OS every time a new VDA is released is not a viable solution

[Glenn Eriksen]

Hi all,

Since this is a generic error, I just wanted to share my same issue earlier today. Long story short, there is a very easy fix! We are a MSP, and have many customers on the same local AD domain, but we also have customers which is isolated with the own local AD domain. Someone had rolled out new Citrix servers couple of months ago to different customers in different domains. Problem was they all pointed to the same RDS licensing servers in our main AD domain! You need to have a RDS Licensing server in each local AD domain, which we have, but someone configured this setting to point to wrong RDS licensing server for another domain. This does not work. I found out when I just RDP to the same offending Citrix server, which then finally gave me a error message saying there is not RDS licensing server, and through my session right out! Puhh! Took me a couple of hours to figure out, but that is why it worked with rolling out new servers, since there is a grace periode and will work for a couple of months before the problem starts... So just a tip if someone else spends way to much time troubleshooting this issue!

 

Happy weekend everyone!

[Ecki]

I have a similar issue on Server 2022, not on 21016/2019, where sessions end up in a grey screen forever. RTM and CU1 are working but starting with CU1.1 (2203.0.1100.x) the issue occurs immediately. The issue is also seen with 2209 and 2212.

Here are my case details:

 

Case Number:

81462687

Status:

Pending - Awaiting Hotfix

Date Submitted:

10/10/2022

Case Subject:

Grey screen after CU1 update

Comment Body:

Case Status:
- The fixes for Server2022 and Win10 multi session will be served on 2023.2C (the third week of February).
- This is known issue - HDX-34091. Fix will be released by Microsoft.

- The gray screen issue you see on Windows 2022 matches known issue caused by Microsoft RDS function. Our engineering involved Microsoft, and MS was working on fix for it. It should be included in KB 5018496, and it should be available for Windows 11 22H2 and Server 2022. It seems the KB is available for Win11, I do not see it for Windows 2022. I will check with our engineering what is the status of this fix.
Anyway this issue has to be addressed by Microsoft.  It is caused by Terminal Service calling PreDisconnect when session starts, and because of this HDX graphic is not properly initialized, so users see gray screen. 

[Helpdesk SES]

Hi,

almost same error as Patrik here: after update to 2203 LTSR CU2 (from CU1) on Windows Server 2022,  connection seems to start but terminates without errors except for the 1030 information from Desktop service in event viewer. This happens on the first and second attempt to launch the session. Strangely enough the connection is established on the third and later attempts.

!!!

 

 

[Jan Klimek1709163519]

We also have the same error  with Server 2022 - as soon as a Citrix server is updated, no connection can be established. Error ID 1030 is stored in the event log. 
Uninstalling and reinstalling the VDA agent does not improve the situation. We had to restore the server from a backup. Server 2019 and 2016 is working fine... Furthermore, the remote desktop licensing servers are also set and work without error messages.

We then set up a completely new server (with Server 2022) and installed version 2209 on it  -  but it worked without any problems.. However, once version 2203 was installed or the server was already in use with an older Citrix VDA version, the connection cannot be re-established after installing a newer agent. 

It would be great to hear something from Citrix in a timely manner here!!

[Daniel Sjoumlgren]

Same Issue here as well. One of six of my catalogs got this.

Windows Server 2022 with CVAD 2203 LTSR CU1 updated to CU2. I can't connect to this VDA since the update and have error 1030. 

[Martin Becker]

We are now hitting this. I have a case opened for it.

[Martin Becker]

I was able to revert to CU1 successfully.

1. Ran vdacleanuputility.exe
2. Removed Citrix Workspace app as it was installed
3. Reran CU1 installer and was able to do so without issue.

Strike that, I cannot connect still.

Citrix case says a fix to the OS will resolve this coming this month. (Feb)

 

[David Markley]

Same here: Server 2022 with latest Windows Updates applied, Move from CVAD 2203 LTSR or 2203 LTSR CU1 to CU2 and cannot connect to VDA with error 1030.

 

Have tried VDA Cleanup Utility, reinstall of original VDA software, registry edits, nothing seems to make this server able to be connected to via Citrix. RDP works fine, but not via Citrix.

[Martin Becker]

I was told that a fix would be coming from Microsoft to address this issue the 3rd week of February.

[Martin Becker]

Patches did not work.

[Jürgen Gasser]

19 hours ago, Martin Becker said:

I installed patches released this week and was able to connect. Did anyone else have this happen?

I also installed the Windows Patches of this week. But the problem still persits here.

Windows OS Version: OS Server 2022 Build 20348.1547

Citrix LTSR 2203 CU2

 

Problem description:
After updating from LTSR 2203 CU1 to CU2, every user who logs on to the relevant VDA gets a grey screen.
The following event was written in the system event log during the VDA update: ID 1005
The Citrix Universal DLL Injection Driver has encountered an unexpected error.
ID 1003
The Citrix Universal DLL Injection Driver has detected an integrity error during process creation. The Citrix Universal DLL Injection Driver has been disabled.

 

[Jürgen Gasser]

7 hours ago, Jürgen Gasser said:

I also installed the Windows Patches of this week. But the problem still persits here.

Windows OS Version: OS Server 2022 Build 20348.1547

Citrix LTSR 2203 CU2

 

Problem description:
After updating from LTSR 2203 CU1 to CU2, every user who logs on to the relevant VDA gets a grey screen.
The following event was written in the system event log during the VDA update: ID 1005
The Citrix Universal DLL Injection Driver has encountered an unexpected error.
ID 1003
The Citrix Universal DLL Injection Driver has detected an integrity error during process creation. The Citrix Universal DLL Injection Driver has been disabled.

 

The following registry Key and a reboot of the VDA resolved my problem:

1. Check if the registry HKLM\SYSTEM\CurrentControlSet\Services\CtxUvi = REG_DWORD “UviEnabled” is created. If it's set to "0",  change it to "1".

[Qihua Liu]

On 2/16/2023 at 9:49 PM, Martin Becker said:

I installed patches released this week and was able to connect. Did anyone else have this happen?

Which patches did you install ? Could you pls share the detail ?
I installed latest updates in Windows 2022 but still gets ConnectionFailure (1030).

[Jürgen Gasser]

4 hours ago, Qihua Liu said:

Which patches did you install ? Could you pls share the detail ?
I installed latest updates in Windows 2022 but still gets ConnectionFailure (1030).

Hi, I installed the windows updates KB5022842 (OS Build 20348.1547 for Server 2022

[Qihua Liu]

9 minutes ago, Jürgen Gasser said:

Hi, I installed the windows updates KB5022842 (OS Build 20348.1547 for Server 2022

Thanks for reply.
Same update was installed but not helpful. The Citrix session still drops immediately when connecting Windows 2022 desktop.

[Jan Klimek1709163519]

5 hours ago, Qihua Liu said:

Thanks for reply.
Same update was installed but not helpful. The Citrix session still drops immediately when connecting Windows 2022 desktop.

 

Also in my case, KB5022842 does not fix the error. The connection continues to break with error 1030

[Qihua Liu]

On 1/11/2023 at 6:15 PM, Eckart Gutzeit said:

I have a similar issue on Server 2022, not on 21016/2019, where sessions end up in a grey screen forever. RTM and CU1 are working but starting with CU1.1 (2203.0.1100.x) the issue occurs immediately. The issue is also seen with 2209 and 2212.

Here are my case details:

 

Case Number:

81462687

Status:

Pending - Awaiting Hotfix

Date Submitted:

10/10/2022

Case Subject:

Grey screen after CU1 update

Comment Body:

Case Status:
- The fixes for Server2022 and Win10 multi session will be served on 2023.2C (the third week of February).
- This is known issue - HDX-34091. Fix will be released by Microsoft.

- The gray screen issue you see on Windows 2022 matches known issue caused by Microsoft RDS function. Our engineering involved Microsoft, and MS was working on fix for it. It should be included in KB 5018496, and it should be available for Windows 11 22H2 and Server 2022. It seems the KB is available for Win11, I do not see it for Windows 2022. I will check with our engineering what is the status of this fix.
Anyway this issue has to be addressed by Microsoft.  It is caused by Terminal Service calling PreDisconnect when session starts, and because of this HDX graphic is not properly initialized, so users see gray screen. 

Hello,
Any news for this case ?

[Martin Becker]

Side note if running VMware ESX, if you are setting your Server 2022 boxes to secure boot, Microsoft Monthly patch KB5022842 breaks secure boot. VMware has a fix and is posted here: https://kb.vmware.com/s/article/90947

 

Update: the hotfixes were a fluke apparently.

[Jan Klimek1709163519]

On 2/22/2023 at 1:58 PM, JAKOB JENSEN1709163176 said:

We had same issue : 

 

After updating a VDA from Citrix 2203 LTSR to CU2, connections can no longer be established. After an app is started, the connection is immediately terminated. No error message appears for the end user either.  There is nothing interesting in the Windows event log except an information from the Citrix Desktop Service:

ID 1030
The Citrix Desktop Service has detected that a user session has been terminated. Session 'a37c579b-6cd5-4277-ad1e-4c4c6f9cb263' for user 'test' was terminated; cause: ConnectionFailure.

 

Opened case with Citrix and following resolved our issue.

1.  Disabled Antivirus on VDA

2.  Uninstalled VDA

3. Ran cleanup VDA cleanup tool

4. Rebooted VDA

5. Ran Microsoft Powershell script provided by Citrix

6. Rebooted VDA

7. Ran VDA 2203 LTSR CU2 installation again.

 

Problem resolved 

  

 

Hi Jakob,

could you tell us which Powershell script from Citrix you are refering to? 

 

[Jan Klimek1709163519]

Hi Jakob,

I just tested your guide and was able to successfully update to CU2 and also to version 2212 without getting error 1030 again.

 

Thank you so much for sharing your knowledge with us!