PVS Services crash after updates

[Dennis van der Velde]

Hello all,

 

Since installing microsoft updates KB5009546 and KB5008877 on our provisioning 2109 machines, the Citrix PVS soap server service doesn't start anymore, when trying to start this service it crashes right away, when this happens the Citrix PVS API service also stops working.

 

Has anyone else seen this behaviour or know any workaround/fix? We have created a citrix case and are currently uninstalling the updates to see if that works.

[Carl Fallis]

We have just started to see this at a number of customer sites,  to work around this issue please set the following register key on the PVS server and reboot or restart the soap server:

 

HKLM\Software\Citrix\Provisioning services. Value Name: SkipForestLevelTrusts (DWORD) Value: 1

 

You can also recreate the issue by using Powershell  (thanks  to Citrix Support for being on top of this issue): 

$forestName = [System.DirectoryServices.ActiveDirectory.Domain]::GetComputerDomain().Forest.Name
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest((New-Object System.DirectoryServices.ActiveDirectory.DirectoryContext -Args @([System.DirectoryServices.ActiveDirectory.DirectoryContextType]::Forest, $forestName)))
$Forest = [System.DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()
$Forest.Domains

$forest.GetAllTrustRelationships()

This will cause an unhandled exception.   We are still investigating the issue.

[Patrick Dyke]

We did not install this months patches on any PVS servers yet (1912 CU4) but all of our DCs were updated, and we encountered this issue. The registry key above resolved the issue.

[Karthik R Makam]

Microsoft had documented couple of issues in the 'known issue in update' section and stated as being investigated.

 

E.g. issue:

 

- After installing this update, IP Security (IPSEC) connections that contain a Vendor ID might fail. VPN connections using Layer 2 Tunneling Protocol (L2TP) or IP security Internet Key Exchange (IPSEC IKE) might also be affected.

 

- After installing this update on domain controllers (DCs), affected versions of Windows Servers might restart unexpectedly.

Note: On Windows Server 2016 and later, you are more likely to be affected when DCs are using Shadow Principals in Enhanced Security Admin Environment (ESAE) or environments with Privileged Identity Management (PIM).

Now, they have released non security optional updates that resolve these issues

[David Ross1709151782]

Just to throw a small spanner into the mix. None of my windows servers have that update applied.

 

Yet the soap service wasn't starting having upgraded from 1912 CU2 to CU4.

 

I've applied the reg fix and low and behold it's sorted.

 

My envrio is a lab however and not in production

[Manfred Martin]

Seems Microsoft fixed it for Windows 2016 and higher OS with .NET v4.8
Microsoft fixes Windows Active Directory bug caused by Jan updates

[Dennis van der Velde]

Quick update, uninstalling these updates does not seem to work ?

[Dennis van der Velde]

Thanks Carl, this register entry works ?

[Carl Fallis]

52 minutes ago, Patrick Dyke said:

We did not install this months patches on any PVS servers yet (1912 CU4) but all of our DCs were updated, and we encountered this issue. The registry key above resolved the issue.

Thanks Patrick we thought this was the case after some recent testing but this just helps us verify it!   

 

So this issue was reproduced in your environment with just the DCs updated with the MS Patch! 

[Christoph Quast]

45 minutes ago, Patrick Dyke said:

We did not install this months patches on any PVS servers yet (1912 CU4) but all of our DCs were updated, and we encountered this issue. The registry key above resolved the issue.

Same here.
Thank you all for helping to solve this problem.

[Carl Fallis]

But do we know if it resolves the issue with GetAllTrustRelationships that is the main question for PVS, I will try to get someone to test it. 

[Justin Annand]

We updated half of our PVS servers and encountered the issue. The DC's were updated with the out of band patch and it still had the problem. So we applied the registry key.

 

Note to others the registry key needs the space between provisioning and services removed. Or just paste the below.

REG ADD "HKLM\Software\Citrix\Provisioningservices" /v "SkipForestLevelTrusts" /d 1 /t reg_dword /f

 

 

[Ralf Muumlnz]

Not only Server 2016 is affected! This happens also when these updates are installed on 2012 R2 Domain Controllers. and 2012R2 PVS-Servers!
DC and PVS Server: KB5009624 (2022-01 Rollup Win2012R2) and KB5010794 (2022-01 Out-Of Band Update Win2012R2)

On PVS-Server only: KB5009721 (2022-01 Net Rollup 2012R2)
Installing these 3 Updates on PVS Server only does not cause this problem! They started fine after applying the updates as long as the DC are not updated.

After we updated the domain Controllers with KB5009624 and KB5010794 the Citrix PVS soap server service and Citrix PVS API service do not start when rebooting server. Event 1000 and .net event 1026 come up. Registry entry solves it.

[NEH Beheer]

Same here. As soon as our Domain Controllers (2012R2) got patched with KB5010794 and KB5009713 and rebooted, soon after half the provisioning servers (W2016) pool where patched en rebooted. Those PVS servers got the issue with soapserver and PvsApi.

Carls solution worked for us, thank you.

 

[Chris Haag]

We had the original issue last week when our domain controllers were patched. The suggested reg key fixed it. Today we had another stack of pvs servers where we couldn't start the soap service after they rebooted. The reg key no longer worked. Fixed it this time using the json file mentioned here:

https://support.citrix.com/article/CTX231194

Don't include  "Context:" in the file. I think that's supposed to read "Contents:"

 

[Peter Beijer1709162656]

Same issue on our 2109 environment, running on Server 2019. Mentioned MS updates were NOT installed, still SOAP service crashing. After adding HKLM\Software\Citrix\ProvisioningServices. Value Name: SkipForestLevelTrusts (DWORD) Value: 1 and restarting service issue was resolved.

[Richard Gardner1709161653]

Some of our domain controllers have had the Microsoft patch which has caused the issue with our non-production PVS servers and the workaround allowed the SOAP/API services to start again.

 

However we have noticed that we cannot see any of our 'Forest type' trusted domains when attempting to carry out tasks such as creating a machine account. We can still see our 'External type' trusted domains.

 

Unfortunately we cannot reverse the workaround to see if this has caused the loss of the trusted domains because the service would not start meaning no console.

It does seem like it might be caused by the workaround given its name 'SkipForestLevelTrusts' but Citrix do not provide any information on what this actually does or what affect this might have on PVS environments which provide targets on multiple domains.

 

Is anyone else experiencing this?

 

We have raised a call with Citrix as we are assuming this will soon affect our Production environment.

[Karthik R Makam]

The issue is not seen when .NET patches made available on 7th Jan is installed. https://support.citrix.com/article/CTX338561 is updated with the validated KB's

[Yonathan Duerr]

Hello All

 

We had the same issue with our Provisioning Server running on MS Server 2019. Installing the Following Microsoft KB solved the problem.

 

2022-02 Update for .NET Framework 3.5 and 4.8 for Windows Server 2019 for x64 (KB5011257)

https://www.catalog.update.microsoft.com/Search.aspx?q=KB5011257