Jump to content
Welcome to our new Citrix community!
  • 2

Citrix vms crashes tdica.sys blue screen 2203


Björn Schläfli

Question

Our vmware support team has upgraded our esx to 7.0.3. Since then we have noticed crashes on our vdas. "the guest operating system has failed" event on esx. 

PVS 2203 LTSR

Virtual Apps & Desktops 2203 LTSR

VDA 2203 LTSR CU1 (W2k16)

 

The affected servers are rebooted automatically.

 

Sometimes there is a bnistack 158 error logged in event log of the vda server.

 

Anyone else with these issues?

Link to comment

Recommended Posts

  • 0

Hi Bjoern,

 

no such problems here so far. Been running 7.0.3 since it came out. I'll share the exact build versions I'm running so you can compare:

 

image.png.e19ca8c86e88ea45a8c12b10c5c1a490.png

 

image.thumb.png.c94a6068e36295b1d6c9d48d9351ea1e.png

 

-Virtual Apps & Desktops 2203 LTSR

-VDA 2203 LTSR CU1 (2019)

 

image.thumb.png.173d77998f05a4190ef8b484b3915786.png

 

I hope this helps. From troubleshooting PVS in the past I think I remember that BNI stack is related to PVS so I would troubleshoot that first.

 

PS: Since the functionality of PVS is so mission-critical I have chosen not to upgrade that part to 2203 yet in our environment. Let's first see if Citrix can fix the VDA problems introduced in 2203 ?

 

 

Link to comment
  • 0

Hi Andy (once again ? ),

 

thank you for your answer. 

 

I use PVS 2203 since june. The issue with the crashes startet two weeks ago. So it should not be related. We use VMware Tools 12.0.0. Maybe an update can help. 

Do you have auto recovery (vmware HA options for the cluster) enabled? It's disabled for our cluster but the VM HA recovery does take effekt anyway.

 

Have a good day!

Link to comment
  • 0

No, my VMware host servers are not clustered. In our design the pool of VDA VM's are assigned fixed to their respective VMware host machines without clustering in order to avoid any of the typical problems that can occur at that level. So no VMware HA.

 

The redundancy for these dedicated Citrix host servers is handled at the level of the Citrix Delivery groups that are each spread over VDA VM's on different standalone VMware host servers. This design eliminates any potential issues at VM cluster level so I can't compare that.

 

I'm just about to start testing and releasing (if no problems) the all new VMware tools 12.1 since they fix an important vulnerability. So you may as well test with those as well perhaps.

 

https://docs.vmware.com/en/VMware-Tools/12.1/rn/VMware-Tools-1210-Release-Notes.html

 

This release resolves CVE-2022-31676. For more information on this vulnerability and its impact on VMware products see: 

 

https://www.vmware.com/security/advisories/VMSA-2022-0024.html

 

 

 

Link to comment
  • 0

We recently updated to 2203 CU1, PVS and TDA and are on the same versions of VMWare components.  We immediately began experiencing crashes on restart of our app layered and provisioned XenApp servers. I found it was due to AV. We still have to review our AV exceptions so I do not know specifically which object is causing the crash.

Link to comment
  • 0

I was able to catch the crash and it seems to be Citrix related. Blue screen with tdica.sys system thread exception. 

Found another person with this issue. https://www.reddit.com/r/Citrix/comments/u4jovd/issues_with_2203_ltsr/

 

Since Andy has no crashes with VDA 2203 CU1 (as in my site), the difference between his environment and mine is the os (we use w2k16) and VMware Tools (we use 12.0 at the moment).

Image 1272.png

Link to comment
  • 0
5 hours ago, Björn Schläfli said:

What kind of AV software do you use Andy?

 

Trendmicro Apex One.

 

On 8/30/2022 at 9:11 AM, Björn Schläfli said:

I was able to catch the crash and it seems to be Citrix related. Blue screen with tdica.sys system thread exception. 

Found another person with this issue. https://www.reddit.com/r/Citrix/comments/u4jovd/issues_with_2203_ltsr/

 

Since Andy has no crashes with VDA 2203 CU1 (as in my site), the difference between his environment and mine is the os (we use w2k16) and VMware Tools (we use 12.0 at the moment).

 

 

Today I also launched a new VDA build in QA using the newest Vmware tools after first upgrading host servers successfully to the latest VMware ESX version:

 

image.png.beccac1baa6c806bf37bed6b2c3ecc42.png

 

image.thumb.png.2c3a38e9ae48c63045793f749aab155f.png

 

No major problems detected at first glance. Gradual rollout in production and end user results soon to follow

 

Link to comment
  • 0

Hi Andy, 

 

thank you for your information.

 

I now have a guess what could be the trigger. In one site the crashes started this Monday. On Monday I allowed HTML5 access in StoreFront. In the other site html5 was already enabled but used very rarely.

I've disabled html5 a few minutes ago and hope the crashes stop now. 

 

Do you have html5 access enabled?

 

Link to comment
  • 0

Hi, yes I am having this same issue with VDA2203 and VDA 2203 CU1.
Our environment is Windows 2016, Citrix MCS, Vmware ESXi 6.7 & Vmware Tools 11.0.0.14549434

VDA 2112 working fine no problems.
As soon as I update our master MCS image to VDA 2203 we start to experience random server blue screens TdIca.sys
Nothing changes except the VDA version from 2112 -> 2203

I have tried upgrade of the VDA and full uninstall of the old VDA and fresh install of the new VDA, Windows Updates etc. 
I have also tried VDA 2206 but this VDA version has a problem where it does not apply the Time Zone Redirection policy for us so this rules out the use of this VDA version straight away.

This forces us to stay on VDA 2112 because this is the only stable VDA for us and all the VDA's released after this version (2203, 2203 CU1, 2206) have caused us problems

PPDC-XA7-12 Blue Screen.png

Link to comment
  • 0
1 hour ago, Aaron Atkinson said:

Greetings,

 

I encountered a BSOD on tdica.sys today. Our environment is as follows:

Citrix Hypervisor 8.2

Windows 2019 VDAs

CVAD 2203 CU1

 

I'll be escalating to support this afternoon.

Escalated to Citrix and here is what they said:

 

Yes, currently there is a known issue affecting 2203/2206 Server OS VDAs. Engineering team is working on a fix.
As  for now the workaround is to rollback to previous version.

For more information please check https://support.citrix.com/article/CTX463756/2203-or-2206-server-os-vda-may-experience-bsod-on-tdicasys

 

When I tried the link above, I received a message that the content is restricted. ?

Link to comment
  • 0
On 9/8/2022 at 7:34 AM, Björn Schläfli said:

Citrix sent me a private fix (new tdica.sys) but it's necessary to allow testsigned drivers with bcdedit. So it's unuseable for me in production.

If you need the private fix my Citrix case number is 81386211.

 

Workarounds are:

Back to vda 2112 and older

disable html5 access in StoreFront

Hey there, 

we are also encountering these bluescreen issues in our environment. You have written, that disabling HTML5 access in StoreFront will work around this problem. Could you please describe where you have disabled this?

While looking at some 7.6 guides to enable HTML5-access, it says that it should be the deployment option of Workspace App, which has to be set to something else than "Install locally" (https://support.citrix.com/article/CTX223503/how-to-enable-receiver-for-html5-for-internalexternal-users). We never changed this setting away from "Install locally", so is it already disabled in our environment, or is there an option somewhere else? 

We have also tried to run Citrix Cleanup Utility and reinstalling VDAs without any success. We only want to downgrade as last option.

Thanks!

Regards,

David 

Link to comment
  • 0
12 minutes ago, Björn Schläfli said:

I've set Install locally in receiver for web settings. That's it. No bsod since then. Strange it's working for me and not for you.

Okay, I see.

We are using HTTP Basic Auth to connect to Citrix using storebrowse with scripts. Maybe this triggers BSOD the same like HTML5-client does for you. Regardless in this manner we downgrade until there is a patch. Thanks for your input!

Link to comment
  • 0

Hello everyone,

 

we had the same issue.

Our environment:

Windows Server 2016

Citrix Version 2203 LTSR CU1

VMWare Tool 12.1 

PVS 2203 LTSR CU1

 

We had 2, or 3 BSODS with tdica.sys a week. 

Last Tuesday I followed Bjoerns advice and deactivated the HTML5 Client in Storefront, since this day no BSOD. I will reactivate it when a fix is available.

Many thanks for this tip!

 

Link to comment
  • 0

I have been working with Citrix on this issue as we have had sessions hanging and freezing due to TDICA.sys.  The article is now available to view, they removed the restrictions. https://support.citrix.com/article/CTX463756/2203-or-2206-server-os-vda-may-experience-bsod-on-tdicasys

 

We have access to an unsigned driver to resolve this issue which we cannot use in production. Citrix sent an email this morning that the signed version will be available this Friday (21st of October @Bjoern Schlaefli

 

We will put the signed driver in testing once received and I will feed back.

 

 

Link to comment
  • 0
On 10/19/2022 at 11:38 AM, Clinton Dunn1709152451 said:

I have been working with Citrix on this issue as we have had sessions hanging and freezing due to TDICA.sys.  The article is now available to view, they removed the restrictions. https://support.citrix.com/article/CTX463756/2203-or-2206-server-os-vda-may-experience-bsod-on-tdicasys

 

We have access to an unsigned driver to resolve this issue which we cannot use in production. Citrix sent an email this morning that the signed version will be available this Friday (21st of October @Bjoern Schlaefli

 

We will put the signed driver in testing once received and I will feed back.

 

 

Hi Clinton, We are running into the exact same issue - and Citrix provided us an unsigned driver (I guess it is because it's from August). Would you mind give me more information (Your Case number, mail) that we can once more escalate this correctly? Best, Hannes

Link to comment
  • 0

we recently started updating from 1912.1000 to 2203.1000 and at least one person is blue screening several times a day .  I stumbled upon this page and known issue however in our case the blue screen is showing afd.sys as the failed file (analyzing further points to our Nutanix NIC driver).  However we've had that driver on this system and all others for a while w/out issue and only since updating the VDA agent this started.  think this private fix might apply to our situation as well or no?

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...